Verification failing after last renewal

Help
1

I have a server that is accessed by a python script. As of yesterday, the same day that certbot updated my certificate, the script no fails with the error message:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618)

Looking at the apache config on datafind.ligo.org I see

$ grep SSLCertificateFile /etc/httpd/conf.d/00-datafind.ligo.org-https.conf 
  SSLCertificateFile      "/etc/pki/tls/certs/datafind.ligo.uwm.edu_fullchain.pem"

which points at the fullchain version of the certificate that is valid until Mar 08 2021

$ openssl x509 -noout -enddate -in /etc/pki/tls/certs/datafind.ligo.uwm.edu_fullchain.pem
notAfter=Mar  8 13:00:24 2021 GMT

Checking with openssl (see below), it also reports a failure to verify the end entity certificate:

Verify return code: 21 (unable to verify the first certificate)

Since the full chain cert should be passing the end entity and the intermediate certs, I'm guessing that perhaps something changed with the root and the client host cert chains do not have the appropriate root for the new chain? Can anyone shed light? Thanks!

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: datafind.ligo.org

I ran this command: openssl s_client -connect datafind.ligo.org:443 -prexit

It produced this output:

CONNECTED(00000003)
depth=0 CN = datafind.ligo.uwm.edu
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = datafind.ligo.uwm.edu
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
 0 s:/CN=datafind.ligo.uwm.edu
   i:/C=US/O=Let's Encrypt/CN=R3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFQjCCBCqgAwIBAgISA5YJyyGYROL9WfNl1gAE1D2BMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMDEyMDgxMzAwMjRaFw0yMTAzMDgxMzAwMjRaMCAxHjAcBgNVBAMT
FWRhdGFmaW5kLmxpZ28udXdtLmVkdTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAL7qKrSNDqGpaUcFcVdO/OjohKjUBX/Wc7IVtU1f/WUshhwCtTzMozFn
qg4McLP7GOthLnV+UmRSbURDsYJNWQj/WI2HCifqNfQRkXErRbUeENAbhOd6i3b7
iCxbtUQ+EAXJRd6uVik2pnPb12b2x918Y8OZpIswXhfR8qiFRIUloDlGghjQrFfl
fu1zAHY8FFeGY2cokLYO/hqGTLrzCXVsN4PBFeW3Jffq0WGcKsYvGGn2toIOhMXi
tirNcXFI3pNOUjtEmPE5f2AtDhyiQv11Xpye+FEHpelRM3nmbAuSOx++UoKJ+tdJ
ZEBAx0Z/+PJfX5DPpwixElUOEVQw0dkCAwEAAaOCAmIwggJeMA4GA1UdDwEB/wQE
AwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw
ADAdBgNVHQ4EFgQUvEbo38sPkfRkBQWlfMIDnKLXEoYwHwYDVR0jBBgwFoAUFC6z
F7dYVsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVo
dHRwOi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxl
bmNyLm9yZy8wMwYDVR0RBCwwKoIRZGF0YWZpbmQubGlnby5vcmeCFWRhdGFmaW5k
LmxpZ28udXdtLmVkdTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEB
ATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQMG
CisGAQQB1nkCBAIEgfQEgfEA7wB2AESUZS6w7s6vxEAH2Kj+KMDa5oK+2MsxtT/T
M5a1toGoAAABdkKnOOEAAAQDAEcwRQIgNsPV6CBGZkIPIIlFQr3Fm2HGLlW9l6ge
1FISTi4IZgoCIQCHwwZ177nHFGUgf6ldSmSYG/KzvQu0hfxuKAwshqvckQB1APZc
lC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAABdkKnOSAAAAQDAEYwRAIg
cI+mNoeH+ngwG/3FXbqPOhCbpFcpCy0bRNRfCyykSpUCIHUKYi4bzAsqGnW5jH9e
9zASJPmRRtbOd9hoG7cvVl1PMA0GCSqGSIb3DQEBCwUAA4IBAQBQVmK+ok/Joo0p
BXqA+VtiYODnJqX06lBQu+J9IlqyVygTTdzlNViH6geNFYtJ2kNePUwWEEUgHd1Y
HDIsQyjUHciLxn2Ltxk5ox0tpXBQ55JVRE636+VH3Y6/ZPNxECmkMjb0v4AiCOjP
MmIEJVfqDDwNOufeC1zXjqa3/JgKaKgiShhGtpjg/ULvyBFsBQj4z9CGWvmULyBo
ftkm6HZ+eTRSudtYee4dQjU36xP4kt+K+VhQnAVwv6RuXfZkvWeWTIV+9lt2qWJt
9p2oTLuNrMUvVuIjekB6ZVD9lTWHBDvP0HqRj9LbZpNJiJLuqoZXDH7Y3VEqAeIi
M2lBRAIe
-----END CERTIFICATE-----
subject=/CN=datafind.ligo.uwm.edu
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2041 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 5D2CB1A1F4944EDDBE9420A511A73539670B3357A0E30810690AA1C6EF5E8EA5
    Session-ID-ctx: 
    Master-Key: 0CCFF86C17C2CB42775FF34B8AD4AD0B99CB906F4AE514BA68F4EA19AEF7D6FECA2100BED91B6EBD556CC3B05E73FDF0
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 66 f3 3e d4 48 01 8a de-65 a4 93 56 64 8a bf e0   f.>.H...e..Vd...
    0010 - 52 c5 9c ab 07 55 f1 68-53 b3 bc 03 f7 a3 cf bf   R....U.hS.......
    0020 - 4e d4 3b e7 81 34 39 8d-7d 25 38 4c 29 f1 f8 ca   N.;..49.}%8L)...
    0030 - 37 00 fa ec b7 bd 40 47-24 db 4c 08 76 3c 5f 0f   7.....@G$.L.v<_.
    0040 - 77 82 ee d2 5c fd c6 49-77 88 f3 da 1b 30 c2 b5   w...\..Iw....0..
    0050 - 79 6a 41 09 9d a3 7e 7d-a5 0f 7a 97 4d ff f2 99   yjA...~}..z.M...
    0060 - fa d7 eb a6 0b 9a 49 c8-50 fb a2 4b 07 5c b6 f9   ......I.P..K.\..
    0070 - c9 31 72 b3 a6 95 a2 8e-cb f5 96 21 aa 9c 5f a6   .1r........!.._.
    0080 - 15 1d de f1 88 42 46 c3-27 a7 f1 6a 96 8d fd 65   .....BF.'..j...e
    0090 - d5 c1 35 18 52 c7 a0 54-03 bb d1 38 d0 a8 87 3d   ..5.R..T...8...=
    00a0 - 2a 79 b3 a5 45 cc 56 07-7a a0 61 d8 4d 1f 44 85   *y..E.V.z.a.M.D.
    00b0 - 13 2d ff c5 08 6c 89 3e-ff d0 66 08 f7 50 43 ff   .-...l.>..f..PC.

    Start Time: 1607548350
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---
read:errno=0
---
Certificate chain
 0 s:/CN=datafind.ligo.uwm.edu
   i:/C=US/O=Let's Encrypt/CN=R3
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFQjCCBCqgAwIBAgISA5YJyyGYROL9WfNl1gAE1D2BMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMDEyMDgxMzAwMjRaFw0yMTAzMDgxMzAwMjRaMCAxHjAcBgNVBAMT
FWRhdGFmaW5kLmxpZ28udXdtLmVkdTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCC
AQoCggEBAL7qKrSNDqGpaUcFcVdO/OjohKjUBX/Wc7IVtU1f/WUshhwCtTzMozFn
qg4McLP7GOthLnV+UmRSbURDsYJNWQj/WI2HCifqNfQRkXErRbUeENAbhOd6i3b7
iCxbtUQ+EAXJRd6uVik2pnPb12b2x918Y8OZpIswXhfR8qiFRIUloDlGghjQrFfl
fu1zAHY8FFeGY2cokLYO/hqGTLrzCXVsN4PBFeW3Jffq0WGcKsYvGGn2toIOhMXi
tirNcXFI3pNOUjtEmPE5f2AtDhyiQv11Xpye+FEHpelRM3nmbAuSOx++UoKJ+tdJ
ZEBAx0Z/+PJfX5DPpwixElUOEVQw0dkCAwEAAaOCAmIwggJeMA4GA1UdDwEB/wQE
AwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw
ADAdBgNVHQ4EFgQUvEbo38sPkfRkBQWlfMIDnKLXEoYwHwYDVR0jBBgwFoAUFC6z
F7dYVsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVo
dHRwOi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxl
bmNyLm9yZy8wMwYDVR0RBCwwKoIRZGF0YWZpbmQubGlnby5vcmeCFWRhdGFmaW5k
LmxpZ28udXdtLmVkdTBMBgNVHSAERTBDMAgGBmeBDAECATA3BgsrBgEEAYLfEwEB
ATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3BzLmxldHNlbmNyeXB0Lm9yZzCCAQMG
CisGAQQB1nkCBAIEgfQEgfEA7wB2AESUZS6w7s6vxEAH2Kj+KMDa5oK+2MsxtT/T
M5a1toGoAAABdkKnOOEAAAQDAEcwRQIgNsPV6CBGZkIPIIlFQr3Fm2HGLlW9l6ge
1FISTi4IZgoCIQCHwwZ177nHFGUgf6ldSmSYG/KzvQu0hfxuKAwshqvckQB1APZc
lC/RdzAiFFQYCDCUVo7jTRMZM7/fDC8gC8xO8WTjAAABdkKnOSAAAAQDAEYwRAIg
cI+mNoeH+ngwG/3FXbqPOhCbpFcpCy0bRNRfCyykSpUCIHUKYi4bzAsqGnW5jH9e
9zASJPmRRtbOd9hoG7cvVl1PMA0GCSqGSIb3DQEBCwUAA4IBAQBQVmK+ok/Joo0p
BXqA+VtiYODnJqX06lBQu+J9IlqyVygTTdzlNViH6geNFYtJ2kNePUwWEEUgHd1Y
HDIsQyjUHciLxn2Ltxk5ox0tpXBQ55JVRE636+VH3Y6/ZPNxECmkMjb0v4AiCOjP
MmIEJVfqDDwNOufeC1zXjqa3/JgKaKgiShhGtpjg/ULvyBFsBQj4z9CGWvmULyBo
ftkm6HZ+eTRSudtYee4dQjU36xP4kt+K+VhQnAVwv6RuXfZkvWeWTIV+9lt2qWJt
9p2oTLuNrMUvVuIjekB6ZVD9lTWHBDvP0HqRj9LbZpNJiJLuqoZXDH7Y3VEqAeIi
M2lBRAIe
-----END CERTIFICATE-----
subject=/CN=datafind.ligo.uwm.edu
issuer=/C=US/O=Let's Encrypt/CN=R3
---
No client certificate CA names sent
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2041 bytes and written 446 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 5D2CB1A1F4944EDDBE9420A511A73539670B3357A0E30810690AA1C6EF5E8EA5
    Session-ID-ctx: 
    Master-Key: 0CCFF86C17C2CB42775FF34B8AD4AD0B99CB906F4AE514BA68F4EA19AEF7D6FECA2100BED91B6EBD556CC3B05E73FDF0
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 66 f3 3e d4 48 01 8a de-65 a4 93 56 64 8a bf e0   f.>.H...e..Vd...
    0010 - 52 c5 9c ab 07 55 f1 68-53 b3 bc 03 f7 a3 cf bf   R....U.hS.......
    0020 - 4e d4 3b e7 81 34 39 8d-7d 25 38 4c 29 f1 f8 ca   N.;..49.}%8L)...
    0030 - 37 00 fa ec b7 bd 40 47-24 db 4c 08 76 3c 5f 0f   7.....@G$.L.v<_.
    0040 - 77 82 ee d2 5c fd c6 49-77 88 f3 da 1b 30 c2 b5   w...\..Iw....0..
    0050 - 79 6a 41 09 9d a3 7e 7d-a5 0f 7a 97 4d ff f2 99   yjA...~}..z.M...
    0060 - fa d7 eb a6 0b 9a 49 c8-50 fb a2 4b 07 5c b6 f9   ......I.P..K.\..
    0070 - c9 31 72 b3 a6 95 a2 8e-cb f5 96 21 aa 9c 5f a6   .1r........!.._.
    0080 - 15 1d de f1 88 42 46 c3-27 a7 f1 6a 96 8d fd 65   .....BF.'..j...e
    0090 - d5 c1 35 18 52 c7 a0 54-03 bb d1 38 d0 a8 87 3d   ..5.R..T...8...=
    00a0 - 2a 79 b3 a5 45 cc 56 07-7a a0 61 d8 4d 1f 44 85   *y..E.V.z.a.M.D.
    00b0 - 13 2d ff c5 08 6c 89 3e-ff d0 66 08 f7 50 43 ff   .-...l.>..f..PC.

    Start Time: 1607548350
    Timeout   : 300 (sec)
    Verify return code: 21 (unable to verify the first certificate)
---

My web server is (include version):

$ httpd -V
Server version: Apache/2.4.6 (Scientific Linux)
Server built:   Aug  7 2019 11:32:41
Server's Module Magic Number: 20120211:24
Server loaded:  APR 1.4.8, APR-UTIL 1.5.2
Compiled using: APR 1.4.8, APR-UTIL 1.5.2
Architecture:   64-bit
Server MPM:     prefork
  threaded:     no
    forked:     yes (variable process count)
Server compiled with....
 -D APR_HAS_SENDFILE
 -D APR_HAS_MMAP
 -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
 -D APR_USE_SYSVSEM_SERIALIZE
 -D APR_USE_PTHREAD_SERIALIZE
 -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
 -D APR_HAS_OTHER_CHILD
 -D AP_HAVE_RELIABLE_PIPED_LOGS
 -D DYNAMIC_MODULE_LIMIT=256
 -D HTTPD_ROOT="/etc/httpd"
 -D SUEXEC_BIN="/usr/sbin/suexec"
 -D DEFAULT_PIDLOG="/run/httpd/httpd.pid"
 -D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
 -D DEFAULT_ERRORLOG="logs/error_log"
 -D AP_TYPES_CONFIG_FILE="conf/mime.types"
 -D SERVER_CONFIG_FILE="conf/httpd.conf"

The operating system my web server runs on is (include version):

$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	Scientific
Description:	Scientific Linux release 7.5 (Nitrogen)
Release:	7.5
Codename:	Nitrogen

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): Yes (obviously)

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

$ certbot --version
certbot 1.0.0
2

You're not sending the intermediate certificate at all. You're also sporting a very unusual location for the certificate when using certbot. How do you handle the issued certificate by certbot to get it into that locatiion?

1 Like
3

This file should contain two certificates concatenated to each other (your certificate + 1 intermediate).

Apache is behaving like it only contains 1 certificate.

You should check that it contains the same contents as the fullchain.pem in /etc/letsencrypt/live/. Perhaps symlinking it would be easier.

1 Like
4

You can review your progress with this online tool:
SSL Server Test: datafind.ligo.uwm.edu (Powered by Qualys SSL Labs)

It now shows:

  • missing intermediate certificate
  • outdated weak/old ciphers and protocols

Both of which are corrected within the Apache configuration files.

5

First, let me say that I did not set this up, I inherited it without documentation. The configuration is managed via puppet. I believe the certificate files are moved with the deploy scripts triggered by the certbot hooks:

$ sudo cat /etc/letsencrypt/renewal-hooks/deploy/standard-services 
[sudo] password for ******:
#!/bin/sh

set -e

if [ -f /etc/os-release ]; then
  . /etc/os-release
  case $ID in
    debian|ubuntu)
      ssl_path=/etc/ssl
      ;;
    rhel|centos|scientific)
      ssl_path=/etc/pki/tls
      ;;
  esac
fi

for domain in ${RENEWED_DOMAINS}; do
  # Make sure the certificate and private key files are
  # never world readable, even just for an instant while
  # we're copying them into daemon_cert_root.
  umask 077

  ssl_cert=${ssl_path}/certs/${domain}_cert.pem
  ssl_chain=${ssl_path}/certs/letsencrypt_chain.pem
  ssl_cert_fullchain=${ssl_path}/certs/${domain}_fullchain.pem
  ssl_key=${ssl_path}/private/${domain}_key.pem

  cp "${RENEWED_LINEAGE}/cert.pem" "${ssl_cert}"
  cp "${RENEWED_LINEAGE}/chain.pem" "${ssl_chain}"
  cp "${RENEWED_LINEAGE}/fullchain.pem" "${ssl_cert_fullchain}"
  cp "${RENEWED_LINEAGE}/privkey.pem" "${ssl_key}"

  if [ "$ID" = "debian" ]; then
    chown root:ssl-cert ${ssl_cert} ${ssl_key} ${ssl_chain} ${ssl_cert_fullchain}
    chmod 0640 ${ssl_key}
  fi

  chmod 0644 ${ssl_cert} ${ssl_chain} ${ssl_cert_fullchain}
done
6

This file does indeed contain both certificates and is identical to

/etc/letsencrypt/live/datafind.ligo.uwm.edu/fullchain.pem

It looks as though Apache is not sending the full chain, so it is probably a web server configuration issue (as @rg305 pointed out). Thanks to all for your help, I will start trying to figure out what is misconfigured in Apache now.

7

Depending on the version of Apache, it might want those two included parts in separate files.

Yup, I should have noticed that before:

image

2 Likes
8

Apache <2.4.8 requires SSLCertificateChainFile indeed.

1 Like
9

Thanks all. I eventually figured out how to add the SSLCertificateChainFile directive to my Apache config (but it was made considerably difficult by the, to me, arcane puppet config for the server which had configuration elements in two different paths of the git repo in which the puppet configs are stored) and all is working as expected again. As a bonus, I removed the deprecated SSL transports from the config and got the server to an A+ on SSLlabs (for the time being). Thanks again!

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.