Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
- The same renewal process has been working fine for several years
My domain is: hausworth.com
I ran this command: sudo certbot renew
It produced this output:
My web server is (include version): Apache/2.4.46
The operating system my web server runs on is (include version): amazon-linux-2
My hosting provider, if applicable, is:
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.11.0
Have you recently started using AWS elb in front of your server?
1 Like
Hi @MikeMcQ , I've had an ALB for while now, hasn't been a problem until now. The only change I can think of is how AWS treats ipv4 vs ipv6 now, but not sure if that could impact the certbot
you only have ipv4 in the dns for you domain so no ipv6 issue
do you know why lets debug test shows 403 for test challenges? Do those show in your apache logs or something elb does?
Both ip need to be able to respond correctly to Let's Encrypt server requests to you
Please show the output of sudo certbot certificates and sudo apachectl -t -D DUMP_VHOSTS
Thanks @MikeMcQ . What's confusing to me is that nothing in the ALB set up has changed since December, and certbot renew ran fine then (as you can see on the ssllabs). I've been trying to see where 403s are coming from, but no luck so far. There are no errors in the Apache logs, /.well-known/acme-challenge exists and has the proper privileges etc
[ec2-user@ip-172-31-27-219 ~]$ sudo apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:80 is a NameVirtualHost
default server ip-172-31-27-219.ec2.internal (/etc/httpd/conf.d/hausworth.conf:1)
port 80 namevhost ip-172-31-27-219.ec2.internal (/etc/httpd/conf.d/hausworth.conf:1)
port 80 namevhost hausworth.com (/etc/httpd/conf.d/hausworth_bak.conf:2)
alias www.hausworth.com
*:443 is a NameVirtualHost
default server hausworth.com (/etc/httpd/conf.d/hausworth-le-bak.conf:2)
port 443 namevhost hausworth.com (/etc/httpd/conf.d/hausworth-le-bak.conf:2)
alias www.hausworth.com
port 443 namevhost www.hausworth.com (/etc/httpd/conf.d/hausworth-le-ssl.conf:2)
alias hausworth.com
port 443 namevhost hausworth.com (/etc/httpd/conf.d/hausworth_bak.conf:21)
alias www.hausworth.com
port 443 namevhost ip-172-31-27-219.ec2.internal (/etc/httpd/conf.d/ssl.conf:56)
[ec2-user@ip-172-31-27-219 ~]$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Found the following certs:
Certificate Name: hausworth.com
Serial Number: 4f2c21e1ef5379d6ce3f5d0c072daed19bf
Key Type: RSA
Domains: hausworth.com www.hausworth.com
Expiry Date: 2025-03-21 18:19:05+00:00 (VALID: 18 days)
Certificate Path: /etc/letsencrypt/live/hausworth.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/hausworth.com/privkey.pem
[ec2-user@ip-172-31-27-219 ~]$
1 Like
One other thing: I saw a suggestion to add the rule at the ALB level to pass through requests to /.well-known/acme-challenge instead of redirecting to HTTPS, which I did, but it had no effect (again, that wasn't in place in December, but certbot ran fine then)
I find all queries to curl -Ii http://hausworth.com or with additional paths and filenames
return a 403 Forbidden - HTTP | MDN
$ curl -Ii http://hausworth.com/.well-known/acme-challenge/sometestfile
HTTP/1.1 403 Forbidden
Server: awselb/2.0
Date: Sun, 02 Mar 2025 22:16:07 GMT
Content-Type: text/html
Content-Length: 118
Connection: keep-alive
$ curl -Ii http://hausworth.com/.well-known/acme-challenge/
HTTP/1.1 403 Forbidden
Server: awselb/2.0
Date: Sun, 02 Mar 2025 22:16:10 GMT
Content-Type: text/html
Content-Length: 118
Connection: keep-alive
$ curl -Ii http://hausworth.com/.well-known/acme-challenge
HTTP/1.1 403 Forbidden
Server: awselb/2.0
Date: Sun, 02 Mar 2025 22:16:12 GMT
Content-Type: text/html
Content-Length: 118
Connection: keep-alive
$ curl -Ii http://hausworth.com/.well-known/
HTTP/1.1 403 Forbidden
Server: awselb/2.0
Date: Sun, 02 Mar 2025 22:16:15 GMT
Content-Type: text/html
Content-Length: 118
Connection: keep-alive
$ curl -Ii http://hausworth.com/.well-known
HTTP/1.1 403 Forbidden
Server: awselb/2.0
Date: Sun, 02 Mar 2025 22:16:17 GMT
Content-Type: text/html
Content-Length: 118
Connection: keep-alive
$ curl -Ii http://hausworth.com/
HTTP/1.1 403 Forbidden
Server: awselb/2.0
Date: Sun, 02 Mar 2025 22:16:21 GMT
Content-Type: text/html
Content-Length: 118
Connection: keep-alive
$ curl -Ii http://hausworth.com
HTTP/1.1 403 Forbidden
Server: awselb/2.0
Date: Sun, 02 Mar 2025 22:16:22 GMT
Content-Type: text/html
Content-Length: 118
Connection: keep-alive
Edit
And now a clue changing the User Agent: to Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36
Now there is a response of HTTP/1.1 301 Moved Permanently and a redirect.
$ curl -Ii http://hausworth.com -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
HTTP/1.1 301 Moved Permanently
Server: awselb/2.0
Date: Sun, 02 Mar 2025 22:27:08 GMT
Content-Type: text/html
Content-Length: 134
Connection: keep-alive
Location: https://hausworth.com:443/
$ curl -Ii http://hausworth.com/.well-known/acme-challenge/sometestfile -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36"
HTTP/1.1 302 Found
Date: Sun, 02 Mar 2025 22:28:43 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 23
Connection: keep-alive
Server: Apache/2.4.46 () OpenSSL/1.0.2k-fips
X-DNS-Prefetch-Control: off
Expect-CT: max-age=0
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=15552000; includeSubDomains
X-Download-Options: noopen
X-Content-Type-Options: nosniff
X-Permitted-Cross-Domain-Policies: none
Referrer-Policy: no-referrer
X-XSS-Protection: 0
Access-Control-Allow-Origin: *
Location: /
Vary: Accept,Accept-Encoding
Set-Cookie: hwSearchSettings=%7B%22numBedrooms%22%3A%223%22%2C%20%22lat%22%3A43.70%2C%20%22long%22%3A-79.40%2C%20%22zoom%22%3A12%2C%20%22state%22%3A%22ON%22%2C%20%22area%22%3A%22Toronto%22%20%7D; Max-Age=900; Path=/; Expires=Sun, 02 Mar 2025 22:43:43 GMT; HttpOnly
Set-Cookie: hwsecret=s%3AgPWTWGINMea8AqcFf0pob7NzI20iHQNV.ugAjJZ8PAxzFrKgBdQNuRYnQuzPR0wjoBzCN%2FUqv%2FW4; Domain=hausworth.com; Path=/; Expires=Sun, 02 Mar 2025 23:28:43 GMT; HttpOnly
I suspect this issue is cause by a firewall. The user agent "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" needs to be allowed.
1 Like
That's possible, if AWS ACL added it to the block list... let me see if I can add the allow rule...
1 Like
@Bruce5051 Thanks to your suggestion I was able to resolve my issue! If anyone else is running certbot on AWS with a Web ACL on, you might have to create a custom rule to allow user-agent https://www.letsencrypt.org requests through, otherwise they are rejected as bad bots
2 Likes
Hi @Andytoc,
This show the certificate presently being served https://decoder.link/sslchecker/hausworth.com/443
and here SSL Server Test: hausworth.com (Powered by Qualys SSL Labs)
While the certificate is fine and recent; the certificate chain is incomplete, in this instance R10 is missing from the certificate chain. This cause extra download for the clients. Sometime be it is problematic not having a proper certificate chain.
And actually the issued certificate is being sent twice and R10 not at all.
$ openssl s_client -showcerts -servername hausworth.com -connect hausworth.com:443 < /dev/null
CONNECTED(00000003)
depth=0 CN = hausworth.com
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = hausworth.com
verify error:num=21:unable to verify the first certificate
verify return:1
depth=0 CN = hausworth.com
verify return:1
---
Certificate chain
0 s:CN = hausworth.com
i:C = US, O = Let's Encrypt, CN = R10
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 2 23:17:03 2025 GMT; NotAfter: May 31 23:17:02 2025 GMT
-----BEGIN CERTIFICATE-----
MIIE/TCCA+WgAwIBAgISAwP+TAGXSMzmWN2Qg94h6h+9MA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTAwHhcNMjUwMzAyMjMxNzAzWhcNMjUwNTMxMjMxNzAyWjAYMRYwFAYDVQQD
Ew1oYXVzd29ydGguY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
msKsFjwedXEtPbl6v03pFnUWFWx+HD8SfH1CRDpvQPjoiWKlEzwEWtweApNYj9nl
58XMtFkBxNZ+frT2PP8Hzr3vQ2aQiPQutG5T8C/7JvKME4nqLPRUOkMbl8jjCwEz
nWtp242YlZUYljwv+/ixkP/7Lmm9whzmWaXCQrBncmHSyY/WR0jDC0CB1G5GQG2J
iL+TkT51UY7ii/sFaygxO5KaVE6zVj3qMMYF1ABS8ROVnK4iINkogpXkVNhUYiyb
S5SPU8JGDJD8B9Piom6OnKwGanNEZya8Mh3JvPYkYPW+UquKP+JhfBzjFDvOBcO+
IeV2obYq5bhTdyQcOo427QIDAQABo4ICJDCCAiAwDgYDVR0PAQH/BAQDAgWgMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1Ud
DgQWBBQ4ju8XIsDLmuGcqx9NnhyfY3JtjTAfBgNVHSMEGDAWgBS7vMNHpeS8qcbD
pHIMEI2iNeHI6DBXBggrBgEFBQcBAQRLMEkwIgYIKwYBBQUHMAGGFmh0dHA6Ly9y
MTAuby5sZW5jci5vcmcwIwYIKwYBBQUHMAKGF2h0dHA6Ly9yMTAuaS5sZW5jci5v
cmcvMCsGA1UdEQQkMCKCDWhhdXN3b3J0aC5jb22CEXd3dy5oYXVzd29ydGguY29t
MBMGA1UdIAQMMAowCAYGZ4EMAQIBMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHUA
5tIxY0B3jMEQQQbXcbnOwdJA9paEhvu6hzId/R43jlAAAAGVWVuoVwAABAMARjBE
AiBp0r4w8LVaxKxla768UaQMh1Cy905YL942w2dYSQkwJQIgYbpDbDYpQNKxVWU2
nUCSw68DItvt7x4CGjWtyiQ72cgAdwCi4wrkRe+9rZt+OO1HZ3dT14JbhJTXK14b
LMS5UKRH5wAAAZVZW6hTAAAEAwBIMEYCIQD2ggmsE1HMdj2SGI7Fp8L2qDGmTgL4
2EZ5XLQeqE2uPgIhAJCXgrYXegNvNgOxP7UkCT4kWltiMqwG9e39Mr7M5FyEMA0G
CSqGSIb3DQEBCwUAA4IBAQBKywmNhTv98VbBLwF+YbLt4itgTtMOZrxVGouOytjm
+J5HKNu6kBxPVGkKx7i5TpBx0grVuvAsaRbxZntJE48/otNQ0AmipRQ9ApufGOz3
eqY2W+zOAxZITBnFjCEBlyH/3QLkG6EySj9J0RTkw/k/2uz2q1e5DkC52AKbafRx
d9He/yYiFM6LpkO4qAv1bm1wLQLPkeybv3wEN3ks+9g5Gf43itfO4uYg/Iy1FKuT
JTa9u6Dbm7NM+C6Yd37gelCD7wzXTRJR9Pba/WSU7NPvhs8tLTAXZ5k2SDlHQdH/
Q10B3ffAdjcCxwyLNP7Xd9ID00RtVAyFEtJrQDoDIwjA
-----END CERTIFICATE-----
1 s:CN = hausworth.com
i:C = US, O = Let's Encrypt, CN = R10
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 2 23:17:03 2025 GMT; NotAfter: May 31 23:17:02 2025 GMT
-----BEGIN CERTIFICATE-----
MIIE/TCCA+WgAwIBAgISAwP+TAGXSMzmWN2Qg94h6h+9MA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTAwHhcNMjUwMzAyMjMxNzAzWhcNMjUwNTMxMjMxNzAyWjAYMRYwFAYDVQQD
Ew1oYXVzd29ydGguY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
msKsFjwedXEtPbl6v03pFnUWFWx+HD8SfH1CRDpvQPjoiWKlEzwEWtweApNYj9nl
58XMtFkBxNZ+frT2PP8Hzr3vQ2aQiPQutG5T8C/7JvKME4nqLPRUOkMbl8jjCwEz
nWtp242YlZUYljwv+/ixkP/7Lmm9whzmWaXCQrBncmHSyY/WR0jDC0CB1G5GQG2J
iL+TkT51UY7ii/sFaygxO5KaVE6zVj3qMMYF1ABS8ROVnK4iINkogpXkVNhUYiyb
S5SPU8JGDJD8B9Piom6OnKwGanNEZya8Mh3JvPYkYPW+UquKP+JhfBzjFDvOBcO+
IeV2obYq5bhTdyQcOo427QIDAQABo4ICJDCCAiAwDgYDVR0PAQH/BAQDAgWgMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1Ud
DgQWBBQ4ju8XIsDLmuGcqx9NnhyfY3JtjTAfBgNVHSMEGDAWgBS7vMNHpeS8qcbD
pHIMEI2iNeHI6DBXBggrBgEFBQcBAQRLMEkwIgYIKwYBBQUHMAGGFmh0dHA6Ly9y
MTAuby5sZW5jci5vcmcwIwYIKwYBBQUHMAKGF2h0dHA6Ly9yMTAuaS5sZW5jci5v
cmcvMCsGA1UdEQQkMCKCDWhhdXN3b3J0aC5jb22CEXd3dy5oYXVzd29ydGguY29t
MBMGA1UdIAQMMAowCAYGZ4EMAQIBMIIBBAYKKwYBBAHWeQIEAgSB9QSB8gDwAHUA
5tIxY0B3jMEQQQbXcbnOwdJA9paEhvu6hzId/R43jlAAAAGVWVuoVwAABAMARjBE
AiBp0r4w8LVaxKxla768UaQMh1Cy905YL942w2dYSQkwJQIgYbpDbDYpQNKxVWU2
nUCSw68DItvt7x4CGjWtyiQ72cgAdwCi4wrkRe+9rZt+OO1HZ3dT14JbhJTXK14b
LMS5UKRH5wAAAZVZW6hTAAAEAwBIMEYCIQD2ggmsE1HMdj2SGI7Fp8L2qDGmTgL4
2EZ5XLQeqE2uPgIhAJCXgrYXegNvNgOxP7UkCT4kWltiMqwG9e39Mr7M5FyEMA0G
CSqGSIb3DQEBCwUAA4IBAQBKywmNhTv98VbBLwF+YbLt4itgTtMOZrxVGouOytjm
+J5HKNu6kBxPVGkKx7i5TpBx0grVuvAsaRbxZntJE48/otNQ0AmipRQ9ApufGOz3
eqY2W+zOAxZITBnFjCEBlyH/3QLkG6EySj9J0RTkw/k/2uz2q1e5DkC52AKbafRx
d9He/yYiFM6LpkO4qAv1bm1wLQLPkeybv3wEN3ks+9g5Gf43itfO4uYg/Iy1FKuT
JTa9u6Dbm7NM+C6Yd37gelCD7wzXTRJR9Pba/WSU7NPvhs8tLTAXZ5k2SDlHQdH/
Q10B3ffAdjcCxwyLNP7Xd9ID00RtVAyFEtJrQDoDIwjA
-----END CERTIFICATE-----
---
Server certificate
subject=CN = hausworth.com
issuer=C = US, O = Let's Encrypt, CN = R10
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3111 bytes and written 379 bytes
Verification error: unable to verify the first certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 21 (unable to verify the first certificate)
---
DONE
1 Like