Renew Certs Error

Help
1

I keep receiving this error every time I try to renew a certificate for the last week:

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Renewing an existing certificate
Attempting to renew cert (themud.org) from /etc/letsencrypt/renewal/themud.org.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v01.api.letsencrypt.org', port=443): Read timed out. (read timeout=45). Skipping.

Checked to see if I can connect to the server and I can, I made sure my website is publicly accessible and it is, rebooted webserver, router and dns server just in case still won't renew.

so I'm out of ideas here, anyone else have some?

2

Hi @adam-themud,

This looks like an error from Certbot failing to reach Let's Encrypt's ACME server at acme-v01.api.letsencrypt.org. Are you able to share a traceroute/mtr to that address? Does openssl s_client -connect acme-v01.api.letsencrypt.org:443 </dev/null succeed for you?

many moons ago I worked on an LPC intermud2 implementation. Glad to see there's still MUDs out there :slight_smile:

3

openssl s_client -connect acme-v01.api.letsencrypt.org:443 </dev/null

openssl s_client -connect acme-v01.api.letsencrypt.org:443 </dev/null
CONNECTED(00000003)
depth=3 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=2 C = US, O = IdenTrust, CN = IdenTrust Commercial Root CA 1
verify return:1
depth=1 C = US, O = IdenTrust, OU = TrustID Server, CN = TrustID Server CA A52
verify return:1
depth=0 CN = *.api.letsencrypt.org, O = INTERNET SECURITY RESEARCH GROUP, L = Mountain View, ST = California, C = US
verify return:1

Certificate chain
0 s:/CN=*.api.letsencrypt.org/O=INTERNET SECURITY RESEARCH GROUP/L=Mountain View/ST=California/C=US
i:/C=US/O=IdenTrust/OU=TrustID Server/CN=TrustID Server CA A52
1 s:/C=US/O=IdenTrust/OU=TrustID Server/CN=TrustID Server CA A52
i:/C=US/O=IdenTrust/CN=IdenTrust Commercial Root CA 1
2 s:/C=US/O=IdenTrust/CN=IdenTrust Commercial Root CA 1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Server certificate
-----BEGIN CERTIFICATE-----
MIIHDzCCBfegAwIBAgIQfwAAAQAAAU4w1QtkQskueDANBgkqhkiG9w0BAQsFADBa
MQswCQYDVQQGEwJVUzESMBAGA1UEChMJSWRlblRydXN0MRcwFQYDVQQLEw5UcnVz
dElEIFNlcnZlcjEeMBwGA1UEAxMVVHJ1c3RJRCBTZXJ2ZXIgQ0EgQTUyMB4XDTE1
MDYyNjE3MDU0NVoXDTE4MDYyNTE3MDU0NVowgYUxHjAcBgNVBAMTFSouYXBpLmxl
dHNlbmNyeXB0Lm9yZzEpMCcGA1UEChMgSU5URVJORVQgU0VDVVJJVFkgUkVTRUFS
Q0ggR1JPVVAxFjAUBgNVBAcTDU1vdW50YWluIFZpZXcxEzARBgNVBAgTCkNhbGlm
b3JuaWExCzAJBgNVBAYTAlVTMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAlkLBpvVuOTZrpaPkzmAx+5OcK6gffKPon8E6ktJmNsjfrh3ATdgKt121ccD+
7I3gBcZu5euG1ldynO/wx29oeGhBiiWx9Bkm1brFKDLvH56ztO29xEo5ZQnxV7w0
25NvjBca5J2tRiAIjcdjMITE1usp5rrNLNM2LIs1RutP7bexvuNv2dGyJ0uNIWI4
Ym2Tt5rnUTx7Vsz0M4nz2i6sSB9ajrSa1h4hFrZYqIU/zpl20f7So8oZDYCZN4WY
xYgbUuEneWxFigElkJbNk0UeVN26tdMGHdIWCabgoZSy80X+pFvxCgdbLQzS0LXQ
orqkGsMYO0+fjjmLj0742IHYswIDAQABo4IDozCCA58wDgYDVR0PAQH/BAQDAgWg
MIICJwYDVR0gBIICHjCCAhowggELBgpghkgBhvkvAAYDMIH8MEAGCCsGAQUFBwIB
FjRodHRwczovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xp
Y3kvdHMvMIG3BggrBgEFBQcCAjCBqhqBp1RoaXMgVHJ1c3RJRCBTZXJ2ZXIgQ2Vy
dGlmaWNhdGUgaGFzIGJlZW4gaXNzdWVkIGluIGFjY29yZGFuY2Ugd2l0aCBJZGVu
VHJ1c3QncyBUcnVzdElEIENlcnRpZmljYXRlIFBvbGljeSBmb3VuZCBhdCBodHRw
czovL3NlY3VyZS5pZGVudHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMv
MIIBBwYGZ4EMAQICMIH8MEAGCCsGAQUFBwIBFjRodHRwczovL3NlY3VyZS5pZGVu
dHJ1c3QuY29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvMIG3BggrBgEFBQcCAjCB
qhqBp1RoaXMgVHJ1c3RJRCBTZXJ2ZXIgQ2VydGlmaWNhdGUgaGFzIGJlZW4gaXNz
dWVkIGluIGFjY29yZGFuY2Ugd2l0aCBJZGVuVHJ1c3QncyBUcnVzdElEIENlcnRp
ZmljYXRlIFBvbGljeSBmb3VuZCBhdCBodHRwczovL3NlY3VyZS5pZGVudHJ1c3Qu
Y29tL2NlcnRpZmljYXRlcy9wb2xpY3kvdHMvMB0GA1UdDgQWBBQvAQatcXq62MSt
v2WLHPKwErqzgDBFBgNVHR8EPjA8MDqgOKA2hjRodHRwOi8vdmFsaWRhdGlvbi5p
ZGVudHJ1c3QuY29tL2NybC90cnVzdGlkY2FhNTIuY3JsMIGEBggrBgEFBQcBAQR4
MHYwMAYIKwYBBQUHMAGGJGh0dHA6Ly9jb21tZXJjaWFsLm9jc3AuaWRlbnRydXN0
LmNvbTBCBggrBgEFBQcwAoY2aHR0cDovL3ZhbGlkYXRpb24uaWRlbnRydXN0LmNv
bS9jZXJ0cy90cnVzdGlkY2FhNTIucDdjMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggr
BgEFBQcDAjAfBgNVHSMEGDAWgBSiViQ80NQVuei/eKMTEFhILhZU4TA1BgNVHREE
LjAsghUqLmFwaS5sZXRzZW5jcnlwdC5vcmeCE2FwaS5sZXRzZW5jcnlwdC5vcmcw
DQYJKoZIhvcNAQELBQADggEBAIgV29HRYYk2RfrkE6KOeAT6K3AJsKgNe+i9Pdo7
YLdiDGMv/kKyyTNsq4l8IEaY1kw3aqkyT44D8vp6yGbbNl1uAE4VnaPeChLOBK0G
xz3CjtoZPqLxmieft8jybIVWZKZlIIsZrSMaEvrmRTGYartBSuxqOzIjCNxvkkRt
MYDNYRhiy6NPMtA0kpDQCN6tOggXdCm19kmzKI8pUvQGCnGscXJvlQIeaZva0GWf
UOuPael/zohUUsm2lJowlIli939RdxUal0hwv8tD12qAyZAF2/+4y61ds4svGD9B
ukNnsAzS3EXNqq5qWFqRQyiTZfCcjaXEdtySyNAUiM6eFTY=
-----END CERTIFICATE-----
subject=/CN=*.api.letsencrypt.org/O=INTERNET SECURITY RESEARCH GROUP/L=Mountain View/ST=California/C=US
issuer=/C=US/O=IdenTrust/OU=TrustID Server/CN=TrustID Server CA A52

No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 5917 bytes and written 415 bytes

New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 1EFB2566D4A7F690010EABC7D3071D17ADF302C5F64446092ABE0960D9AD917A
Session-ID-ctx:
Master-Key: 4EF0CCA135C306AB39106701042C94AE1A595DEA4BE31E752AB20B5CFDD0F498F2B78AFEF47B10E407C689CB479AC675
Key-Arg : None
Krb5 Principal: None
PSK identity: None
PSK identity hint: None
TLS session ticket lifetime hint: 7200 (seconds)
TLS session ticket:
0000 - 00 00 03 d5 01 d0 68 ba-73 6c 13 fd 91 01 b0 a4 ......h.sl......
0010 - 9c 9f b1 8a 2b cb 16 2f-66 26 9c 8b 12 7a 5a d5 ....+../f&...zZ.
0020 - 53 37 72 42 4b 5a eb 16-4b 58 59 f7 b3 ec 0c 8d S7rBKZ..KXY.....
0030 - d0 35 4d de e3 49 e3 bb-48 cb 40 86 48 dd c7 63 .5M..I..H.@.H..c
0040 - 01 8c 05 c3 93 69 24 82-aa 06 6a 58 66 a4 68 be .....i$...jXf.h.
0050 - 1d 5d 20 50 86 01 8d 67-83 4e 6c 0f 94 d4 3b 10 .] P...g.Nl...;.
0060 - fb df 31 06 02 aa 39 ae-64 cb 9e 7c 34 ed 87 86 ..1...9.d..|4...
0070 - 70 e5 7b 81 26 56 00 59-9c d6 2c 22 aa 1b 1d 61 p.{.&V.Y..,"...a
0080 - f0 cd ac e3 d9 ac fd be-fa 1d 02 2e d6 ac 43 69 ..............Ci
0090 - c2 5f dd 7d d1 be 55 0c-e7 ee d0 48 fc 96 92 67 ._.}..U....H...g

Start Time: 1514930644
Timeout   : 300 (sec)
Verify return code: 0 (ok)

DONE

Yes is succeeded.

Yes there is quite a few still around, you should drop us a line at somepoint https://www.lpmuds.net

4

Traceroute:

traceroute to acme-v01.api.letsencrypt.org (122.252.44.37), 30 hops max, 60 byte packets
1 gateway.sunnydale (10.1.1.1) 0.296 ms 0.240 ms 0.270 ms
2 lo0.bras1.mel10.on.ii.net (150.101.32.127) 9.967 ms 9.961 ms 11.117 ms
3 be11.cr2.mel11.on.ii.net (150.101.35.163) 11.872 ms 11.867 ms 11.863 ms
4 be22.cr2.mel10.on.ii.net (150.101.34.40) 11.625 ms ae10.cr1.mel4.on.ii.net (150.101.35.207) 11.958 ms be22.cr2.mel10.on.ii.net (150.101.34.40) 11.615 ms
5 ae0.cr1.mel8.on.ii.net (150.101.33.11) 11.837 ms ae19.cr1.mel8.on.ii.net (150.101.40.134) 11.969 ms ae0.cr1.mel8.on.ii.net (150.101.33.11) 11.941 ms
6 as20940.vic.ix.asn.au (218.100.78.36) 49.847 ms 49.671 ms 44.728 ms
7 a122-252.44-37.deploy.akamaitechnologies.com (122.252.44.37) 11.172 ms 11.129 ms 11.122 ms

5

Is it possible that the ACME client is trying to use IPv6 while the other tools are using IPv4? (Though I don’t think that would be likely to produce that particular socket error.)

6

I can disable IPv6 and try but its never given me issues before?

7

You could try

curl -4 https://acme-v01.api.letsencrypt.org/directory
curl -6 https://acme-v01.api.letsencrypt.org/directory
8

curl -6 https://acme-v01.api.letsencrypt.org/directory

{
"Mj6rGDKAscY": "Adding random entries to the directory",
"key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
"meta": {
"terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf"
},
"new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
"new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
"new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
"revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
}

curl -4 https://acme-v01.api.letsencrypt.org/directory

{
"key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
"meta": {
"terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf"
},
"new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
"new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
"new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
"revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert",
"zTDH_eJ3YOY": "Adding random entries to the directory"
}

9

Would it make a difference if my IPv4 and IPv6 address changed after the last renewal of the certificates over 90 days ago?

10

No, that shouldn’t have any effect.

@cpu, any theories on whether this is more of a client-side or more of a server-side problem?

11

Log File traceback:

2018-01-03 00:38:51,338:WARNING:certbot.renewal:Attempting to renew cert (themud.org) from /etc/letsencrypt/renewal/themud.org.conf produced an unexpected error: HTTPSConnectionPool(host='acme-v01.api.letsencrypt.org', port=443): Read ti
med out. (read timeout=45). Skipping.
2018-01-03 00:38:51,338:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/renewal.py", line 425, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/main.py", line 743, in renew_cert
_get_and_save_cert(le_client, config, lineage=lineage)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/main.py", line 80, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/renewal.py", line 297, in renew_cert
new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/client.py", line 318, in obtain_certificate
self.config.allow_subset_of_names)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/auth_handler.py", line 66, in get_authorizations
self.authzr[domain] = self.acme.request_domain_challenges(domain)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/acme/client.py", line 213, in request_domain_challenges
typ=messages.IDENTIFIER_FQDN, value=domain), new_authzr_uri)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/acme/client.py", line 192, in request_challenges
response = self.net.post(self.directory.new_authz, new_authz)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/acme/client.py", line 709, in post
return self._post_once(*args, **kwargs)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/acme/client.py", line 720, in _post_once
response = self._send_request('POST', url, data=data, **kwargs)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/acme/client.py", line 630, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/requests/sessions.py", line 488, in request
resp = self.send(prep, **send_kwargs)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/requests/sessions.py", line 609, in send
r = adapter.send(request, **kwargs)
File "/opt/eff.org/certbot/venv/lib/python2.7/site-packages/requests/adapters.py", line 499, in send
raise ReadTimeout(e, request=request)
ReadTimeout: HTTPSConnectionPool(host='acme-v01.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)

12

According to the log it connects to acme-v01.api.letsencrypt.org no problem at all only when it attempts to run the renew code is when it gets a timeout

13

Ok mounting the webroot on another machine with internet access via a vpn yields this result.

Plugins selected: Authenticator webroot, Installer None
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.themud.org
http-01 challenge for themud.org
Using the webroot path /mnt/www/themud.org for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. themud.org (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching https://www.themud.org/.well-known/acme-challenge/wymR6qLh939zhEz-y7VTMFcbwyE0_4qmX6B4YFe_iPs: Timeout, www.themud.org (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching https://www.themud.org/.well-known/acme-challenge/KWzh-HaM9IJ2g4RK3kdfJkcIEuTsC9Hk8EQ07X_nenI: Timeout

IMPORTANT NOTES:

files are being created on the webserver, but the remote server cannot connect to me.

14

Can you connect to https://www.themud.org/? Over IPv6? For me it opens a TCP connection and then dies shortly afterwards.

http://www.themud.org/ seems to work well enough, but it redirects to the HTTPS site.

(The DNS servers look iffy, too, by the way. ns1.themud.org's IPv4 address 150.101.219.57 works but not the other 3 IPs.)

15

the site fails because of jquery is trying to go via 443 and the certificate is expired.

Domain only has 2 name servers, not 3 so whats strange can you tell me the ones you are seeing?

16

I was just downloading one URL with curl. The handshake (on IPv6) didn't get far enough for the certificate to be an issue.

It has two hostnames, each with two IP addresses.

ns1.themud.org.  (signed)  84700  A     150.101.219.57
ns1.themud.org.  (signed)  84701  AAAA  2001:44b8:4104:8000::100
ns2.themud.org.  (signed)  84700  A     150.101.219.58
ns2.themud.org.  (signed)  84701  AAAA  2001:44b8:4104:8000::200

Only the first IP is working.

17

Reset all the DNS records will try again later, seems to not have updated when i changed it a week or so ago.

hopefully this was the cause of the issues

18

OK tried one last thing I turned off SSL so only standard http would work and bam no issue,
looks like the certs will not always renew when too far expired.

or its just an amzing coinsidence

1 Like
19

That shouldn't be an issue, in and of itself. Let's Encrypt is tolerant of certificate issues when validating (perhaps ironically). The issue was that connecting to the site didn't work at all.

(I'm not sure what Apache does when an expired certificate causes OCSP stapling to fail, but I think it handles it well enough.)

Connecting to the site is still an issue, in fact.

Turning off the redirect to HTTPS just avoids the HTTPS problem, whatever it is.

1 Like
20

My gut instinct says this was a server-side problem (with @adam-themud's server, not the ACME server) but I don't have any compelling theories :frowning:

It sounds like the thread is resolved now. Very mysterious! I don't understand why the machine was able to reach the directory endpoint reliably except during the renewal operation. Very strange!