ReadTimeout: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)

[saravanan]

We getting the below error message while creating a certificate. This is occurring regularly. We are unable to provide certificates to customers. please look into it ASAP

root@84d448e12737:/# certbot certonly -n -d san1xxxxxx.com -d san1-d1.xxxxxx,san1-d2.xxxxxx.com --apache --agree-tos -m cloud@xxxxxx.com --noninteractive --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for san1-d1.xxxx.com
http-01 challenge for san1-d2.xxxx.com
http-01 challenge for san1.xxxx.com
Enabled Apache rewrite module
Waiting for verification...
Cleaning up challenges
An unexpected error occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
response.begin()
File "/usr/lib/python3.8/http/client.py", line 307, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.8/socket.py", line 669, in readinto
return self._sock.recv_into(b)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 326, in recv_into
raise timeout("The read operation timed out")
socket.timeout: The read operation timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 400, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3/dist-packages/six.py", line 703, in reraise
raise value
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 423, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 330, in _raise_timeout
raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)
During handling of the above exception, another exception occurred:
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)
Please see the logfiles in /var/log/letsencrypt for more details.
IMPORTANT NOTES:
- Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.

[griffin]

Hello

We've been seeing these errors somewhat frequently lately. Some have suggested that they can be related to a nameserver problem. I'm going to ask someone for their specific input though because I want a definitive answer.

@_az

I elevated this to the LE staff once I had confirmed it was an LE server issue and nothing specifically certbot. This topic says it all.

@lestaff

Sorry for pinging you all, but it has now reached outbreak level. Here are several other recent occurrences.

Browsers not picking up new SSL certificate

v…

Dry-run error messages

i…

Cannot renew my SSL Certificate

P…

[Osiris]

https://letsencrypt.status.io/ is not showing any maintainance in any case.

I'm getting timeouts too

[griffin]

That confirms it then. It's not just some random nameserver issue.

[Osiris]

It's very flaky, it's working now again.

acme-staging-v02.api.letsencrypt.org. 7200 IN CNAME staging.api.letsencrypt.org.
staging.api.letsencrypt.org. 300 IN     CNAME   56a5f4b0bc8146689ec3e272c43525f9.pacloudflare.com.

I didn't realise the Let's Encrypt servers are behind CloudFlare? I thought LE was using a different CDN.

Perhaps not the best choice of action

[griffin]

That's... interesting.

[mbanker]

we noticed the timeouts as well, starting this past friday, but they disappeared by friday night. reappeared this morning, though

[jillian]

Let's Encrypt is aware of this issue and has started investigating. We opened a status page incident and will update when possible. The errors are intermittent and we still see lots of successful requests.

[griffin]

Isn't v2 staging the most affected? I see v1 and v2 production and v1 staging marked as degraded, but not v2 staging.

Fixed.

https://letsencrypt.status.io/

[jillian]

Thanks for catching that- looks like a simple forgotten checkbox when quickly making the status. The affected components on the status page is updated to include Staging V2.

[griffin]

Still showing the same down lower.

Back in business with a watchful eye.

https://letsencrypt.status.io/

[jillian]

Thanks for your patience. The Let's Encrypt team has implemented a fix and is monitoring. The status page reflects this current state.

[griffin]

Well done, staff!

Again, sorry for pinging you guys on a Sunday. Given that it was the staging environment that was flaking, I really didn't think this would get attention until tomorrow. Regardless, my hat's off to you!

Bonus:

Ooo! A testimonial!

@edwardsmarkf

regarding your latest inquiry, you are absolutely correct. 100% success this time!