Issue\renew error:Invalid response 503 During secondary validation

Hi there,

My domain is: orbitaltoday.com

I ran this command: certbot renew -n

When the problem is reproduced, there are no records in the web servers(nginx) access\error-logs from the address(2606:4700:3037::6815:10b8) specified in the error.

You might think that the response "503" returns a Cloudlflare, however, at this point the site is available as usual.

The problem shows up periodically. After some time, i try again renew certificates, and it renewed without any errors:

It produced this output:

Domain: orbitaltoday.com
Type: unauthorized
Detail: During secondary validation: 2606:4700:3037::6815:10b8: Invalid response from https://orbitaltoday.com/.well-known/acme-challenge/tPGHRZ6BV3A-G8d8vx6A1q1nF5TbdQgPzTAuZh-2xR8: 503

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2022-09-26 11:10:25,310:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

My configuration is:

Cloudflare with proxied SSL + Nginx(reverse proxy) + Apache

The operating system my web server runs on is CentOS Linux release 7.9.2009 (Core)

The version of my client is:
certbot 1.11.0

Hello @lmrlmrlmr, welcome to the Let's Encrypt community. :slightly_smiling_face:

Please see New "Service Busy" responses beginning during high load for Let's Encrypt information on HTTP Status code 503

And for reference: 503 Service Unavailable - HTTP | MDN

2 Likes

I don't understand how this applies to my case.

In my case, I get a "503" response, while the site works as usual, while there are no entries in the access-logs and error-logs about requests from the IP address mentioned at certbot error log (2606:4700:3037::6815:10b8)

2 Likes

Oh, my bad.

First link says about 500 responce.

I think, topic could be marked as solved

2 Likes

We aren't serving 503s during high load in production yet (that will be turned on later today, most likely). But I can tell this isn't being served by Let's Encrypt because the detailed error message ("During secondary validation...") is an error that Let's Encrypt returns from its API.

In this case though, it is in fact orbitaltoday.com that is serving a 503, not Let's Encrypt. Unfortunately I don't have any more information than what's already included in the error message. I've looked at our logs and I see you've obtained a cert again, so hopefully it was temporary and you can move on.

Perhaps Cloudflare has some logs that you can view, but I'm not sure about what they have available in their various products.

5 Likes

And a list of issued certificates is here crt.sh | orbitaltoday.com
Latest being today 2022-09-26.

Testing and debugging are best done using the Staging Environment as the Rate Limits are much higher. Rate Limits are per week (rolling).

2 Likes

Adding on to mcpherrinm comment, the 503 is always being returned even for requests not related to Let's Encrypt. You could review the Cloudflare page HERE for suggestions

(many other response headers omitted for readability)

curl -I http://orbitaltoday.com                                             
HTTP/1.1 503 Service Temporarily Unavailable
Date: Mon, 26 Sep 2022 15:25:27 GMT
Server: cloudflare
4 Likes

Presently I get this

$ curl -I http://orbitaltoday.com
HTTP/1.1 301 Moved Permanently
Date: Mon, 26 Sep 2022 15:28:34 GMT
Content-Type: text/html
Connection: keep-alive
Location: https://orbitaltoday.com/
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Strict-Transport-Security: max-age=31536000
Content-Security-Policy: upgrade-insecure-requests
CF-Cache-Status: DYNAMIC
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=y5YbsMCmrNqGkIzUa4TuQx3TJnS46S7pp5ld5uAkkx8E9YwcBIRyevYp5tZmD%2BBhNgTf39F39ZqZizSOnNXeNY3BNWWnLemT%2FGDOwlKQ0qEB7oXTkUWEGex6fWL%2Fh5khtIky"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 750d07b38de08e5d-PDX
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400

$ curl -I https://orbitaltoday.com
HTTP/2 200
date: Mon, 26 Sep 2022 15:28:45 GMT
content-type: text/html; charset=UTF-8
vary: Accept-Encoding
vary: User-Agent
cf-edge-cache: cache,platform=wordpress
link: <https://orbitaltoday.com/wp-json/>; rel="https://api.w.org/"
cache-control: max-age=31557600
expires: Tue, 26 Sep 2023 21:28:44 GMT
x-content-type-options: nosniff
x-xss-protection: 1; mode=block
x-frame-options: SAMEORIGIN
strict-transport-security: max-age=31536000
content-security-policy: upgrade-insecure-requests
cf-cache-status: DYNAMIC
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=qJBwqqNuXkw4TbXv93dOwmzZZb0PPhC%2Bv72UywSumklWn22NRRvX9DrPlFHScjC1Wyqt565KUUo2jdN0MRn%2FblRXfkHf3UlSlMbHm%2BddZL%2FR9BcpA9AFyN4tE804TPXjcNLm"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 750d07f488f6efce-PDX
alt-svc: h3=":443"; ma=86400, h3-29=":443"; ma=86400
3 Likes

Interesting. Because I still see 503 with both http and https. Could there be some sort of geographic block in Cloudflare to prevent US East Coast (me) vs you (US West Coast)?

4 Likes

I am US West Coast (Oregon), I have Comcast Xfinity for an ISP.

And then I just tried this:

$ traceroute orbitaltoday.com
traceroute: Warning: orbitaltoday.com has multiple addresses; using 172.67.215.74
traceroute to orbitaltoday.com (172.67.215.74), 64 hops max, 40 byte packets
 1  EdgeRouter-4 (192.168.1.1)  0.377 ms  0.306 ms  0.303 ms
 2  96.120.60.137 (96.120.60.137)  8.04 ms  8.031 ms  8.809 ms
 3  162.151.125.157 (162.151.125.157)  10.757 ms  8.753 ms  8.846 ms
 4  ae-2-rur02.beaverton.or.bverton.comcast.net (68.85.243.154)  8.652 ms  7.068 ms  9.604 ms
 5  96.216.60.245 (96.216.60.245)  8.111 ms  8.122 ms  10.954 ms
 6  68.85.243.197 (68.85.243.197)  20.616 ms  15.939 ms  14.542 ms
 7  69.252.236.134 (69.252.236.134)  10.756 ms  13.785 ms  20.365 ms
 8  172.67.215.74 (172.67.215.74)  10.835 ms  9.478 ms  9.99 ms

This is the interesting message from above Warning: orbitaltoday.com has multiple addresses
certainly not a bad thing but helpful for debugging.

2 Likes

Yes, I could tell from the Cloudflare response header. The "-PDX" is the Cloudflare Edge Server ID (usually airport)

3 Likes

That is standard for Cloudflare CDN (and many other CDNs)

4 Likes

Right but different endpoints might be having different responses.

2 Likes

Ah, I see. That would be a significant failure in the Cloudflare Edge servers to behave that way. I think that would be front page news :slight_smile:

The 503's are almost certainly some kind of origin server misconfiguration or perhaps other security setting in the CDN. That's something best resolved by working with Cloudflare community (which I previously linked to).

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.