Certbot renew with nginx module - returns error 404 for challenge response

Help
1

Hello,
We are running certbot script on Debian 8 server with nginx, using the nginx auto renewal module.
Currently managing 705 domains on a single server.
certbot is running in cron but for some of the domains it return error 404 when trying to renew a certificate on a challenge response stage.

When manually testing the URL, for instance:
http://domain.com/.well-known/acme-challenge/cC_xDfk5yr92dDOkPbFnqVta3zaSB5FGU-asspQrdFY
it works but the script thinks it isn’t.

I assume it’s checking the URL too early before the nginx module creates it due to big number of domains maybe or other bug.

Is it possible to set a retry to sleep (delay) inside the challenge response test?

Meanwhile we are just relaunching the certbot several times until it renews the certificate, sometimes we need to restart nginx to make it work.

I ran this command:
/home/letsencrypt/certbot-auto renew

It produced this output:
IMPORTANT NOTES:

My web server is (include version):
nginx version: nginx/1.15.9

The operating system my web server runs on is (include version):
Debian GNU/Linux 8

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
/home/letsencrypt/certbot-auto --version
certbot 0.31.0

2

{
“identifier”: {
“type”: “dns”,
“value”: “www.bioovit.info”
},
“status”: “invalid”,
“expires”: “2019-03-10T14:44:22Z”,
“challenges”: [
{
“type”: “tls-alpn-01”,
“status”: “invalid”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/challenge/rmvuQfzG2fiU-uiX7yDufWq2snSIJkBcxOALgZWBYyE/13196648288”,
“token”: “3dc5_KnE4Dp9i1Gzabsh2T3bWNROfx3LOl4UZx40NP4”
},
{
“type”: “http-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Invalid response from http://www.bioovit.info/.well-known/acme-challenge/kLY4hYWg0IabGLLS8x0eVcylQCIy_jevQaRjm8Ux9Fg [185.18.205.221]: “\u003c!DOCTYPE HTML PUBLIC \”-//IETF//DTD HTML 2.0//EN\”\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp"",
“status”: 403
},
“url”: “https://acme-v02.api.letsencrypt.org/acme/challenge/rmvuQfzG2fiU-uiX7yDufWq2snSIJkBcxOALgZWBYyE/13196648289”,
“token”: “kLY4hYWg0IabGLLS8x0eVcylQCIy_jevQaRjm8Ux9Fg”,
“validationRecord”: [
{
“url”: “http://www.bioovit.info/.well-known/acme-challenge/kLY4hYWg0IabGLLS8x0eVcylQCIy_jevQaRjm8Ux9Fg”,
“hostname”: “www.bioovit.info”,
“port”: “80”,
“addressesResolved”: [
“185.18.205.221”
],
“addressUsed”: “185.18.205.221”
}
]
},
{
“type”: “dns-01”,
“status”: “invalid”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/challenge/rmvuQfzG2fiU-uiX7yDufWq2snSIJkBcxOALgZWBYyE/13196648291”,
“token”: “_x3sgzG_ETh39MU4on0hr3N8KsDS65_fa2gGKiCqSmk”
}
]
}
2019-03-03 16:44:50,824:DEBUG:acme.client:Storing nonce: xGFtnBpvJaq-GhN06vg6t0jkb6fPKNtdWPAxsHu1srY
2019-03-03 16:44:50,828:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: www.bioovit.info
Type: unauthorized
Detail: Invalid response from http://www.bioovit.info/.well-known/acme-challenge/kLY4hYWg0IabGLLS8x0eVcylQCIy_jevQaRjm8Ux9Fg [185.18.205.221]: “\n\n404 Not Found\n\n

Not Found

\n<p”

Domain: bioovit.info
Type: unauthorized
Detail: Invalid response from http://bioovit.info/.well-known/acme-challenge/LKe4J2O14nso9lO3avRE3AM_ICbb5p_1pKdxvwu4aes [185.18.205.221]: “\n\n404 Not Found\n\n

Not Found

\n<p”

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2019-03-03 16:44:50,830:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 168, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 239, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. www.bioovit.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.bioovit.info/.well-known/acme-challenge/kLY4hYWg0IabGLLS8x0eVcylQCIy_jevQaRjm8Ux9Fg [185.18.205.221]: “\n\n404 Not Found\n\n

Not Found

\n<p”, bioovit.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://bioovit.info/.well-known/acme-challenge/LKe4J2O14nso9lO3avRE3AM_ICbb5p_1pKdxvwu4aes [185.18.205.221]: “\n\n404 Not Found\n\n

Not Found

\n<p”

2019-03-03 16:44:50,831:DEBUG:certbot.error_handler:Calling registered functions
2019-03-03 16:44:50,831:INFO:certbot.auth_handler:Cleaning up challenges
2019-03-03 16:45:35,590:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/bin/letsencrypt”, line 11, in
sys.exit(main())
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 1365, in main
return config.func(config, plugins)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 1119, in run
certname, lineage)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 121, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py”, line 410, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py”, line 353, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py”, line 389, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 168, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 239, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. www.bioovit.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.bioovit.info/.well-known/acme-challenge/kLY4hYWg0IabGLLS8x0eVcylQCIy_jevQaRjm8Ux9Fg [185.18.205.221]: “\n\n404 Not Found\n\n

Not Found

\n<p”, bioovit.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://bioovit.info/.well-known/acme-challenge/LKe4J2O14nso9lO3avRE3AM_ICbb5p_1pKdxvwu4aes [185.18.205.221]: “\n\n404 Not Found\n\n

Not Found

\n<p”
3

image
image.png1492×280 43.5 KB

4

Well, that's an interesting theory.

Certbot, after injecting a location {} block for the challenge response, basically does this, which sends SIGHUP to nginx:

nginx -s reload

I haven't looked deeply into it, but I'm pretty sure that sending a signal is asynchronous - the above command doesn't wait around for nginx to actually reload the config and re-fork the worker processes.

So it sounds like a viable explanation to me.

One way you could try to confirm it is by running Certbot with --debug-challenges, which basically pauses the Certbot process immediately after it makes the alterations to the nginx configuration. It would give you an opportunity to see whether an extra 5 second delay or whatever, completely eliminates the issue.

If that fixes it, I would then suggest opening a bug for this - Issues · certbot/certbot · GitHub

5

Just one thing to add. If you have a very large number of domains, it could be worth just setting up the challenge response statically, as shown in this wiki: https://github.com/Neilpang/acme.sh/wiki/Stateless-Mode

You would have to adapt it for Certbot, but avoiding graceful reloads (or 700+ touches to the filesystem for webroot authenticator) might be a good idea for a large deployment.

Adapting it should be fairly easy, once you have the static response implemented in nginx, you would be able to renew using a no-operation manual auth hook like:

-a manual --manual-auth-hook "/bin/true" --manual-cleanup-hook "/bin/true"

or something.

2 Likes
6

Hello,

It’s better with the debug flag but yet not perfect:

Processing /etc/letsencrypt/renewal/www.djozbatish.info.conf

Cert is due for renewal, auto-renewing…

Plugins selected: Authenticator nginx, Installer nginx

Renewing an existing certificate

Performing the following challenges:

http-01 challenge for djozbatish.info

http-01 challenge for www.djozbatish.info

Using default address 80 for authentication.

Using default address 80 for authentication.

Waiting for verification…

Challenges loaded. Press continue to submit to CA. Pass “-v” for more info about

challenges.

Cleaning up challenges

Attempting to renew cert (www.djozbatish.info) from /etc/letsencrypt/renewal/www.djozbatish.info.conf produced an unexpected error: Failed authorization procedure. djozbatish.info (http-01): urn:ietf:params:acme:error:unauthorized :: The
client lacks sufficient authorization :: Invalid response from http://djozbatish.info/.well-known/acme-challenge/kcin4JvBDvxBzJbrvrL5-UAmXBUGzyk6wx4UlSlBBBU [185.18.205.221]: “\n\n404 Not
Found\n\n

Not Found

\n<p”, www.djozbatish.info (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.djozbatish.info/.well-known/acme-challenge/dOE3voeGsCIohsHKbPDL6wmwcd0eTeBIe6lvgoaDmuw
[185.18.205.221]: “\n\n404 Not Found\n\n

Not Found

\n<p”. Skipping.
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.