Issue with certbot automatically renewing certificates

Hi

I am having issues with using certbot-auto on my server (nginx, ubuntu, details below) for renewing certificates, as it fails in verifying the domain (details below).

  • I can do the renewal properly by verifying my domain manually.
  • I have also copied the nginx config modified by certbot-auto while it was trying to verify the domain, and then went through the manual process using the same config (with modified challenge tokens), and that works as well.

This is the modification certbot makes to the nginx config during the challenge:

server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot

        listen 80;
        server_name bf.ecomply.io;

location = /.well-known/acme-challenge/<challenge_url>{default_type text/plain;return 200 <challenge_token>;} # managed by Certbot

}

The issue seems to be that certbot-auto tries to fetch the proper response from https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url> while the auto-generated modified config serves the appropriate challenge token on https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url>.

Is this a bug in latest version of certbot (or the nginx plugin)? Or am I doing something wrong?

My domain is:

bf.ecomply.io

I ran this command:

sudo certbot

It produced this output:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for bf.ecomply.io
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. bf.ecomply.io (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url> [82.165.97.140]: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><cen"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: bf.ecomply.io
   Type:   unauthorized
   Detail: Invalid response from
   https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url>
   [82.165.97.140]: "<html>\r\n<head><title>502 Bad
   Gateway</title></head>\r\n<body
   bgcolor=\"white\">\r\n<center><h1>502 Bad
   Gateway</h1></center>\r\n<hr><cen"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version):

nginx/1.10.3 (Ubuntu)

The operating system my web server runs on is (include version):

Ubuntu 16.04.6 LTS

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

certbot 0.31.0

Hi @loreanvictor

your authenticator isn’t visible.

What’s the content of the config file in

/etc/letsencrypt/renewal

The error:

http is redirected to https, there is a bad gateway.

But checking your domain there is no redirect ( https://check-your-website.server-daten.de/?q=bf.ecomply.io ):

Domainname Http-Status redirect Sec. G
• http://bf.ecomply.io/
82.165.97.140 200 0.053 H
• https://bf.ecomply.io/
82.165.97.140 502 3.540 S
Bad Gateway
• http://bf.ecomply.io/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
82.165.97.140 404 0.047 A
Not Found
Visible Content: 404 Not Found nginx

But a bad gateway https.

Looks like the standard handling of --nginx doesn’t work. Or you use already another authenticator.

There is no file bf.ecomply.io in the aforementioned folder.

Yes seems to be the case. Curious thing is I can interrupt the automatic process, and continue with doing it manually, and everything works. i.e. I can initiate the process, copy the modified nginx config (or just the modified segment I put in the original post), then do manual verification using that bit, and then verification works fine as the request is not redirected to https then.

Another curious thing that happens is that if I do as described, then renewing the certificates of a domain through the automatic process starts to work properly again.

P.S. I’ve had this issue on multiple domains on the same server, and I’ve fixed it for them via the aforementioned process. However I intentionally did not fix bf.ecomply.io to keep it as a viable test case for the aforementioned behavior.

Do you have any guesses as to why this is happening? I know I can renew certificates manually, however that takes way more time and energy for multiple domains, and since the automatic process always worked before, I am wondering what has happened here.

There is no newer certificate ( https://check-your-website.server-daten.de/?q=bf.ecomply.io#ct-logs ):

Last certificate is from 2019-05-10, there is no newer certificate visible.

If you use the standard nginx, a redirect is skipped because of the code certbot inserts. But your error message shows https, that’s not possible. Use --webroot instead.

that doesn’t seem correct. for example this is within the output of sudo certbot certificates:

  Certificate Name: lapras.ecomply.io
    Domains: lapras.ecomply.io
    Expiry Date: 2019-10-23 09:26:27+00:00 (VALID: 83 days)
    Certificate Path: /etc/letsencrypt/live/lapras.ecomply.io/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/lapras.ecomply.io/privkey.pem

considering the usual 90days of validation, this matches the date I manually renewed the certificate on the domain.

That’s what I meant. I can use certbot --manual, and use the code snippet that certbot inserts into nginx configs to properly conduct the verification. But for whatever reason certbot is unable to do this on its own.

Anyways --webroot also is not efficient. I suspect if this happens again I will add a script to gloss over the steps in the process that certbot fails at. Though still cannot understand why that is, i.e. is it a certbot bug or server misconfiguration.

That’s

another domain name.

Please check that domain name with the online tool to see, if there are certificates via CT-logs visible.

Ok sorry for the miscommunication. To clarify:

  • Yes this is another domain name. It had the same issue, which I could work-around via described method (using certbot --manual and using the nginx config snippet inserted by certbot for verification)
  • bf.ecomply.io is another domain name on the same machine with the issue. I purposefully did not utilize the work-around I used for lapras.ecomply.io for it to be able to study the issue further.

Did you mean http:// in the second case?

The Let’s Encrypt validator bot always starts with the http:// version, but is willing to follow a redirect to https:// if one is encountered. Do you have anything that would be generating such a redirect?

Could you share the Certbot output from a failed renewal attempt?

1 Like

yes, sorry for the typo.

Not one that I could find. I checked all the nginx configs, and even tried manually verifying by just adding the code-snippet certbot generates in nginx configs, and succeeded, which should mean that there is no such redirect (or that in manual verification certbot DOES NOT follow redirects to https but in automatic verification it does, can that be the case?):

server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot

        listen 80;
        server_name bf.ecomply.io;

location = /.well-known/acme-challenge/<challenge_url>{default_type text/plain;return 200 <challenge_token>;} # managed by Certbot

}

sure:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for bf.ecomply.io
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. bf.ecomply.io (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url> [82.165.97.140]: "<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><cen"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: bf.ecomply.io
   Type:   unauthorized
   Detail: Invalid response from
   https://bf.ecomply.io/.well-known/acme-challenge/<challenge_url>
   [82.165.97.140]: "<html>\r\n<head><title>502 Bad
   Gateway</title></head>\r\n<body
   bgcolor=\"white\">\r\n<center><h1>502 Bad
   Gateway</h1></center>\r\n<hr><cen"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
1 Like

You may have a .htaccess or an application specific redirect.

And you use a proxy configuration, that doesn’t work. So fix your “bad gateway”.

1 Like

I couldn’t find any of those.

Isn’t this the proxied server not responding properly? Anyhow I highly suspect it does not directly relate to the issue at hand since in that case it should have not been working with manual verification using certbot’s auto-generated nginx configs.

Certbot isn’t the one verifying the challenges—that’s the Let’s Encrypt bot, which doesn’t run on your computer but in Let’s Encrypt datacenters. They don’t know¹ what kind of Certbot verification mode or method you’re using.

Do you have web server logs and proxy logs related to these requests?

¹ Certbot does tell the Let’s Encrypt service that it’s Certbot and what authenticator it’s running with, for debugging and statistical purposes, but Let’s Encrypt doesn’t routinely use this information for any purpose and it doesn’t affect how challenges are validated.

1 Like

There is no actual proxy server behind this. We took off the actual proxy a while back (thats why I’m using this domain for testing this issue to begin with). But I ran certbot again on the domain and these are nginx access and error logs respectively:

# Access logs
18.197.227.110 - - [02/Aug/2019:11:28:56 +0200] "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1" 404 136 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.14.255.131 - - [02/Aug/2019:11:28:56 +0200] "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1" 404 136 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
34.222.229.130 - - [02/Aug/2019:11:28:57 +0200] "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1" 404 136 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [02/Aug/2019:11:28:57 +0200] "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1" 404 136 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
# Error logs
2019/08/02 11:28:56 [error] 9881#9881: *52330 open() "/usr/share/nginx/html/.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI" failed (2: No such file or directory), client: 3.14.255.131, server: baylda-fragebogen.ecomply.io, request: "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1", host: "bf.ecomply.io"
2019/08/02 11:28:56 [error] 9881#9881: *52331 open() "/usr/share/nginx/html/.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI" failed (2: No such file or directory), client: 18.197.227.110, server: baylda-fragebogen.ecomply.io, request: "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1", host: "bf.ecomply.io"
2019/08/02 11:28:56 [error] 9881#9881: *52332 open() "/usr/share/nginx/html/.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI" failed (2: No such file or directory), client: 34.222.229.130, server: baylda-fragebogen.ecomply.io, request: "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1", host: "bf.ecomply.io"
2019/08/02 11:28:57 [error] 9881#9881: *52333 open() "/usr/share/nginx/html/.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI" failed (2: No such file or directory), client: 66.133.109.36, server: baylda-fragebogen.ecomply.io, request: "GET /.well-known/acme-challenge/9VeqzLSNEuCrMXM3Z_BYLi74gq1PQiaOfQ5yypK48XI HTTP/1.1", host: "bf.ecomply.io"

Seems like nginx is trying to fallback on some internal default configuration (I couldn’t find the address /usr/share/nginx anywhere in actual nginx configuration files) as none of its rules apply to the incoming request.

1 Like

That’s interesting.

Earlier you wrote

  • I have also copied the nginx config modified by certbot-auto while it was trying to verify the domain, and then went through the manual process using the same config (with modified challenge tokens), and that works as well.

How did you get this configuration, and then how did you integrate it into your setup during the manual verification?

1 Like

Well basically I ran certbot as usual (sudo certbot). While verification (which usually takes a few seconds) I copied the nginx config (e.g. /etc/nginx/sites-available/bf.ecomply.io), which during that period bears this additional segment:

server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot

        listen 80;
        server_name bf.ecomply.io;

location = /.well-known/acme-challenge/<challenge_url>{default_type text/plain;return 200 <challenge_token>;} # managed by Certbot

}

then when the verification fails, I would run manual verification, and when certbot is waiting me to place the proper token on the proper path on my webserver, I would simply paste this snippet back into the same nginx config file, replacing the path and the token with the new values that certbot has provided.

1 Like

Could we see the whole output of sudo certbot certificates?

I’m also kind of confused that you’re running sudo certbot rather than sudo certbot renew in order to renew your certificates. Does the latter work any better for you, perhaps?

1 Like

sure thing:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: ecomply.io
    Domains: ecomply.io baylda-fragebogen.ecomply.io bf.ecomply.io bulb.ecomply.io charizard.ecomply.io deploy-snorlax.ecomply.io lapras.ecomply.io mystic.ecomply.io snorlax.ecomply.io www.ecomply.io zeppelin.ecomply.io
    Expiry Date: 2019-08-08 10:45:53+00:00 (VALID: 2 days)
    Certificate Path: /etc/letsencrypt/live/ecomply.io/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/ecomply.io/privkey.pem
  Certificate Name: lapras.ecomply.io
    Domains: lapras.ecomply.io
    Expiry Date: 2019-10-23 09:26:27+00:00 (VALID: 77 days)
    Certificate Path: /etc/letsencrypt/live/lapras.ecomply.io/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/lapras.ecomply.io/privkey.pem
  Certificate Name: mickey.ecomply.io
    Domains: mickey.ecomply.io
    Expiry Date: 2019-10-27 16:53:03+00:00 (VALID: 82 days)
    Certificate Path: /etc/letsencrypt/live/mickey.ecomply.io/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mickey.ecomply.io/privkey.pem
  Certificate Name: mystic.ecomply.io
    Domains: mystic.ecomply.io
    Expiry Date: 2019-09-03 23:26:36+00:00 (VALID: 28 days)
    Certificate Path: /etc/letsencrypt/live/mystic.ecomply.io/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mystic.ecomply.io/privkey.pem
  Certificate Name: zeppelin.ecomply.io
    Domains: zeppelin.ecomply.io
    Expiry Date: 2019-07-30 13:05:16+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/zeppelin.ecomply.io/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/zeppelin.ecomply.io/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

that is mostly because the domains expire at different times and certbot renew, as it stands, does not work with renewal of certificates using domain names:

Currently, the renew verb is capable of either renewing all installed certificates that are due to be renewed or renewing a single certificate specified by its name. If you would like to renew specific certificates by their domains, use the certonly command instead. The renew verb may provide other options for selecting certificates to renew in the future.

renewing all domains at once is also not a good idea for my case since unforeseen issues during renewal process can lead to problematic certificates for domains that did not need a renewal in the first place (though I suspect the worst-case scenario should be certificates simply not being renewed, no piece of software has predictable behavior 100% of the time so its better be safe than sorry).

1 Like

@joohoi, can you understand why certbot --nginx would fail to renew but manually copying the configuration file that it creates back into the nginx configuration would succeed with certbot --manual?

1 Like

ok I just did the same operation on a couple of other domains on the same server with the same process and issues. I realized that for domain x.ecomply.io, going through the process as described above results in valid certificates in /etc/letsencrypt/live/x.ecomply.io. however, when editing the nginx config after the verification process (mainly to get the verification snippet out), I realized that the certificate was configured to be read from /etc/letsencrypt/ecomply.io beforehand (which I had to fix).

additionally, I noticed that while sudo certbot reads the nginx configs to list the domains and act accordingly, sudo certbot certonly --manual asks for the domain. is it possible that the issue is with certbot --nginx trying to read the nginx config and subsequently doing something according to the certificates located in /etc/letsencrypt/live/ecomply.io, which causes it to fail, while certbot --manual ignores all that and subsequently succeeds?

p.s. the content of /etc/letsencrypt/live/ecomply.io is also generated by certbot --nginx, though perhaps a relatively older version.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.