Impossible to renew certs if DNS used

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
I can not provide it.

I ran this command:
It is cron or manual renewal
sudo certbot renew --dry-run

It produced this output:

Processing /etc/letsencrypt/renewal/test.tv.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for api.test.tv
http-01 challenge for test.tv
http-01 challenge for www.test.tv
Waiting for verification…
Cleaning up challenges
…

Domain: test.tv
Type: unauthorized
Detail: Invalid response from
https://test.tv/.well-known/acme-challenge/N9mwBk_5P23c2S3kxck3eUv7C1aSNxH3jk-qN1hdNJw
[2606:4700:30::6812:3dee]: “\n\n\n<meta
charset=“utf-8”>\n<meta http-equiv=“X-UA-Compatible”
content=“IE=edge”>\n<meta name=“viewport””

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

  • The following errors were reported by the server:

My web server is (include version):
nginx version: nginx/1.10.3

The operating system my web server runs on is (include version):
debian 9 stretch

My hosting provider, if applicable, is:
abelohost

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
simple shell, ssh console

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 0.28.0

As a result, the site is down already about a week, I noticed it only now.
It seems cron is run only after certs expired, which leads to this.
As everything is set up for SSL and only SLL, cert bot requires some http I think.
But nginx, CDN, caches, firewalls are forbidding it.

Direct link test https://api.test.tv/.well-known/acme-challenge/h348_2Nrr8qQfNvuyVfc42triCwdM2BnLq9R_gCnijg

[2019-10-29 04:25:53] app.ERROR: EXCEPTION /var/www/test_api_prod/vendor/symfony/http-kernel/EventListener/RouterListe
ner.php. Message: No route found for "GET /.well-known/acme-challenge/h348_2Nrr8qQfNvuyVfc42triCwdM2BnLq9R_gCnijg". Cod
e: 0 [] []
2

Well, the one useful detail you provided was the IP address, which points to Cloudflare.

From that we can guess that this is probably a case of Certbot’s nginx authenticator not working with domains that are going through Cloudflare’s proxy (this bug). In that case, the best workaround is generally to use the webroot authenticator instead.

I’m not sure how your thread title (“if DNS used”) relates to what you have posted. Did you want to use certbot-dns-cloudflare rather than HTTP authentication?

3

I do not use cloudflares plugin, should I?
Will plugin work with cron to auto renew?
What webroot authenticator is?
Is it working for cron?

4

You don’t need to, but you could.

If I were you, I would switch to the webroot authenticator to try and avoid the Cloudflare/nginx bug.

Modifying each nginx virtual host to include this block:

location /.well-known/acme-challenge/ {
  root /var/www/letsencrypt;
}

Then:

systemctl reload nginx
mkdir -p /var/www/letsencrypt

and try renewal:

certbot renew -a webroot -w /var/www/letsencrypt --dry-run
5

I do not have /var/www/letsencrypt;

test@srv49167738:~$ whereis letsencrypt
letsencrypt: /usr/bin/letsencrypt /etc/letsencrypt /usr/share/man/man1/letsencrypt.1.gz

6

Yes. It would be a new directory where Let’s Encrypt challenge responses would get stored temporarily.

The idea is that we are telling Certbot to write the responses to a directory rather than trying to automatically modify your nginx configuration, which doesn’t work so well with Cloudflare.

7

I did as you mentioned, added a new location for every hostname which has letsencrypt cert, created folder and restarted nginx, but unfortunately still got same error.(

Processing /etc/letsencrypt/renewal/test.tv-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for adminpgsql4.test.tv
http-01 challenge for api.test.tv
http-01 challenge for test.tv
http-01 challenge for www.test.tv
Using the webroot path /var/www/letsencrypt for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (test.tv-0001) from /etc/letsencrypt/renewal/test.tv-0001.conf produced an unexpected error: Failed authorization procedure. test.tv (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://test.tv/.well-known/acme-challenge/0v-XgYVZbcyR3xahsbMBQQ9y24yXx5MGC7uw2LE-CI0 [2606:4700:30::6812:3dee]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ", api.test.tv (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://api.test\"no-js ", adminpgsql4.test.tv (http-01): urn:ietf:params:acme:error:dns :: DNS problem: NXDOMAIN looking up A for adminpgsql4.test.tv, www.test.tv (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.test.tv/.well-known/acme-challenge/aOSlQmkGdhreWI0IiixiC-9nxR1ydq0MgZ9IwOKJXYw [2606:4700:30::6812:3dee]: "<!DOCTYPE html>\n<!--[if lt IE 7]> <html class=\"no-js ie6 oldie\" lang=\"en-US\"> <![endif]-->\n<!--[if IE 7]>    <html class=\"no-js ". Skipping.


server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2 ipv6only=on;

        server_name api.test.tv;

        root /var/www/test/public;
        index index.html index.php;

        include /etc/letsencrypt/options-ssl-nginx.conf;

        access_log /var/log/nginx/access.test.api.tv.log;
        error_log /var/log/nginx/error.test.api.tv.log;

        location / {
                try_files $uri /index.php$is_args$args;
        }

        location ~ \.php$ {
                include snippets/fastcgi-php.conf;
                fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
        }

        location ~ /\.ht {
                deny all;
        }

        location /.well-known/acme-challenge/ {
                root /var/www/letsencrypt;
        }
}
8

Can you double check your Cloudflare setup? When I visit your real domain, I’m getting a “520 Origin Error”, which means it’s unable to talk to your webserver.

That would prevent what we’re doing from succeeding.

9

What should I check?

123
123.png1136Ă—878 56.4 KB

10

Firewalls disabled, for now, there is no any.

12354
12354.png1116Ă—779 43.5 KB

11

I don’t know what would cause that Cloudflare error. It’s between your origin and Cloudflare.

Here is their suport page for that error: https://community.cloudflare.com/t/community-tip-fixing-error-520-web-server-is-returning-an-unknown-error/44205

Maybe try make a local HTTP request on your server to see that nginx is sending working responses:

curl -X GET -I localhost
curl -X GET -Ik https://localhost

Or maybe taking your SSL setting off “Full (Strict)” could help.

12 
bonbon@debian:~$ curl -X GET -I test.tv
HTTP/1.1 520 Origin Error
Date: Tue, 29 Oct 2019 08:54:37 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
Set-Cookie: __cfduid=d8e56b45f0f80c141b2033fed35d82ce71572339277; expires=Wed, 28-Oct-20 08:54:37 GMT; path=/; domain=.test.tv; HttpOnly
Cache-Control: no-store, no-cache
CF-Cache-Status: DYNAMIC
X-Content-Type-Options: nosniff
Alt-Svc: h3-23=":443"; ma=86400
Set-Cookie: cf_ob_info=520:52d3ed04ad1f8aea:KBP; path=/; expires=Tue, 29-Oct-19 08:55:07 GMT
Set-Cookie: cf_use_ob=80; path=/; expires=Tue, 29-Oct-19 08:55:07 GMT
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Server: cloudflare
CF-RAY: 52d3ed04ad1f8aea-KBP

bonbon@debian:~$ curl -X GET -I https://test.tv
HTTP/2 200 
date: Tue, 29 Oct 2019 08:54:44 GMT
content-type: text/html
set-cookie: __cfduid=df5c98cea1b91cc76178b4995dc42f56b1572339284; expires=Wed, 28-Oct-20 08:54:44 GMT; path=/; domain=.test.tv; HttpOnly
last-modified: Sat, 24 Aug 2019 03:07:08 GMT
strict-transport-security: max-age=15552000; includeSubDomains; preload
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
cf-cache-status: DYNAMIC
alt-svc: h3-23=":443"; ma=86400
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 52d3ed2dda80824d-KBP
13

I used localhost intentionally becuse I don’t want the request to go through Cloudflare.

14

I also tried running this command

sudo certbot renew -a webroot -w /var/www/letsencrypt --dry-run

with DNS SSL Flex and Disabled, same error. Location with letsencrypt new root is saved, nginx restarted long ago.

15

I understand it didn’t work. Right now, your entire site is broken (the 520 Origin Error), not just the SSL renewal.

16

What do you mean? For production site?

17

Yes, the h****.tv site is erroring for me from all locations.

18

Of course, because cert is outdated. but https connection is used, and because of autdated cert it can not renew or open site.

19

From what I saw, your most recent certificate expires in November. And if you set your Cloudflare setting to “Flexible”, then the expiry wouldn’t matter.

Something else is wrong. The actual localhost requests might help.

20

Site was working fine for 3 months. What you want to do with localhost?