Certbot Dry Run Failures on Apache with NGINX - DNS Related

jpetitte · March 29, 2017, 3:15am

Please fill out the fields below so we can help you better.

My domain is:
www.petiteng.com
www.petitteengineering.com

I ran this command:
sudo certbot renew --dry-run
as well as
sudo certbot renew --dry-run --standalone (certbot was originally attempting to stop the nginx server on its own, but I stopped nginx already. Adding --standalone seemed to stop that behavior. I did this to try to make the environment similar to a server that IS working and can renew certs.)

Both commands produced similar output(below).

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/www.petiteng.com.conf

Cert not due for renewal, but simulating renewal for dry run
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.petiteng.com
Waiting for verification…
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/www.petiteng.com.conf produced an unexpected error: Failed authorization procedure. www.petiteng.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to 24.125.106.205:443 for TLS-SNI-01 challenge. Skipping.

Processing /etc/letsencrypt/renewal/www.petitteengineering.com.conf

Cert not due for renewal, but simulating renewal for dry run
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.petitteengineering.com
Waiting for verification…
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/www.petitteengineering.com.conf produced an unexpected error: Failed authorization procedure. www.petitteengineering.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to 24.125.106.205:443 for TLS-SNI-01 challenge. Skipping.
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/www.petiteng.com/fullchain.pem (failure)
/etc/letsencrypt/live/www.petitteengineering.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: www.petiteng.com
Type: connection
Detail: Failed to connect to 24.125.106.205:443 for TLS-SNI-01
challenge

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you’re using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.
The following errors were reported by the server:

Domain: www.petitteengineering.com
Type: connection
Detail: Failed to connect to 24.125.106.205:443 for TLS-SNI-01
challenge

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you’re using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.

My operating system is (include version):
Ubuntu 16.04.2 LTS

My web server is (include version):
nginx version: nginx/1.10.3

My hosting provider, if applicable, is:
I host myself.

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

In an attempt to isolate the problem, I created a duplicate server on the same network (i will call it S2) that runs a dummy “hellow world” response when hit on port 80 (just so I know when it is accessible from outside my network). I forwarded all port 80 and 443 traffic to S2 instead of the problem server, and I pointed a new domain (www.bebopreview.com) to petitteng.mynetgear.com (which is my dynamic DNS hostname I use to ensure my router has a pseudo static IP). S2 worked flawlessly. It obtained a cert, and can renew a cert (dry-run). When I directed www.petitteengineering.com to the dynamic DNS hostname, and changed port forwarding for 80 and 443 to my problem server, I could obtain new certs for www.petiteng.com and www.petitteengineering.com, but the renewal dry run still failed.

When I have nginx running, the www.petitteengineering.com functions properly over ssl. Nothing seems odd.

I’m a bit stumped.

pgporada · March 29, 2017, 4:11am

I can’t hit port 80 or 443 on petitteng.mynetgear.com which resolves to 24.125.106.205. Are you sure your port forwarding and DNS entries are correct?

$ telnet petitteng.mynetgear.com. 80
Trying 24.125.106.205...
^C

$ telnet petitteng.mynetgear.com. 443
Trying 24.125.106.205...
^C

jpetitte · March 29, 2017, 4:53am

That’s a good question.

DNS is configured as follows:

Type	                  Host         Value                                     TTL
CNAME Record	           www        petitteng.mynetgear.com.                   1 min

URL Redirect Record	        @          http://www.petitteengineering.com
Unmasked

For port forwarding, the server has an inet addr:192.168.1.2
And the forwarding entry in the router is:

#         service name        External Start Port      Internal Start Port    internal IP address  
3	            HTTP                   80,443           80,443                     192.168.1.2

I tried a few telnet experiments based on yours (I am not that familiar with telnet…but I gave it a go). The results seem different than yours for some reason.

$ telnet www.petitteengineering.com. 80
Trying 24.125.106.205...
Connected to petitteng.mynetgear.com.
Escape character is '^]'..

$ telnet www.petitteengineering.com. 443
Trying 24.125.106.205...
Connected to petitteng.mynetgear.com.
Escape character is '^]'.

$ telnet petitteengineering.com. 80
Trying 192.64.119.79...
Connected to petitteengineering.com.
Escape character is '^]'.

$ telnet petitteengineering.com. 443
Trying 192.64.119.79...
telnet: Unable to connect to remote host: Connection refused

$ telnet petitteng.mynetgear.com. 80
Trying 24.125.106.205...
Connected to petitteng.mynetgear.com.
Escape character is '^]'.

$ telnet petitteng.mynetgear.com. 443
Trying 24.125.106.205...
Connected to petitteng.mynetgear.com.
Escape character is '^]'.

24.125.106.205 is the IP of my router, but I don’t know what 192.64.119.79 is…

As a side note, the application that I’m running (an instance of the Ghost blogging platform), seems to perform a permanent redirect to the ssl instance (https://www.petitteengineering.com). This may mean nothing, but I figure more details are better.

What do you think?

mnordhoff · March 29, 2017, 5:23am

It's how the "URL Redirect Record" works: They create an A record for their own IP, 192.64.119.79, which runs a web server, which does the HTTP redirect.

By the way, due to how the URL Redirect Record works, you can't get a Let's Encrypt certificate for petitteengineering.com using HTTP-01 or TLS-SNI-01 validation. You can use DNS-01 validation.

(You can use any validation method for www.petitteengineering.com.)

jpetitte · March 29, 2017, 1:44pm

I see. Interesting. Well that clears up a previous question I had about obtaining a cert for petitteengineering.com. I think I should be OK without one for that at the moment.

I was looking through the logs, and it appears that a successful cerbot renew took place at the beginning of February. I’m not sure what changed between then and now so that the operation now fails.

I performed another experiment. I now have 2 domains pointed to petitteng.mynetgear.com. (www.bebopreview.com and www.petitteengineering.com) In my internal network, I directed my router to now forward http traffic first to my test server, and then to my main server. I attempted obtaining a new certificate for both domains on each server (when the router was pointing traffic at it). I was hoping I could have both domains with a certified on the same server, and then test renewal. This, however, did not go as expected. Results summary:

Test server

www.petitteengineering.com failed
www.bebopreview.com was successful

Main server

www.petitteengineering.com was successful
www.bebopreview.com failed

I’m struggling to understand the patterns here. Maybe this is an entirely separate issue from the renewal problem. Am I not allowed to obtain a cert on one server, then immediately get a cert for it on another server?

Details are below

(Test server on same network as main server, 192.168.1.63)

$ sudo certbot certonly --standalone -d www.petitteengineering.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for www.petitteengineering.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.petitteengineering.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to 24.125.106.205:443 for TLS-SNI-01 challenge

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.petitteengineering.com
   Type:   connection
   Detail: Failed to connect to 24.125.106.205:443 for TLS-SNI-01
   challenge

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.



$ sudo certbot certonly --standalone -d www.bebopreview.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for www.bebopreview.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0011_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0011_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/www.bebopreview.com/fullchain.pem. Your cert
   will expire on 2017-06-27. To obtain a new or tweaked version of
   this certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

(Main server that serves the website, 192.168.1.2)

$ sudo certbot certonly --standalone -d www.petitteengineering.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/www.petitteengineering.com.conf)

What would you like to do?
-------------------------------------------------------------------------------
1: Keep the existing certificate for now
2: Renew & replace the cert (limit ~5 per 7 days)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.petitteengineering.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0012_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0012_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/www.petitteengineering.com/fullchain.pem.
   Your cert will expire on 2017-06-27. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le


$ sudo certbot certonly --standalone -d www.bebopreview.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for www.bebopreview.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.bebopreview.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to 24.125.106.205:443 for TLS-SNI-01 challenge

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.bebopreview.com
   Type:   connection
   Detail: Failed to connect to 24.125.106.205:443 for TLS-SNI-01
   challenge

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

schoen · March 30, 2017, 10:16pm

Hi @jpetitte,

HTTP traffic, or HTTPS traffic? The TLS-SNI-01 challenge that you're using relies on port 443 being forwarded to the internal host.

There's no such policy, so something else must be going wrong. Are you sure the port forwarding policies that you're using on your router are exactly consistent? Is the router possibly sometimes inspecting or proxying HTTPS traffic before deciding where to forward it, or does it forward it totally unmodified and uninspected?

jpetitte · March 31, 2017, 3:31am

Hi @schoen!

It’s definitely configured to forward both HTTP and HTTPS. I was having identical failures (as far as I could tell) using http-01 with port 80.

The answer to your last question could be interesting. I’m using a Netgear Nighthawk R7000. I have not heard of these filtering specific traffic, and my assumption was/is that is does not inspect it. I don’t see any settings that indicate this behavior. I’ll probably have to call Netgear and get their opinion.

I called my ISP (Comcast), and they insisted that they are not restricting or modifying traffic, though I don’t know how much to believe them. They suspected it could be the modem, but I haven’t made any changes to that modem between the previous successful certificate renewals and now. The Netgear R7000 has probably received a firmware update (I install them regularly). Maybe that is the culprit…

jpetitte · March 31, 2017, 1:26pm

@schoen

Just a heads up that I updated the firmware this morning, and I found the following in the router documentation:

NAT Filtering. Network Address Translation (NAT) determines how the router processes inbound traffic. Secured NAT protects computers on the LAN from attacks from the Internet, but might prevent some Internet games, point-to-point applications, or multimedia applications from working. Open NAT provides a much less secured firewall, but allows almost all Internet applications to work.

This looks interesting. Once I find some time to experiment, I’ll let you know the results!

system · April 30, 2017, 1:26pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certbot renew --dry-run failed with authentication failure [SOLVED] Help	5	3172	April 19, 2017
Cerbot renewal dry run error Server	9	5204	October 21, 2016
Dry run error -- http-01 challenge times out Server	3	2963	April 28, 2018
Problem on certbot dry run Help	4	736	February 16, 2023
Certbot renew dry run works but not actual renew Help	8	5880	December 17, 2017

Certbot Dry Run Failures on Apache with NGINX - DNS Related

Processing /etc/letsencrypt/renewal/www.petiteng.com.conf

Processing /etc/letsencrypt/renewal/www.petitteengineering.com.conf

Related topics