Certbot Dry Run Failures on Apache with NGINX - DNS Related


#1

Please fill out the fields below so we can help you better.

My domain is:
www.petiteng.com
www.petitteengineering.com

I ran this command:
sudo certbot renew --dry-run
as well as
sudo certbot renew --dry-run --standalone (certbot was originally attempting to stop the nginx server on its own, but I stopped nginx already. Adding --standalone seemed to stop that behavior. I did this to try to make the environment similar to a server that IS working and can renew certs.)

Both commands produced similar output(below).

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/www.petiteng.com.conf

Cert not due for renewal, but simulating renewal for dry run
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.petiteng.com
Waiting for verification…
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/www.petiteng.com.conf produced an unexpected error: Failed authorization procedure. www.petiteng.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to 24.125.106.205:443 for TLS-SNI-01 challenge. Skipping.


Processing /etc/letsencrypt/renewal/www.petitteengineering.com.conf

Cert not due for renewal, but simulating renewal for dry run
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.petitteengineering.com
Waiting for verification…
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/www.petitteengineering.com.conf produced an unexpected error: Failed authorization procedure. www.petitteengineering.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to 24.125.106.205:443 for TLS-SNI-01 challenge. Skipping.
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/www.petiteng.com/fullchain.pem (failure)
/etc/letsencrypt/live/www.petitteengineering.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: www.petiteng.com
    Type: connection
    Detail: Failed to connect to 24.125.106.205:443 for TLS-SNI-01
    challenge

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

  • The following errors were reported by the server:

    Domain: www.petitteengineering.com
    Type: connection
    Detail: Failed to connect to 24.125.106.205:443 for TLS-SNI-01
    challenge

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

My operating system is (include version):
Ubuntu 16.04.2 LTS

My web server is (include version):
nginx version: nginx/1.10.3

My hosting provider, if applicable, is:
I host myself.

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

In an attempt to isolate the problem, I created a duplicate server on the same network (i will call it S2) that runs a dummy “hellow world” response when hit on port 80 (just so I know when it is accessible from outside my network). I forwarded all port 80 and 443 traffic to S2 instead of the problem server, and I pointed a new domain (www.bebopreview.com) to petitteng.mynetgear.com (which is my dynamic DNS hostname I use to ensure my router has a pseudo static IP). S2 worked flawlessly. It obtained a cert, and can renew a cert (dry-run). When I directed www.petitteengineering.com to the dynamic DNS hostname, and changed port forwarding for 80 and 443 to my problem server, I could obtain new certs for www.petiteng.com and www.petitteengineering.com, but the renewal dry run still failed.

When I have nginx running, the www.petitteengineering.com functions properly over ssl. Nothing seems odd.

I’m a bit stumped.


#2

I can’t hit port 80 or 443 on petitteng.mynetgear.com which resolves to 24.125.106.205. Are you sure your port forwarding and DNS entries are correct?

$ telnet petitteng.mynetgear.com. 80
Trying 24.125.106.205...
^C

$ telnet petitteng.mynetgear.com. 443
Trying 24.125.106.205...
^C

#3

That’s a good question.

DNS is configured as follows:

Type	                  Host         Value                                     TTL
CNAME Record	           www        petitteng.mynetgear.com.                   1 min

URL Redirect Record	        @          http://www.petitteengineering.com
Unmasked

For port forwarding, the server has an inet addr:192.168.1.2
And the forwarding entry in the router is:

#         service name        External Start Port      Internal Start Port    internal IP address  
3	            HTTP                   80,443           80,443                     192.168.1.2

I tried a few telnet experiments based on yours (I am not that familiar with telnet…but I gave it a go). The results seem different than yours for some reason.

$ telnet www.petitteengineering.com. 80
Trying 24.125.106.205...
Connected to petitteng.mynetgear.com.
Escape character is '^]'..

$ telnet www.petitteengineering.com. 443
Trying 24.125.106.205...
Connected to petitteng.mynetgear.com.
Escape character is '^]'.

$ telnet petitteengineering.com. 80
Trying 192.64.119.79...
Connected to petitteengineering.com.
Escape character is '^]'.

$ telnet petitteengineering.com. 443
Trying 192.64.119.79...
telnet: Unable to connect to remote host: Connection refused

$ telnet petitteng.mynetgear.com. 80
Trying 24.125.106.205...
Connected to petitteng.mynetgear.com.
Escape character is '^]'.

$ telnet petitteng.mynetgear.com. 443
Trying 24.125.106.205...
Connected to petitteng.mynetgear.com.
Escape character is '^]'.

24.125.106.205 is the IP of my router, but I don’t know what 192.64.119.79 is…

As a side note, the application that I’m running (an instance of the Ghost blogging platform), seems to perform a permanent redirect to the ssl instance (https://www.petitteengineering.com). This may mean nothing, but I figure more details are better.

What do you think?


#4

It’s how the “URL Redirect Record” works: They create an A record for their own IP, 192.64.119.79, which runs a web server, which does the HTTP redirect.

By the way, due to how the URL Redirect Record works, you can’t get a Let’s Encrypt certificate for petitteengineering.com using HTTP-01 or TLS-SNI-01 validation. You can use DNS-01 validation.

(You can use any validation method for www.petitteengineering.com.)


#5

I see. Interesting. Well that clears up a previous question I had about obtaining a cert for petitteengineering.com. I think I should be OK without one for that at the moment.

I was looking through the logs, and it appears that a successful cerbot renew took place at the beginning of February. I’m not sure what changed between then and now so that the operation now fails.

I performed another experiment. I now have 2 domains pointed to petitteng.mynetgear.com. (www.bebopreview.com and www.petitteengineering.com) In my internal network, I directed my router to now forward http traffic first to my test server, and then to my main server. I attempted obtaining a new certificate for both domains on each server (when the router was pointing traffic at it). I was hoping I could have both domains with a certified on the same server, and then test renewal. This, however, did not go as expected. Results summary:

Test server

Main server

I’m struggling to understand the patterns here. Maybe this is an entirely separate issue from the renewal problem. Am I not allowed to obtain a cert on one server, then immediately get a cert for it on another server?

Details are below

(Test server on same network as main server, 192.168.1.63)

$ sudo certbot certonly --standalone -d www.petitteengineering.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for www.petitteengineering.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.petitteengineering.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to 24.125.106.205:443 for TLS-SNI-01 challenge

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.petitteengineering.com
   Type:   connection
   Detail: Failed to connect to 24.125.106.205:443 for TLS-SNI-01
   challenge

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.



$ sudo certbot certonly --standalone -d www.bebopreview.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for www.bebopreview.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0011_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0011_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/www.bebopreview.com/fullchain.pem. Your cert
   will expire on 2017-06-27. To obtain a new or tweaked version of
   this certificate in the future, simply run certbot again. To
   non-interactively renew *all* of your certificates, run "certbot
   renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le

(Main server that serves the website, 192.168.1.2)

$ sudo certbot certonly --standalone -d www.petitteengineering.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/www.petitteengineering.com.conf)

What would you like to do?
-------------------------------------------------------------------------------
1: Keep the existing certificate for now
2: Renew & replace the cert (limit ~5 per 7 days)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.petitteengineering.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0012_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0012_csr-certbot.pem

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at
   /etc/letsencrypt/live/www.petitteengineering.com/fullchain.pem.
   Your cert will expire on 2017-06-27. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"
 - If you like Certbot, please consider supporting our work by:

   Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
   Donating to EFF:                    https://eff.org/donate-le


$ sudo certbot certonly --standalone -d www.bebopreview.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for www.bebopreview.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.bebopreview.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to 24.125.106.205:443 for TLS-SNI-01 challenge

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.bebopreview.com
   Type:   connection
   Detail: Failed to connect to 24.125.106.205:443 for TLS-SNI-01
   challenge

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

#6

Hi @jpetitte,

HTTP traffic, or HTTPS traffic? The TLS-SNI-01 challenge that you’re using relies on port 443 being forwarded to the internal host.

There’s no such policy, so something else must be going wrong. Are you sure the port forwarding policies that you’re using on your router are exactly consistent? Is the router possibly sometimes inspecting or proxying HTTPS traffic before deciding where to forward it, or does it forward it totally unmodified and uninspected?


#7

Hi @schoen!

It’s definitely configured to forward both HTTP and HTTPS. I was having identical failures (as far as I could tell) using http-01 with port 80.

The answer to your last question could be interesting. I’m using a Netgear Nighthawk R7000. I have not heard of these filtering specific traffic, and my assumption was/is that is does not inspect it. I don’t see any settings that indicate this behavior. I’ll probably have to call Netgear and get their opinion.

I called my ISP (Comcast), and they insisted that they are not restricting or modifying traffic, though I don’t know how much to believe them. They suspected it could be the modem, but I haven’t made any changes to that modem between the previous successful certificate renewals and now. The Netgear R7000 has probably received a firmware update (I install them regularly). Maybe that is the culprit…


#8

@schoen

Just a heads up that I updated the firmware this morning, and I found the following in the router documentation:

NAT Filtering. Network Address Translation (NAT) determines how the router processes inbound traffic. Secured NAT protects computers on the LAN from attacks from the Internet, but might prevent some Internet games, point-to-point applications, or multimedia applications from working. Open NAT provides a much less secured firewall, but allows almost all Internet applications to work.

This looks interesting. Once I find some time to experiment, I’ll let you know the results!


#9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.