"./certbot-auto renew --dry-run" fails with Timeout (but urls are perfectly accessible)

Help
1

Due to changes in requirements after issuing the initial certificates my apache configuration has changed somewhat. Same hostnames. But all HTTP traffic is now redirected to HTTPS. Additionally, the CN for the cert was discovered to be wrong… as there was no clear documentation originally that the order domains were specified on the command line was significant. I re-ordered the renewal file to hopefully correct this on renewal (as suggested in another thread here that’s been weeks since I saw so I don’t have a link to it).

Apparently one or both of these has sent letsencrypt and/or certbot into a tailspin. What it says in the subject line. Doing a dry run renewal to test my new configuration results in “timedout” for all hostnames. I’m coming up on renewal as well.

My renewal config:

# renew_before_expiry = 30 days
version = 0.16.0
archive_dir = /etc/letsencrypt/archive/annapuma.onsite-crt.com
cert = /etc/letsencrypt/live/annapuma.onsite-crt.com/cert.pem
privkey = /etc/letsencrypt/live/annapuma.onsite-crt.com/privkey.pem
chain = /etc/letsencrypt/live/annapuma.onsite-crt.com/chain.pem
fullchain = /etc/letsencrypt/live/annapuma.onsite-crt.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = d846e0b4dbf439b0363dbbd14c8da189
webroot_path = /var/www/certbot,
rsa_key_size = 4096
[[webroot_map]]
annapuma.onsite-crt.com = /var/www/certbot
phpmyadmin.annapuma.onsite-crt.com = /var/www/certbot
receptiveskepticism.org = /var/www/certbot
www.receptiveskepticism.org = /var/www/certbot
recepskep.onsite-crt.com = /var/www/certbot

My apache vhosts:

# For certbot -JCA
Alias /.well-known /var/www/certbot/.well-known
<Directory "/var/www/certbot/">
	allow from all
</Directory>

# yes I'm aware that's not the "correct" way to use NameVirtualHost directives...
# the correct way doesn't work... -JCA

NameVirtualHost annapuma.onsite-crt.com:80
<VirtualHost 144.217.115.63:80>
	ServerName annapuma.onsite-crt.com
	RedirectPermanent / https://annapuma.onsite-crt.com/
</VirtualHost>

NameVirtualHost annapuma.onsite-crt.com:443
<VirtualHost 144.217.115.63:443>
	ServerName annapuma.onsite-crt.com
	SSLEngine on
	DocumentRoot /var/www/html
	<Directory "/var/www/html">
		allow from all
		Options SymLinksIfOwnerMatch
		AllowOverride FileInfo
	</Directory>
</VirtualHost>

NameVirtualHost phpmyadmin.annapuma.onsite-crt.com:80
<VirtualHost 144.217.115.63:80>
	ServerName phpmyadmin.annapuma.onsite-crt.com
	AssignUserId phpmyadmin phpmyadmin
	RedirectPermanent / https://phpmyadmin.annapuma.onsite-crt.com:8080/
</VirtualHost>

NameVirtualHost phpmyadmin.annapuma.onsite-crt.com:443
<VirtualHost 144.217.115.63:443>
	ServerName phpmyadmin.annapuma.onsite-crt.com
	AssignUserId phpmyadmin phpmyadmin
	RedirectPermanent / https://phpmyadmin.annapuma.onsite-crt.com:8080/
	SSLEngine on
</VirtualHost>

NameVirtualHost phpmyadmin.annapuma.onsite-crt.com:8080
<VirtualHost 144.217.115.63:8080>
	ServerName phpmyadmin.annapuma.onsite-crt.com
	AssignUserId phpmyadmin phpmyadmin
	SSLEngine on
	Include /etc/httpd/conf.d/phpMyAdmin.conf
	DocumentRoot /usr/local/share/phpMyAdmin/
	<Directory "/usr/local/share/phpMyAdmin/setup">
		deny from all
	</Directory>
</VirtualHost>

NameVirtualHost www.receptiveskepticism.org:80
<VirtualHost 144.217.115.63:80>
	AssignUserId recepskep recepskep
	ServerName www.receptiveskepticism.org
	ServerAlias receptiveskepticism.org recepskep.onsite-crt.com
	RedirectPermanent / https://www.receptiveskepticism.org/
</VirtualHost>

NameVirtualHost www.receptiveskepticism.org:443
<VirtualHost 144.217.115.63:443>
	AssignUserId recepskep recepskep
	ServerName www.receptiveskepticism.org
	ServerAlias receptiveskepticism.org recepskep.onsite-crt.com
	SSLEngine on
	DocumentRoot /home/recepskep/www
	<Directory "/home/recepskep/www">
		allow from all
		Options SymLinksIfOwnerMatch
		AllowOverride FileInfo Limit
	</Directory>
</VirtualHost>

certbot-auto output:

 -------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/annapuma.onsite-crt.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: annapuma.onsite-crt.com
   Type:   connection
   Detail: Fetching
   http://annapuma.onsite-crt.com/.well-known/acme-challenge/WaRvkGFKf94uypk7FuuTSmEs3OnCKLk7AmBx-bUq9uc:
   Timeout

   Domain: www.receptiveskepticism.org
   Type:   connection
   Detail: Fetching
   http://www.receptiveskepticism.org/.well-known/acme-challenge/CI2NX-Ic1_-Idx6S9DZ925F6ViGGNP4Qu6-0Vfyr_Lk:
   Timeout

   Domain: recepskep.onsite-crt.com
   Type:   connection
   Detail: Fetching
   http://recepskep.onsite-crt.com/.well-known/acme-challenge/kD--UHGEO-B309AdOz2U1QOEnHSgCt8CcYdwGR9Wq6Q:
   Timeout

   Domain: receptiveskepticism.org
   Type:   connection
   Detail: Fetching
   http://www.receptiveskepticism.org/.well-known/acme-challenge/nsVdvKVWT8gz66077vF_jLUzgyNIC7uIByZaKoKbUI4:
   Timeout

   Domain: phpmyadmin.annapuma.onsite-crt.com
   Type:   connection
   Detail: Fetching
   http://phpmyadmin.annapuma.onsite-crt.com/.well-known/acme-challenge/lpiW5gVdioFAIByVjMOWc-CXqZ7shpSU7OLe8cJ2uYY:
   Timeout

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

A set of links to a simple text file showing that **/.well-known/** is accessible:
works works works

Thoughts?

Showing that my certs are functioning well (other than the incorrect CN:
good good good

2

Edit: I replied about reordering the names being potentially impossible without deleting the cert, but then I looked again and realized that that actually wasn’t the main point of your question, so my reply wasn’t particularly helpful.

3

@cpu, can you see if there’s any reason that this shouldn’t be reachable from the data center? It looks reachable to me…

@JadedDragoon, I assume you didn’t whitelist Let’s Encrypt IP addresses in a firewall or anything?

You could try the real certbot renew and see if you see any difference. There is a rate limit which will prevent you from trying this over and over again, but you’re allowed five attempts per hour under most circumstances so trying a single attempt shouldn’t harm anything. Also, if certbot renew doesn’t manage to get the new certificate, it doesn’t delete the original certificate or change your configuration or anything.

4

Actually, I did whitelist letsencrypt’s IP (the phpmyadmin is hidden behind netfilter rules for all but whitelisted ips)… or at least I whitelisted the IP that was showing up in my logs at the time (66.133.109.36).

Leme try and see if it works now. I was having some issues getting the alias to work correctly (accidentally typed _ instead of - in .well-known and spent an hour scratching my head over it) so I may have gotten hit with the rate limit right when I got it working.

…Nope. And I checked the logs, My last attempt was 9:21 EST. It’s now 10:27 EST. Unless that’s too low a margin of error.

If I have to re-request a new cert I can do that, certainly. But, as you saw, I’ll need to resolve the other issue first. :slight_smile:

5

This was never officially supported and now is more likely to be a source of failures:

Some further discussion can be found at

and the links there.

So, I think that might be the reason that you've started to run into problems.

1 Like
6

Ok, that explains the PHPMyAdmin one. The rest are on ports 80 and 443 which are entirely publicly accessible (as demonstrated above). Also, It doesn’t explain still having this issue when altered my vhosts so that PHPMyAdmin was also on 443… and thus also publicly accessible (desperate times… etc).

But assuming that is the issue, I should be able to use --cert-name to remove the phpmyadmin hostname from the cert and authenticate the others with the webroot method yes? Then perhaps take care of phpmyadmin separately via the DNS method you mention?

7

Yes, you can use --cert-name and specify a new list of domains to be included in the new version of the cert with with -d.

8

It appears to be resolved. I don’t suppose a list of current IPs for letsencrypt VA’s is available so I can purge them from my firewall’s blacklist? I do believe the entire issue was that the 2nd and 3rd VA’s got auto-blacklisted due to me trying too often (an issue that won’t happen again with me only targeting public vhosts). They’ll rotate out of the blacklist in a month regardless, but it’d be good for my peace of mind to purge them now.

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/annapuma.onsite-crt.com/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/annapuma.onsite-crt.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
9

If you look at the items I linked above, they really try to emphasize a negative answer to this. If you use the HTTP-01 or TLS-SNI-01 challenges, you should be prepared to accept inbound challenge connections from anywhere on the Internet. I've been committed enough to this answer that I haven't tried to obtain such a list myself.

1 Like
10

I checked and all of the names except for phpmyadmin.annapuma.onsite-crt.com have successful issuance in staging. This domain has timeouts from the VA and the remote VAs. It sounds like that's explained by firewall rules so I won't ask operations to do any network troubleshooting for now. Please let me know if that becomes necessary.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.