Certbot-auto can install new cert but fails on renewal

[HausInteractive]

We recently started installing letsencrypt certificates on our websites using certbot-auto. The installation process works without a hitch but the renewal process is failing. Both of these processes use the webroot authenticator method.

The error returned suggests that letsencrypt is unable to access the webroot auth file but we do not believe this is accurate. We have done some debugging and can confirm the following:

• Certbot-auto is dropping the authentication file in the webroot successfully
• We are able to wget the file that certbot-auto places in the webroot from remote locations successfully
• The call to check the file is posting successfully to letsencrypt
• Letsencrypt is hitting our server and showing in our access.log file. The IP address of the hit is:
  66.133.109.36 = outbound1.letsencrypt.org and this address is the same as appears in letsencrypt docs
• Whitelisting this address did not improve the outcome

It appears that the timeout is not from picking up the auth file but some part of the process one step further along the chain. We can see the request come in and we serve the file within a second. More details below per the help form:

My domain is: www.nypsychotherapy.com

I ran this command: certbot-auto renew --dry-run

It produced this output:

certbot-auto renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.nypsychotherapy.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.nypsychotherapy.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (www.nypsychotherapy.com) from /etc/letsencrypt/renewal/www.nypsychotherapy.com.conf produced an unexpected error: Failed authorization procedure. www.nypsychotherapy.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://www.nypsychotherapy.com/.well-known/acme-challenge/SyicbfCvfBDLCbjATOiN0O_edcZFvH6CW_3EvQQn9ig: Timeout during connect (likely firewall problem). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.nypsychotherapy.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/www.nypsychotherapy.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.nypsychotherapy.com
   Type:   connection
   Detail: Fetching
   http://www.nypsychotherapy.com/.well-known/acme-challenge/SyicbfCvfBDLCbjATOiN0O_edcZFvH6CW_3EvQQn9ig:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version): Apache/2.2.22 (Debian)

The operating system my web server runs on is (include version): Debian 3.2.93-1

My hosting provider, if applicable, is: Haus Interactive

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

[schoen]

@HausInteractive

Letsencrypt is hitting our server and showing in our access.log file. The IP address of the hit is:
66.133.109.36 = outbound1.letsencrypt.org and this address is the same as appears in letsencrypt docs

Hi @HausInteractive,

Where did you find Let's Encrypt documentation that mentions a particular verification IP address? We've tried to emphasize that there are many, that more than one may be used for a particular verification process (making several connections to your site from several different places on the Internet), that their IP addresses may change without notice, and that we don't support whitelisting these IP addresses. If you're blocking some inbound IP addresses, you might well be blocking another Let's Encrypt validation server.

[HausInteractive]

Thank you for your reply. We were hoping to gain a little more insight about the procedure that certbot is following and what is different between a new cert and a renewal rather than read yet another “turn off your firewall” response but it did prompt us to look deeper and were reminded of our AWS blacklist, which we found one of your servers on.

The ease of spinning up cheap instances has made AWS traffic almost as low of quality as traffic from the Ukraine or China so we have a pretty broad filter against them.

Now that we know what we’re looking for we are better able to fine-tune rules for it. Hopefully some day let’s encrypt can “settle down” somewhere and provide a more solid list of IP address to whitelist. Let’s Encrypt will keep hearing this from its users until it happens. Many providers, such as CDNs provide this. We’ve even seen this from services within AWS.

But most of that is beyond the scope of this topic. For now the problem is solved for us and certificates are renewing. Thanks.

[schoen]

@HausInteractive

Now that we know what we’re looking for we are better able to fine-tune rules for it. Hopefully some day let’s encrypt can “settle down” somewhere and provide a more solid list of IP address to whitelist. Let’s Encrypt will keep hearing this from its users until it happens. Many providers, such as CDNs provide this. We’ve even seen this from services within AWS.

We have a lot of threads about the fact that Let's Encrypt won't do this:

https://community.letsencrypt.org/search?q=whitelist%20ip%20addresses%20order%3Alatest

In particular, the unpredictability of the source IP addresses is a feature rather than a bug for purposes of mitigating BGP attacks against validation. In other threads, we've indicated that people who don't want to allow inbound connections from arbitrary IP addresses for validation purposes should use the DNS-01 challenge method. (Using a CNAME record, you can also allow this challenge to be performed by a third-party DNS provider or DNS server instance of your choice which can be completely separate from any other infrastructure that you use for any other purpose.)