Certbot-auto failing, challenges have failed

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: www.fleet.org, mail.fleet.org

I ran this command: certbot-auto

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?

1: mail.fleet.org
2: www.fleet.org

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter ‘c’ to cancel):
Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.fleet.org
http-01 challenge for www.fleet.org
Waiting for verification…
Challenge failed for domain mail.fleet.org
Challenge failed for domain www.fleet.org
http-01 challenge for mail.fleet.org
http-01 challenge for www.fleet.org
Cleaning up challenges
Some challenges have failed.

My web server is (include version): httpd-2.4.41-6.1.fc30.x86_64 (Apache 2.4)

The operating system my web server runs on is (include version):
Fedora release 30 (Thirty)

My hosting provider, if applicable, is: self

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
Just upgraded to certbot 1.7.0
Last run succesfully under 1.4.0

Can you tell me what changed in the challenge process between 1.4.0 and 1.7.0? I checked all the things listed in the “IMPORTANT NOTES:” and nothing stands out as wrong, other than there is no .well-known directory under the web root after certbot-auto runs.

1 Like
2

Hi,

First of all, I think you want to renew a certificate right?
If so, the correct command should be sudo certbot-auto renew. This will attempt to renew the certificate based on the previous success attempts.

Can you provide more information on your attempt?
If possible, a full output of sudo certbot-auto renew --dry-run would be better.

Thank you

1 Like
3

certbot-auto renew --dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/mail.fleet.org.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.fleet.org
http-01 challenge for www.fleet.org
Waiting for verification...
Challenge failed for domain mail.fleet.org
Challenge failed for domain www.fleet.org
http-01 challenge for mail.fleet.org
http-01 challenge for www.fleet.org
Cleaning up challenges
Attempting to renew cert (mail.fleet.org) from /etc/letsencrypt/renewal/mail.fleet.org.conf produced an unexpected error: Some challenges have failed.. Skipping.

Processing /etc/letsencrypt/renewal/www.fleet.org.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.fleet.org
Waiting for verification...
Challenge failed for domain www.fleet.org
http-01 challenge for www.fleet.org
Cleaning up challenges
Attempting to renew cert (www.fleet.org) from /etc/letsencrypt/renewal/www.fleet.org.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mail.fleet.org/fullchain.pem (failure)
/etc/letsencrypt/live/www.fleet.org/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mail.fleet.org/fullchain.pem (failure)
/etc/letsencrypt/live/www.fleet.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mail.fleet.org/fullchain.pem (failure)
/etc/letsencrypt/live/www.fleet.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: mail.fleet.org
    Type: connection
    Detail: Fetching
    http://mail.fleet.org/.well-known/acme-challenge/5fIZVHRqPm6-EmZVcW75Jd84v7MrfSd0vpLSTEXR2mw:
    Timeout during connect (likely firewall problem)

    Domain: www.fleet.org
    Type: connection
    Detail: Fetching
    http://www.fleet.org/.well-known/acme-challenge/1Rx8GC6q-43jnnUI5EcNsIblEBTy3QdS2yT6DyMR-3I:
    Timeout during connect (likely firewall problem)

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you're using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

  • The following errors were reported by the server:

    Domain: www.fleet.org
    Type: connection
    Detail: Fetching
    http://www.fleet.org/.well-known/acme-challenge/30YlcHEXuJ4UlH9BVUAEccQ4RqylQfVnScawbegvw8s:
    Timeout during connect (likely firewall problem)

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you're using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

  • Your account credentials have been saved in your Certbot
    configuration directory at /etc/letsencrypt. You should make a
    secure backup of this folder now. This configuration directory will
    also contain certificates and private keys obtained by Certbot so
    making regular backups of this folder is ideal.

1 Like
4

Hi,

The error message doesn’t make sense to me, since I’m able to connect to your domain without any issues. Can you check if there’s any firewall in place?

To @JuergenAuer, i also can’t. However i’m reluctant to place the blame onto the ISP (CentryLink) for blocking something (Like IP or User-agent)? So i guess it might be on firewall

2 Likes
5

Hi @anewton81

I can load that file.

So if Letsencrypt has a timeout and @stevenzhu may see the same timeout:

There is a regional filter.

2 Likes
6

I cannot access the challenge files either though I’m assuming they’ve been cleaned-up anyhow so I get a 404. Can you run the following, which pauses during the run so that we can see if the challenge files are properly created and accessible?

certbot-auto renew --dry-run --debug-challenges

Incidentally, per https://crt.sh/?q=fleet.org it looks like there have been certificates with overlapping domains created in the past. Might want to avoid this. It also looks like fleet.org itself isn’t being included in the certificate.

When I go to http://mail.fleet.org I immediately get “under construction”.

When I go to https://mail.fleet.org I immediately get “under construction” and an expired Let’s Encrypt certificate.

When I go to http://www.fleet.org it hangs for awhile then I get redirected to https where I get “under construction” and an expired Let’s Encrypt certificate.

When I got to https://www.fleet.org it hangs for awhile then I get “under construction” and an expired Let’s Encrypt certificate.

Both http://fleet.org and https://fleet.org seem to just hang.

Screenshot_20200821-184207_Samsung Internet
Screenshot_20200821-184207_Samsung Internet1440×3040 1.43 MB

7

Fun fact: For Apache and Nginx authenticators, sometimes certbot will modify the configuration to directly include a line that return the value of the file. So it's defined in the virtual host configuration (might also placed into the directory but i doubt that). (Discovered this when i was reading my certbot log)

1 Like
8

Good to know. :slightly_smiling_face: It wasn’t the value of the file I was concerned about though. I wanted it to pause so it wouldn’t clean up the challenge files so that I could try to access them and check the delay in fetching.

9

Okay, I have run that comand:
certbot-auto renew --dry-run --debug-challenges

and the server is in that state.

A couple of things: The certificate expired today, so you're correct in that it says it has an expired certificate. That's what I get for procrastinating, but this has happened before and I was able to renew successfully.

Second thing: You're right about the overlapping, and here's why: Prior to last December, I had a /29 routable subnet and www.fleet.org and mail.fleet.org had different IP addresses. When I upgraded my DSL, I decided in order to save money, I would take a single IP address (/32 address) and make www.fleet.org a CNAME for mail.fleet.org. If I need to revoke prior certificates or something, please let me know.

As for just plain "fleet.org", that IP address is a dialup line and hasn't been used for quite awhile, so you won't get any response from it.

What do you see on my server with --debug-challenges?

1 Like
10

Do you know what has changed in the challenge-response between version 1.4.0 and 1.7.0? My firewall configuration didn’t change, and it worked last time under 1.4.0.

1 Like
11

I'm too getting time outs here. ICMP, TCP port 80, port 443, UDP, protocol doesn't matter, it all ends at 12 67-42-136-90.albq.qwest.net (67.42.136.90) 192.972 ms 192.961 ms 192.953 ms.

Revoking certificates is almost never a solution to any problem, only when the private key has been or might have been leaked.

This is also not related to Let's Encrypt. There is literally nothing that certbot or Let's Encrypt could do to make connections from third parties (like the users on this Community) to your IP fail. There are only non-Let's Encrypt related things causing that, like (regional) firewalls, failing NAT router portmaps and such things.

2 Likes
12

Sorry, I had shut down the service last night, it’s back up, now.

When those of you who say “regional firewall”, what do you mean? My DSL modem is bridged with a public IP address, ports 80 & 443 are open to the world (and several responders here were able to get to it yesterday). Do I also need ICMP open to the world? Because it isn’t at the moment…

1 Like
13

Okay, hold the presses! I dug a little deeper this morning (after a refreshing sleep overnight) and figured it out. Short story: Firewall rule, as some responders said (thank you for that, it was the first place I looked today).

Long story if you want it: My mail server has a process where it throttles, then blocks an IP address (or complete CIDR, if the spammer is using multiple addresses in the same subnet) if it receives too many emails from that source (or if too many messages are flagged as spam by users). I had decided early on, since this is a “leaf node”, that any “badness” from an IP (or CIDR) merited blocking ALL ports from those IP addresses. This decision may have been short-sighted on my part. In any case, on January 11th, I renewed my certificates (needed help then, too, for a different issue - thanks, you all!). On Jan 17, someone spammed multiple addresses, probably a lot of them, from many IPs in the the 66.0.0.0/8 network, resulting in that CIDR getting blocked. That didn’t seem to create a problem for my renewals in March or May, but this time LetsEncrypt was sending challenges from 66.133.109.36, which were getting blocked because of the spam rule.

I found it, decided it was overly aggressive, and removed it. Happiness and joy once again. And now I have something else to think about – targeting ports of bad actors and not just the entire IP. Or maybe something else more reasonable.

I welcome comments on this either here or in PM, if anyone desires to discuss. I’m always up for learning and also contributing!

Thanks again for all your help!

3 Likes
14

That's simple.

Don't block or create a blocking rule if /.well-known/acme-challenge is checked.

Happy to read you have found your blocking filter :+1:

1 Like
15

Sorry to leave you hanging yesterday. I got tied up and couldn’t get back on. Glad you found your snafu. :smiley: What I was trying to test turned out to be unrelated anyhow. As usual, @JuergenAuer was on the right trail and suggested a workable solution (unless someone tries to (D)DOS you by requesting your Let’s Encrypt challenge files for the moments they exist every couple of months, which is just silly :stuck_out_tongue:).

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.