Suddenly Certbot Stop Working - Some challenges have failed

Help
1

Hello Team,

linkjoy product helps users to create shorten their links. User can add their domain as custom domain in linkjoy. So they can use their domains to create short links.

So, First users have to add CNAME of linkjoy in their DNS and then they have to add domin in our platform. When users add domin, we generate Nginx file for that domain and generate SSL through "sudo certbot --nginx -n --redirect -d domain_name" command. All this is done with PHP script in queue (We use laravel)

This is working fine for around 230 domains. But suddenly certbot stop working without any changes in code, server or nginx configuration.

When we run "sudo certbot --nginx -n --redirect -d domain_name", we got following error.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for ssltesting.notionjoy.io
Waiting for verification...
Challenge failed for domain ssltesting.notionjoy.io
http-01 challenge for ssltesting.notionjoy.io
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

Now, this happens with every new domain and even get the same error when we try to renew the old certificates.

Please help us to fix this error.

My domain is: ssltesting.notionjoy.io

I ran this command: ssltesting.notionjoy.io

It produced this output:

My web server is (include version): nginx/1.18.0

The operating system my web server runs on is (include version): Ubuntu 20.04.1 LTS

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.40.0

2

Maybe the problem was only temporary.
Albeit slow, I was able to reach http://ssltesting.notionjoy.io/ and heard the redirection.
Let's Debug does point out another possible problem:
Let's Debug (letsdebug.net)

4

Hello,

We haven't changed any Cloudflare configurations in a long time. We set "Full SSL" on cloudflare side because we generate self-signed certificate for each domain. I also have attached a screenshot of the configuration.

It was working fine before 2-3 days and now we are not able to generate SSL for new domain.

Screenshot from 2021-09-01 22-58-56
Screenshot from 2021-09-01 22-58-561366×616 57.9 KB

5

Please try again with --dry-run

6

Still not working with --dry-run.
Command - sudo certbot --nginx certonly -n --redirect -d ssltesting.notionjoy.io --dry-run

Output -
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for ssltesting.notionjoy.io
Waiting for verification...
Challenge failed for domain ssltesting.notionjoy.io
http-01 challenge for ssltesting.notionjoy.io
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

7

Again, LE fails to reach the redirection.

Are you running something like fail2ban?

8

No, we are not use anything like fail2ban.

10

When LE call "http://ssltesting.notionjoy.io/.well-known/acme-challenge/_uZDe_rxN6U82DsqdZlkxCFGghorHKHy5HJyEb_6VeY" URL, we got "Not found' page. Actually, laravel gives this error. Means request is reached to our server and not able to complete verification

1 Like
11

OK, we are on the same page now.

Can we see the nginx config?
sudo nginx -T

12

As I mentioned, we have around 200+ domains and we generate separate Nginx fire for each domain. I have attached a file that has output of the given command.

Nginx.txt (396.3 KB)

13

Let's test the access directly, with:

mkdir /var/www/linkjoy/public/.well-known
mkdir /var/www/linkjoy/public/.well-known/acme-challenge
echo "test-file-1234" > /var/www/linkjoy/public/.well-known/acme-challenge/Test-File-1234

Then try:
http://ssltesting.notionjoy.io/.well-known/acme-challenge/Test-File-1234

14

As you mentioned, I have created directory and the file gets download when we hit "http://ssltesting.notionjoy.io/.well-known/acme-challenge/Test-File-1234" URL.

15

OK, that is great news!

The means you don't need to be limited to using the --nginx authenticator method.
You can now also use --webroot -w /var/www/linkjoy/public instead.

16

Try:

sudo certbot certonly \
--webroot -w /var/www/linkjoy/public \
-d ssltesting.notionjoy.io --dry-run
17

Command - sudo certbot certonly --webroot -w /var/www/linkjoy/public -d ssltesting.notionjoy.io --dry-run

Outout -
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for ssltesting.notionjoy.io
Using the webroot path /var/www/linkjoy/public for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:

  • The dry run was successful.

But if I run this command without --dry-run, I get following output
sudo certbot certonly --webroot -w /var/www/linkjoy/public -d ssltesting.notionjoy.io
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for ssltesting.notionjoy.io
Using the webroot path /var/www/linkjoy/public for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:

  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/ssltesting.notionjoy.io/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/ssltesting.notionjoy.io/privkey.pem
    Your cert will expire on 2021-12-01. To obtain a new or tweaked
    version of this certificate in the future, simply run certbot
    again. To non-interactively renew all of your certificates, run
    "certbot renew"

  • If you like Certbot, please consider supporting our work by:

    Donating to ISRG / Let's Encrypt: Donate - Let's Encrypt
    Donating to EFF: Support EFF's Work on Let's Encrypt | Electronic Frontier Foundation

So, we are able to generate certificate successfully.
Now, how do I reflect the certificate in nginx file?

18

You are a step ahead of me.
Let's see the cert, with:
certbot certificates

And to answer:

As you would with any other cert...???
Have you used a cert in nginx before?
Please show the output of:
nginx -T | grep -Ei 'ssl_cert|server_name'

I would expect to see something like:

ssl_certificate     /etc/letsencrypt/live/ssltesting.notionjoy.io/fullchain.pem
ssl_certificate_key /etc/letsencrypt/live/ssltesting.notionjoy.io/privkey.pem
19

Please find attachment
Cert.txt (68.6 KB)

20

We do not get the expected result for ssltesting.notionjoy.io domain.
PFA
EI.txt (67.7 KB)

Output be like

We do not get the expected result for ssltesting.notionjoy.io domain.

Output is something like this :

server_name ssltesting.notionjoy.io;
server_name stephen.sharedhomies.com;
ssl_certificate /etc/letsencrypt/live/stephen.sharedhomies.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/stephen.sharedhomies.com/privkey.pem; # managed by Certbot

21

OK, so how are the secure vhost configs normally created?

22

We just used "sudo certbot --nginx -d example.com" command to get certificate and reflact it in nginx file of that domain.