Certbot can't access/create the .well-known/acme-challenge folder/file

Note: I'm using Nginx and I'm getting a 404 for the challenge.

I had an issue before with certain subdomains and not others not working, but I decided to just give up and create a new subdomain for the site and a brand new site file.

server {
        listen 80;
        root /var/www/wikifamily;
        index index.php;

        server_name globalwiki.rfx.fi
                www.globalwiki.rfx.fi;

          location ~* \.php$ {
                 fastcgi_pass unix:/run/php/php7.3-fpm.sock;
                 include         fastcgi_params;
                 fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                 fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
          }
}

Above is what I started with and below are two of the possibilities that I tried and neither one of them works.

server {
        listen 80;
        root /var/www/wikifamily;
        index index.php;

        server_name globalwiki.rfx.fi
                www.globalwiki.rfx.fi;

          location ~* \.php$ {
                 fastcgi_pass unix:/run/php/php7.3-fpm.sock;
                 include         fastcgi_params;
                 fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                 fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
          }
        location ~ /.well-known {
                default_type "text/plain";
                root /var/www/wikifamily;
        }
}

server {
        listen 80;
        root /var/www/wikifamily;
        index index.php;

        server_name globalwiki.rfx.fi
                www.globalwiki.rfx.fi;

          location ~* \.php$ {
                 fastcgi_pass unix:/run/php/php7.3-fpm.sock;
                 include         fastcgi_params;
                 fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                 fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
          }
          location ~ /.well-known {
              allow all;
          }
}

This is what happens:

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 47 48
Requesting a certificate for www.globalwiki.rfx.fi and globalwiki.rfx.fi
Performing the following challenges:
http-01 challenge for globalwiki.rfx.fi
http-01 challenge for www.globalwiki.rfx.fi
Waiting for verification...
Challenge failed for domain globalwiki.rfx.fi
Challenge failed for domain www.globalwiki.rfx.fi
http-01 challenge for globalwiki.rfx.fi
http-01 challenge for www.globalwiki.rfx.fi
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: globalwiki.rfx.fi
   Type:   unauthorized
   Detail: Invalid response from
   http://globalwiki.rfx.fi/.well-known/acme-challenge/BfjbNnSYE-rgDCguskhj85gxq10brQbjsVxRnkEztjU
   [192.241.132.174]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   Domain: www.globalwiki.rfx.fi
   Type:   unauthorized
   Detail: Invalid response from
   http://www.globalwiki.rfx.fi/.well-known/acme-challenge/hyRtsa82_AZOkqeeiifJe2a8mTsKZ6XZg7AoowUPNAA
   [192.241.132.174]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

The domain records work fine because I can navigate to the site and have it work. Something is wrong with this "well known acme" system. The only thing that I can figure is something is wrong with location ~* \.php$ { and somehow it's preventing this "well known acme challenge" from working.

Edit: That's not the problem because I checked another site and it's the same and works fine with a cert. This particular site is just cursed for some reason and I can't figure out why when everything shows that it should be working.

1 Like

Hello :slightly_smiling_face:

It's my understanding that certbot doesn't actually create the /.well-known/acme-challenge/ directory structure in nginx, but rather adds an exception to the nginx configuration to serve the challenge files.

Try adding --debug-challenges to your certbot command so that certbot will pause before submitting verification requests to the CA. This will let you take a look around.

2 Likes

Thanks. I just created a folder in the web directory and chown'd it for www-data. I hit the rate limit of failed challenges for the hour but I'll try when I can to see if it works. I'll also try running that flag and see if it gives any useful information for why it doesn't work.

1 Like

You are using --dry-run I hope? That uses the staging environment with much higher rate limits for testing.

2 Likes

I didn't do that but I'll add that so it won't count against the rate limit.

I just got this:

certbot --nginx --debug-challenges --dry-run
--dry-run currently only works with the 'certonly' or 'renew' subcommands ('run')

1 Like

Silly, huh? :grin:

--dry-run doesn't work with run (the default command). Go figure...

I'll help you out there.

2 Likes

Test acquisition with:
certbot certonly --nginx --debug-challenges --dry-run

Once acquisition is working, acquire and install with:
certbot --nginx --keep

2 Likes

Thanks. That's what I just ran.

2 Likes

I didn't see certonly in the command you posted.

2 Likes

I didn't use certonly correctly the first time. I had to correct it.

It just said "The dry run was successful."

 certbot certonly --nginx --debug-challenges --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Account registered.

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: 62x54r.ru
2: 7.62x54r.ru
3: blog.oss
4: chatbook.oss
5: foxnic.oss
6: www.foxnic.oss
7: opennic.pol
8: opennic.sct
9: opennic.rfs
10: opennic.mag
11: opennic.srb
12: opennic.ssr
13: www.opennic.pol
14: www.opennic.srb
15: www.opennic.sct
16: www.opennic.rfs
17: www.opennic.mag
18: www.opennic.ssr
19: r3df0x.com
20: analytics.r3df0x.com
21: antislavists.r3df0x.com
22: mail.dns.blacklist.r3df0x.com
23: cccrwiki.r3df0x.com
24: www.cccrwiki.r3df0x.com
25: coc.r3df0x.com
26: code-of-conduct.r3df0x.com
27: codeofconduct.r3df0x.com
28: conservativesjw.r3df0x.com
29: global.r3df0x.com
30: guns.r3df0x.com
31: i.r3df0x.com
32: metawiki.r3df0x.com
33: www.metawiki.r3df0x.com
34: opennic.r3df0x.com
35: www.opennic.r3df0x.com
36: resources.r3df0x.com
37: shitlordsinaction.r3df0x.com
38: sturmovik.r3df0x.com
39: www.sturmovik.r3df0x.com
40: template.r3df0x.com
41: thesexynerd.r3df0x.com
42: www.r3df0x.com
43: register.rf.o
44: rfx.fi
45: cccrwiki.rfx.fi
46: www.cccrwiki.rfx.fi
47: globalwiki.rfx.fi
48: www.globalwiki.rfx.fi
49: metawiki.rfx.fi
50: www.metawiki.rfx.fi
51: opennic.rfx.fi
52: www.opennic.rfx.fi
53: sturmovik.rfx.fi
54: www.sturmovik.rfx.fi
55: voynawiki.rfx.fi
56: www.voynawiki.rfx.fi
57: www.rfx.fi
58: volk.ssr
59: www.volk.ssr
60: wiki.oss
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 47 48
Simulating a certificate request for www.globalwiki.rfx.fi and globalwiki.rfx.fi
Performing the following challenges:
http-01 challenge for globalwiki.rfx.fi
http-01 challenge for www.globalwiki.rfx.fi
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Cleaning up challenges

IMPORTANT NOTES:
 - The dry run was successful.

That makes it sound like it should work.

1 Like

Did it pause for the debug?

I concur that if the dry run works then this should work:

certbot certonly --nginx

It that works then this is the next step:

certbot --nginx --keep

(I'm segregating the acquisition and installation steps here.)

1 Like

I ran the certbot --nginx --keep command and it had the same failure.

I'll try running those two commands and see if it works.

Edit: I'm at the rate so I'll try soon later.

1 Like

If the staging works and the production doesn't then I'm beginning to suspect a DNS issue where the LE staging server is possibly seeing a different version of your webserver than the LE production server.

2 Likes

Are you using a DigitalOcean load balancer, perchance?

If so, you might want to read this and save yourself many headaches:

2 Likes

It's on Digital Ocean. I checked resolv.conf:

 cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 206.189.193.106
nameserver 2001:4860:4860::8844
nameserver 2001:4860:4860::8888 
nameserver 8.8.8.8

I think that's a server I made to resolve OpenNIC domains but certbot worked in the past with it.

1 Like

I only asked because many people find DigitalOcean's integrated Let's Encrypt solution to be far easier to manage than certbot.

2 Likes

I might just use that then. It looks like the LB takes all the traffic from https and then sends it over http to the web server?

1 Like

That's correct and known as "SSL termination at the LB".

Upside:
No need to configure certificates for each backend worker.

Downside:
Puts burden of decryption on LB.

2 Likes

I'm going to try setting it up for the site that I'm working on.

1 Like

Excellent. Keep us posted.

:+1:

I've gotta run for now, but if you need more immediate help my colleagues are more than eager.

2 Likes