Certbot can't access/create the .well-known/acme-challenge folder/file

Note: I'm using Nginx and I'm getting a 404 for the challenge.

I had an issue before with certain subdomains and not others not working, but I decided to just give up and create a new subdomain for the site and a brand new site file.

server {
        listen 80;
        root /var/www/wikifamily;
        index index.php;

        server_name globalwiki.rfx.fi
                www.globalwiki.rfx.fi;

          location ~* \.php$ {
                 fastcgi_pass unix:/run/php/php7.3-fpm.sock;
                 include         fastcgi_params;
                 fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                 fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
          }
}

Above is what I started with and below are two of the possibilities that I tried and neither one of them works.

server {
        listen 80;
        root /var/www/wikifamily;
        index index.php;

        server_name globalwiki.rfx.fi
                www.globalwiki.rfx.fi;

          location ~* \.php$ {
                 fastcgi_pass unix:/run/php/php7.3-fpm.sock;
                 include         fastcgi_params;
                 fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                 fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
          }
        location ~ /.well-known {
                default_type "text/plain";
                root /var/www/wikifamily;
        }
}

server {
        listen 80;
        root /var/www/wikifamily;
        index index.php;

        server_name globalwiki.rfx.fi
                www.globalwiki.rfx.fi;

          location ~* \.php$ {
                 fastcgi_pass unix:/run/php/php7.3-fpm.sock;
                 include         fastcgi_params;
                 fastcgi_param   SCRIPT_FILENAME    $document_root$fastcgi_script_name;
                 fastcgi_param   SCRIPT_NAME        $fastcgi_script_name;
          }
          location ~ /.well-known {
              allow all;
          }
}

This is what happens:

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 47 48
Requesting a certificate for www.globalwiki.rfx.fi and globalwiki.rfx.fi
Performing the following challenges:
http-01 challenge for globalwiki.rfx.fi
http-01 challenge for www.globalwiki.rfx.fi
Waiting for verification...
Challenge failed for domain globalwiki.rfx.fi
Challenge failed for domain www.globalwiki.rfx.fi
http-01 challenge for globalwiki.rfx.fi
http-01 challenge for www.globalwiki.rfx.fi
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: globalwiki.rfx.fi
   Type:   unauthorized
   Detail: Invalid response from
   http://globalwiki.rfx.fi/.well-known/acme-challenge/BfjbNnSYE-rgDCguskhj85gxq10brQbjsVxRnkEztjU
   [192.241.132.174]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   Domain: www.globalwiki.rfx.fi
   Type:   unauthorized
   Detail: Invalid response from
   http://www.globalwiki.rfx.fi/.well-known/acme-challenge/hyRtsa82_AZOkqeeiifJe2a8mTsKZ6XZg7AoowUPNAA
   [192.241.132.174]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

The domain records work fine because I can navigate to the site and have it work. Something is wrong with this "well known acme" system. The only thing that I can figure is something is wrong with location ~* \.php$ { and somehow it's preventing this "well known acme challenge" from working.

Edit: That's not the problem because I checked another site and it's the same and works fine with a cert. This particular site is just cursed for some reason and I can't figure out why when everything shows that it should be working.

Hello :slightly_smiling_face:

It's my understanding that certbot doesn't actually create the /.well-known/acme-challenge/ directory structure in nginx, but rather adds an exception to the nginx configuration to serve the challenge files.

Try adding --debug-challenges to your certbot command so that certbot will pause before submitting verification requests to the CA. This will let you take a look around.

Thanks. I just created a folder in the web directory and chown'd it for www-data. I hit the rate limit of failed challenges for the hour but I'll try when I can to see if it works. I'll also try running that flag and see if it gives any useful information for why it doesn't work.

You are using --dry-run I hope? That uses the staging environment with much higher rate limits for testing.

I didn't do that but I'll add that so it won't count against the rate limit.

I just got this:

certbot --nginx --debug-challenges --dry-run
--dry-run currently only works with the 'certonly' or 'renew' subcommands ('run')

Silly, huh? :grin:

--dry-run doesn't work with run (the default command). Go figure...

I'll help you out there.

Test acquisition with:
certbot certonly --nginx --debug-challenges --dry-run

Once acquisition is working, acquire and install with:
certbot --nginx --keep

Thanks. That's what I just ran.

I didn't see certonly in the command you posted.

I didn't use certonly correctly the first time. I had to correct it.

It just said "The dry run was successful."

 certbot certonly --nginx --debug-challenges --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Account registered.

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: 62x54r.ru
2: 7.62x54r.ru
3: blog.oss
4: chatbook.oss
5: foxnic.oss
6: www.foxnic.oss
7: opennic.pol
8: opennic.sct
9: opennic.rfs
10: opennic.mag
11: opennic.srb
12: opennic.ssr
13: www.opennic.pol
14: www.opennic.srb
15: www.opennic.sct
16: www.opennic.rfs
17: www.opennic.mag
18: www.opennic.ssr
19: r3df0x.com
20: analytics.r3df0x.com
21: antislavists.r3df0x.com
22: mail.dns.blacklist.r3df0x.com
23: cccrwiki.r3df0x.com
24: www.cccrwiki.r3df0x.com
25: coc.r3df0x.com
26: code-of-conduct.r3df0x.com
27: codeofconduct.r3df0x.com
28: conservativesjw.r3df0x.com
29: global.r3df0x.com
30: guns.r3df0x.com
31: i.r3df0x.com
32: metawiki.r3df0x.com
33: www.metawiki.r3df0x.com
34: opennic.r3df0x.com
35: www.opennic.r3df0x.com
36: resources.r3df0x.com
37: shitlordsinaction.r3df0x.com
38: sturmovik.r3df0x.com
39: www.sturmovik.r3df0x.com
40: template.r3df0x.com
41: thesexynerd.r3df0x.com
42: www.r3df0x.com
43: register.rf.o
44: rfx.fi
45: cccrwiki.rfx.fi
46: www.cccrwiki.rfx.fi
47: globalwiki.rfx.fi
48: www.globalwiki.rfx.fi
49: metawiki.rfx.fi
50: www.metawiki.rfx.fi
51: opennic.rfx.fi
52: www.opennic.rfx.fi
53: sturmovik.rfx.fi
54: www.sturmovik.rfx.fi
55: voynawiki.rfx.fi
56: www.voynawiki.rfx.fi
57: www.rfx.fi
58: volk.ssr
59: www.volk.ssr
60: wiki.oss
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 47 48
Simulating a certificate request for www.globalwiki.rfx.fi and globalwiki.rfx.fi
Performing the following challenges:
http-01 challenge for globalwiki.rfx.fi
http-01 challenge for www.globalwiki.rfx.fi
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Cleaning up challenges

IMPORTANT NOTES:
 - The dry run was successful.

That makes it sound like it should work.

Did it pause for the debug?

I concur that if the dry run works then this should work:

certbot certonly --nginx

It that works then this is the next step:

certbot --nginx --keep

(I'm segregating the acquisition and installation steps here.)

I ran the certbot --nginx --keep command and it had the same failure.

I'll try running those two commands and see if it works.

Edit: I'm at the rate so I'll try soon later.

If the staging works and the production doesn't then I'm beginning to suspect a DNS issue where the LE staging server is possibly seeing a different version of your webserver than the LE production server.

Are you using a DigitalOcean load balancer, perchance?

If so, you might want to read this and save yourself many headaches:

It's on Digital Ocean. I checked resolv.conf:

 cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
#     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN

nameserver 206.189.193.106
nameserver 2001:4860:4860::8844
nameserver 2001:4860:4860::8888 
nameserver 8.8.8.8

I think that's a server I made to resolve OpenNIC domains but certbot worked in the past with it.

I only asked because many people find DigitalOcean's integrated Let's Encrypt solution to be far easier to manage than certbot.

I might just use that then. It looks like the LB takes all the traffic from https and then sends it over http to the web server?

That's correct and known as "SSL termination at the LB".

Upside:
No need to configure certificates for each backend worker.

Downside:
Puts burden of decryption on LB.

I'm going to try setting it up for the site that I'm working on.

Excellent. Keep us posted.

:+1:

I've gotta run for now, but if you need more immediate help my colleagues are more than eager.