Unable to renew certs, Invalid response from well-known acme-challenge 404

kcenter · March 8, 2023, 6:43pm

I ran this command: certbot renew

It produced this output:

 - The following errors were reported by the server:

   Domain: cloud.copythis.net
   Type:   unauthorized
   Detail: 2606:4700:3037::6815:35c9: Invalid response from
   https://cloud.copythis.net/index.php/login: "<!DOCTYPE html>\n<html
   class=\"ng-csp\" data-placeholder-focus=\"false\" lang=\"en\"
   data-locale=\"en\" >\n\t<head\n data-requesttoken=\"6b8s"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: kasm.copythis.net
   Type:   unauthorized
   Detail: 2606:4700:3030::ac43:da7b: Invalid response from
   https://kasm.copythis.net/.well-known/acme-challenge/buYPOnpRbAuYZzW0JS_PuP8aoeR7o_7vim83ZElXksY:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: vault.copythis.net
   Type:   unauthorized
   Detail: 2606:4700:3037::6815:35c9: Invalid response from
   https://vault.copythis.net/.well-known/acme-challenge/IAcu7-l6MerKFb7loaVmmz3DvwFutMK7r_G5z7JcDeo:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version): nginx version: nginx/1.18.0

The operating system my web server runs on is (include version):

Linux - Debian 11 - Turnkey ngninx-php-fastcgi CT on Proxmox 6.4

My hosting provider, if applicable, is: self hosted, Cloudflare DNS

I can login to a root shell on my machine (yes or no, or I don't know):

yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 1.12.0

I can still access all the sites just fine. I haven't changed anything to the individual configs in my nginx config file, only added additional servers as I have added more services.

All individual services are ran in either separate VM's or CT

nginx config:

server {
    server_name vault.copythis.net;

    location / {
        proxy_pass http://192.168.1.238:8000; # bitwarden server address
    }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/vault.copythis.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/vault.copythis.net/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = vault.copythis.net) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name vault.copythis.net;
    return 404; # managed by Certbot


}

server {
    server_name kasm.copythis.net;

    location / {
         # The following configurations must be configured when proxying to Kasm Workspaces

         # WebSocket Support
         proxy_set_header        Upgrade $http_upgrade;
         proxy_set_header        Connection "upgrade";

         # Host and X headers
         proxy_set_header        Host $host;
         proxy_set_header        X-Real-IP $remote_addr;
         proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
         proxy_set_header        X-Forwarded-Proto $scheme;

         # Connectivity Options
         proxy_http_version      1.1;
         proxy_read_timeout      1800s;
         proxy_send_timeout      1800s;
         proxy_connect_timeout   1800s;
         proxy_buffering         off;

         # Allow large requests to support file uploads to sessions
         client_max_body_size 10M;

         # Proxy to Kasm Workspaces running locally on 8443 using ssl
         proxy_pass https://192.168.1.231;
     }
 
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/kasm.copythis.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/kasm.copythis.net/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}


server {
    if ($host = kasm.copythis.net) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    server_name kasm.copythis.net;
    return 404; # managed by Certbot


}


server {
   server_name cloud.copythis.net;
   
   location / {
      proxy_pass https://192.168.1.225;
        proxy_set_header        Host $host;
        proxy_set_header        X-Real-IP $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header        X-Forwarded-Proto $scheme;
}


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/cloud.copythis.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/cloud.copythis.net/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}




server {
    if ($host = cloud.copythis.net) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


   server_name cloud.copythis.net;
    listen 80;
    return 404; # managed by Certbot


}

server {
    server_name  tail.copythis.net;
    location / {
        proxy_pass http://192.168.1.220:8080/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_redirect http:// https://;
        proxy_buffering off;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
        add_header Strict-Transport-Security "max-age=15552000; includeSubDomains" always;
    }


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/tail.copythis.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/tail.copythis.net/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}


server {
    if ($host = tail.copythis.net) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    server_name  tail.copythis.net;
    listen 80;
    return 404; # managed by Certbot


}

MikeMcQ · March 8, 2023, 6:59pm

Cloudflare isn't just your DNS provider but you have the DNS proxied to also use the Cloudflare CDN. Was that also your config when you got your certs in Jan?

Because getting certs "behind" a CDN takes extra effort. Do you want to have Cloudflare be your CDN in addition to DNS provider? Or was this a recent change which you don't have completely sorted out yet with your origin server?

kcenter · March 8, 2023, 7:48pm

I am fairly certain Cloudfare proxied was turned on when I originally pulled the certs.

I can try turning proxied off and seeing if it helps.

Edit: I realized I may have not completely answered your question. I have left most of Cloudfare's settings as default, including leaving 'proxied' on for my A record and my CNAME records.

Only change I can think of that I made at all, and it was since setting up these services and pulling certs originally was to add a rule to Cloudflare's Web Application Firewall to block all traffic from outside the USA.

kcenter · March 8, 2023, 10:34pm

Well I found a workaround. I don't know if this was the correct solution or not, but I delete all the certs through certbot

certbot delete

remove all references to the certs from the ngnix sites file, and added to each server block:

    location '/.well-known/acme-challenge' {
         root /var/www/webroot/service;
    }

changing out "service" for the name of each service. Then made each of those folders in /var/www/webroot

restarted nginx, and reran certbot for each service. It successfully pulled certs for each and everything seems to be working correctly now.

We will see if I have issues in the future renewing certificates or not.

_az · March 8, 2023, 10:45pm

Glad to hear it's fixed.

If you were using the --nginx plugin in a version of Certbot older than v1.13.0, it would fail to get a certificate if you had your Cloudflare Encryption Mode set to "Full" or "Full (Strict)".

If you use --webroot, you need to make sure that the location is present in both the HTTP and HTTPS server blocks, because requests may come in on one protocol or the other, depending on your Cloudflare settings.

MikeMcQ · March 9, 2023, 11:19am

You could test that now with

sudo certbot renew --dry-run

(omit sudo if not required)

The --dry-run uses the Staging system and will not affect your current production certs

orange-kush · March 9, 2023, 5:30pm

Hello, I'm having the same problem but couldn't make it work following the steps you provided. I may be missing something. I didn't have DNS when I issued the first certificate, and now i just need to renew them but I'm having this same problem.

After deleting certs I've modified my /etc/nginx/conf.d/my-site.conf file and added the block you mentioned, while commenting any reference to previous certificates.

Once that done I run sudo certbot certonly --domain "my-site" and chose option 3:

3: Saves the necessary validation files to a .well-known/acme-challenge/
directory within the nominated webroot path. A seperate HTTP server must be
running and serving files from the webroot path. HTTP challenge only (wildcards
not supported). (webroot)

When prompted to add the webroot I add the directory I created (/var/www/webroot/my-site), but it still gives me the same error as before, though the Hint now changed to the following one:

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Could you point out if you went through the same process, or you did anything different than me?

Thank you.

MikeMcQ · March 9, 2023, 5:46pm

If the same steps didn't fix your problem then it is not the same one

You should start a new Help topic and answer the questions on the form you will be shown. This will give us info to help you.

orange-kush · March 9, 2023, 5:57pm

Done, thank you.

system · April 8, 2023, 5:57pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Error Invalid response when renew Help	12	3983	March 7, 2019
Certbot renewal unauthorized error, acme-challenge invalid response Help	10	5108	January 17, 2020
Certbot renew bug Help	5	1272	August 30, 2018
Auto renew failed - invalid response Help	8	687	March 16, 2021
Failed authorization procedure Server	20	2150	May 6, 2019

Unable to renew certs, Invalid response from well-known acme-challenge 404

Related topics