Challenges failed for renewal

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
covid-see.com

I ran this command:
sudo certbot renew --dry-run

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/covid-see.cis.unimelb.edu.au.conf


Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Simulating renewal of an existing certificate for covid-see.cis.unimelb.edu.au
Performing the following challenges:
http-01 challenge for covid-see.cis.unimelb.edu.au
Waiting for verification...
Challenge failed for domain covid-see.cis.unimelb.edu.au
http-01 challenge for covid-see.cis.unimelb.edu.au
Cleaning up challenges
Failed to renew certificate covid-see.cis.unimelb.edu.au with error: Some challenges have failed.


Processing /etc/letsencrypt/renewal/covid-see.com.conf


Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Simulating renewal of an existing certificate for covid-see.com and www.covid-see.com
Performing the following challenges:
http-01 challenge for covid-see.com
http-01 challenge for www.covid-see.com
Waiting for verification...
Challenge failed for domain covid-see.com
Challenge failed for domain www.covid-see.com
http-01 challenge for covid-see.com
http-01 challenge for www.covid-see.com
Cleaning up challenges
Failed to renew certificate covid-see.com with error: Some challenges have failed.


All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/fullchain.pem (failure)
/etc/letsencrypt/live/covid-see.com/fullchain.pem (failure)


2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My web server is (include version):
nginx version: nginx/1.21.6

The operating system my web server runs on is (include version):
cat /etc/os-release
NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"
PRETTY_NAME="Red Hat Enterprise Linux"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:redhat:enterprise_linux:7.9:GA:server"
HOME_URL="https://www.redhat.com/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

My hosting provider, if applicable, is:
N/A

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.11.0


The acme-challenge files seem not to be accessible. The server appears to be redirecting despite including this file in nginx.conf:

configuration file /etc/nginx/letsencrypt-acme-challenge.conf:

location ^~ /.well-known/acme-challenge/ {
default_type "text/plain";
root /var/www/html/acme;
break;
}
location = /.well-known/acme-challenge/ {
return 404;
}

Referenced in nginx.conf in a few places:

include /etc/nginx/conf.d/*.conf;

server {
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    server_name www.covid-see.cis.unimelb.edu.au covid-see.cis.unimelb.edu.au;
include /etc/nginx/letsencrypt-acme-challenge.conf;

location / {
        return 301 https://covid-see.com$request_uri;
}
}

server {
    listen 443 ssl; # managed by Certbot
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
add_header Content-Security-Policy upgrade-insecure-requests;
    ssl_certificate /etc/letsencrypt/live/covid-see.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/covid-see.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    server_name www.covid-see.com covid-see.com;
include /etc/nginx/letsencrypt-acme-challenge.conf;

    root /srv/silo-q36/nlp-covid19/COVID-SEE/covidsee/WebApp/client/build;

    index index.html;

   #snip

    location / {
       try_files $uri /index.html;
    }

}

server {
    listen       80 default_server;
    listen       [::]:80 default_server;
server_name covid-see.com www.covid-see.com;

    include /etc/nginx/letsencrypt-acme-challenge.conf;

    root         /var/www/html;

location / {
	#redirect http to https www
        return 301 https://$host$request_uri;
}

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

# snip        

    error_page 404 /404.html;
        location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
        location = /50x.html {
    }
}

I put a test file in /var/www/html/acme but it is being redirected:

% curl -Iki http://covid-see.com/.well-known/acme-challenge/test
HTTP/1.1 301 Moved Permanently
Server: nginx/1.21.6
Date: Mon, 20 Jun 2022 11:10:20 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
Location: https://covid-see.com/.well-known/acme-challenge/test


I suspect I've made a mess of the configuration but I can't figure this out. Please help!

Thank you.

2 Likes

Okay I manually created the /var/www/html/acme/.well-known/acme-challenge folder and moved "test" into it. Now it is at least resolving okay I think:

% curl -Iki http://covid-see.com/.well-known/acme-challenge/test
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Mon, 20 Jun 2022 11:34:11 GMT
Content-Type: text/plain
Content-Length: 16
Last-Modified: Mon, 20 Jun 2022 10:42:36 GMT
Connection: keep-alive
ETag: "62b04f1c-10"
Accept-Ranges: bytes

However sudo certbot renew --dry-run is still failing as previous.

1 Like

Thanks for the detailed thread opening post, makes debugging already a lot easier.

However, I'd very much like to see the contents of the files /etc/letsencrypt/renewal/covid-see.cis.unimelb.edu.au.conf and /etc/letsencrypt/renewal/covid-see.com.conf.

3 Likes
$ more /etc/letsencrypt/renewal/covid-see.cis.unimelb.edu.au.conf
# renew_before_expiry = 30 days
version = 1.6.0
archive_dir = /etc/letsencrypt/archive/covid-see.cis.unimelb.edu.au
cert = /etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/cert.pem
privkey = /etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/privkey.pem
chain = /etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/chain.pem
fullchain = /etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = nginx
installer = nginx
account = b076e501906df108fedced787721e504
server = https://acme-v02.api.letsencrypt.org/directory
$ more /etc/letsencrypt/renewal/covid-see.com.conf
# renew_before_expiry = 30 days
version = 1.6.0
archive_dir = /etc/letsencrypt/archive/covid-see.com
cert = /etc/letsencrypt/live/covid-see.com/cert.pem
privkey = /etc/letsencrypt/live/covid-see.com/privkey.pem
chain = /etc/letsencrypt/live/covid-see.com/chain.pem
fullchain = /etc/letsencrypt/live/covid-see.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = nginx
installer = nginx
account = b076e501906df108fedced787721e504
server = http://acme-v02.api.letsencrypt.org/directory
1 Like

Please show output of:
certbot certificates

and note:
*** 8.8.8.8 can't find www.covid-see.cis.unimelb.edu.au: Non-existent domain

10 Likes

Not sure how it worked before, but chances are your rather complex nginx configuration doesn't work well with the nginx authenticator plugin.

You might want to consider using the webroot authenticator, but keep the nginx installer plugin using the options -a webroot -i nginx. You should then also use the -w /var/www/html/acme option to point Certbot to the correct webroot.

2 Likes

$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Found the following certs:
Certificate Name: covid-see.cis.unimelb.edu.au
Serial Number: 3c11a47ab18de978da1e8a20dd9781874da
Key Type: RSA
Domains: covid-see.cis.unimelb.edu.au
Expiry Date: 2022-04-17 00:23:01+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/fullchain.pem
Private Key Path: /etc/letsencrypt/live/covid-see.cis.unimelb.edu.au/privkey.pem
Certificate Name: covid-see.com
Serial Number: 3192b4a6477006b3b94fb14e228de400c10
Key Type: RSA
Domains: covid-see.com www.covid-see.com
Expiry Date: 2022-04-29 12:40:58+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/covid-see.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/covid-see.com/privkey.pem


1 Like

Thanks I tried this. Please let me know if I'm not doing it correctly:

$ sudo certbot certonly -a webroot -w /var/www/html/acme -i nginx --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer nginx
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: covid-see.com
2: www.covid-see.com
3: covid-see.cis.unimelb.edu.au
4: www.covid-see.cis.unimelb.edu.au
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/covid-see.com.conf)

It contains these names: covid-see.com, www.covid-see.com

You requested these names for the new certificate: covid-see.com,
www.covid-see.com, covid-see.cis.unimelb.edu.au,
www.covid-see.cis.unimelb.edu.au.

Do you want to expand and replace this existing certificate with the new
certificate?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(E)xpand/(C)ancel: 
(E)xpand/(C)ancel: E
Simulating renewal of an existing certificate for covid-see.com and 3 more domains
Performing the following challenges:
http-01 challenge for covid-see.cis.unimelb.edu.au
http-01 challenge for covid-see.com
http-01 challenge for www.covid-see.cis.unimelb.edu.au
http-01 challenge for www.covid-see.com
Using the webroot path /var/www/html/acme for all unmatched domains.
Waiting for verification...
Challenge failed for domain covid-see.com
Challenge failed for domain www.covid-see.cis.unimelb.edu.au
Challenge failed for domain www.covid-see.com
Challenge failed for domain covid-see.cis.unimelb.edu.au
http-01 challenge for covid-see.com
http-01 challenge for www.covid-see.cis.unimelb.edu.au
http-01 challenge for www.covid-see.com
http-01 challenge for covid-see.cis.unimelb.edu.au
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: covid-see.com
   Type:   unauthorized
   Detail: 128.250.59.15: Invalid response from
   http://covid-see.com/.well-known/acme-challenge/p17dcHAkK08m90C-H0ilHS3BvEu5EHKFyQVULvnb4kU:
   503

   Domain: www.covid-see.com
   Type:   unauthorized
   Detail: 128.250.59.15: Invalid response from
   http://www.covid-see.com/.well-known/acme-challenge/f8d1CNZXzm34BBTzasXGlUwctGEmMiRQgHYU7idrawc:
   503

   Domain: covid-see.cis.unimelb.edu.au
   Type:   unauthorized
   Detail: 128.250.59.15: Invalid response from
   http://covid-see.cis.unimelb.edu.au/.well-known/acme-challenge/K62kGFxSLizifMstU3pX7Q9lkqoiT2dgETNDL1Vp-EE:
   503

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: www.covid-see.cis.unimelb.edu.au
   Type:   dns
   Detail: DNS problem: NXDOMAIN looking up A for
   www.covid-see.cis.unimelb.edu.au - check that a DNS record exists
   for this domain; DNS problem: NXDOMAIN looking up AAAA for
   www.covid-see.cis.unimelb.edu.au - check that a DNS record exists
   for this domain

I'm separately working on the DNS issue for www.covid-see.cis.unimelb.edu.au but I'm not too concerned about that one -- mostly using the primary covid-see.com

Thanks

2 Likes

Hm, 503 error, that suggests "service unavailable". Weird. Anything helpful in the nginx logs?

Also, I noticed the test file at http://covid-see.com/.well-known/acme-challenge/test is gone.

3 Likes

Interesting. I manually recreated the .well-known/acme-challenge directory (no idea why it disappeared!!) and put the test file there again.

The logs show
022/06/21 17:48:12 [notice] 21938#21938: signal process started
2022/06/21 17:50:36 [error] 21940#21940: *14 open() "/var/www/letsencrypt/.well-known/acme-challenge/c5OLn_Oa2I-IBySS7qIbZI0fbzQmok_Csqvbg2NvkiI" failed (2: No such file or directory), client: 10.240.58.157, server: covid-see.com, request: "GET /.well-known/acme-challenge/c5OLn_Oa2I-IBySS7qIbZI0fbzQmok_Csqvbg2NvkiI HTTP/1.1", host: "covid-see.com"
2022/06/21 17:54:22 [error] 22256#22256: *3 open() "/var/www/html/sean" failed (2: No such file or directory), client: 124.170.102.13, server: covid-see.com, request: "GET /sean HTTP/1.1", host: "128.250.59.15"
2022/06/21 17:54:37 [error] 22258#22258: *5 open() "/var/www/html/sean" failed (2: No such file or directory), client: 65.154.226.168, server: covid-see.com, request: "GET /sean HTTP/1.1", host: "128.250.59.15"
2022/06/21 17:55:06 [error] 22256#22256: *7 open() "/var/www/html/sean" failed (2: No such file or directory), client: 49.184.174.255, server: covid-see.com, request: "GET /sean HTTP/1.1", host: "128.250.59.15"
2022/06/21 17:55:17 [
84.174.255, server: covid-see.com, request: "GET /sean2 HTTP/1.1", host: "128.250.59.15"
2022/06/21 17:55:34 [notice] 22361#22361: signal process started

Interestingly, I just tried this again

sudo certbot certonly -a webroot -w /var/www/html/acme -i nginx --dry-run

and don't see an error in the log for that, although the challenge failed again (same Invalid Response/503 error). Oh wait, maybe there is:

2022/06/21 18:02:34 [error] 22376#22376: *18 open() "/var/www/letsencrypt/.well-known/acme-challenge/1NDL7Q-PD8YDU3QUB2XSHXPA9DF7UV71" failed (2: No such file or directory), client: 173.248.191.182, server: covid-see.com, request: "GET /.well-known/acme-challenge/1NDL7Q-PD8YDU3QUB2XSHXPA9DF7UV71 HTTP/1.1", host: "covid-see.com"
2022/06/21 18:02:34 [error] 22378#22378: *19 open() "/var/www/letsencrypt/.well-known/acme-challenge/IJNL0310JS973AW1XB6_K1NEU_13QLHQ" failed (2: No such file or directory), client: 173.248.191.182, server: covid-see.com, request: "GET /.well-known/acme-challenge/IJNL0310JS973AW1XB6_K1NEU_13QLHQ HTTP/1.1", host: "www.covid-see.com"

1 Like

Where does that directory come from? It is not in the configuration files you've posted.

2 Likes

Ha! Good catch. Looks like one of my colleagues trying to be helpful went in there and started editing things (without warning me).

I'm trying to revert. The really strange thing is that I am getting challenge failed, but no error in the log.
#confused :confused:

1 Like

Can we have a look at this file?:

8 Likes

Sorry for the delay. Yes, it is:

location ^~ /.well-known/acme-challenge/ {
    default_type "text/plain";
    root         /var/www/html/acme;
}
location = /.well-known/acme-challenge/ {
    return 404;
}

This seems to be working; if I go to http://covid-see.com/.well-known/acme-challenge/test I see the contents of the file in /var/www/html/acme/.well-known/acme-challenge/test.

However, the challenge is still failing.

What is interesting is that when I run the cerbot renew command
sudo certbot certonly -a webroot -w /var/www/html/acme -i nginx

there is no evidence of any access to the nginx server in the access or error logs. I would have expected to see something there.

When I checked I see the following, so it looks like it is hitting a 403 error even though it is reporting a 503 error:

{
  "identifier": {
    "type": "dns",
    "value": "covid-see.com"
  },
  "status": "invalid",
  "expires": "2022-07-02T10:53:22Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "128.250.59.15: Invalid response from http://covid-see.com/.well-known/acme-challenge/UypD66FJWko1Ld6lpbd_zY6lz9o0AiSICAHGU4eVuTI: 503",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/123491213226/i3IXBA",
      "token": "UypD66FJWko1Ld6lpbd_zY6lz9o0AiSICAHGU4eVuTI",
      "validationRecord": [
        {
          "url": "http://covid-see.com/.well-known/acme-challenge/UypD66FJWko1Ld6lpbd_zY6lz9o0AiSICAHGU4eVuTI",
          "hostname": "covid-see.com",
          "port": "80",
          "addressesResolved": [
            "128.250.59.15"
          ],
          "addressUsed": "128.250.59.15"
        }
      ],
      "validated": "2022-06-25T10:53:23Z"
    }
  ]
}
1 Like

Does that return 404 to all requests?
If so, I'd change that to allow the challenge requests access to the required files.

This doesn't explain, nor correct, the 503 errors but would likely cause a problem once that part is corrected.

8 Likes

Sorry, but I'm not going to be able to help any further:

3 Likes

Yes, but it is an "=" comparison so only exactly that uri. Seems unnecessary but harmless. When = is matched it also causes an immediate end to location matching

10 Likes

Thanks, I followed the instructions from this related post

My understanding is that the previous statement should allow the challenge requests access:

location ^~ /.well-known/acme-challenge/ {
    default_type "text/plain";
    root         /var/www/html/acme;
}
2 Likes

I appreciate your help so far.

I have no idea why you would get a Phishing response for that? There is literally nothing in that subdirectory apart from the single test file I put there; not even the certificate challenge files that I guess should be there!

1 Like

@nanana There is (almost certainly) a Palo Alto Networks firewall blocking requests. It is issuing the http error 503. You should have your network people look at the Application Rules and be sure that the "acme-challenge" is allowed both inbound and outbound.

We have seen this problem regularly with this brand of firewalls the past couple months. The vendor changed the default for the rule.

Here are sample requests that show the failure:

curl -I -m10 covid-see.com/.well-known/acme-challenge/test
(see http 200 ok for your test file - using curl default user-agent string)
HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Sun, 26 Jun 2022 13:44:33 GMT
Content-Type: text/plain
Content-Length: 29
Last-Modified: Tue, 21 Jun 2022 07:49:42 GMT
Connection: keep-alive
ETag: "62b17816-1d"
Accept-Ranges: bytes

curl -I -m10 covid-see.com/.well-known/acme-challenge/test -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"  
(request fails 503 when using user-agent string for the Let's Encrypt server)
HTTP/1.1 503 Service Unavailable
Content-Type: text/html; charset=UTF-8
Content-Length: 1091
Connection: close
P3P: CP="CAO PSA OUR"
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
11 Likes