Cannot renew with certbot: Failed to renew certificate www.apkdcx.com with error: Some challenges have failed

entropy67 · April 23, 2024, 3:29am

Hi LE team, I'm trying to renew my ssl certificate (it was expired a month ago). However, I got the following error. Could anyone help to provide any suggestions on how to debug on this? Thanks a lot for your time and attention!

My domain is: www.apkdcx.com

I ran this command: certbot renew --dry-run -v

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/www.apkdcx.com.conf

Certificate is due for renewal, auto-renewing...

Plugins selected: Authenticator webroot, Installer None

Simulating renewal of an existing certificate for www.apkdcx.com

Performing the following challenges:

http-01 challenge for www.apkdcx.com

Using the webroot path /var/www/certbot for all unmatched domains.

Waiting for verification...

Challenge failed for domain www.apkdcx.com

http-01 challenge for www.apkdcx.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:

Domain: www.apkdcx.com

Type: unauthorized

Detail: 8.134.195.218: Invalid response from https://www.apkdcx.com/.well-known/acme-challenge/vgFJ8hEqGgCw3CDZu3UTa__j1S6Y_pK6EQ9ZGHxSTW8: "<!doctype html><html lang="en"><meta charset="utf-8"/><link rel="icon" href="./favicon.ico"/><meta name="viewport" content"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Cleaning up challenges

Failed to renew certificate www.apkdcx.com with error: Some challenges have failed.

All simulated renewals failed. The following certificates could not be renewed:

/etc/letsencrypt/live/www.apkdcx.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): nginxinc/nginx-unprivileged:1-alpine

The operating system my web server runs on is (include version): Ubuntu 22.04.3 LTS

My hosting provider, if applicable, is: Unknown

I can login to a root shell on my machine (yes or no, or I don't know): Yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.8.0

Additional information:

It seems Niginx returned 200 when challenged:

proxy | 66.133.109.36 - - [22/Apr/2024:07:39:33 +0000] "GET /.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk HTTP/1.1" 302 138 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
proxy | 13.53.235.28 - - [22/Apr/2024:07:39:33 +0000] "GET /.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk HTTP/1.1" 302 138 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
proxy | 18.143.92.227 - - [22/Apr/2024:07:39:33 +0000] "GET /.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk HTTP/1.1" 302 138 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
proxy | 18.218.184.140 - - [22/Apr/2024:07:39:33 +0000] "GET /.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk HTTP/1.1" 302 138 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
proxy | 18.143.92.227 - - [22/Apr/2024:07:39:33 +0000] "GET /.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk HTTP/1.1" 200 667 "http://www.apkdcx.com/.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
proxy | 66.133.109.36 - - [22/Apr/2024:07:39:33 +0000] "GET /.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk HTTP/1.1" 200 667 "http://www.apkdcx.com/.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
proxy | 13.53.235.28 - - [22/Apr/2024:07:39:33 +0000] "GET /.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk HTTP/1.1" 200 667 "http://www.apkdcx.com/.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4dWgQyk" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

Here is the trace in the log file:

Link: https://acme-staging-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: _O0fw7ZkabkErWy6X3DOE9QUFu6UyNCHIqHbWYltFnt3MDuvyfA
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"identifier": {
"type": "dns",
"value": "www.apkdcx.com"
},
"status": "invalid",
"expires": "2024-04-29T07:39:31Z",
"challenges": [
{
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietf:params:acme:error:unauthorized",
"detail": "8.134.195.218: Invalid response from https://www.apkdcx.com/.well-known/acme-challenge/ETVm_dv7IQnxhi5nY6LvmHVT1dfhoNAV2hqz4d
WgQyk: "\u003c!doctype html\u003e\u003chtml lang=\"en\"\u003e\u003chead\u003e\u003cmeta charset=\"utf-8\"/\u003e\u003clink rel=\"icon
\" href=\"./favicon.ico\"/\u003e\u003cmeta name=\"viewport\" content"",
"status": 403
},
k3ADaW3PyWSbYmMs
2024-04-22 07:50:42,274:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2024-04-22 07:50:42,275:ERROR:certbot._internal.renewal:Failed to renew certificate www.apkdcx.com with error: Some challenges have failed.
2024-04-22 07:50:42,276:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
File "/opt/certbot/src/certbot/certbot/_internal/renewal.py", line 540, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1550, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 131, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/opt/certbot/src/certbot/certbot/_internal/renewal.py", line 399, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 428, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

MikeMcQ · April 23, 2024, 4:11am

Welcome to the community @entropy67

It looks like you reject requests that cannot handle javascript. See sample ACME Challenge response below. You'll see that test response looks like the text in the error message from Certbot. The Let's Encrypt Server must be sent the challenge token and not require it to handle Javascript or any other scripting language.

Ideally you would handle the challenge in your port 80 server block. We can help you set that up if you show that server block. Please place 3 backticks before and after to maintain formatting like:
```
nginx server block for port 80 for domain www.apkdcx.com
```

Here is result of a sample ACME HTTP challenge. You really should reply with a 404 in this case but the data is what is more important right now.

Also, this is HTTPS request because you redirect HTTP to HTTPS. Best if the challenge is not redirected but I needed to do this so you can see the real reason for the challenge failing.

curl -ik https://www.apkdcx.com/.well-known/acme-challenge/Test404
HTTP/1.1 200 OK
Server: nginx/1.25.3
(other headers omitted)

<!doctype html><html lang="en"><head>
<meta charset="utf-8"/>
<link rel="icon" href="./favicon.ico"/>
<meta name="viewport" content="width=device-width,initial-scale=1"/><meta name="theme-color" content="#000000"/>
<meta name="description" content="Web site created using create-react-app"/><link rel="apple-touch-icon" href="./logo.JPG"/><link rel="manifest" href="./manifest.json"/><title>湛汽长途车票管理平台</title>
<script defer="defer" src="./static/js/main.2fd72cf1.js"></script><link href="./static/css/main.2c3fdfbb.css" rel="stylesheet"></head><body>
<noscript>You need to enable JavaScript to run this app.</noscript><div id="root"></div></body></html>

entropy67 · April 23, 2024, 5:19am

Hi Mike @MikeMcQ

Thank you so much for your prompt and detailed response. It is very helpful. Your explanation is very clear and totally make sense to me.

Here is how I setup the nginx server

port 80

server {
    listen 80;
    server_name apkdcx.com www.apkdcx.com;
    server_tokens off;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://www.apkdcx.com$request_uri;
    }
}

port 443

server {
    listen 443 ssl;

    server_name  www.apkdcx.com;

    ssl_certificate /etc/nginx/ssl/live/www.apkdcx.com/fullchain.pem;
    ssl_certificate_key /etc/nginx/ssl/live/www.apkdcx.com/privkey.pem;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location /static {
        alias /vol/static;
    }

    location /api {
        uwsgi_pass           ${APP_HOST}:${APP_PORT};
        include              /etc/nginx/uwsgi_params;
        client_max_body_size 10M;
    }

    location /dashboard {
        proxy_pass http://client/;
    }

    location / {
        proxy_pass http://client/;
    }
}

It would be very very helpful if you could guide me on how to set up the nginx server. Really appreciate your time and help.

MikeMcQ · April 23, 2024, 8:37am

Your port 80 server block looks perfect. But, for some reason it is not the one processing HTTP requests to your domain. You should check the dns to make sure it is the correct public ip for your server. The text in the error message talks about a react app. That mean anything to you?

Your port 443 server block should have its server_name line changed so that it includes both of your domain names. You have it correct in your port 80 block.

Look at the result below and you will see that HTTP challenge requests are getting redirected to HTTPS. That should not be happening if your port 80 server block that you showed is the one doing the processing. I know the result says OK but ignore that. It is mostly just testing the communications and some other items. It can communicate to your server but the response isn't the one needed

rg305 · April 24, 2024, 2:14am

This looks more like it was redirected to HTTPS:

I must agree.

We should review the entire nginx config, with:

`nginx -T`

rg305 · April 24, 2024, 2:15am

After the redirection problem is addressed...
This also looks inconsistent:

entropy67 · April 25, 2024, 5:23am

@rg305 @MikeMcQ Thank you so much for your reply! Appreciate it a lot!!

You should check the dns to make sure it is the correct public ip for your server.

I checked the ip address and it is the correct public ip for our server.

The text in the error message talks about a react app. That mean anything to you?

I think this is expected. Any general URL (without prefix /api) will be directed to our React frontend service.

Your port 443 server block should have its server_name line changed so that it includes both of your domain names.

That's a great catch. I will update it. But to me it seems not to be the root cause of the problem.

We should review the entire nginx config, with: nginx -T

Here is the full output of nginx -T:

/ $ nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

configuration file /etc/nginx/nginx.conf:

worker_processes auto;

error_log /var/log/nginx/error.log notice;
pid /tmp/nginx.pid;

events {
worker_connections 1024;
}

http {
proxy_temp_path /tmp/proxy_temp;
client_body_temp_path /tmp/client_temp;
fastcgi_temp_path /tmp/fastcgi_temp;
uwsgi_temp_path /tmp/uwsgi_temp;
scgi_temp_path /tmp/scgi_temp;

include       /etc/nginx/mime.types;
default_type  application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

keepalive_timeout  65;

#gzip  on;

include /etc/nginx/conf.d/*.conf;

}

configuration file /etc/nginx/mime.types:

types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;

text/mathml                                      mml;
text/plain                                       txt;
text/vnd.sun.j2me.app-descriptor                 jad;
text/vnd.wap.wml                                 wml;
text/x-component                                 htc;

image/avif                                       avif;
image/png                                        png;
image/svg+xml                                    svg svgz;
image/tiff                                       tif tiff;
image/vnd.wap.wbmp                               wbmp;
image/webp                                       webp;
image/x-icon                                     ico;
image/x-jng                                      jng;
image/x-ms-bmp                                   bmp;

font/woff                                        woff;
font/woff2                                       woff2;

application/java-archive                         jar war ear;
application/json                                 json;
application/mac-binhex40                         hqx;
application/msword                               doc;
application/pdf                                  pdf;
application/postscript                           ps eps ai;
application/rtf                                  rtf;
application/vnd.apple.mpegurl                    m3u8;
application/vnd.google-earth.kml+xml             kml;
application/vnd.google-earth.kmz                 kmz;
application/vnd.ms-excel                         xls;
application/vnd.ms-fontobject                    eot;
application/vnd.ms-powerpoint                    ppt;
application/vnd.oasis.opendocument.graphics      odg;
application/vnd.oasis.opendocument.presentation  odp;
application/vnd.oasis.opendocument.spreadsheet   ods;
application/vnd.oasis.opendocument.text          odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                 pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                 xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                 docx;
application/vnd.wap.wmlc                         wmlc;
application/wasm                                 wasm;
application/x-7z-compressed                      7z;
application/x-cocoa                              cco;
application/x-java-archive-diff                  jardiff;
application/x-java-jnlp-file                     jnlp;
application/x-makeself                           run;
application/x-perl                               pl pm;
application/x-pilot                              prc pdb;
application/x-rar-compressed                     rar;
application/x-redhat-package-manager             rpm;
application/x-sea                                sea;
application/x-shockwave-flash                    swf;
application/x-stuffit                            sit;
application/x-tcl                                tcl tk;
application/x-x509-ca-cert                       der pem crt;
application/x-xpinstall                          xpi;
application/xhtml+xml                            xhtml;
application/xspf+xml                             xspf;
application/zip                                  zip;

application/octet-stream                         bin exe dll;
application/octet-stream                         deb;
application/octet-stream                         dmg;
application/octet-stream                         iso img;
application/octet-stream                         msi msp msm;

audio/midi                                       mid midi kar;
audio/mpeg                                       mp3;
audio/ogg                                        ogg;
audio/x-m4a                                      m4a;
audio/x-realaudio                                ra;

video/3gpp                                       3gpp 3gp;
video/mp2t                                       ts;
video/mp4                                        mp4;
video/mpeg                                       mpeg mpg;
video/quicktime                                  mov;
video/webm                                       webm;
video/x-flv                                      flv;
video/x-m4v                                      m4v;
video/x-mng                                      mng;
video/x-ms-asf                                   asx asf;
video/x-ms-wmv                                   wmv;
video/x-msvideo                                  avi;

}

configuration file /etc/nginx/conf.d/default.conf:

upstream client {
server client:3000;
}

server {
listen 8000;
server_name apkdcx.com www.apkdcx.com;
server_tokens off;

location /.well-known/acme-challenge {
    root /var/www/certbot;
}

rewrite ^(.*)$ https://www.apkdcx.com$1;

}

server {
listen 443 ssl;

server_name  www.apkdcx.com;
ssl_certificate /etc/nginx/ssl/live/www.apkdcx.com/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/live/www.apkdcx.com/privkey.pem;

location /static {
    alias /vol/static;
}

location /api {
    uwsgi_pass           backend:9000;
    include              /etc/nginx/uwsgi_params;
    client_max_body_size 10M;
}

location /dashboard {
    proxy_pass http://client/;
}

location / {

proxy_pass http://client/;
}
}

configuration file /etc/nginx/uwsgi_params:

uwsgi_param QUERY_STRING $query_string;
uwsgi_param REQUEST_METHOD $request_method;
uwsgi_param CONTENT_TYPE $content_type;
uwsgi_param CONTENT_LENGTH $content_length;
uwsgi_param REQUEST_URI $request_uri;
uwsgi_param PATH_INFO $document_uri;
uwsgi_param DOCUMENT_ROOT $document_root;
uwsgi_param SERVER_PROTOCOL $server_protocol;
uwsgi_param REMOTE_ADDR $remote_addr;
uwsgi_param REMOTE_PORT $remote_port;
uwsgi_param SERVER_ADDR $server_addr;
uwsgi_param SERVER_PORT $server_port;
uwsgi_param SERVER_NAME $server_name;

Again, thank you so much for your time and input! This is already very very helpful.

rg305 · April 25, 2024, 6:15am

I only see two vhosts: One serving port 8000 and the other port 443.
Where do the HTTP ACME challenge requests get served?
I suspect it was intended to be served by this block:

But as we can see from the failure message:

the error was to an HTTPS request.
That means the HTTP request was redirected to HTTPS.
Which means...
Either:

the port 8000 vhost isn't serving the HTTP requests [being served elsewhere]
OR
the port 8000 vhost isn't configured correctly to serve the HTTP requests [misconfigured]

I'm betting on an oversight/misconfiguration that's affecting vhosts with location with rewrite.
And it might be something simple like adding "^" to the location statement.
In any case, we can test/fix this easily [without having to involve certbot] by placing a test text file in the expected challenge location and review the nginx logs and make changes to that vhost until we can reach that file.

rg305 · April 25, 2024, 6:26am

I would try it this way:

server {
  listen 8000;
  server_name apkdcx.com www.apkdcx.com;
  server_tokens off;

  #location /.well-known/acme-challenge {
  location ^~ /.well-known/acme-challenge/ {
    root /var/www/certbot;
  }

  #rewrite ^(.*)$ https://www.apkdcx.com$1;
  return 301 https://www.apkdcx.com$request_uri;
}

MikeMcQ · April 25, 2024, 12:40pm

The rewrite statement has to appear in its own location block. Without that it happens for every request

entropy67:

server {
listen 8000;

server_name apkdcx.com www.apkdcx.com;
server_tokens off;
location /.well-known/acme-challenge {
    root /var/www/certbot;
}

location / {
    rewrite ^(.*)$ https://www.apkdcx.com$1;
}
}

rg305 · April 25, 2024, 1:02pm

OR
Will switching to return work?:

MikeMcQ · April 25, 2024, 1:19pm

I prefer return but it still needs its own location block. Otherwise that statement applies to every request.

entropy67 · April 28, 2024, 11:21pm

@MikeMcQ @rg305

Thank you so much for the detailed explanation and suggestions! I have fixed the problem by revising the redirect statement to the following, as @rg305 suggested,

location ^~ /.well-known/acme-challenge/ {
         root /var/www/certbot;
    }

Thank you very very much for your help. Our website is now back to use.

MikeMcQ · April 29, 2024, 12:35am

I"m not sure it's all working as you intended. Some, sure, but not all of it.

Your www.apkdcx.com domain uses a current cert. But, requests to https://apkdcx.com don't work because your cert only has the www subdomain in it. You should have both names in the cert.

Also, your HTTP server block still redirects all requests even acme-challenge requests. You should review my post #12 again

Tests below are for your root domain but same result with www subdomain

curl -I apkdcx.com
HTTP/1.1 301 Moved Permanently
Server: nginx
Location: https://www.apkdcx.com

curl -I apkdcx.com/.well-known/acme-challenge/Test404
HTTP/1.1 301 Moved Permanently
Server: nginx
Location: https://www.apkdcx.com

entropy67 · April 29, 2024, 1:48am

Hi @MikeMcQ Mike. Thanks a lot for pointing that out. Actually I wasn't sure how I fixed it because I changed a lot to the configure and somehow it worked after several days.

I just did some experiments and figured out the key change I made is putting

rewrite ^(.*)$ https://www.apkdcx.com$1;

into a block, which explicitly is

location / {
        rewrite ^(.*)$ https://www.apkdcx.com$1;
    }

The full config for http request is

server {
    listen 8000;
    server_name apkdcx.com www.apkdcx.com;
    server_tokens off;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
    
    location / {
        rewrite ^(.*)$ https://www.apkdcx.com$1;
    }
}

So @MikeMcQ you are absolutely right. It will always redirect if we didn't use a block.

The remaining step is to apply for certificate for apkdcx.com but that shouldn't be a problem. Thank you so much!