Renewal of certificates fail with 404 error

My domain is: swk.nl and www.swk.nl

I ran this command: sudo certbot renew --nginx

It produced this output:

Processing /etc/letsencrypt/renewal/swk.nl.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Simulating renewal of an existing certificate for swk.nl and www.swk.nl
Performing the following challenges:
http-01 challenge for swk.nl
http-01 challenge for www.swk.nl
Waiting for verification...
Challenge failed for domain swk.nl
Challenge failed for domain www.swk.nl
http-01 challenge for swk.nl
http-01 challenge for www.swk.nl

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: swk.nl
  Type:   unauthorized
  Detail: 136.144.238.184: Invalid response from https://swk.nl/.well-known/acme-challenge/JpV3AQ3BfGP8rpo4CckzT-FK_BiCTqNg02NmY2u34Po: 404

  Domain: www.swk.nl
  Type:   unauthorized
  Detail: During secondary validation: 136.144.238.184: Invalid response from https://swk.nl/.well-known/acme-challenge/zFNqf-AY1t2RQb49qru3kX7ggQ3rNuANVtkNJtIGQA4: 404

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate swk.nl with error: Some challenges have failed.

My web server is (include version): nginx version: nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 20

My hosting provider, if applicable, is: TransIP

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.28.0

I have tried adding this:

location /.well-known/acme-challenge/ {
        allow all;
        root /var/www/certbot;
    }

But it does not help. I can see the 404 errors in my access log. DNS is set up correctly.

Hm, weird. For your www subdomain, the validation server returns an error terug containing "During secondary validation (...)". This is due to the fact Let's Encrypt tries to validate the challenge from multiple locations. And "During secondary validation" means the primary validation location is just fine, but at least two of the three secondary validation servers are not OK..

But for some reason this doesn't happen with your apex domain?

Anyway, let's just check the basics for now. Could you please show the output of sudo nginx -T ?

6 Likes

For www.swk.nl is find this:


For swk.nl is find this:


The tool I use are:

Hope this information is of some help to you @freddiehenk :slight_smile:

1 Like

Using this tool https://www.redirect-checker.org/ the redirects look ok.

2 Likes

This is the ouput of sudo nginx -T:

 configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {
	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	client_max_body_size 64M;

	# server_tokens off;

	server_names_hash_bucket_size 128;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}


#mail {
#	# See sample authentication script at:
#	# http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
# 
#	# auth_http localhost/auth.php;
#	# pop3_capabilities "TOP" "USER";
#	# imap_capabilities "IMAP4rev1" "UIDPLUS";
# 
#	server {
#		listen     localhost:110;
#		protocol   pop3;
#		proxy      on;
#	}
# 
#	server {
#		listen     localhost:143;
#		protocol   imap;
#		proxy      on;
#	}
#}

# configuration file /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:
load_module modules/ngx_http_image_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-modsecurity.conf:
load_module /etc/nginx/modules/ngx_http_modsecurity_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:
load_module modules/ngx_http_xslt_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-mail.conf:
load_module modules/ngx_mail_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-stream.conf:
load_module modules/ngx_stream_module.so;

# configuration file /etc/nginx/mime.types:

types {
    text/html                             html htm shtml;
    text/css                              css;
    text/xml                              xml;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
    application/atom+xml                  atom;
    application/rss+xml                   rss;

    text/mathml                           mml;
    text/plain                            txt;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/x-component                      htc;

    image/png                             png;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/x-icon                          ico;
    image/x-jng                           jng;
    image/x-ms-bmp                        bmp;
    image/svg+xml                         svg svgz;
    image/webp                            webp;

    application/font-woff                 woff;
    application/java-archive              jar war ear;
    application/json                      json;
    application/mac-binhex40              hqx;
    application/msword                    doc;
    application/pdf                       pdf;
    application/postscript                ps eps ai;
    application/rtf                       rtf;
    application/vnd.apple.mpegurl         m3u8;
    application/vnd.ms-excel              xls;
    application/vnd.ms-fontobject         eot;
    application/vnd.ms-powerpoint         ppt;
    application/vnd.wap.wmlc              wmlc;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/x-7z-compressed           7z;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-perl                    pl pm;
    application/x-pilot                   prc pdb;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            der pem crt;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xspf+xml                  xspf;
    application/zip                       zip;

    application/octet-stream              bin exe dll;
    application/octet-stream              deb;
    application/octet-stream              dmg;
    application/octet-stream              iso img;
    application/octet-stream              msi msp msm;

    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
    application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

    audio/midi                            mid midi kar;
    audio/mpeg                            mp3;
    audio/ogg                             ogg;
    audio/x-m4a                           m4a;
    audio/x-realaudio                     ra;

    video/3gpp                            3gpp 3gp;
    video/mp2t                            ts;
    video/mp4                             mp4;
    video/mpeg                            mpeg mpg;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-m4v                           m4v;
    video/x-mng                           mng;
    video/x-ms-asf                        asx asf;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;
}

# configuration file /etc/nginx/conf.d/webp.conf:
map $http_accept $webp_suffix {
    default   "";
    "~*webp"  ".webp";
}

# configuration file /etc/nginx/sites-enabled/swk.nl:

server {
    index index.html;
    server_name swk.nl www.swk.nl;
    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;
    
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains;";
    add_header X-Frame-Options "SAMEORIGIN";
    add_header Referrer-Policy "no-referrer";

    access_log /var/log/nginx/swk.nl.access.log;
    error_log /var/log/nginx/swk.nl.error.log;

    location /.well-known/acme-challenge/ {
        allow all;
        root /var/www/certbot;
    }

    if ($host = www.swk.nl) {
        return 301 https://swk.nl$request_uri;
    }
    
    # deny access to PHP files, this prevents unnecessary lockouts
    location ~ \.php$ {
        deny all;
    }

    location / {
        proxy_pass http://127.0.0.1:3107;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header Range "";
        proxy_cache_bypass $http_upgrade;
    }

    ssl_certificate /etc/letsencrypt/live/swk.nl/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/swk.nl/privkey.pem; # managed by Certbot

}

server {
    location /.well-known/acme-challenge/ {
        allow all;
        root /var/www/certbot;
    }

    if ($host = www.swk.nl) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = swk.nl) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    server_name swk.nl www.swk.nl;
    listen 80;
    listen [::]:80;
    return 404; # managed by Certbot




}

Definitely some sites cannot access your server.

1 Like

But I can ping it manually from here:

$ ifconfig
em0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 00:25:90:f3:1a:38
        inet 147.28.0.62 netmask 0xffffff00 broadcast 147.28.0.255
        inet 147.28.0.34 netmask 0xffffffff broadcast 147.28.0.34
        inet6 fe80::225:90ff:fef3:1a38%em0 prefixlen 64 scopeid 0x1
        inet6 2001:418:1::62 prefixlen 64
        inet6 2001:418:1::34 prefixlen 128
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
em1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=481249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER,NOMAP>
        ether 00:25:90:f3:1a:39
        media: Ethernet autoselect
        status: no carrier
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
$
$
$ ping 136.144.238.184
PING 136.144.238.184 (136.144.238.184): 56 data bytes
64 bytes from 136.144.238.184: icmp_seq=0 ttl=52 time=144.034 ms
64 bytes from 136.144.238.184: icmp_seq=1 ttl=52 time=143.991 ms
64 bytes from 136.144.238.184: icmp_seq=2 ttl=52 time=143.980 ms
64 bytes from 136.144.238.184: icmp_seq=3 ttl=52 time=144.032 ms
64 bytes from 136.144.238.184: icmp_seq=4 ttl=52 time=143.986 ms
64 bytes from 136.144.238.184: icmp_seq=5 ttl=52 time=144.029 ms
64 bytes from 136.144.238.184: icmp_seq=6 ttl=52 time=143.989 ms
64 bytes from 136.144.238.184: icmp_seq=7 ttl=52 time=144.026 ms
^C
--- 136.144.238.184 ping statistics ---
8 packets transmitted, 8 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 143.980/144.008/144.034/0.022 ms

This may be applicable How to know the location of CA to enable access to my server - #2 by Osiris
And as you may have noticed @Osiris knows his stuff.

Best of luck! :crossed_fingers:

1 Like

Maybe you have only partial remote connectivity from what I read above.

That is a very small "network" ("/32"). [strange]

8 Likes

But this is the systems main IPv4 Address

FreeBSD 13.1-RELEASE releng/13.1-n250148-fc952ac2212 GENERIC

psg.com  147.28.0.62       +1 206 356 8341
dual Xeon/HTT 2.66GHz, 4GB, 250GB mirrored
unauthorized access/use strictly forbidden

ping psg.com

Pinging psg.com [147.28.0.62] with 32 bytes of data:
Reply from 147.28.0.62: bytes=32 time=15ms TTL=53
Reply from 147.28.0.62: bytes=32 time=13ms TTL=53
Reply from 147.28.0.62: bytes=32 time=15ms TTL=53
Reply from 147.28.0.62: bytes=32 time=13ms TTL=53

Ping statistics for 147.28.0.62:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 13ms, Maximum = 15ms, Average = 14ms

And now they can. I believe you were just suffering from a DNS propagation situation. Let know if your issues are resolved now.

I see the multiple IPs [in an overlapping network] and it doesn't make that other IP any less "strange".
It's a "/32" IP [which is strange all on its' own] that is within a covered "/24" [making it even stranger].

7 Likes

Thanks for your replies, but unfortunately not. The 404's are logged in our access logs, which means it is not a DNS issue since the requests are received by Nginx.

2 Likes

How many do you see? You should see 4 requests per attempt.

Also, you could try to switch to the webroot authenticator by running:

certbot renew --dry-run -i nginx -a webroot -w /var/www/certbot

Notice the --dry-run, which will do a test run on the staging environment. If that works, you can remove it for an actual renewal.

6 Likes

I just ran the command as you suggested, it first tries diepezakken-staging.website-lab.nl. In the access logs of this domain I see the following:

172.70.206.59 - - [08/Jul/2022:06:38:43 +0000] "GET /.well-known/acme-challenge/UzIxTaBoV-UrUpMEK6aUZXSXJhGs5IoA-g8QqqJ7SO8 HTTP/1.1" 404 11930 "http://diepezakken-staging.website-lab.nl/.well-known/acme-challenge/UzIxTaBoV-UrUpMEK6aUZXSXJhGs5IoA-g8QqqJ7SO8" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
172.70.251.132 - - [08/Jul/2022:06:38:43 +0000] "GET /.well-known/acme-challenge/UzIxTaBoV-UrUpMEK6aUZXSXJhGs5IoA-g8QqqJ7SO8 HTTP/1.1" 404 11925 "http://diepezakken-staging.website-lab.nl/.well-known/acme-challenge/UzIxTaBoV-UrUpMEK6aUZXSXJhGs5IoA-g8QqqJ7SO8" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
108.162.245.41 - - [08/Jul/2022:06:38:43 +0000] "GET /.well-known/acme-challenge/UzIxTaBoV-UrUpMEK6aUZXSXJhGs5IoA-g8QqqJ7SO8 HTTP/1.1" 404 11925 "http://diepezakken-staging.website-lab.nl/.well-known/acme-challenge/UzIxTaBoV-UrUpMEK6aUZXSXJhGs5IoA-g8QqqJ7SO8" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
172.70.175.10 - - [08/Jul/2022:06:38:43 +0000] "GET /.well-known/acme-challenge/UzIxTaBoV-UrUpMEK6aUZXSXJhGs5IoA-g8QqqJ7SO8 HTTP/1.1" 404 11925 "http://diepezakken-staging.website-lab.nl/.well-known/acme-challenge/UzIxTaBoV-UrUpMEK6aUZXSXJhGs5IoA-g8QqqJ7SO8" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Note: this domain uses Cloudflare, swk.nl does not.

Including the --webroot -w part?

That site wasn't present in the sudo nginx -T output above, so no clue if the webroot is correct...

5 Likes

Yes, this is the command I ran: sudo certbot renew --dry-run -i nginx -a webroot -w /var/www/certbot

And this is the output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/diepezakken-staging.website-lab.nl.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for diepezakken-staging.website-lab.nl

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: diepezakken-staging.website-lab.nl
  Type:   unauthorized
  Detail: 2606:4700:3034::6815:36fd: Invalid response from https://diepezakken-staging.website-lab.nl/.well-known/acme-challenge/Y2CU0RIfyrbj6kOD2zOn842b0qYoWMbw77TAq0AMmFo: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Failed to renew certificate diepezakken-staging.website-lab.nl with error: Some challenges have failed.

This is the server block for diepezakken-staging.website-lab.nl:

# configuration file /etc/nginx/sites-enabled/diepezakken-staging.website-lab.nl:

server {
    index index.html;
    server_name diepezakken-staging.website-lab.nl www.diepezakken-staging.website-lab.nl;

    modsecurity on;
    modsecurity_rules_file /etc/nginx/modsec/main.conf;

    access_log /var/log/nginx/diepezakken-staging.website-lab.access.log;
    error_log /var/log/nginx/diepezakken-staging.website-lab.error.log;

    if ($host = www.diepezakken-staging.website-lab.nl) {
        return 301 https://diepezakken-staging.website-lab.nl$request_uri;
    }

    location / {
        proxy_pass http://127.0.0.1:3101;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header Range "";
        proxy_cache_bypass $http_upgrade;
    }

    listen [::]:443 ssl; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/diepezakken-staging.website-lab.nl/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/diepezakken-staging.website-lab.nl/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = diepezakken-staging.website-lab.nl) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;
    server_name diepezakken-staging.website-lab.nl www.diepezakken-staging.website-lab.nl;
    return 404; # managed by Certbot


}

# configuration file /etc/letsencrypt/options-ssl-nginx.conf:
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file. Contents are based on https://ssl-config.mozilla.org

ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;
ssl_session_tickets off;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;

ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";

If you would like to make that vhost work with the webroot plugin, you'd probably need to add the acme-challenge location block in that vhost too.

5 Likes

That was half of the solution. The other half was that I had to create the /var/www/certbot folder and change the ownership to www-data:

mkdir -p /var/www/certbot
chown -R www-data:www-data /var/www/certbot

Strange though that this worked automagically before, I am pretty sure that I did not need to do this before.

1 Like