Certbot renewal: Failed authorization procedure (403 Forbidden)

Help
1

Hi

my letsencrypt certificate renewal script does suddenly not work anymore (was working flawlessly for months). I am trying to figure out what happened. Similar Issues have been reported and I had a look at them, but none of them solved my problem.

This is the error msg of "certbot renew":

Attempting to renew cert (odroid-online.dedyn.io) from /etc/letsencrypt/renewal/odroid-online.dedyn.io.conf produced an unexpected error: Failed authorization procedure. odroid-online.dedyn.io (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://odroid-online.dedyn.io/.well-known/acme-challenge/q5RLpI79VFAULANW5MR_NyJh8yqIi2jA58WYHFfZuf8: "<html>
<head><title>403 Forbidden</title></head>
<body bgcolor="white">
<center><h1>403 Forbidden</h1></center>
<hr><center>". Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/odroid-online.dedyn.io/fullchain.pem (failure)

I was following the suggestions here: Unable to verify my domain because of acme challenge
and here: Can't renew certificate. DNS problem: Failed authorization procedure - #6 by sahsanu

The webroot-path is: /var/www/letsencrypt as specified in /etc/letsencrypt/renewal/odroid-online.dedyn.io.conf and the /etc/nginx/conf.d/letsencrypt.conf
I created the "test" file in /var/www/letsencrypt/.well_known/acme_challenge/. However when I try to access this file via firefox (i.e. https://odroid-online.dedyn.io/.well_known/acme-challenge/test) I get a 404 not found.

The nginx-server error.log recorded the following:

2018/04/16 23:39:11 [error] 929#929: *52416 open() "/var/www/nextcloud/.well-known/acme-challenge/test" failed (2: No such file or directory), client: 127.0.0.1, server: 127.0.0.1, request: "GET /.well-known/acme-challenge/test HTTP/1.1", host: "odroid-online.dedyn.io"

Why does the error msg not state the webroot-path? (i.e. var/www/nextcloud/.well-known/acme-challenge/test instead of var/www/letsencrypt/.well-known/acme-challenge/test)

I didn't change anything at my router, i.e. the port-forwarding for HTTP (80) and HTTPS (443) for the server is still active and the server-firewall is also still forwarding both ports.

I also compared my /var/www/nextcloud/.htaccess file with one from a previous installation. There is a difference (but I didn't modify this):

diff nextcloud/.htaccess nextcloud-old/.htaccess 
83,84c83,84
< ErrorDocument 403 //
< ErrorDocument 404 //
---
> ErrorDocument 403 /core/templates/403.php
> ErrorDocument 404 /core/templates/404.php

any clues?

My web server is (include version): nginx/1.13.0

The operating system my web server runs on is (include version): Ubuntu 16.04.4 LTS

I can login to a root shell on my machine (yes or no, or I don't know): yes

2

Hi @SpontEIN,

Apparently your webroot path in your renewal configuration file is not correct. Could you post what the nginx configuration says about this path?

3

Hi, the nginx.conf includes serveral *.conf files and they somehow have contradicting root-path entries:

grep -r "root" *.conf
gateway.conf:root /var/www;
letsencrypt.conf:root /var/www/letsencrypt;
nextcloud.conf: root /var/www/nextcloud/;

For the setup of my nextcloud I followed this very nice tutorial (https://www.c-rieger.de/nextcloud-installation-guide-advanced/) which suggested the above setup and so far it was working fine.

4

Hmmm, do you think you could post the contents of all three files?

5

It seems your entire site (odroid-online.dedyn.io) is returning 403 access forbidden.

6

hmm … well I can still access my nextcloud … also from outside the LAN …

7

I wouldn’t know why, but all of these links show 403:
http://odroid-online.dedyn.io/
https://odroid-online.dedyn.io/
http://odroid-online.dedyn.io/nextcloud
https://odroid-online.dedyn.io/nextcloud
http://odroid-online.dedyn.io/cloud
https://odroid-online.dedyn.io/cloud
http://odroid-online.dedyn.io/nc
https://odroid-online.dedyn.io/nc

8

letsencrypt.conf

server {
listen 127.0.0.1:81;
server_name 127.0.0.1;
charset utf-8;
location ^~ /.well-known/acme-challenge {
default_type text/plain;
root /var/www/letsencrypt;
}
}

gateway.conf

server {
listen 80 default_server;
server_name odroid-online.dedyn.io 192.168.178.8;
charset utf-8;
root /var/www;
if ($allowed_country = yes) {
set $exclusions 1;
}
if ($exclusions = "0") {
return 403;
}
location ^~ /.well-known/acme-challenge {
proxy_pass http://127.0.0.1:81;
proxy_redirect off;
}
location / {
return 301 https://$host$request_uri;
}
}
server {
listen 443 ssl http2 default_server;
server_name odroid-online.dedyn.io 192.168.178.8;
charset utf-8;
include /etc/nginx/ssl.conf;
if ($allowed_country = yes) {
set $exclusions 1;
}
if ($exclusions = "0") {
return 403;
}
location ^~ / {
client_max_body_size 10240M;
proxy_connect_timeout 300;
proxy_send_timeout 300;
proxy_read_timeout 300;
send_timeout 300;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_pass http://127.0.0.1:82;
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_redirect off;
proxy_buffering off;
proxy_request_buffering off;
}
# static files                                                                                                                                        
location ^~ /loleaflet {
proxy_pass https://localhost:9980;
proxy_set_header Host $http_host;
}

# WOPI discovery URL                                                                                                                                  
location ^~ /hosting/discovery {
proxy_pass https://localhost:9980;
proxy_set_header Host $http_host;
}

# websockets, download, presentation and image upload                                                                                                 
location ^~ /lool {
proxy_pass https://localhost:9980;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header Host $http_host;
}
}

nextcloud.conf

fastcgi_cache_path /usr/local/tmp/cache levels=1:2 keys_zone=NEXTCLOUD:100m inactive=60m;
fastcgi_cache_key $scheme$request_method$host$request_uri;
map $request_uri $skip_cache {
 default 1;
 ~*/thumbnail.php 0;
 ~*/apps/galleryplus/ 0;
 ~*/apps/gallery/ 0;
}
server {
 listen 127.0.0.1:82;
 server_name 127.0.0.1;
 charset utf-8;
 proxy_set_header Strict-Transport-Security "max-age=15768000; includeSubDomains; always;";
 proxy_set_header X-Content-Type-Options "nosniff; always;";
 proxy_set_header X-XSS-Protection "1; mode=block; always;";
 proxy_set_header X-Robots-Tag none;
 proxy_set_header X-Download-Options noopen;
 proxy_set_header X-Permitted-Cross-Domain-Policies none;
 root /var/www/nextcloud/;
 location = /robots.txt {
 allow all;
 log_not_found off;
 access_log off;
 }
 location = /.well-known/carddav { return 301 $scheme://$host/remote.php/dav; }
 location = /.well-known/caldav { return 301 $scheme://$host/remote.php/dav; }
 location /.well-known/acme-challenge { }
 location ^~ /{
 client_max_body_size 10240M;
 fastcgi_buffer_size 256k;
 fastcgi_buffers 64 64k; 
 fastcgi_busy_buffers_size 3840k;
 fastcgi_temp_file_write_size 256k;
 gzip on;
 gzip_vary on;
 gzip_comp_level 4;
 gzip_min_length 256;
 gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
 gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;
 fastcgi_cache_key $http_cookie$request_method$host$request_uri;
 fastcgi_cache_use_stale error timeout invalid_header http_500;
 fastcgi_ignore_headers Cache-Control Expires Set-Cookie;
 error_page 403 /core/templates/403.php;
 error_page 404 /core/templates/404.php;
 location / {
 rewrite ^ /index.php$uri;
 }
 location ~ ^/(?:build|tests|config|lib|3rdparty|templates|data)/ {
 deny all;
 }
 location ~ ^/(?:\.|autotest|occ|issue|indie|db_|console) {
 deny all;
 }
 location ~ ^/(?:index|remote|public|cron|core/ajax/update|status|ocs/v[12]|updater/.+|ocs-provider/.+|core/templates/40[34])\.php(?:$|/) {
 fastcgi_split_path_info ^(.+\.php)(/.*)$;
 include fastcgi_params;
 fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
 fastcgi_param PATH_INFO $fastcgi_path_info;
 fastcgi_pass php-handler;
 fastcgi_cache_bypass $skip_cache;
 fastcgi_no_cache $skip_cache;
 fastcgi_cache NEXTCLOUD;
 fastcgi_cache_valid 60m;
 fastcgi_cache_methods GET HEAD;
 fastcgi_param modHeadersAvailable true; 
 fastcgi_param front_controller_active true;
 fastcgi_intercept_errors on;
 fastcgi_request_buffering off;
 fastcgi_read_timeout 3600;
 fastcgi_send_timeout 600;
 fastcgi_connect_timeout 600; 
 fastcgi_param PHP_VALUE "upload_max_filesize = 10240M
 post_max_size = 10250M
 max_execution_time = 3600"; 
 fastcgi_param REMOTE_ADDR $http_x_real_ip;
 }
 location ~ ^/(?:updater|ocs-provider)(?:$|/) {
 try_files $uri/ =404;
 index index.php;
 }
 location ~* \.(?:css|js)$ {
 try_files $uri /index.php$uri$is_args$args;
 proxy_set_header Cache-Control "public, max-age=7200";
 proxy_set_header Strict-Transport-Security "max-age=15768000; includeSubDomains; always;";
 proxy_set_header X-Content-Type-Options "nosniff; always;";
 proxy_set_header X-XSS-Protection "1; mode=block; always;";
 proxy_set_header X-Robots-Tag none;
 proxy_set_header X-Download-Options noopen;
 proxy_set_header X-Permitted-Cross-Domain-Policies none;
 access_log off;
 }
 location ~* \.(?:svg|gif|png|html|ttf|woff|ico|jpg|jpeg)$ {
 try_files $uri /index.php$uri$is_args$args;
 access_log off;
 log_not_found off;
 expires 30d;
}
}
}
9

this is the correct link:

https://odroid-online.dedyn.io/

not sure why you get a 403 ...

EDIT: I have some GEOIP restrictions ... that might be the reason

10

ARGH … mega fail … I just recalled that I modified the GeoIP settings recently. For some reason I decided to block US IPs and the letsencrypt server is located somewhere in the US …

I added US IPs again and now it’s working … face palm

issue can be closed :wink:

11

It’s not guaranteed that the validation server will always connect from the US. The list of IP addresses is intentionally not published and in the future multiple IP addresses may be used for each validation. For reliable renewals you should disable geoip blocking (at least for /.well-known/acme-challenge/* paths during the renewal process).

2 Likes
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.