Certbor Renew fails: Remote PerformValidation RPCs failed

leononame · 2019-01-31 16:27:01 UTC

I ran this command: certbot renew --dry-run

It produced this output:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/drone.lho.io.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for drone.lho.io
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (drone.lho.io) from /etc/letsencrypt/renewal/drone.lho.io.conf produced an unexpected error: Failed authorization procedure. drone.lho.io (http-01): urn:ietf:params:acme:error:serverInternal :: The server experienced an internal error :: Remote PerformValidation RPCs failed. Skipping.

My web server is (include version): nginx/1.10.3

The operating system my web server runs on is: Debian 9.7, Linux 4.9.0-8-amd64

My hosting provider, if applicable, is: hetzner.de

I can login to a root shell on my machine: Yes

I’m using a control panel to manage my site: No

The version of my client is: certbot 0.28.0

More information:
My bare nginx config looks like this:

# HTTPS Server
server {
    listen 80;
    listen [::]:80;
    server_name drone.lho.io;
    
    error_log /var/log/nginx/drone_error.log;

    # lets encrypt redirect
    location /.well-known {
            alias /var/www/letsencrypt/.well-known;
    }

    location / {
        proxy_pass http://127.0.0.1:8082/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Nginx-Proxy true;
        proxy_redirect off;
    }
    # lets encrypt will add settings here for ssl!
}

After installing the certificate, cerbot added some stuff:

# HTTPS Server
server {
    listen 80;
    listen [::]:80;
    server_name drone.lho.io;

    error_log /var/log/nginx/drone_error.log;

    # lets encrypt redirect
    location /.well-known {
            alias /var/www/letsencrypt/.well-known;
    }

    location / {
        proxy_pass http://127.0.0.1:8082/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Nginx-Proxy true;
        proxy_redirect off;
    }
    # lets encrypt will add settings here for ssl!

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/drone.lho.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/drone.lho.io/privkey.pem; # managed by Certbot
ssl_session_cache shared:le_nginx_SSL:1m; # managed by Certbot
ssl_session_timeout 1440m; # managed by Certbot

ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # managed by Certbot
ssl_prefer_server_ciphers on; # managed by Certbot

ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-SHA ECDHE-ECDSA-AES256-SHA ECDHE-ECDSA-AES128-SHA256 ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-RSA-AES128-SHA ECDHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-SHA384 DHE-RSA-AES128-GCM-SHA256 DHE-RSA-AES256-GCM-SHA384 DHE-RSA-AES128-SHA DHE-RSA-AES256-SHA DHE-RSA-AES128-SHA256 DHE-RSA-AES256-SHA256 EDH-RSA-DES-CBC3-SHA"; # managed by Certbot

    if ($scheme != "https") {
        return 301 https://$host$request_uri;
    } # managed by Certbot
}

This used to work fine. However, I reset the config file and reinstalled the certificate using certbot --authenticator webroot -w /var/www/letsencrypt/ --installer nginx. The configuration file this time looked a little different:

# HTTPS Server
server {
    server_name drone.lho.io;

    error_log /var/log/nginx/drone_error.log;

    # lets encrypt redirect
    location /.well-known {
            alias /var/www/letsencrypt/.well-known;
    }

    location / {
        proxy_pass http://127.0.0.1:8082/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Nginx-Proxy true;
        proxy_redirect off;
    }
    # lets encrypt will add settings here for ssl!
    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/drone.lho.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/drone.lho.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = drone.lho.io) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;
    listen [::]:80;
    server_name drone.lho.io;
    return 404; # managed by Certbot
}

Renewal failed with the same log output described above. However, if I don’t force HTTPS redirects, the renewal works. The config file is as follows:

# HTTPS Server
server {
    listen 80;
    listen [::]:80;
    server_name drone.lho.io;

    error_log /var/log/nginx/drone_error.log;

    # lets encrypt redirect
    location /.well-known {
            alias /var/www/letsencrypt/.well-known;
    }

    location / {
        proxy_pass http://127.0.0.1:8082/;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_set_header X-Nginx-Proxy true;
        proxy_redirect off;
    }
    # lets encrypt will add settings here for ssl!

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/drone.lho.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/drone.lho.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Edit: For clarification: Reinstallation of certificate works, just as well as removing the certificate and installing a completely new one does (you’ll notice the certificate was issued today because I also tried removing everything and installing a new certificate).

The config file for renewal looks like this:

# renew_before_expiry = 30 days
version = 0.28.0
archive_dir = /etc/letsencrypt/archive/drone.lho.io
cert = /etc/letsencrypt/live/drone.lho.io/cert.pem
privkey = /etc/letsencrypt/live/drone.lho.io/privkey.pem
chain = /etc/letsencrypt/live/drone.lho.io/chain.pem
fullchain = /etc/letsencrypt/live/drone.lho.io/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = nginx
account = XXX
webroot_path = /var/www/letsencrypt,
server = https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]
drone.lho.io = /var/www/letsencrypt

_az · 2019-01-31 20:01:56 UTC

This error technically should not happen, but it usually coincides with network or request timeout conditions.

The first thing I’d try is to fix/drop your IPv6/AAAA record - https://letsdebug.net/drone.lho.io/20433?debug=y . Again, Let’s Encrypt should in theory not be fatally affected by it, but it’s probably good to fix it up for regular visitors.

The other thing I would suggest is trying to use the nginx authenticator - you may as well, since you are using the nginx installer anyway. The reason this may help is that Certbot will inject the challenge response directly into your port 80 virtualhost, which will bypass any complications relating to your reverse proxying, webroot or your HTTPS redirect.

certbot renew --cert-name drone.lho.io \
-a nginx -i nginx --dry-run

leononame · 2019-01-31 20:48:57 UTC

Wow, thanks! I fixed the IPv6 records and it actually resolved the problem. My copy/paste had just missed a number when I set up a new server.

However, changing to the nginx authenticator also worked fine. Thank you very much