Timeout after connect (your server may be slow or overloaded)

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: prsnl-server.com & files.prsnl-server.com (+ www)

I ran this command: certbot renew --dry-run

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/prsnl-server.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for files.prsnl-server.com
http-01 challenge for prsnl-server.com
Using default addresses 80 and [::]:80 ipv6only=on for authentication.
Using default addresses 80 and [::]:80 ipv6only=on for authentication.
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (prsnl-server.com-0001) from /etc/letsencrypt/renewal/prsnl-server.com-0001.conf produced an unexpected error: Failed authorization procedure. prsnl-server.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://prsnl-server.com/.well-known/acme-challenge/aQgEESSbz75sHxSU92van7uiEB5tJCE0JEb7uZXNjOk: Timeout after connect (your server may be slow or overloaded), files.prsnl-server.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://files.prsnl-server.com/.well-known/acme-challenge/k5FCBjEsGBOCwQXP3OSjMpdZI94PvGntjq-0QYZa2Wo: Timeout after connect (your server may be slow or overloaded). Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/prsnl-server.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for files.prsnl-server.com
http-01 challenge for prsnl-server.com
http-01 challenge for www.files.prsnl-server.com
http-01 challenge for www.prsnl-server.com
Using default addresses 80 and [::]:80 ipv6only=on for authentication.
Using default addresses 80 and [::]:80 ipv6only=on for authentication.
Using default addresses 80 and [::]:80 ipv6only=on for authentication.
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (prsnl-server.com) from /etc/letsencrypt/renewal/prsnl-server.com.conf produced an unexpected error: Failed authorization procedure. files.prsnl-server.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://files.prsnl-server.com/.well-known/acme-challenge/yL5r6WkVRxsDYD-FpaX-AzDU-Xttl6BiJ_P5UvgqQb8: Timeout after connect (your server may be slow or overloaded), www.prsnl-server.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://www.prsnl-server.com/.well-known/acme-challenge/VeSzCY6pbNYXeBl-x4nrtcsZrXJhwDns4rAbWt9cYhs: Timeout after connect (your server may be slow or overloaded), prsnl-server.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://prsnl-server.com/.well-known/acme-challenge/_uvcHTfzq25WnHJDKABcBoaya8fZ1a746DjGQXew6eg: Timeout after connect (your server may be slow or overloaded), www.files.prsnl-server.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://www.files.prsnl-server.com/.well-known/acme-challenge/P3znZYgulNo8D3mlxmgR_STtFMyv3e_3SlvhryORkUU: Timeout after connect (your server may be slow or overloaded). Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.files.prsnl-server.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.files.prsnl-server.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (www.files.prsnl-server.com) from /etc/letsencrypt/renewal/www.files.prsnl-server.com.conf produced an unexpected error: Failed authorization procedure. www.files.prsnl-server.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://www.files.prsnl-server.com/.well-known/acme-challenge/wUoMb7anALS_GO1pePv8ASGEi0_N1JsY4ywxv2C_HQc: Timeout after connect (your server may be slow or overloaded). Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.prsnl-server.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.prsnl-server.com
Using default addresses 80 and [::]:80 ipv6only=on for authentication.
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (www.prsnl-server.com) from /etc/letsencrypt/renewal/www.prsnl-server.com.conf produced an unexpected error: Failed authorization procedure. www.prsnl-server.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://www.prsnl-server.com/.well-known/acme-challenge/_jV54H2HuS5MRFo3_7C5_xc85T7PPyJZXK6zHrPLuYU: Timeout after connect (your server may be slow or overloaded). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/prsnl-server.com/fullchain.pem (failure)
  /etc/letsencrypt/live/www.files.prsnl-server.com/fullchain.pem (failure)
  /etc/letsencrypt/live/www.prsnl-server.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/prsnl-server.com/fullchain.pem (failure)
  /etc/letsencrypt/live/www.files.prsnl-server.com/fullchain.pem (failure)
  /etc/letsencrypt/live/www.prsnl-server.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
4 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: files.prsnl-server.com
   Type:   connection
   Detail: Fetching
   http://files.prsnl-server.com/.well-known/acme-challenge/yL5r6WkVRxsDYD-FpaX-AzDU-Xttl6BiJ_P5UvgqQb8:
   Timeout after connect (your server may be slow or overloaded)

   Domain: www.prsnl-server.com
   Type:   connection
   Detail: Fetching
   http://www.prsnl-server.com/.well-known/acme-challenge/VeSzCY6pbNYXeBl-x4nrtcsZrXJhwDns4rAbWt9cYhs:
   Timeout after connect (your server may be slow or overloaded)

   Domain: prsnl-server.com
   Type:   connection
   Detail: Fetching
   http://prsnl-server.com/.well-known/acme-challenge/_uvcHTfzq25WnHJDKABcBoaya8fZ1a746DjGQXew6eg:
   Timeout after connect (your server may be slow or overloaded)

   Domain: www.files.prsnl-server.com
   Type:   connection
   Detail: Fetching
   http://www.files.prsnl-server.com/.well-known/acme-challenge/P3znZYgulNo8D3mlxmgR_STtFMyv3e_3SlvhryORkUU:
   Timeout after connect (your server may be slow or overloaded)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - The following errors were reported by the server:

   Domain: prsnl-server.com
   Type:   connection
   Detail: Fetching
   http://prsnl-server.com/.well-known/acme-challenge/aQgEESSbz75sHxSU92van7uiEB5tJCE0JEb7uZXNjOk:
   Timeout after connect (your server may be slow or overloaded)

   Domain: files.prsnl-server.com
   Type:   connection
   Detail: Fetching
   http://files.prsnl-server.com/.well-known/acme-challenge/k5FCBjEsGBOCwQXP3OSjMpdZI94PvGntjq-0QYZa2Wo:
   Timeout after connect (your server may be slow or overloaded)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - The following errors were reported by the server:

   Domain: www.files.prsnl-server.com
   Type:   connection
   Detail: Fetching
   http://www.files.prsnl-server.com/.well-known/acme-challenge/wUoMb7anALS_GO1pePv8ASGEi0_N1JsY4ywxv2C_HQc:
   Timeout after connect (your server may be slow or overloaded)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - The following errors were reported by the server:

   Domain: www.prsnl-server.com
   Type:   connection
   Detail: Fetching
   http://www.prsnl-server.com/.well-known/acme-challenge/_jV54H2HuS5MRFo3_7C5_xc85T7PPyJZXK6zHrPLuYU:
   Timeout after connect (your server may be slow or overloaded)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

I have done some research and it seems like port 80 isn’t open, so here is my ufw status:

Status: active

To                         Action      From
--                         ------      ----
Nginx HTTP                 ALLOW       Anywhere                  
OpenSSH                    ALLOW       Anywhere                  
443/tcp                    ALLOW       Anywhere                  
5000                       ALLOW       Anywhere                  
8200                       ALLOW       Anywhere                  
22/tcp                     ALLOW       Anywhere                  
22                         ALLOW       Anywhere                  
Nginx Full                 ALLOW       Anywhere                  
5001                       ALLOW       Anywhere                  
Apache Full                ALLOW       Anywhere                  
80                         ALLOW       Anywhere                  
80/tcp                     ALLOW       Anywhere                  
Nginx HTTP (v6)            ALLOW       Anywhere (v6)             
OpenSSH (v6)               ALLOW       Anywhere (v6)             
443/tcp (v6)               ALLOW       Anywhere (v6)             
5000 (v6)                  ALLOW       Anywhere (v6)             
8200 (v6)                  ALLOW       Anywhere (v6)             
22/tcp (v6)                ALLOW       Anywhere (v6)             
22 (v6)                    ALLOW       Anywhere (v6)             
Nginx Full (v6)            ALLOW       Anywhere (v6)             
5001 (v6)                  ALLOW       Anywhere (v6)             
Apache Full (v6)           ALLOW       Anywhere (v6)             
80 (v6)                    ALLOW       Anywhere (v6)             
80/tcp (v6)                ALLOW       Anywhere (v6) 

Nginx config, I have been trying out a bunch of things here, so problem could be in here:

# HTTP — redirect all traffic to HTTPS
#server {
    listen 80;
    listen [::]:80;
    server_name prsnl-server.com;
    return 301 https://$host$request_uri;
#}

# HTTPS — proxy all requests to the Node app
server {
    # Enable HTTP/2
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name prsnl-server.com;

    # Use the Let’s Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/prsnl-server.com-0001/privkey.pem; # managed by Certbot

    # Include the SSL configuration from cipherli.st
    include snippets/ssl-params.conf;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-NginX-Proxy true;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_cache_bypass $http_upgrade;
        proxy_redirect off;
       # proxy_pass http://my_http_servers;
        proxy_pass http://localhost:5000;
       # proxy_set_header Upgrade $http_upgrade;
       # proxy_set_header Connection "upgrade";
       # proxy_http_version 1.1;
       # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       # proxy_set_header Host $host;
    }


}

# HTTPS — proxy all requests to the Node app
server {
    # Enable HTTP/2
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name www.prsnl-server.com;

    # Use the Let’s Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/www.prsnl-server.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/www.prsnl-server.com/privkey.pem; # managed by Certbot

    # Include the SSL configuration from cipherli.st
    include snippets/ssl-params.conf;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-NginX-Proxy true;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_cache_bypass $http_upgrade;
        proxy_redirect off;
        # proxy_pass http://my_http_servers;
        proxy_pass http://localhost:5000;
        # proxy_set_header Upgrade $http_upgrade;
        # proxy_set_header Connection "upgrade";
        # proxy_http_version 1.1;
        # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # proxy_set_header Host $host;
    }

}

server {
    server_name files.prsnl-server.com;

    location / {
        proxy_pass https://prsnl.ams3.cdn.digitaloceanspaces.com/;
        proxy_hide_header      Strict-Transport-Security;
        proxy_cache            example-cache;
        proxy_cache_valid      200 1440m;
        proxy_cache_use_stale  error timeout updating http_500 http_502 http_503 http_504;
        proxy_cache_revalidate on;
        proxy_cache_lock       on;
        proxy_ignore_headers   Set-Cookie;
        add_header             X-Cache-Status $upstream_cache_status;
    }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/prsnl-server.com-0001/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    server_name www.files.prsnl-server.com;

    location / {
        proxy_pass https://prsnl.ams3.cdn.digitaloceanspaces.com/;
        proxy_hide_header      Strict-Transport-Security;
        proxy_cache            example-cache;
        proxy_cache_valid      200 1440m;
        proxy_cache_use_stale  error timeout updating http_500 http_502 http_503 http_504;
        proxy_cache_revalidate on;
        proxy_cache_lock       on;
        proxy_ignore_headers   Set-Cookie;
        add_header             X-Cache-Status $upstream_cache_status;
    }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/www.files.prsnl-server.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/www.files.prsnl-server.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

My web server is (include version): nginx/1.14.0 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 18.04.3 (LTS) x64

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no, DigitalOcean control panel, but that’s it.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0

I wonder if the timeouts are related to the way you are proxying (i.e. Certbot is not properly preparing nginx to intercept the acme-challenge requests and it’s hitting your backends).

Do the timeouts still happen if we take nginx out of the equation?

certbot renew -a standalone --dry-run \
--pre-hook "service nginx stop" \
--post-hook "service nginx start"

Seems like you are already on to something. Crazy.

certbot renew -a standalone --dry-run \
> --pre-hook "service nginx stop" \
> --post-hook "service nginx start"

Returns the following:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/prsnl-server.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Error while running nginx -c /etc/nginx/nginx.conf -t.
nginx: [emerg] duplicate listen options for [::]:443 in /etc/nginx/sites-enabled/default:123
nginx: configuration file /etc/nginx/nginx.conf test failed
Plugins selected: Authenticator standalone, Installer None
Running pre-hook command: service nginx stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for files.prsnl-server.com
http-01 challenge for prsnl-server.com
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/prsnl-server.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Error while running nginx -c /etc/nginx/nginx.conf -t.
nginx: [emerg] duplicate listen options for [::]:443 in /etc/nginx/sites-enabled/default:123
nginx: configuration file /etc/nginx/nginx.conf test failed
Plugins selected: Authenticator standalone, Installer None
Pre-hook command already run, skipping: service nginx stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.files.prsnl-server.com
http-01 challenge for www.prsnl-server.com
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/prsnl-server.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.files.prsnl-server.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Error while running nginx -c /etc/nginx/nginx.conf -t.
nginx: [emerg] duplicate listen options for [::]:443 in /etc/nginx/sites-enabled/default:123
nginx: configuration file /etc/nginx/nginx.conf test failed
Plugins selected: Authenticator standalone, Installer None
Pre-hook command already run, skipping: service nginx stop
Renewing an existing certificate
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.files.prsnl-server.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.prsnl-server.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Error while running nginx -c /etc/nginx/nginx.conf -t.
nginx: [emerg] duplicate listen options for [::]:443 in /etc/nginx/sites-enabled/default:123
nginx: configuration file /etc/nginx/nginx.conf test failed
Plugins selected: Authenticator standalone, Installer None
Pre-hook command already run, skipping: service nginx stop
Renewing an existing certificate
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.prsnl-server.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)
Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem (success)
/etc/letsencrypt/live/prsnl-server.com/fullchain.pem (success)
/etc/letsencrypt/live/www.files.prsnl-server.com/fullchain.pem (success)
/etc/letsencrypt/live/www.prsnl-server.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: service nginx start
Hook command "service nginx start" returned error code 1
Error output from service:
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.

Oh, OK. I think this is simpler than it looks.

I think the problem is that your nginx configuration is not valid. Check with:

nginx -t

Certbot renewal won’t work if the nginx configuration is not valid to begin with, because it’s not able to apply the required changes it needs to make for verification.

Fixing up the config and then trying to renew the usual way should work … I hope.

True. You are right:

nginx: [emerg] duplicate listen options for [::]:443 in /etc/nginx/sites-enabled/default:123
nginx: configuration file /etc/nginx/nginx.conf test failed

However, while there is indeed a duplicate entry for listening to port 443, the configuration is for the www and non-www address. I’m not entirely sure how I can still listen to port 443, handling both the www and non-www address to my site. Any idea?

You can certainly listen on port 443 in multiple virtualhosts with different server names. I’m not sure what’s going wrong without seeing the config, could you post the full config as reported by nginx -T?

Sure. It’s in the OP, I removed some commented fragments, but this is everything:

# HTTP — redirect all traffic to HTTPS
server {
    listen 80;
    listen [::]:80;
    server_name prsnl-server.com;
    return 301 https://$host$request_uri;
}

#upstream my_http_servers {
    #ip_hash;
#    server 127.0.0.1:5000;      # httpServer1 listens to port 444
#    server 127.0.0.1:5001;      # httpServer2 listens to port 445
#}

# HTTPS — proxy all requests to the Node app
server {
    # Enable HTTP/2
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name prsnl-server.com;

    # Use the Let’s Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/prsnl-server.com-0001/privkey.pem; # managed by Certbot

    # Include the SSL configuration from cipherli.st
    include snippets/ssl-params.conf;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-NginX-Proxy true;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_cache_bypass $http_upgrade;
        proxy_redirect off;
       # proxy_pass http://my_http_servers;
        proxy_pass http://localhost:5000;
       # proxy_set_header Upgrade $http_upgrade;
       # proxy_set_header Connection "upgrade";
       # proxy_http_version 1.1;
       # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
       # proxy_set_header Host $host;
    }


}

# HTTPS — proxy all requests to the Node app
server {
    # Enable HTTP/2
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name www.prsnl-server.com;

    # Use the Let’s Encrypt certificates
    ssl_certificate /etc/letsencrypt/live/www.prsnl-server.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/www.prsnl-server.com/privkey.pem; # managed by Certbot

    # Include the SSL configuration from cipherli.st
    include snippets/ssl-params.conf;

    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-NginX-Proxy true;
        proxy_ssl_session_reuse off;
        proxy_set_header Host $http_host;
        proxy_cache_bypass $http_upgrade;
        proxy_redirect off;
        # proxy_pass http://my_http_servers;
        proxy_pass http://localhost:5000;
        # proxy_set_header Upgrade $http_upgrade;
        # proxy_set_header Connection "upgrade";
        # proxy_http_version 1.1;
        # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # proxy_set_header Host $host;
    }

}



server {
    server_name files.prsnl-server.com;

    location / {
        proxy_pass https://prsnl.ams3.cdn.digitaloceanspaces.com/;
        proxy_hide_header      Strict-Transport-Security;
        proxy_cache            example-cache;
        proxy_cache_valid      200 1440m;
        proxy_cache_use_stale  error timeout updating http_500 http_502 http_503 http_504;
        proxy_cache_revalidate on;
        proxy_cache_lock       on;
        proxy_ignore_headers   Set-Cookie;
        add_header             X-Cache-Status $upstream_cache_status;
    }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/prsnl-server.com-0001/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    server_name www.files.prsnl-server.com;

    location / {
        proxy_pass https://prsnl.ams3.cdn.digitaloceanspaces.com/;
        proxy_hide_header      Strict-Transport-Security;
        proxy_cache            example-cache;
        proxy_cache_valid      200 1440m;
        proxy_cache_use_stale  error timeout updating http_500 http_502 http_503 http_504;
        proxy_cache_revalidate on;
        proxy_cache_lock       on;
        proxy_ignore_headers   Set-Cookie;
        add_header             X-Cache-Status $upstream_cache_status;
    }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/www.files.prsnl-server.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/www.files.prsnl-server.com/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

Line 123 is listen [::]:443 ssl ipv6only=on; # managed by Certbot under the www.files.prsnl-server.com config.

Right. You have two virtualhosts with server_name www.files.prsnl-server.com that listen on port 443.

The one created by Certbot (L123), and the one created preceded by the # HTTPS — proxy all requests to the Node app comment.

You’ll probably want to merge them into one. I’m not sure how you ended up with two for the same domain, to be honest.

Hmm. Not too sure if I understand correctly. Aren’t the server_name www.files.prsnl-server.com and # HTTPS — proxy all requests to the Node app different routes?

The last one is listening to server_name prsnl-server.com, another domain name. Right?

The good news: I commented out the last configuration (which gave the error), nginx test ran successful, and dry-run ran successful as well:

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/prsnl-server.com-0001/fullchain.pem (success)
  /etc/letsencrypt/live/prsnl-server.com/fullchain.pem (success)
  /etc/letsencrypt/live/www.files.prsnl-server.com/fullchain.pem (success)
  /etc/letsencrypt/live/www.prsnl-server.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)

I misread, you’re right.

I think it might be the ipv6only=on that produces the error. If I remember right, nginx will complain if it appears more than once for the same bind address.

This was supposed to be fixed in a fairly early version of Certbot (so it doesn’t generate config containing that property multiple times) so I’m unsure why it would have happened to you.

Somewhat unrelated, are you aware that you can fit all of these names on a single certificate? e.g.

certbot --nginx -d files.prsnl-server.com -d www.files.prsnl-server.com

That might prevent you having more virtual host blocks than you need, make things a little simpler.

Hmm, it’s not too bad. I have, for now, taken out the www for the files domain. I only use it internally, so other people don’t need to visit the www, I don’t point to the www subdomain, so I don’t expect many issues. I’ll mark your answer as the solution, as you have helped me a lot and fixed the solution.

Regarding the single cert, I didn’t know. Can I then use the same certificate, without having to include the www subdomain in my virtual host block?

Also, another small question: can I rerun my cert renewal now, making sure the auto renewal will still be running?

--dry-run does perform renewal, it just skips the installation step (and also uses the test Let’s Encrypt CA instead of the live one).

You can do a live renewal with --force-renewal, but it’s usually better to avoid doing that, as you can quickly run into trouble with rate limits.

Not exactly sure what you’re asking here, but I think so, yes. In general, I find it best to just match your certificate names to how you organize your virtualhosts (so whatever your server_names are in that virtualhost, create a matching certificate with the same list of domains).

Hmm. OK. I’ll leave config as is for now.

I tried renewing, but I don’t get the verification. It’s just stuck for a few minutes:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/prsnl-server.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for files.prsnl-server.com
http-01 challenge for prsnl-server.com
Using default addresses 80 and [::]:80 ipv6only=on for authentication.
Waiting for verification...

Then, when I quit:

^CCleaning up challenges
^[[AExiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1272, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 452, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1193, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 310, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 353, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 389, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 168, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 215, in _poll_challenges
    time.sleep(min_sleep)
KeyboardInterrupt
Please see the logfiles in /var/log/letsencrypt for more details.

Log:

2020-07-11 11:04:52,645:DEBUG:certbot.error_handler:Calling registered functions
2020-07-11 11:04:52,645:INFO:certbot.auth_handler:Cleaning up challenges
2020-07-11 11:04:53,838:ERROR:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1272, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 452, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1193, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 310, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 353, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 389, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 168, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 219, in _poll_challenges
    aauthzrs, index, chall_update[index])
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 251, in _handle_check
    updated_authzr, _ = self.acme.poll(original_aauthzr.authzr)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 694, in poll
    response = self._post_as_get(authzr.uri)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 800, in _post_as_get
    return self._post(*new_args, **kwargs)  # pylint: disable=star-args
  File "/usr/lib/python3/dist-packages/acme/client.py", line 96, in _post
    return self.net.post(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1204, in post
    return self._post_once(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1217, in _post_once
    response = self._send_request('POST', url, data=data, **kwargs)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1120, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 520, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 630, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 383, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.6/http/client.py", line 1356, in getresponse
    response.begin()
  File "/usr/lib/python3.6/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.6/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
  File "/usr/lib/python3.6/ssl.py", line 1012, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/lib/python3.6/ssl.py", line 874, in read
    return self._sslobj.read(len, buffer)
  File "/usr/lib/python3.6/ssl.py", line 631, in read
    v = self._sslobj.read(len, buffer)
KeyboardInterrupt

I’m still a bit worried the autorenewal will not work as well.

Could you find the order URL from the log corresponding to that renewal attempt? It should look something like https://acme-v02.api.letsencrypt.org/acme/order/XXXX/YYYY

You’ll probably want this:

Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/5809362364:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNjkyMjg4NDkiLCAibm9uY2UiOiAiMDAwMnltUXlOeko0aVNlOXJvQ3FIYVp2WXdQeERaMTdXMEZTbUlDd2VjX2pTckkiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2F1dGh6LXYzLzU4MDkzNjIzNjQifQ",
  "signature": "vbmx1xocE_Q4p0Tpw6OD-WmIUsmISQkaNDXrButkOR0jHvqFUKdTM7oXlzXfWjKxBKrVu81HBxGJeAXc3hKNpeg4_7Rb77q_n1FeGvzVuAwmG02lySgRQlpx_pGAsCr1VQRygMcbG-Y1TlBg5qC-XZoS5eQzrc0u4PEoXUssDYy7mTQSPIYjrImyURwDKlLXGv8Onm1EzcRV8sptTu_4X5qbuLKHMhYeuW68k-IVUNTkwZwwq-LDfpeSZEaA9tL6YOc6Pd9e41Oc7B8WVtw8ifMyVOrOQ_FhB4DlXEctdTueip-nAgXw_ZxdyxqRgGZCQcc6uXBo3D7XiFM0oi_8sQ",
  "payload": ""
}
2020-07-11 11:04:52,645:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 380, in _make_request
    httplib_response = conn.getresponse(buffering=True)
TypeError: getresponse() got an unexpected keyword argument 'buffering'

There should be one a bit further up that is /acme/order, not /acme/authz-v3. I want to see why your Certbot was stuck in a polling loop and that would reveal which authz it got stuck on.

https://acme-v02.api.letsencrypt.org/acme/order/69228849/4155381350

Doesn’t look like the nginx authenticator is working. The validation requests went to your S3 proxy & Node.js backends again

nginx -t isn’t complaining?

Nope. Nginx seems happy:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Just restarted nginx service and still same result.

Could it be that my server is responding to the requests and trying to route them to the nodejs routes for prsnl-server.com and towards the file storage system (DigitalOcean object spaces) for files.prsnl-server.com?

nginx.conf:

user www-data;
worker_processes auto;
worker_rlimit_nofile 1000000;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        multi_accept on;
        use epoll;
	worker_connections 1000000;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

        client_max_body_size 50M;
	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 120;
        keepalive_requests 10000;
	types_hash_max_size 2048;
	# server_tokens off;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}

Yes, that’s what’s happening. The Certbot nginx authenticator is supposed to intercept the requests and respond to them. But evidently it’s not working.

Could you dump your full config using the below command into a file and post it? I’d like to run Certbot against it and see what’s happening.

nginx -T > nginx-full.conf