Unable to auto renew LetsEncrypt Certificate for 4 years due to Nginx running

BilboBaggins · January 8, 2024, 9:31pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: www.TheWeylandCorporation.com

I ran this command: certbot renew --force-renewal

It produced this output: Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/theweylandcorporation.com.conf

Plugins selected: Authenticator standalone, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for theweylandcorporation.com
http-01 challenge for www.theweylandcorporation.com
Cleaning up challenges
Attempting to renew cert (theweylandcorporation.com) from /etc/letsencrypt/renew al/theweylandcorporation.com.conf produced an unexpected error: Problem binding to port 80: Could not bind to IPv4 or IPv6.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/theweylandcorporation.com/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/theweylandcorporation.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

My web server is (include version): site runs Apache 2.4.41 and Nginx 1.18.0

The operating system my web server runs on is (include version): Ubuntu 20.04.6 LTS

My hosting provider, if applicable, is: NA

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Certbot 0.40.0

Note: I can only renew Letsencrypt if Nginix is manually shut down first, then the renew command works. I then have to start Nginx up again after the renew every time.

rg305 · January 8, 2024, 9:41pm

Hi @BilboBaggins, and welcome to the LE community forum

I'm glad you finally found us [after 4 years of trouble].

There is no need to use the force.
It will not forcibly renew a cert that can't be renewed.

Let's have a look at this file:

And the full nginx config, with the output of:

`nginx -T`

[note: that's a capital T]

This could use an update:

See: Certbot Instructions | Certbot (eff.org)

schoen · January 8, 2024, 9:55pm

This looks like you originally used the --standalone method (which assumes that there isn't a running web server using port 80). The renewals might work better with the --nginx method (and as @rg305 says, ideally also with a Certbot upgrade!).

BilboBaggins · January 8, 2024, 10:00pm

4 years I know tell me about it, everything became much harder for me to process things like I used to since I had a stroke in couple years ago. Still trying to get back.

You mentioned you wanted to look at - "Processing /etc/letsencrypt/renewal/theweylandcorporation.com.conf
And the full nginx config, with the output of:

`nginx -T`

[note: that's a capital T]"

Apologies, but what command do I need to run to get you that info you asked?

I have managed to update certbot it is now version 2.8.0

rg305 · January 8, 2024, 10:03pm

From the CLI [via SSH], do the following [and post whatever it returns]:
cat /etc/letsencrypt/renewal/theweylandcorporation.com.conf

Then [seapartly] also from the same CLI prompt, do [and post all that it returns]:
nginx -T

BilboBaggins · January 8, 2024, 10:11pm

Done as you said

# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/theweylandcorporation.com
cert = /etc/letsencrypt/live/theweylandcorporation.com/cert.pem
privkey = /etc/letsencrypt/live/theweylandcorporation.com/privkey.pem
chain = /etc/letsencrypt/live/theweylandcorporation.com/chain.pem
fullchain = /etc/letsencrypt/live/theweylandcorporation.com/fullchain.pem
# Options used in the renewal process
[renewalparams]
account = dfa06c6ba98440312ef939f2eae3e1f1
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = standalone

BilboBaggins · January 8, 2024, 10:13pm

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;
        more_set_headers "Server: cloudflare";
        # server_names_hash_bucket_size 64;
         server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}


#mail {
#       # See sample authentication script at:
#       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#       # auth_http localhost/auth.php;
#       # pop3_capabilities "TOP" "USER";
#       # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#       server {
#               listen     localhost:110;
#               protocol   pop3;
#               proxy      on;
#       }
#
#       server {
#               listen     localhost:143;
#               protocol   imap;
#               proxy      on;
#       }
#}

# configuration file /etc/nginx/modules-enabled/10-mod-http-ndk.conf:
load_module modules/ndk_http_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:
load_module modules/ngx_http_auth_pam_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-cache-purge.conf:
load_module modules/ngx_http_cache_purge_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-dav-ext.conf:
load_module modules/ngx_http_dav_ext_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-echo.conf:
load_module modules/ngx_http_echo_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-fancyindex.conf:
load_module modules/ngx_http_fancyindex_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-geoip.conf:
load_module modules/ngx_http_geoip_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-geoip2.conf:
load_module modules/ngx_http_geoip2_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-headers-more-filter.conf:
load_module modules/ngx_http_headers_more_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:
load_module modules/ngx_http_image_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-lua.conf:
load_module modules/ngx_http_lua_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-perl.conf:
load_module modules/ngx_http_perl_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-subs-filter.conf:
load_module modules/ngx_http_subs_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-uploadprogress.conf:
load_module modules/ngx_http_uploadprogress_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-upstream-fair.conf:
load_module modules/ngx_http_upstream_fair_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:
load_module modules/ngx_http_xslt_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-mail.conf:
load_module modules/ngx_mail_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-nchan.conf:
load_module modules/ngx_nchan_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-stream.conf:
load_module modules/ngx_stream_module.so;

# configuration file /etc/nginx/mime.types:

types {
    text/html                             html htm shtml;
    text/css                              css;
    text/xml                              xml;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
    application/atom+xml                  atom;
    application/rss+xml                   rss;

    text/mathml                           mml;
    text/plain                            txt;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/x-component                      htc;

    image/png                             png;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/x-icon                          ico;
    image/x-jng                           jng;
    image/x-ms-bmp                        bmp;
    image/svg+xml                         svg svgz;
    image/webp                            webp;

    application/font-woff                 woff;
    application/java-archive              jar war ear;
    application/json                      json;
    application/mac-binhex40              hqx;
    application/msword                    doc;
    application/pdf                       pdf;
    application/postscript                ps eps ai;
    application/rtf                       rtf;
    application/vnd.apple.mpegurl         m3u8;
    application/vnd.ms-excel              xls;
    application/vnd.ms-fontobject         eot;
    application/vnd.ms-powerpoint         ppt;
    application/vnd.wap.wmlc              wmlc;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/x-7z-compressed           7z;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-perl                    pl pm;
    application/x-pilot                   prc pdb;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            der pem crt;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xspf+xml                  xspf;
    application/zip                       zip;

    application/octet-stream              bin exe dll;
    application/octet-stream              deb;
    application/octet-stream              dmg;
    application/octet-stream              iso img;
    application/octet-stream              msi msp msm;

    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
    application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

    audio/midi                            mid midi kar;
    audio/mpeg                            mp3;
    audio/ogg                             ogg;
    audio/x-m4a                           m4a;
    audio/x-realaudio                     ra;

    video/3gpp                            3gpp 3gp;
    video/mp2t                            ts;
    video/mp4                             mp4;
    video/mpeg                            mpeg mpg;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-m4v                           m4v;
    video/x-mng                           mng;
    video/x-ms-asf                        asx asf;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;
}

# configuration file /etc/nginx/conf.d/redirect.conf:
server {
    server_name www.theweylandcorporation.com;
    return 301 $scheme://theweylandcorporation.com:8080$request_uri;
}

server {
    server_name theweylandcorporation.com;
    return 301 $scheme://theweylandcorporation.com:8080$request_uri;
}


# configuration file /etc/nginx/sites-enabled/default:
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# In most cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or Wordpress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration
#
server {
        listen 80 default_server;
        listen [::]:80 default_server;

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Note: You should disable gzip for SSL traffic.
        # See: https://bugs.debian.org/773332
        #
        # Read up on ssl_ciphers to ensure a secure configuration.
        # See: https://bugs.debian.org/765782
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        root /var/www/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm;

        server_name _;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

        # pass PHP scripts to FastCGI server
        #
        #location ~ \.php$ {
        #       include snippets/fastcgi-php.conf;
        #
        #       # With php-fpm (or other unix sockets):
        #       fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
        #       # With php-cgi (or other tcp sockets):
        #       fastcgi_pass 127.0.0.1:9000;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #       deny all;
        #}
}


# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that
# to sites-enabled/ to enable it.
#
#server {
#       listen 80;
#       listen [::]:80;
#
#       server_name example.com;
#
#       root /var/www/example.com;
#       index index.html;
#
#       location / {
#               try_files $uri $uri/ =404;
#       }
#}

BilboBaggins · January 8, 2024, 10:51pm

Apologies if I double posted, I am unsure about the forum layout structure, it looks like a double post but I'm not 100% sure if should delete anything to reduce clutter.

rg305 · January 8, 2024, 11:12pm

That is problematic [as was suspected].

And these are also a bit problematic.

BilboBaggins:

# configuration file /etc/nginx/conf.d/redirect.conf:
server {
    server_name www.theweylandcorporation.com;
    return 301 $scheme://theweylandcorporation.com:8080$request_uri;
}

server {
    server_name theweylandcorporation.com;
    return 301 $scheme://theweylandcorporation.com:8080$request_uri;
}

The first can be overcome with a proper one-time certbot request.
The second will need some editing to ensure certbot can work with it.

Do you know how to edit a file?

BilboBaggins · January 8, 2024, 11:18pm

I think, I know how to use filezilla and then as long as i know the file path I can find it and I believe open the file with notepad to edit then save , is that correct?

rg305 · January 8, 2024, 11:33pm

That could work.
Get the file:

edit it like so:

server {
    server_name www.theweylandcorporation.com;
    location /.well-known/acme-challenge {
        root /acme-challenge;
    }
    return 301 $scheme://theweylandcorporation.com:8080$request_uri;
}

server {
    server_name theweylandcorporation.com;
    location /.well-known/acme-challenge {
        root /acme-challenge;
    }
    return 301 $scheme://theweylandcorporation.com:8080$request_uri;
}

Copy it back to the same location.
And then create that /acme-challenge folder, with:
mkdir /acme-challenge

After all that, try:

certbot certonly --nginx -d theweylandcorporation.com -d www.theweylandcorporation.com --dry-run

BilboBaggins · January 9, 2024, 3:37pm

Right hopefully I did it correctly, After following your instructions, I ran the command and select option (U)?

“certbot certonly --nginx -d theweylandcorporation.com -d www.theweylandcorporation.com --dry-run”

Saving debug log to /var/log/letsencrypt/letsencrypt.log

An RSA certificate named theweylandcorporation.com already exists. Do you want

to update its key type to ECDSA?

(U)pdate key type/(K)eep existing key type: (U)pdate key type/(K)eep existing key type: u

Simulating renewal of an existing certificate for theweylandcorporation.com and www.theweylandcorporation.com

An unexpected error occurred:

No such authorization

Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Then I ran it again to try the other option (K)” certbot certonly --nginx -d theweylandcorporation.com -d www.theweylandcorporation.com --dry-run”

Saving debug log to /var/log/letsencrypt/letsencrypt.log

An RSA certificate named theweylandcorporation.com already exists. Do you want

to update its key type to ECDSA?

(U)pdate key type/(K)eep existing key type: k

Simulating renewal of an existing certificate for theweylandcorporation.com and www.theweylandcorporation.com

The dry run was successful.

BilboBaggins · January 9, 2024, 7:08pm

I know you said there is not need to use force, but i rant a test to force renew and this was the output:

certbot renew --force-renewal
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/theweylandcorporation.com.conf

Renewing an existing certificate for theweylandcorporation.com and www.theweylan dcorporation.com
Failed to renew certificate theweylandcorporation.com with error: Could not bind TCP port 80 because it is already in use by another process on this system (suc h as a web server). Please stop the program in question and then try again.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/theweylandcorporation.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

MikeMcQ · January 9, 2024, 7:42pm

That is expected at this stage of the problem solving. You have not yet gotten your certificate renewal profile repaired. Only --dry-run testing was done to test nginx and Certbot interaction.

It was very odd to have your migration to ECDSA fail but retaining the RSA cert work. Still, the next step is to issue the below (note no --dry-run). I would try it again with upgrading to ECDSA but if that fails try it without upgrade.

certbot certonly --nginx -d theweylandcorporation.com -d www.theweylandcorporation.com

Just let us know what happens. Please do not try renew with --force-renewal after those commands. If the above works, go ahead and try certbot renew --dry-run

BilboBaggins · January 9, 2024, 7:54pm

Right so I ran the commands you shared :certbot certonly --nginx -d theweylandcorporation.com -d www.theweylandcorporation.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log

An RSA certificate named theweylandcorporation.com already exists. Do you want
to update its key type to ECDSA?

(U)pdate key type/(K)eep existing key type: (U)pdate key type/(K)eep existing key type: U
Renewing an existing certificate for theweylandcorporation.com and ww.theweylandcorporation.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/theweylandcorporation.com/fullchain.pem
Key is saved at: /etc/letsencrypt/live/theweylandcorporation.com/privkey .pem
This certificate expires on 2024-04-08.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: Donate - Let's Encrypt
Donating to EFF: Support EFF's Work on Let's Encrypt | Electronic Frontier Foundation

certbot renew --dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/theweylandcorporation.com.conf

Simulating renewal of an existing certificate for theweylandcorporation.com and www.theweylandcorporation.com

Congratulations, all simulated renewals succeeded:
/etc/letsencrypt/live/theweylandcorporation.com/fullchain.pem (success)

MikeMcQ · January 9, 2024, 8:20pm

Excellent! A fresh cert was issued and your renewal is now working with viable settings.

That bad news is that HTTPS to your domain is not using the cert from today.

Are you still re-configuring your system? Do you need help sorting out this problem?

If so, let's start with seeing output of these two

certbot certificates
nginx -T

BilboBaggins · January 9, 2024, 8:25pm

Ok not sure what happened here think i ran your command again and this happened:

certbot renew --dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/theweylandcorporation.com.conf

Simulating renewal of an existing certificate for theweylandcorporation.com and www.theweylandcorporation.com

Encountered exception during recovery: certbot.errors.MisconfigurationError: nginx restart failed:

nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)