Cannot get certificate with custom Nginx config

Help
1

My domain is: wheelodex.org

I ran this command:

sudo certbot run \
        --nginx \
        --rsa-key-size 4096 \
        --expand \
        --cert-name wheelodex \
        --email REDACTED \
        --domains www.wheelodex.org,wheelodex.org \
        --non-interactive \
        --agree-tos

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Plugins selected: Authenticator nginx, Installer nginx

Cert is due for renewal, auto-renewing...

Renewing an existing certificate

Performing the following challenges:

http-01 challenge for wheelodex.org

http-01 challenge for www.wheelodex.org

nginx: [warn] conflicting server name "wheelodex.org" on 0.0.0.0:80, ignored

nginx: [warn] conflicting server name "www.wheelodex.org" on 0.0.0.0:80, ignored

nginx: [warn] conflicting server name "wheelodex.org" on [::]:80, ignored

nginx: [warn] conflicting server name "www.wheelodex.org" on [::]:80, ignored

Waiting for verification...

Cleaning up challenges

Failed authorization procedure. www.wheelodex.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.wheelodex.org/.well-known/acme-challenge/dLKv4nsTNS7d80I4G934B1XM5vrchidJl_lVdbnH9Ds: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\\n<title>404 Not Found</title>\\n<h1>Not Found</h1>\\n<p>The requested URL was"

IMPORTANT NOTES:

- The following errors were reported by the server:

Domain: www.wheelodex.org
Type: unauthorized

Detail: Invalid response from

http://www.wheelodex.org/.well-known/acme-challenge/dLKv4nsTNS7d80I4G934B1XM5vrchidJl_lVdbnH9Ds:

"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2

Final//EN\">\\n<title>404 Not Found</title>\\n<h1>Not

Found</h1>\\n<p>The requested URL was

To fix these errors, please make sure that your domain name was

entered correctly and the DNS A/AAAA record(s) for that domain

contain(s) the right IP address.

My web server is (include version): nginx-full 1.14.0-0ubuntu1.2

The operating system my web server runs on is (include version): Ubuntu Bionic 18.04.1

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

My site is configured via an Ansible playbook that runs the command listed above every time it’s run in order to ensure a certificate is present, and I trust certbot’s systemd files to take care of certificate renewals. The last time the above command was run successfully was 2018 Nov 14. Today, I got an e-mail that my site’s cert was about to expire, and the systemd logs showed that the renewal job was failing for some unspecified reason. I tried running sudo certbot renew, but that failed with the output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/wheelodex.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Cert is due for renewal, auto-renewing...

Plugins selected: Authenticator nginx, Installer nginx

Renewing an existing certificate

Performing the following challenges:

tls-sni-01 challenge for wheelodex.org

tls-sni-01 challenge for www.wheelodex.org

Waiting for verification...

Cleaning up challenges

Attempting to renew cert (wheelodex) from /etc/letsencrypt/renewal/wheelodex.conf produced an unexpected error: Failed authorization procedure. www.wheelodex.org (tls-sni-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect validation certificate for tls-sni-01 challenge. Requested 0f441e7c6ddafce814f175c886c06ea5.053d4ebc12491e1f302f6f72e26fb1f6.acme.invalid from [2604:a880:800:a1::c5d:8001]:443. Received 1 certificate(s), first certificate had names "30cbd04dcd1f74c6151b25ba6ebdb445.ccc945d59857aa9580a9d5e58881ac25.acme.invalid, dummy". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/wheelodex/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/wheelodex/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.wheelodex.org
   Type:   unauthorized
   Detail: Incorrect validation certificate for tls-sni-01 challenge.
   Requested
   0f441e7c6ddafce814f175c886c06ea5.053d4ebc12491e1f302f6f72e26fb1f6.acme.invalid
   from [2604:a880:800:a1::c5d:8001]:443. Received 1 certificate(s),
   first certificate had names
   "30cbd04dcd1f74c6151b25ba6ebdb445.ccc945d59857aa9580a9d5e58881ac25.acme.invalid,
   dummy"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

After some Googling, I then tried sudo certbot renew --preferred-challenges http with the result:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/wheelodex.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Cert is due for renewal, auto-renewing...

Plugins selected: Authenticator nginx, Installer nginx

Renewing an existing certificate

Performing the following challenges:

http-01 challenge for wheelodex.org

http-01 challenge for www.wheelodex.org

nginx: [warn] conflicting server name "www.wheelodex.org" on 0.0.0.0:80, ignored

nginx: [warn] conflicting server name "wheelodex.org" on 0.0.0.0:80, ignored

nginx: [warn] conflicting server name "www.wheelodex.org" on [::]:80, ignored

nginx: [warn] conflicting server name "wheelodex.org" on [::]:80, ignored

Waiting for verification...

Cleaning up challenges

Attempting to renew cert (wheelodex) from /etc/letsencrypt/renewal/wheelodex.conf produced an unexpected error: Failed authorization procedure. www.wheelodex.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.wheelodex.org/.well-known/acme-challenge/fFdwvIwNBSCByd4u5acHyrjLl9tLld5fsCvF3DBIsvE: "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2 Final//EN\">\n<title>404 Not Found</title>\n<h1>Not Found</h1>\n<p>The requested URL was". Skipping.
All renewal attempts failed. The following certs could not be renewed:

/etc/letsencrypt/live/wheelodex/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:

/etc/letsencrypt/live/wheelodex/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1 renew failure(s), 0 parse failure(s)

**IMPORTANT NOTES:**

- The following errors were reported by the server:

Domain: www.wheelodex.org

Type: unauthorized

Detail: Invalid response from

http://www.wheelodex.org/.well-known/acme-challenge/fFdwvIwNBSCByd4u5acHyrjLl9tLld5fsCvF3DBIsvE:

"<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 3.2

Final//EN\">\n<title>404 Not Found</title>\n<h1>Not

Found</h1>\n<p>The requested URL was"

To fix these errors, please make sure that your domain name was

entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I then ran sudo apt-get install to upgrade various Certbot packages:

  • certbot — upgraded from 0.26.1-1+ubuntu18.04.1+certbot+2 to 0.28.0-1+ubuntu18.04.1+certbot+4
  • python-certbot-nginx — upgraded from 0.25.0-2+ubuntu18.04.1+certbot+1 to 0.28.0-1+ubuntu18.04.1+certbot+3
  • python3-acme — upgraded from 0.26.0-1+ubuntu18.04.1+certbot+1 to 0.28.0-1+ubuntu18.04.1+certbot+3
  • python3-certbot — upgraded from 0.26.1-1+ubuntu18.04.1+certbot+2 to 0.28.0-1+ubuntu18.04.1+certbot+4
  • python3-certbot-nginx — upgraded from 0.25.0-2+ubuntu18.04.1+certbot+1 to 0.28.0-1+ubuntu18.04.1+certbot+3
  • python3-josepy — upgraded from 1.1.0-1 to 1.1.0-2+ubuntu18.04.1+certbot+1

and then tried running my Ansible playbook, which failed with the error message listed in the section above.

Past experience with running my Ansible playbook has led me to believe that Certbot edits the Nginx configuration in some manner in order to ensure things work (I don’t know how it edits them, since the configuration is immediately overwritten with my configuration by a subsequent Ansible task); however, based on filestamps, it appears that today’s Certbot runs are leaving the Nginx config alone, which may be behind the failures. (Incidentally, my Nginx config redirects everything to https://www.wheelodex.org, which is then handled by uWSGI; this may be another reason why everything is failing.)

How do I get Certbot to work again?

2

The duplicate virtual hosts might be the culprit – if Certbot edits one server block, but Nginx ignores it and uses the other, validation will fail.

Can you try fixing that issue?

And if it doesn’t help, post the Nginx configuration, which can be shown by “sudo nginx -T”?

3

Fix it how? By reducing the number of server {} blocks? I don't know how to do that while preserving the redirection behavior. Plus, as I said, part of the problem appears to be that Certbot isn't editing any server {} blocks in the first place.

Regardless, the full Nginx config:

# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 65;
	types_hash_max_size 2048;
	# server_tokens off;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}


#mail {
#	# See sample authentication script at:
#	# http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
# 
#	# auth_http localhost/auth.php;
#	# pop3_capabilities "TOP" "USER";
#	# imap_capabilities "IMAP4rev1" "UIDPLUS";
# 
#	server {
#		listen     localhost:110;
#		protocol   pop3;
#		proxy      on;
#	}
# 
#	server {
#		listen     localhost:143;
#		protocol   imap;
#		proxy      on;
#	}
#}

# configuration file /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:
load_module modules/ngx_http_auth_pam_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-dav-ext.conf:
load_module modules/ngx_http_dav_ext_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-echo.conf:
load_module modules/ngx_http_echo_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-geoip.conf:
load_module modules/ngx_http_geoip_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:
load_module modules/ngx_http_image_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-subs-filter.conf:
load_module modules/ngx_http_subs_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-upstream-fair.conf:
load_module modules/ngx_http_upstream_fair_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:
load_module modules/ngx_http_xslt_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-mail.conf:
load_module modules/ngx_mail_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-stream.conf:
load_module modules/ngx_stream_module.so;

# configuration file /etc/nginx/mime.types:

types {
    text/html                             html htm shtml;
    text/css                              css;
    text/xml                              xml;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
    application/atom+xml                  atom;
    application/rss+xml                   rss;

    text/mathml                           mml;
    text/plain                            txt;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/x-component                      htc;

    image/png                             png;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/x-icon                          ico;
    image/x-jng                           jng;
    image/x-ms-bmp                        bmp;
    image/svg+xml                         svg svgz;
    image/webp                            webp;

    application/font-woff                 woff;
    application/java-archive              jar war ear;
    application/json                      json;
    application/mac-binhex40              hqx;
    application/msword                    doc;
    application/pdf                       pdf;
    application/postscript                ps eps ai;
    application/rtf                       rtf;
    application/vnd.apple.mpegurl         m3u8;
    application/vnd.ms-excel              xls;
    application/vnd.ms-fontobject         eot;
    application/vnd.ms-powerpoint         ppt;
    application/vnd.wap.wmlc              wmlc;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/x-7z-compressed           7z;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-perl                    pl pm;
    application/x-pilot                   prc pdb;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            der pem crt;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xspf+xml                  xspf;
    application/zip                       zip;

    application/octet-stream              bin exe dll;
    application/octet-stream              deb;
    application/octet-stream              dmg;
    application/octet-stream              iso img;
    application/octet-stream              msi msp msm;

    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
    application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

    audio/midi                            mid midi kar;
    audio/mpeg                            mp3;
    audio/ogg                             ogg;
    audio/x-m4a                           m4a;
    audio/x-realaudio                     ra;

    video/3gpp                            3gpp 3gp;
    video/mp2t                            ts;
    video/mp4                             mp4;
    video/mpeg                            mpeg mpg;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-m4v                           m4v;
    video/x-mng                           mng;
    video/x-ms-asf                        asx asf;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;
}

# configuration file /etc/nginx/sites-enabled/wheelodex.conf:
server {
    listen *:80;
    listen *:443 ssl;
    listen [::]:80;
    listen [::]:443 ssl;
    server_name wheelodex.org;

    ssl on;
    ssl_certificate     /etc/letsencrypt/live/wheelodex/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/wheelodex/privkey.pem;

    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH !EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";

    return 301 https://www.wheelodex.org$request_uri;
}


server {
    listen *:80 default_server;
    listen [::]:80 default_server;
    server_name www.wheelodex.org;

    return 301 https://www.wheelodex.org$request_uri;
}

server {
    listen *:443 ssl;
    listen [::]:443 ssl;
    server_name www.wheelodex.org;

    ssl on;
    ssl_certificate     /etc/letsencrypt/live/wheelodex/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/wheelodex/privkey.pem;

    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_ciphers "EECDH+ECDSA+AESGCM EECDH+aRSA+AESGCM EECDH+ECDSA+SHA384 EECDH+ECDSA+SHA256 EECDH+aRSA+SHA384 EECDH+aRSA+SHA256 EECDH+aRSA+RC4 EECDH !EDH+aRSA !RC4 !aNULL !eNULL !LOW !3DES !MD5 !EXP !PSK !SRP !DSS";

    root /var/www/html;

    location / {
        if (-f $document_root/active_maintenance.html) {
            return 503;
        }
        try_files $uri @wheelodex;
    }

    location @wheelodex {
        include uwsgi_params;
        uwsgi_pass unix:/tmp/wheelodex.sock;
    }

    error_page 503 /active_maintenance.html;

    location = /active_maintenance.html { }
}

# configuration file /etc/nginx/uwsgi_params:

uwsgi_param  QUERY_STRING       $query_string;
uwsgi_param  REQUEST_METHOD     $request_method;
uwsgi_param  CONTENT_TYPE       $content_type;
uwsgi_param  CONTENT_LENGTH     $content_length;

uwsgi_param  REQUEST_URI        $request_uri;
uwsgi_param  PATH_INFO          $document_uri;
uwsgi_param  DOCUMENT_ROOT      $document_root;
uwsgi_param  SERVER_PROTOCOL    $server_protocol;
uwsgi_param  REQUEST_SCHEME     $scheme;
uwsgi_param  HTTPS              $https if_not_empty;

uwsgi_param  REMOTE_ADDR        $remote_addr;
uwsgi_param  REMOTE_PORT        $remote_port;
uwsgi_param  SERVER_PORT        $server_port;
uwsgi_param  SERVER_NAME        $server_name;
4

By removing any duplicate server blocks, but you don't seem to have any...

For the validation step, Certbot makes temporary changes to the Nginx configuration, and then removes them a few seconds later.

What it was up to should be logged in /var/log/letsencrypt/letsencrypt.log.

If you run Certbot without --non-interactive and with --debug-challenges, it will pause in the middle so you can examine the configuration.

If it successfully obtains a certificate, for the installation step, it makes permanent changes to the configuration to use the certificate. But since your configuration already does, it shouldn't have anything left to do (other than reload Nginx).

You need to remove "ssl on;" -- it's deprecated, and it enables SSL on all ports in the server block, which in this case includes port 80. Port 80 should be really malfunctioning right now.

(You already have ssl on the appropriate listen directives, so you don't need to do anything else.)

I dunno if it's causing any of the problems you have now, but it should cause a bunch of other problems at the least.

5

Removing ssl on seems to have been sufficient to get Certbot working again. Thanks!

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.