Opaque error message

lewisl · September 13, 2018, 9:22pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: the prequel.org

I ran this command: certbot certainly --cert-name theprequel.org -d theprequel.org, www.theprequel.org

It produced this output: Requested domain is not a FQDN because it contains an empty label.

My web server is (include version): nginx

The operating system my web server runs on is (include version): ubuntu 16.04 its

My hosting provider, if applicable, is: linode

I can login to a root shell on my machine (yes or no, or I don’t know): yes

Further, I looked for empty labels:
fgrep -r … /etc found tons of things, all of which were comment lines from various ini files, conf files, etc.
fgrep -r … /etc/letsencrypt returned nothing.
fgrep -r … /etc/nginx returned nothing.

So, not only is the error message opaque (of no value) it would also appear to be inaccurate in this case. If the error message were useful, it would list the domain names and files where empty labels were found.

lewisl · September 13, 2018, 9:28pm

Fixed by adding --nginx option in the command. You’d think it would discover that, but it doesn’t. Best to update documenatinom that the plug-in option is ALWAYS needed.

JuergenAuer · September 13, 2018, 9:29pm

Hi @lewisl

looks curious. there is a space.

Try using two -d options

certbot certonly --cert-name theprequel.org -d theprequel.org -d www.theprequel.org

or without the space

certbot certonly --cert-name theprequel.org -d theprequel.org,www.theprequel.org

And use certonly - not certainly

lewisl · September 13, 2018, 9:30pm

No, there is a comma. I redid it and I thought --nginx fixed it. But, I also used -d in front of both. So, that probably fixed it. Should be documented that way. So many problems would be solved by accurate documentation.

_az · September 13, 2018, 9:41pm

It really helps to provide the command you ran verbatim and not just an approximation, since details matter (such as the -d syntax).

(Edit: I’m also not convinced that --nginx is necessary, Certbot does try to detect your webserver if you just run certbot -d example.org).

For the criticism of documentation, it would help to understand where you arrived from. Starting at https://certbot.eff.org and choosing nginx as your web server, you are shown to pass nginx as an authenticator to the the certbot command.

If you started somewhere else, maybe it’s wrong, but please tell us where!

lewisl · September 13, 2018, 9:51pm

Well, I spoke too soon. I got positive acknowledgment:
certbot --nginx --cert-name theprequel.org -d theprequel.org -d www.theprequel.org

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Plugins selected: Authenticator nginx, Installer nginx

Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org

You are updating certificate theprequel.org to include new domain(s):

www.theprequel.org

You are also removing previously included domain(s):

(None)

Did you intend to make this change?

(U)pdate cert/©ancel: U

Renewing an existing certificate

Performing the following challenges:

tls-sni-01 challenge for www.theprequel.org

tls-sni-01 challenge for theprequel.org

Waiting for verification…

Cleaning up challenges

Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/theprequel.org-ssl.conf

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.

1: No redirect - Make no further changes to the webserver configuration.

2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for

new sites, or if you’re confident your site works on HTTPS. You can undo this

change by editing your web server’s configuration.

Select the appropriate number [1-2] then [enter] (press ‘c’ to cancel): 1

Your existing certificate has been successfully renewed, and the new certificate

has been installed.

The new certificate covers the following domains: https://theprequel.org and

https://www.theprequel.org

You should test your configuration at:

https://www.ssllabs.com/ssltest/analyze.html?d=theprequel.org

https://www.ssllabs.com/ssltest/analyze.html?d=www.theprequel.org

IMPORTANT NOTES:

Congratulations! Your certificate and chain have been saved at:

/etc/letsencrypt/live/theprequel.org/fullchain.pem

Your key file has been saved at:

/etc/letsencrypt/live/theprequel.org/privkey.pem

Your cert will expire on 2018-11-30. To obtain a new or tweaked

version of this certificate in the future, simply run certbot again

with the “certonly” option. To non-interactively renew all of

your certificates, run “certbot renew”

I then deleted the certificate exception from my browser. I cannot get a connection at www.theprequel.org. I ran the test from Qualys labs. theprequel.org passed.

www.theprequel.org failed with: Certificate name mismatch.

So, a server with both domains was NOT installed. And to confirm, here is what certbot certificates reports:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Found the following certs:
Certificate Name: theprequel.org
Domains: theprequel.org
Expiry Date: 2018-11-30 15:51:38+00:00 (VALID: 77 days)
Certificate Path: /etc/letsencrypt/live/theprequel.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/theprequel.org/privkey.pem

So, what happened? Why would certbot report success and actually fail? You guys need something like certbot doctor because the whole thing, as simple as it is, is really fragile.

lewisl · September 13, 2018, 9:55pm

So, once again certbot messed up the config. Here is what we have in /etc/letsencrypt/live

root@lewlin1:/etc/letsencrypt/live/theprequel.org# ls -la

total 12

drwxr-xr-x 2 root root 4096 Sep 13 21:26 **.**

drwx------ 3 root root 4096 Sep 13 21:09 **..**

lrwxrwxrwx 1 root root 43 Sep 13 21:26 **cert.pem** -&gt; ../../archive/theprequel.org-0001/cert1.pem
lrwxrwxrwx 1 root root 44 Sep 13 21:26 **chain.pem** -&gt; ../../archive/theprequel.org-0001/chain1.pem
lrwxrwxrwx 1 root root 48 Sep 13 21:26 **fullchain.pem** -&gt; ../../archive/theprequel.org-0001/fullchain1.pem
lrwxrwxrwx 1 root root 46 Sep 13 21:26 **privkey.pem** -&gt; ../../archive/theprequel.org-0001/privkey1.pem
-rw-r--r-- 1 root root 682 Sep 1 16:51 README

It’s auto renamed the domain certs and the wrong cert is being used. In other words, the installation failed.

lewisl · September 13, 2018, 10:03pm

The nginx server block seems to be ok:

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name www.theprequel.org theprequel.org;
    root /var/www/prequel/ghost/system/nginx-root;
    ssl_certificate /etc/letsencrypt/live/theprequel.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/theprequel.org/privkey.pem; # managed by Certbot
    include /etc/nginx/snippets/ssl-params.conf;
    
    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_pass http://127.0.0.1:2368;
        
    }
    
    location ~ /.well-known {
        allow all;
    }
    
    client_max_body_size 50m;



}

So, maybe it is the renaming of the server files?

JuergenAuer · September 13, 2018, 10:18pm

Use

certbot certificates

to see what you have. Find the certificate you’ve created today:

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:www.theprequel.org&lu=cert_search

Change your nginx-configuration, so you use this certificate.

Then use

certbot delete ...

to remove the other certificates.

mnordhoff · September 13, 2018, 10:27pm

Can you also post “ls -alR /etc/letsencrypt/{archive,live,renewal}”?

lewisl · September 13, 2018, 10:28pm

I ran that Google test. It shows I have certificates issued on June 23 and September 13 (today). Both cover 2 domains: www.theprequel.org, theprequel.org.

How does it find the earlier one because I don’t think it is installed any more.

Despite the Google test showing both domains and positive response to getting a new certificate today that confirmed two domains were covered, this is what certbot certificates shows:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: theprequel.org
    Domains: theprequel.org
    Expiry Date: 2018-11-30 15:51:38+00:00 (VALID: 77 days)
    Certificate Path: /etc/letsencrypt/live/theprequel.org/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/theprequel.org/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

So, not both domains. I don’t get this. It is fragile and non-deterministic. How to make this work? I am following the documentation, not the many erroneous posts in the forum.

JuergenAuer · September 13, 2018, 10:32pm

Certificate transparency logs are read only. It's impossible to remove entries.

There must be a certificate with two names theprequel.org www.theprequel.org, created today.

If not, you may have two different certbot installations.

lewisl · September 13, 2018, 10:32pm

There is definitely something wrong with the certbot config:

Google verification says the expiration is December 12, 2018.
Certbot says the expiration is November 30, 2018.

So we are talking about two different certificates here. What’s real?

I agree I should have a cert with two different names. But, certbot doesn’t report that. And, https://www.theprequel.org can’t get through because of a name mismatch. So, what Google verification thinks is installed is not really installed.

I have restarted nginx several times.

Flummoxed.

lewisl · September 13, 2018, 10:37pm

I need to do this at another site but I am not going to try until there is a deterministically reliable process know/proven to work. How can we verify my config that nginx is pointing to the right place? I fear that all the symlinks are messed up again.

mnordhoff · September 13, 2018, 10:40pm

Both certificates are real. Here's the Google site's record of the certificate expiring 2018-11-30:

https://transparencyreport.google.com/https/certificates/Z5IRDimtrULXYWf%2BiGhzvVhzZvbNnY3SiJaOKaBbeqw%3D

The "ls" you posted earlier shows that something has happened to /etc/letsencrypt/. Did you rename or delete anything in it?

lewisl · September 13, 2018, 10:42pm

I did not post the latter. All I do is ls -la.

I did manual changes in /etc/letsencrypt. I learn that is a terrible no-no because of its fragility. I have been working to get it back to something correct. I followed advice from yesterday to do that. The thing is that accessing https://theprequel.org correctly uses https and finds some sort of valid certificate. But, which one? There can only be the most recently installed.

lewisl · September 13, 2018, 10:55pm

Here you go:

root@lewlin1:/etc/letsencrypt/live/theprequel.org# ls -alR /etc/letsencrypt/{archive,live,renewal}
/etc/letsencrypt/archive:
total 16
drwx------ 4 root root 4096 Sep  1 16:51 .
drwxr-xr-x 9 root root 4096 Sep 13 22:30 ..
drwxr-xr-x 2 root root 4096 Sep 13 21:26 theprequel.org
drwxr-xr-x 2 root root 4096 Sep  1 16:51 theprequel.org-0001

/etc/letsencrypt/archive/theprequel.org:
total 40
drwxr-xr-x 2 root root 4096 Sep 13 21:26 .
drwx------ 4 root root 4096 Sep  1 16:51 ..
-rw-r--r-- 1 root root 2179 Jun 23 20:45 cert1.pem
-rw-r--r-- 1 root root 2179 Sep 13 21:26 cert2.pem
-rw-r--r-- 1 root root 1647 Jun 23 20:45 chain1.pem
-rw-r--r-- 1 root root 1647 Sep 13 21:26 chain2.pem
-rw-r--r-- 1 root root 3826 Jun 23 20:45 fullchain1.pem
-rw-r--r-- 1 root root 3826 Sep 13 21:26 fullchain2.pem
-rw-r--r-- 1 root root 1704 Jun 23 20:45 privkey1.pem
-rw-r--r-- 1 root root 1704 Sep 13 21:26 privkey2.pem

/etc/letsencrypt/archive/theprequel.org-0001:
total 24
drwxr-xr-x 2 root root 4096 Sep  1 16:51 .
drwx------ 4 root root 4096 Sep  1 16:51 ..
-rw-r--r-- 1 root root 2151 Sep  1 16:51 cert1.pem
-rw-r--r-- 1 root root 1647 Sep  1 16:51 chain1.pem
-rw-r--r-- 1 root root 3798 Sep  1 16:51 fullchain1.pem
-rw-r--r-- 1 root root 1704 Sep  1 16:51 privkey1.pem

/etc/letsencrypt/live:
total 12
drwx------ 3 root root 4096 Sep 13 21:09 .
drwxr-xr-x 9 root root 4096 Sep 13 22:30 ..
drwxr-xr-x 2 root root 4096 Sep 13 21:26 theprequel.org

/etc/letsencrypt/live/theprequel.org:
total 12
drwxr-xr-x 2 root root 4096 Sep 13 21:26 .
drwx------ 3 root root 4096 Sep 13 21:09 ..
lrwxrwxrwx 1 root root   43 Sep 13 21:26 cert.pem -> ../../archive/theprequel.org-0001/cert1.pem
lrwxrwxrwx 1 root root   44 Sep 13 21:26 chain.pem -> ../../archive/theprequel.org-0001/chain1.pem
lrwxrwxrwx 1 root root   48 Sep 13 21:26 fullchain.pem -> ../../archive/theprequel.org-0001/fullchain1.pem
lrwxrwxrwx 1 root root   46 Sep 13 21:26 privkey.pem -> ../../archive/theprequel.org-0001/privkey1.pem
-rw-r--r-- 1 root root  682 Sep  1 16:51 README

/etc/letsencrypt/renewal:
total 12
drwxr-xr-x 2 root root 4096 Sep 13 21:26 .
drwxr-xr-x 9 root root 4096 Sep 13 22:30 ..
-rw-r--r-- 1 root root  527 Sep 13 21:26 theprequel.org.conf

This makes clear that there are two certificates installed. "live" links to the wrong directory. It appears I should rmdir /etc/letsencrypt/archive/theprequel.org-0001 do you agree?

The symbolic links in live should be to ../../archive/theprequel.org/cert2.pem
and /chain2.pem
and /fullchain2.pem
and /privkey2.pem

(where the last three start with ../../archive/theprequel.org as shown first)

I can do this manually but is there a way certbot can fix it?

I know you want to blame me for messing it up but it got messed up by a prior renewal (before that renewal I didn't alter anything).

mnordhoff · September 13, 2018, 11:18pm

Yyye -- probably. Yes, but I didn't get enough sleep, so I don't trust my judgment enough to say anything absolute.

Certbot has a "certbot update_symlinks" command that can fix some symlink issues. The problem is I'm not sure which types of issues it knows how to fix.

You could:

Back up /etc/letsencrypt.
Run certbot update_symlinks.
If it didn't work, delete the symlinks and run certbot update_symlinks again, to see if it recreates them.
If it still didn't work, fix them manually.

mnordhoff · September 13, 2018, 11:20pm

Also, you may want to double check that /etc/letsencrypt/renewal/theprequel.org.conf is entirely correct. Among other things, it contains references to one of the two /etc/letsencrypt/archive/ directories.

lewisl · September 13, 2018, 11:20pm

Cool. Great job. Running the multiple directory listing made the problem clear.

I’ll try the command after backing up. If that doesn’t work, I’ll fix it. Seems like that is right.

The bigger question is whether certbot can subsequently “just work”.