My domain is sambidb.com. This is my first ever attempt to get a certificate, and diving into the deep end of the pool by trying to get a wildcard one (I don’t even have a website at the root domain, although I might eventually - I’m currently only running a database webapp on subdomains). My server is a CentOS 7 VPS on Linode.
I updated certbot, installed python2-certbot-dns-linode, got an API key from the Linode Manager interface and put it in a file with chmod 600, and then ran this command:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-linode, Installer nginx
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for sambidb.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Waiting 60 seconds for DNS changes to propagate
Waiting for verification...
Cleaning up challenges
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Failed authorization procedure. sambidb.com (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: No TXT record found at _acme-challenge.sambidb.com
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: sambidb.com
Type: unauthorized
Detail: No TXT record found at _acme-challenge.sambidb.com
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
I looked at my DNS, and indeed, there is no TXT record. I would be happy to make it, but I don’t know the value it needs, and I got the impression from the tutorial that the Linode plugin was supposed to create it for me. (That’s the whole point of making the command wait 60 seconds for propagation, right?) My webapp works fine, so I would think that means that the DNS A record, nginx configuration, etc. are correct. What should I look for? (I tried reading letsencrypt.log, but I don’t know enough to recognize clues in it.)
Linode specifically takes a really long time to propagate DNS changes out to their nameservers (relative to other DNS providers). Last I checked, there's a note at the bottom of the DNS management page that states:
Changes made to a master zone will take effect in our nameservers every quarter hour.
That means after certbot writes the TXT record to your zone, it needs at least a 15 minute sleep time before it attempts to finalize the order. So up that propagation seconds value to at least 900 and try again. In my own personal testing with Linode, I'd still fail occasionally at 15 minutes and upped my own delay to 17 minutes.
You could check the sync independently of certbot.
As examples: nslookup -q=txt _acme-challenge.sambidb.com 8.8.8.8 nslookup -q=txt _acme-challenge.sambidb.com 1.1.1.1
(or from any other DNS server IP of your choosing)
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-linode, Installer nginx
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for sambidb.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Waiting 1000 seconds for DNS changes to propagate
Waiting for verification...
Resetting dropped connection: acme-v02.api.letsencrypt.org
Resetting dropped connection: acme-v02.api.letsencrypt.org
Cleaning up challenges
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Failed authorization procedure. sambidb.com (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: No TXT record found at _acme-challenge.sambidb.com
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: sambidb.com
Type: unauthorized
Detail: No TXT record found at _acme-challenge.sambidb.com
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
And looking at my DNS in the Linode interface, I never see any TXT records. Next idea?
Can you test the Linode DNS authenticator API independently of certbot?
Do you see any traffic from your system out to Linode systems?
try: curl https://api.linode.com/
No, I didn’t, because I don’t even understand what it does. Be kind to me - I’m a newbie.
Apparently there is something wrong with it, as your suggested curl https://api.linode.com/ command returned: {"ACTION":"","DATA":{},"ERRORARRAY":[{"ERRORMESSAGE":"Authentication failed","ERRORCODE":4}]}
That doesn’t sound good, but I have no idea what to do about it.
When I set up the API credentials (or attempted to - apparently I failed), I followed the instructions in the “Credentials” section of this page: https://certbot-dns-linode.readthedocs.io/en/stable/ I used the Linode Manager to generate a key, copied the contents into a file like they said to do:
[root](14:50:55)[/etc/letsencrypt]$ cat /root/.certbot/linode.ini
# Linode API credentials used by Certbot
dns_linode_key = 5rU...[the rest redacted, of course]...LUz
And I dutifully restricted the permissions:
[root](15:05:17)[/etc/letsencrypt]$ ls -la /root/.certbot
total 12
drwx------ 2 root root 4096 Nov 1 15:41 .
dr-xr-x---. 10 root root 4096 Nov 2 07:29 ..
-rw------- 1 root root 123 Nov 1 15:41 linode.ini
But I don’t even know how the curl command would know to look in that file for my key. Sorry, I’m out of my league - I’m a programmer, not a server admin.
I see nothing wrong with this command line request.
Sadly, I don't know of anyway to troubleshoot their API...
Maybe adding -vvv and reviewing the LE logs thereafter.
After adding quotes around the key (a shot in the dark, since the plugin doc page that the official certbot doc page point to do not use quotes in their example), the certificate for *.sambidb.com went through!
But after doing my happy dance, I ran the command again for another wildcard certificate I need (different domain, same server - this time including the naked domain in the same cert). I had learned that the default propagation time for the Linode plugin is 960, so I dropped that switch from the command, figuring 16 minutes is long enough. Alas, I got the “no TXT record” error again. I increased the propagation time to 1100 seconds and got a different error:
[root](17:03:32)[/etc/letsencrypt]$ certbot -a dns-linode --dns-linode-credentials /root/.certbot/linode.ini --dns-linode-propagation-seconds 1100 -i nginx -d "kizunadb.com" -d "*.kizunadb.com" --server https://acme-v02.api.letsencrypt.org/directory
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-linode, Installer nginx
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for kizunadb.com
dns-01 challenge for kizunadb.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Waiting 1100 seconds for DNS changes to propagate
Waiting for verification...
Resetting dropped connection: acme-v02.api.letsencrypt.org
Cleaning up challenges
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
Starting new HTTPS connection (1): api.linode.com
An unexpected error occurred:
The server experienced an internal error :: Problem getting authorization
Please see the logfiles in /var/log/letsencrypt for more details.
I reduced the time back to 1000, and it finally succeeded. I don’t think the quotes around the key or the small differences in waiting time are really the issue - it seems that it’s just flaky, and you have to try it several times before you get lucky.
I’ve learned a lot more along the way (waiting 16-17 minutes to see if a command worked will give you a lot of time to read documentation!). Here is some info for people coming along later and reading this:
The suggestion of using nslookup to check for the TXT record is not likely to reveal anything unless your timing is perfect, because as soon as certbot tests for it, it removes the record.
The certonly suggestion is also a distractor in this case. The Linode docs use that in their example, but they are describing a more manual technique. The nginx installer plugin works like a champ, so there is no need to get the cert first and then manually edit the nginx config files. Once the auth succeeds, certbot will list all the server directives in your nginx config files and ask you which ones you want it to modify, and also ask if you want it to add redirection from HTTP to HTTPS (I can’t imagine anyone not wanting that). That part of the process works smoothly.
Since it cleared up without any change in my command, I assume it was some sort of fluke. But it was frustrating to have it fail repeatedly before succeeding.
Question: Will it have to do this authentication and test again when certificates are renewed? I haven’t needed to renew yet, so I don’t know how smooth or difficult that is.
I had assumed I could set up a cron job to renew, but if I’m going to get errors sometimes, I don’t know if it can be set up to nicely repeat upon failure or something. I haven’t begun to study how to do renewals yet - I was just happy to get things going in the first place. If you have any suggestions about this, or want to point me to your favorite tutorial, I’m all ears.
It’s suggested to run Certbot’s renew command twice a day at random times.
When you installed the package, it might have set up a systemd timer or cron job to do that, or something similar.
By default it tries to renew certificates that will expire in less than 30 days, so it should have about 60 chances to succeed. If you can minimize failures, it should be enough.
Does "it" have a suggestion how to randomly schedule a cron job? All I can think of would be to do 14 weekly ones or 60 monthly ones at different (but of course not actually random) times.
I'm on CentOS, not Ubuntu, so I didn't even know what a PPA was (I know now), and it doesn't look like anything similar was installed on my server. I just now added the cron file you suggested. But my server does have systemd, and I get the impression that's the better method - I just don't really know what systemd is or how it works.
Reading elsewhere (especially Certbot - ArchWiki), it appears that I need a timer and a service. You've shared the timer. So I've created the /etc/systemd/service/certbot-renewal.service file the way they suggested. But I'm not sure about a couple things they said:
If you do not use a plugin to manage the web server configuration automatically, the web server has to be reloaded manually to reload the certificates each time they are renewed.
I did use a plugin for Linode when I got the initial certificate - does that mean I'm "using a plugin"? Sorry, but I don't really understand the interplay between different things.
And:
Note: Before adding a timer, check that the service is working correctly and is not trying to prompt anything.
It made no mention of how to do that. Based on this page, I ran systemd-analyze verify certbot-renewal.service and it returned nothing - is that good? I feel like nothing is really running yet - all I've done is create a file.
(Sorry this is getting a bit off-topic, but you obviously know this stuff well, so it's tempting to pick your brain.)
I don't know how it's set up on CentOS. Like, Certbot might not be in /usr/bin.
Not exactly. The important thing is that you used "-i nginx", using the Nginx plugin to install the certificate and configure Nginx. When you do that (or the same with the Apache plugin), Certbot automatically reloads the web server after renewing.
If you had manually configured it, you would have had to add a hook to reload the web server.
Reading more docs, I learned that in addition to /etc/systemd/system, there is /lib/systemd/system (yeah, I know, that should be basic knowledge, but…). Some installation I did the first time apparently put a certbot-renew.service and certbot-renew.timer in there. Yay! Its timer has OnCalendar=daily and RandomizedDelaySec=6hours, so I’ll keep it like that and see how it goes. I’ll set a reminder for myself to check at around the 61-day mark to see if it worked.