LetsEncrypt + DANE

Help
1

(What’s a “holiday”?)

And what is DANE? DNSSEC at your registrar makes sure you get the correct IP address when you want to connect, while a DANE TLSA record (signed by DNSSEC) will give information about the SSL/TLS certificate you can expect at the specific service you are connecting to. So perfect forward secrecy for Apache, Postfix, stunnel, etc.

I’ve just about got LE set up for DANE, but I don’t know some functions of certbot. Here’s what I have so far:

DANE with LetsEncrypt is a problem because LE certs expire every 90 days, but the public cert must be hashed at the registrar. So force LE to use my CSR, for the same public cert hash every time. (Unchanging cert) Make a point to manually change certs once/year.

- Edit /etc/letsencrypt/csr-quantum-equities.com.conf

[req]
default_bits = 4096
distinguished_name = req_distinguished_name
req_extensions = req_ext
[req_distinguished_name]
countryName = US
stateOrProvinceName = Washington
localityName = Seattle
organizationName = ''
commonName = quantum-equities.com
emailAddress = ‘’
[req_ext]
subjectAltName = @alt_names
[alt_names]
DNS.1 = mail.quantum-equities.com
DNS.2 = www.quantum-equities.com
… other domains

    - # openssl req -new -sha512 -key /etc/letsencrypt/live/quantum-equities.com/privkey.pem -out /etc/letsencrypt/csr-quantum-equities.com.csr -subj '/CN=quantum-equities.com' -config /etc/letsencrypt/csr-quantum-equities.com.conf

- Use this CSR to obtain an LE cert:
	# systemctl stop httpd
	# certbot certonly --csr /etc/letsencrypt/csr-quantum-equities.com.csr -d quantum-equities.com
		2: Spin up a temporary webserver (standalone)
	# mv 0000_cert.pem /etc/letsencrypt/archive/quantum-equities.com/cert1.pem
	# mv 0000_chain.pem /etc/letsencrypt/archive/quantum-equities.com/chain1.pem
	# mv 0001_chain.pem /etc/letsencrypt/archive/quantum-equities.com/fullchain1.pem
	... other domains

So far, so good. Now I have my LE certs in place, but how to renew? I needed certonly, to get my initial certs with a custom --csr, so I doubt that certbot will be happy with a simple run. (although it does work with --dryrun) My understanding is that certs fetched with certonly, can not be renewed with a run.

Assuming that is not true, I can just update with a regular run and --csr to specify the same cert over and over so DANE doesn’t get upset.

Assuming the above is true, I’ll need to make a script of some sort.

2

There’s a thread in this forum by a DNSSEC/DANE expert, a couple years back, explaining best practices here. Search for “3 1 1 Dane”, I think.

3

The most recent LE+DANE thread I could find is here.

But there’s also this more recent post which describes several certbot options I’ve never seen, which I suspicion are new.

I think it’s likely that certbot is under heavy development and may have some useful new options WRT DANE (and other functions). That’s the reason for this thread. {trawl, trawl}

4

You are indeed correct that you need certonly in order to do the renewal with a custom CSR.

As for auto renew scripts, as I posted in an earlier thread, you might try mine or any of several other examples online to see if they can be modified to suit your needs.

5

It's complex. And clearly not for RHEL. It'll take me a while to check every command for translation. There are so many things that can go wrong, and I have to make sure I'm aware of it if they do.

Why ECDSA? That was promulgated with input from one of those three-letter agencies.

It will become a larger and larger issue as DANE is more accepted, and clearly the LE devs did not intend for certs to go a year or more with no renewal. Hopefully tomorrow they'll chime in.

6

Modifying the script for RHEL/CentOS variants should mostly be a matter of changing the paths to the certificates and keys. But of course, reviewing foreign code before using it is best practice.

As for why I use ECDSA, simply for the smaller, more efficient key sizes. I’d gladly exchange the NIST curves for Ed25519, but that isn’t supported in TLS or x509 standards yet, and probably won’t be available for new certificates without an update to the CA/B Forum Baseline requirements.

Also, to ensure you don’t have to update your DNS records every 90 days, you could just renew the certificate using the same private key. That’s basically what I do, and rotate the key annually. I feel that’s a decent security/convenience trade-off, but if you desire more frequent key changes, you might be able to automate the DNS updates as well, depending on who your provider is and if their API supports it.

7

So nothing. No input from devs?

8

This was the thread I was trying to point you to: Please avoid “3 0 1” and “3 0 2” DANE TLSA records with LE certificates. It seems like the missing piece for you is easy automated renewal without rotating keys. There’s a Certbot issue open for that.

1 Like
9

This is correct. To renew the certificate just return the same certbot command you ran with --csr in the first place. There is no need to regenerate the CSR every time, as Let's Encrypt ignores the stuff in it that could possibly change anyway.

When people say certbot doesn't support renewing certificates issued with --csr, they're referring to the fact that certbot renew doesn't go over all your old CSRs and renew them. You just have to renew them manually or put your full CSR command into a cron job instead of running certbot renew (or in addition to it, if you have certificates issued without --csr).

@schoen was working on a pull request for key reuse but it is not finished yet. At the current time, using --csr is the only way to achieve key reuse for HPKP and DANE.

RSA has been promulgated by more three letter agencies for longer than ECDSA so by this argument you should prefer ECDSA to RSA. :stuck_out_tongue_winking_eye:

In reality, they have conflicting goals in securing their own stuff and getting into ours. One of their goals in undermining standards processes is to trick you into using less secure stuff because you think the more secure stuff is less secure because they say it's more secure. Don't fall into their reverse psychology trap! Look at what other cryptography people say and base your judgements on that, not whether the government approves of it or not.

10

Hm, I'm using a systemd timer. (They're encouraging us to move to that over cron)

So my letsencrypt.service would look like:

[Unit]
Description=Service for SSL cert renewals

[Service]
#EnvironmentFile=/etc/sysconfig/certbot
Type=oneshot
ExecStart=/usr/bin/certbot certonly --csr /etc/letsencrypt/csr-quantum-equities.com.csr -d quantum-equities.com
ExecStart=/usr/bin/certbot certonly --csr /etc/letsencrypt/csr-delphi-real-estate.com.csr -d delphi-real-estate.com --apache

# Security
PrivateTmp=yes
InaccessibleDirectories=/bin /boot /home /media /mnt /opt /root /sbin /sys
ReadOnlyDirectories=/lib /lib64 /usr
CapabilityBoundingSet=~CAP_SYS_PTRACE
DeviceAllow=/dev/null rw
NoNewPrivileges=yes

(I can get away with multiple ExecStarts since this is Type=oneshot)

Is this all I need, and not a script?

Is it correct that even with just -d {domain}.com, that since the csr includes also www.{domain}.com and mail.{domain}.com, that all three would be in the cert?

And would --apache operate to restart Apache afterward?

lol, fair enough.

In making my statement about ECDSA and RSA above, I am relying on the Word Of Schnier. I'll stick with RSA for now.

11

You shouldn’t need to pass -d at all, as certbot should ignore it and use what’s in the CSR.

You need --apache in the first ExecStart line as well, or --webroot -w /var/www/html, or whichever authentication method you are using.

You also want to set WorkingDirectory to the directory you want your certificate to be stored in, since certbot saves the certificate in the current working directory. systemd would otherwise default to using the root user’s home directory, which you make inaccessible later in your unit file.

Also remember that this will issue a certificate every time, so the corresponding timer unit should be set to run every two months and you should follow up on any errors if it fails. It’s possible the scripts linked to upthread implement some retrying logic to reduce manual effort.

12

YIKES, lots of gotchas there.

So I’ll remove all the -d’s and will add --apache to each line.

Problem with setting WorkingDirectory is that it’s different for each domain. I don’t know that I can have multiple ones in one .service . A different .service for each domain? Hm, seems like certbot has --chain-path ${chain} --cert-path ${cert} options. Would those work per line? And would this be the /live/ directory or the /archive/?

I’ll change my .timer to:

[Unit]
Description=Timer for SSL cert renewals

[Timer]
OnCalendar=01-03,03-03,05-03,07-03,09-03,11-03
Persistent=true

[Install]
WantedBy=timers.target

There doesn’t seem to be a ‘bimonthly’ option. This will trigger at 00:00:00 which is fine with me, but if I need multiple .service files I’d better use RandomizedDelaySec=.

13

Yup, that's what you want.

It looks a lot like the files in /live/, except they aren't symlinks to /archive/. Your private key would be missing unless you manually place it there.

I wouldn't recommend reusing /etc/letsencrypt/{live, archive}. Even though certbot should ignore directories that don't have a corresponding renewal configuration file it's possible this could still confuse some code paths in certbot. Instead just use a directory of your own creation.

You can abbreviate this slightly and this makes it easier to specify a time other than midnight if you want:

OnCalendar=*-01,03,05,09,11-03 00:00:00

RandomizedDelaySec is a great idea even if you have one unit because Let's Encrypt's infrastructure tends to be hit the hardest on the hour. Using it helps them out and slightly reduces the chance you'll run into temporary outages or overcapacity issues.

1 Like
14

Nice, thanks. So now my .timer is:

[Unit]
Description = Timer for SSL cert renewals

[Timer]
# Realtime timer
OnCalendar = *-01,03,05,09,11-03 00:00:00
RandomizedDelaySec = 3600
Persistent = true

[Install]
WantedBy = timers.target

So I’d like to date the certs, and get an email when something goes wrong. This points to the need for a script.

And I think a sensible place to put the certs is /etc/pki/tls/certs, chains, and private.

Proposal (shamefully plagiarized from here):

DATE=$(date +%Y-%m-%d)

##
# certbot handling
#
# first it cannot replace certs, so ensure new locations (date suffix)
# each time mean the certificate is unique each time.  Next, it's
# really chatty, so the only way to tell if there was a failure is to
# check whether the certificates got updated and then get cron to
# email the log
##

# Quantum-Equities.com
DIR=/etc/pki/tls
DIR2=/etc/letsencrypt/live/quantum-equities.com
CERT="${dir}/certs/cert-${date}.pem"
FULLCHAIN="${dir}/chains/fullchain-${date}.pem"
CHAIN="${dir}/chains/chain-${date}.pem"
CSR=/etc/letsencrypt/csr-quantum-equities.com.csr
OUT=/tmp/certbot-QE.out

certbot certonly --webroot --csr ${CSR} --preferred-challenges http-01 -w /srv/www/QE --fullchain-path ${FULLCHAIN} --chain-path ${CHAIN} --cert-path ${CERT} > ${OUT} 2>&1

if [ ! -f ${FULLCHAIN} -o ! -f ${CHAIN} -o ! -f ${CERT} ]; then
    cat ${out}
    exit 1;
fi

# link into place

# cert only (apache needs)
ln -sf ${CERT} ${DIR2}/cert.pem
# cert with chain (stunnel needs)
ln -sf ${FULL} ${DIR2}/fullchain.pem
# chain only (apache needs)
ln -sf ${CHAIN} ${DIR}/chain.pem

# ... next domain ...

# reload the services
systemctl reload HTTPD
systemctl reload postfix
systemctl restart stunnel4

I don’t understand the need for these things – the docs aren’t clear to me:
–webroot
–preferred-challenges http-01
-w /srv/www/QE
} ${OUT} 2>&1

Unfortunately I can’t test it without disrupting the existing certs. Although, since they aren’t getting changed maybe it wouldn’t hurt?

15

I believe that I finally have my update script right, but I can’t test it because, “There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: mail.quantum-equities.com,quantum-equities.com,www.quantum-equities.com”

I thought it was an hour wait, but joke’s on me.

!/bin/bash

DATE=$(date +%Y-%m-%d)

##
# certbot handling
#
# first it cannot replace certs, so ensure new locations (date suffix)
# each time mean the certificate is unique each time.  Next, it's
# really chatty, so the only way to tell if there was a failure is to
# check whether the certificates got updated and then get cron to
# email the log
##

# Quantum-Equities.com
DIR=/etc/pki/tls
DIR2=/etc/letsencrypt/live/quantum-equities.com
CERT="${DIR}/certs/quantum-equities.com_cert-${DATE}.pem"
FULLCHAIN="${DIR}/chains/quantum-equities.com_fullchain-${DATE}.pem"
CHAIN="${DIR}/chains/quantum-equities.com_chain-${DATE}.pem"
CSR=/etc/letsencrypt/csr-quantum-equities.com.csr
OUT=/tmp/certbot-QE.out

certbot certonly --webroot --csr ${CSR} --preferred-challenges http-01 -w /srv/www/QE --fullchain-path ${FULLCHAIN} --chain-path ${CHAIN} --cert-path ${CERT} > ${OUT} 2>&1

if [ ! -f ${FULLCHAIN} -o ! -f ${CHAIN} -o ! -f ${CERT} ]; then
    cat /tmp/certbot-QE.out | mail -s "TLS Cert Update Fail for quantum-equities.com" postmaster@quantum-equities.com
fi

# link into place

# cert only (apache needs)
ln -sf ${CERT} ${DIR2}/cert.pem
# cert with chain (stunnel needs)
ln -sf ${FULL} ${DIR2}/fullchain.pem
# chain only (apache needs)
ln -sf ${CHAIN} ${DIR}/chain.pem
16

How long does it take to reset? I’ve deleted my certs and chains so am dead right now.

17

In your service file, you use --apache, which tells certbot to use TLS-SNI-01 authentication with the running Apache webserver. The --webroot option tells certbot to use HTTP-01 authentication by placing a file in the directory /srv/www/QE which must be the one used to serve files for the domain. (The --preferred-challenges option is redundant and unnecessary.)

The --apache method is easier to use and doesn't require your script to know where the public web files are kept for particular domains. If that's what you were using before you can replace all three of these options with just this one.

> ${OUT} redirects the standard output stream to the filename stored in $OUT. 2>&1 redirects the standard error stream (file descriptor #2) to the standard output stream (file description #1), so ultimately the output of both streams is saved in the log file $OUT.

The wait is two weeks.

You can work around the issue by adding an additional domain name to the certificate, or splitting off the mail subdomain to its own certificate. Any difference in the set of domain names will ensure it is not counted as a duplicate certificate.

Keep in mind there is also a 20 certificates per registered domain limit that isn't so easy to work around. If you need to test things, pass --dry-run to certbot, which instructs it to use the staging servers Let's Encrypt runs that issue untrusted test certificates.

2 Likes
18

Thank you T.C. Very clear and helpful responses to my questions.

I’ll revert to --apache and restore my cert and chains from backup, and hope my script works when the time comes.

19

Make sure to test your script before the time comes, also as @Patches mentioned always make use of the --dry-run flag

20

Here is my experience with DANE and Let’s Encrypt
"DANE technology — security thru DNS" - https://kostikov.co/tehnologiya-dane-bezopasnostj-cherez-dns
Article in Russian but I believe config language is the same everywhere :wink: