Candidate second email

[jsha]

Hi folks,

I’ve been working on the wording for our second round of TLS-SNI deprecation emails, plus some of the underlying code. My current thinking is that it’s better / easier to include the affected IP addresses in the email rather than the domains. There are a lot of accounts that have a large number of domains, but typically they only use a single IP address.

The plan for multiple accounts that have the same email address is to consolidate into a single email, and include up to ten IP addresses.

Feedback on both the message content and the overall plan are welcome! Thanks to everyone for the hard work you’ve been doing answering questions.

---

Hello,

Action may be required to prevent your Let’s Encrypt certificate renewals from breaking.

If you already received a similar e-mail, this one contains updated information.

Your Let’s Encrypt client used ACME TLS-SNI-01 domain validation to issue a certificate in the past 60 days. Below is a list of names and IP addresses validated (max of one per account):

example.com (10.100.10.100) on 2018-11-24
example.net (10.11.10.11) on 2018-11-26

TLS-SNI-01 validation is reaching end-of-life. It will stop working temporarily on February 13th, 2019, and permanently on March 13th 2019. Any certificates issued before then will continue to work for 90 days after their issuance date.

You need to update your ACME client to use an alternative validation method (HTTP-01, DNS-01 or TLS-ALPN-01) before this date or your certificate renewals will break and existing certificates will start to expire.

Our staging environment already has TLS-SNI-01 disabled, so if you’d like to test whether your system will work after February 13, you can run against staging: https://letsencrypt.org/docs/staging-environment/.

If you’re a Certbot user, you can find more information here: How to stop using TLS-SNI-01 with Certbot

Our forum has many threads on this topic. Please search to see if your question has been answered, then open a new thread if it has not: https://community.letsencrypt.org/

For more information about the TLS-SNI-01 end-of-life please see our API announcement:
https://community.letsencrypt.org/t/february-13-2019-end-of-life-for-all-tls-sni-01-validation-support/74209

Thank you,
Let’s Encrypt Staff

[mnordhoff]

@jsha

My current thinking is that it’s better / easier to include the affected IP addresses in the email rather than the domains. There are a lot of accounts that have a large number of domains, but typically they only use a single IP address.

The plan for multiple accounts that have the same email address is to consolidate into a single email, and include up to ten IP addresses.

Would it work to include up to 10 sample domains? If people have multiple ACME clients, or multiple servers behind NAT, it might help them pinpoint it.

[_az]

@jsha

My current thinking is that it’s better / easier to include the affected IP addresses in the email rather than the domains

I'm not sure if I am interpreting this right, but wouldn't this potentially be unhelpful for the many domains behind CDNs and AWS ELB/NAT-like endpoints?

I agree with @mnordhoff that it would be very helpful to have upto 10 certificate names as well. This would give the recipient some hope of identifying which "logical group" of domains are affected.

[jsha]

@mnordhoff

Would it work to include up to 10 sample domains?

It's possible, but I'm trying to balance out implementation time and complexity. If I include domains, I have to choose some method to pick 10 of them, and most methods are likely to leave out some useful information. If I go with IP addresses, they are in theory more information-dense, and any given strategy to pick 10 should be fine. Doing both definitely adds a certain amount of complexity and implementation time.

@_az

I’m not sure if I am interpreting this right, but wouldn’t this potentially be unhelpful for the many domains behind CDNs and AWS ELB/NAT-like endpoints?

I think domains behind CDNs in general can't use TLS-SNI, unless the validation is performed by the CDN itself.

[_az]

Oh, good point. Doesn’t seem so bad then.

Edit: but in general I do think that upto 10 domains + total count would is definitely still be much more helpful.

[mnordhoff]

I'm having a hard time thinking of what people who don't already know what's going on need to be told. But...

@jsha

Your Let’s Encrypt client used ACME TLS-SNI-01 domain validation to issue a certificate in the past 60 days on these IP addresses:

[up to ten IP addresses per user]

I think it should clarify when some IP addresses are being left out. People might think they have only ten servers using TLS-SNI and not investigate their others.

@jsha

If I include domains, I have to choose some method to pick 10 of them, and most methods are likely to leave out some useful information.

Whichever 10 are fastest to extract from the database?

[_az]

One nit about ambiguity: is it the IP address that the VA connected to that is reported, or the IP address of the ACME client that requested the certificate?

[mnordhoff]

My thinking was that a list of domain names might be most useful to people who don’t have a lot of domains or servers. E.g. “Is it my Synology or my home server?” It’s theoretically easy to check only a few servers, but sometimes they don’t have good logs making it actually harder to figure out what’s going on.

Someone with hundreds of web servers and thousands of domains should usually know what their systems are doing anyway.

[jsha]

@_az

is it the IP address that the VA connected to that is reported, or the IP address of the ACME client that requested the certificate?

Current plan is that it would be the IP address the VA connected to, since it's easiest to collect that information alongside the list of validations that used TLS-SNI-01 - it's in the validation logs.

To be really precise, for each given account, I'll have the most recent IP address validated using TLS-SNI-01 (a limitation of our log querying software). Then I'll be merging accounts with the same email address, so if you have clients on three different IP addresses (each with a different account ID), you'll get a single email with three IP addresses. I think it's hard to convey that nuance really precisely in the email, but definitely interested in better ways to talk about it.

[mnordhoff]

@jsha

Please make sure to search for similar problems before starting a new thread:

I'd like to hear what other people think, but this sentence concerns me. In some ways it's easier to manage a flood of repetitive threads than a smaller flood of people necroing other, semi-related threads.

Edit: Confession: Discourse's tool to split threads is fantastic, but coming up with titles is hard!

[jsha]

@mnordhoff

In some ways it’s easier to manage a flood of repetitive threads than a smaller flood of people necroing other, semi-related threads.

That's a good point, I hadn't thought of that! What do other folks think?

Also, what documentation can we add that will make it easier to point people at help?

[_az]

@jsha

Your Let’s Encrypt client used ACME TLS-SNI-01 domain validation to issue a certificate in the past 60 days on these IP addresses:

Wrt the IP address thing, I'd perhaps update this copy to:

In the past 60 days, your Let’s Encrypt client used ACME TLS-SNI-01 domain validation to issue certificates for domains hosted on these IP addresses:

@jsha

Also, what documentation can we add that will make it easier to point people at help?

I think the post at How to stop using TLS-SNI-01 with Certbot could be improved with an example of how to actually perform a dry-run and identify that TLS-SNI is indeed not being used during that run.

There was also one thread where dry-run gave a false positive result on staging due to cached authz (port 80 was clearly inaccessible but there was a previously valid http-01 authz). Maybe it's asking too much, but killing the authz caching on staging for a few days could help? I'd be pretty mad if Certbot lied to me like that and my cert expired :|.

I think perhaps that the Help template could temporarily be modified to include an explicit prompt for certbot --version, since it seems necessary to ask that in every Help thread.

In a similar vein, How to stop using TLS-SNI-01 with Certbot could perhaps promote certbot --version to a proper styled code block, with example output indicating the correct version, so that it doesn't get buried in the prose.

Re: necro, I think it's possible to greatly reduce that problem by making @bmw's post a bit more "step by step" in nature, to match the audience who are looking for very prescriptive advice/instructions.

[jsha]

@_az

the post at How to stop using TLS-SNI-01 with Certbot could be improved with an example of how to actually perform a dry-run and

I'll work on getting this added.

@_az

killing the authz caching on staging for a few days could help?

This is a cool idea! I think it's doable; I'll check with the team.

@_az

Help template could temporarily be modified to include an explicit prompt for certbot --version

Done! I think I phrased this in a way that covers multiple clients, so it's a pretty good candidate for keeping long term.

@_az

promote certbot --version to a proper styled code block

I like this. For now I put --version in a code block to make it stand out, but I'll rework it some more in a bit.

@_az

I think it’s possible to greatly reduce that problem by making @bmw’s post a bit more “step by step” in nature

[jsha]

After some more thinking and experimenting I think the way that makes sense to organize this information is that we’ll report one domain and one IP address per account. This limits the size and complexity, is consistent and easy to explain, and since most of the time one account is operated by one piece of software, that should be sufficient to inform people about all the places they need to update.

I’ve edited the sample post to reflect this. What do you think?

[bmw]

@jsha

Our staging environment already has TLS-SNI-01 disabled, so if you’d like to test whether your system will work fine after the deprecation date, you can run against staging.

I think it may be worth providing a link about the staging environment here. Staging Environment - Let's Encrypt works, but I kind of want to remove the suggestion to use --staging with Certbot in favor of --dry-run if we're going to be including a link to this page in the email. (If you naively use --staging, you can end up with staging certificates installed in Apache/Nginx!)

@jsha

I think it’s possible to greatly reduce that problem by making @bmw’s post a bit more “step by step” in nature

Thanks @jsha. I like your post a lot better than mine. I'm tempted to delete all/most of my post before the email goes out in favor of people reading jsha's. What do people think?

[jsha]

@bmw

I think it may be worth providing a link about the staging environment here. Staging Environment - Let's Encrypt works, but I kind of want to remove the suggestion to use --staging with Certbot in favor of --dry-run if we’re going to be including a link to this page in the email.

That's a great idea. Can you send a PR at GitHub - letsencrypt/website: Let's Encrypt Website and Documentation?

@bmw

Thanks @jsha. I like your post a lot better than mine. I’m tempted to delete all/most of my post before the email goes out in favor of people reading jsha’s. What do people think?

Thanks! I would be totally fine if you wanted to hoist my post into yours with an edit, then we could just delete mine. Feel free to do that directly.

[bmw]

@jsha

That’s a great idea. Can you send a PR at GitHub - letsencrypt/website: Let's Encrypt Website and Documentation?

bmw:

Don't tell people to use --staging.

@jsha

Thanks! I would be totally fine if you wanted to hoist my post into yours with an edit, then we could just delete mine. Feel free to do that directly.

Great. I'm going to give people on the Certbot team who worked on that post with me a brief chance to object, but if they don't, I'll do that.

[bmw]

I also wanted to add that I think this is a good idea:

@jsha

killing the authz caching on staging for a few days could help?

This is a cool idea! I think it’s doable; I’ll check with the team.

I'm not sure how many people will have reusable staging authzs and how significant the increased load would be on Let's Encrypt to disable authz reuse in staging to some degree, but I think authz reuse really negatively affects people's ability to test here.

With Certbot, if you test against staging but have a valid authz, all that was tested is it is possible for Certbot to set up the challenge (e.g. standalone could bind to port 80, we could modify your Apache/Nginx config, etc.). You could have bad firewall rules, your ISP is blocking port 80, etc. and you would have no idea.

I would expect other clients to have similar problems here.

[jsha]

Yep, we’ve disabled authz reuse in staging as of today.

[mnordhoff]

@jsha

Yep, we’ve disabled authz reuse in staging as of today .

I'm not seeing it yet. I did a --dry-run on a certificate, and then I issued two staging certificates for a new hostname, and it still did normal authz reuse.

E.g.:

https://acme-staging-v02.api.letsencrypt.org/acme/order/5707995/20873270
https://acme-staging-v02.api.letsencrypt.org/acme/order/5707995/20873389
https://acme-staging-v02.api.letsencrypt.org/acme/authz/7YmNRiwOX0ztD0yWNnipuavUpfd-nHFZ3TV_1jW6fQk

Edit: The --dry-run was:

https://acme-staging-v02.api.letsencrypt.org/acme/order/5707995/20870483
https://acme-staging-v02.api.letsencrypt.org/acme/authz/GlvZwP6rNTtBlOW82jZrhRswyZk8fKRNgVgTlWDN4do
https://acme-staging-v02.api.letsencrypt.org/acme/authz/xNntDCuXKLFWhF_0WABE2FjO2Uj1AUDJv-DJFlPKAJw

Edit: The --dry-run's authzs' original order was:

https://acme-staging-v02.api.letsencrypt.org/acme/order/5707995/19095710