Candidate second email

After some more thinking and experimenting I think the way that makes sense to organize this information is that we’ll report one domain and one IP address per account. This limits the size and complexity, is consistent and easy to explain, and since most of the time one account is operated by one piece of software, that should be sufficient to inform people about all the places they need to update.

I’ve edited the sample post to reflect this. What do you think?

1 Like

I think it may be worth providing a link about the staging environment here. https://letsencrypt.org/docs/staging-environment/ works, but I kind of want to remove the suggestion to use --staging with Certbot in favor of --dry-run if we’re going to be including a link to this page in the email. (If you naively use --staging, you can end up with staging certificates installed in Apache/Nginx!)

Thanks @jsha. I like your post a lot better than mine. I’m tempted to delete all/most of my post before the email goes out in favor of people reading jsha’s. What do people think?

That’s a great idea. Can you send a PR at https://github.com/letsencrypt/website?

Thanks! :blush: I would be totally fine if you wanted to hoist my post into yours with an edit, then we could just delete mine. Feel free to do that directly.

Great. I’m going to give people on the Certbot team who worked on that post with me a brief chance to object, but if they don’t, I’ll do that.

I also wanted to add that I think this is a good idea:

I’m not sure how many people will have reusable staging authzs and how significant the increased load would be on Let’s Encrypt to disable authz reuse in staging to some degree, but I think authz reuse really negatively affects people’s ability to test here.

With Certbot, if you test against staging but have a valid authz, all that was tested is it is possible for Certbot to set up the challenge (e.g. standalone could bind to port 80, we could modify your Apache/Nginx config, etc.). You could have bad firewall rules, your ISP is blocking port 80, etc. and you would have no idea.

I would expect other clients to have similar problems here.

1 Like

Yep, we’ve disabled authz reuse in staging as of today.

3 Likes

I’m not seeing it yet. I did a --dry-run on a certificate, and then I issued two staging certificates for a new hostname, and it still did normal authz reuse.

E.g.:

https://acme-staging-v02.api.letsencrypt.org/acme/order/5707995/20873270
https://acme-staging-v02.api.letsencrypt.org/acme/order/5707995/20873389
https://acme-staging-v02.api.letsencrypt.org/acme/authz/7YmNRiwOX0ztD0yWNnipuavUpfd-nHFZ3TV_1jW6fQk

Edit: The --dry-run was:

https://acme-staging-v02.api.letsencrypt.org/acme/order/5707995/20870483
https://acme-staging-v02.api.letsencrypt.org/acme/authz/GlvZwP6rNTtBlOW82jZrhRswyZk8fKRNgVgTlWDN4do
https://acme-staging-v02.api.letsencrypt.org/acme/authz/xNntDCuXKLFWhF_0WABE2FjO2Uj1AUDJv-DJFlPKAJw

Edit: The --dry-run‘s authzs’ original order was:

https://acme-staging-v02.api.letsencrypt.org/acme/order/5707995/19095710

1 Like

Hey @mnordhoff,

You’re right. I saw your comment and double checked myself and saw the same results. I checked the V1 API as well and it had the intended effect there. I filed a bug to fix the ACME v2 valid authz reuse: https://github.com/letsencrypt/boulder/issues/4026

Hopefully the majority of people using TLS-SNI-01 with staging are using ACME v1 in the meantime.

Thanks again for flagging this @mnordhoff!

4 Likes

Quick update here: We deployed a fix outside of the usual cycle to make sure this was addressed. The staging environment is now properly not reusing valid authorizations for both the V1 and the V2 API.

Thanks!

3 Likes

Verified fixed. :sparkles:

2 Likes

I’ve updated the draft post, mainly to make this sentence simpler and clearer:

TLS-SNI-01 validation in the production environment is reaching end-of-life. It will stop working temporarily on February 13th, 2019, and permanently on March 13th 2019. Any certificates issued before then will continue to work for 90 days after their issuance date.

(It used to try and explain how to check your certificate’s NotAfter date from browser developer tools).

I also added a link for staging.

Heads up: This second round of emails will start going out today. We’ll be spacing them out slowly just in case someone reports a significant last-minute problem.

4 Likes

We hit a snag with the email sending yesterday but we’ve restarted it today.

Well, I know what’s going on, but I have a lot of domains, and the last email didn’t give me even a hint which one(s) had a problem. So, I don’t care whether I get IP or domain name, just make sure I get at least one.

To give a “recipient’s perspective” on the second email:

It does now say which domain name and which IP address were affected (which is good), but crucially doesn’t say whether the same challenge will be attempted next time. In my case I got an email for one domain that I don’t believe will fail (I’ve just tried dry-run and it says it could renew using http-01) and didn’t (yet) for one that I believe will fail next time (it did tls_sni_01 last time and dry-run fails for all challenge types).

Perhaps if there’s a “third email” it should explicitly suggest dry-run as a way of checking? Of course, assuming people are checking for renewals regularly they should get 30 days notice of a problem when a renewal fails.

The harder problem to solve is explaining to people what they need to do next once they’ve found (e.g. by a renewal dry run) that they do have a problem. The email does link to this help site; there’s lots of information here but maybe encourage people to post “solved problems” here too (e.g. “I was previously using tls_sni on XYZ server and I fixed it by doing A, B and C”).

1 Like

Isn’t the “fix” always to just request a brand new certificate? At least that’s what I did, and it worked for the one I knew was using the deprecated method. Got the email today and found that there was indeed a second.

Still, your point is sound. If that is the simple solution, mentioning it in the email would help

You may be right, but you are assuming (not necessarily correct) that all the sites have static IP. ^here are also those sites that use dynamic DNS and their IP changes. By the time you send the IP the respective domains already may have another IP because thy got rebooted or for any other cause. The only bullet proof method is to target domains and not IP. After all the certificate is per domain not per IP.

Bogdan

Hi all! I think this thread is likely to become a catch-all support thread, so I’m going to close it in favor of separate threads about specific problems. I appreciate the feedback on possible improvements if we need another round! Note that issuing another certificate is not necessarily enough; you need to also make sure your client is fully up-to-date, as described in the Certbot post linked from the email.

1 Like

As a heads-up, we’re going to start sending out weekly batches of this notification email, only to the people who validated using TLS-SNI-01 during the last week. Each batch will be much smaller (~75k accounts, as of this week), and the data will be fresher since it only covers a week. If there are any changes you’d like to recommend, let me know – for instance by starting a new thread and tagging me.

4 Likes