New/old topic: certificate delays at Squarespace

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: MIT.EDU

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is: Squarespace

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

======

I am a network admin at the above domain’s institution. A number of our users have
departmental and group websites hosted at Squarespace. Starting about a year or so ago, we’ve been seeing chronic issues with delays in securing those sites with SSL certificates when the site owner wants to use an MIT.EDU name for the site.

Squarespace help staff flail around for a long time, asking for microscopic changes in the DNS info we’ve created (compliant with their requirements), and occasionally they’ll indicate, when pressed, a claim that Let’s Encrypt certificate requests are failing.

On rare occasions, the certificate is issued within a few hours, on other occasions, the site limps along as “insecure” for months.

When I’ve done the search on this site for certificate issuances for MIT.EDU, I can’t see anything that would indicated a frequency of issuance which would trip the limits. Beyond that I have no visibility into the process. Squarespace won’t even talk to us about specific sites since we’re not the site owners, the individual lab or group owns the site.

Is anyone here able to tell me whether there’s in fact a rate limit we’re hitting for MIT.EDU?

Ron

2

Hi Ron

there are a lof of certificates created:

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:true;domain:mit.edu&lu=cert_search

1875 from Letsencrypt.

Perhaps a higher rate limit may be helpful:

If you are a large hosting provider or organization working on a Let’s Encrypt integration, we have a rate limiting form that can be used to request a higher rate limit. It takes a few weeks to process requests, so this form is not suitable if you just need to reset a rate limit faster than it resets on its own.

1 Like
3

Hello Juergen, thanks so much for the reply. The volume of activity you saw
doesn’t show up in searches against crt.sh . I’ll check out the rate limiting form
as well.

Ron

4

I’ve looked through the transparency logs, and it doesn’t make sense to me.

Is it the case that Let’s Encrypt limits certificates based on the total number issued to a domain by all issuers, or just based on the number which Let’s Encrypt issues? Also, when I zoomed in on the search for issuances by Let’s Encrypt, one listing for 40 included only one certificate for an MIT.EDU site.

I’m confused.

Ron

5

Only certificates issued by Let's Encrypt are factored into the rate limiting decisions. Can you share the search query you used for crt.sh?

6

I visit crt.sh and enter “mit.edu” in the search box, and look at the results.

Ron

7

You might want to search with

https://crt.sh/?Identity=%.mit.edu&iCAID=16418

which includes all, and only, current Let’s Encrypt certificates.

1 Like
8

What does it mean when a particular DNS name appears multiple times in a row? That seems to happen quite a bit according to that search result.

Ron

9

Hi Ron,

I think we are close to figuring out what is happening with the domain mit.edu and the certificates per registered domain rate limit. I have reached out to our contact at Squarespace and will give you an update as soon as I hear back!

Thanks so much for posting this.

Best,
JP

3 Likes
10

@hoffmann, FYI, a search for "%.mit.edu" excludes any certificates for just mit.edu with no subdomains. Luckily, there aren't any such certificates -- from Let's Encrypt -- so it's not a problem.

A search for %mit.edu would include both, but also match other irrelevant domains, like, for example, www.summit.edu (if they have any logged certificates).

Let's Debug and Google's CT website both handle this differently.

https://tools.letsdebug.net/cert-search?m=domain&q=mit.edu&d=168

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:true;domain:mit.edu&lu=cert_search

It depends. You have to click on them to find out. There might actually be multiple certificates.

But, also, to implement Certificate Transparency, Let's Encrypt -- and most CAs -- sort of issue two certificates: First they issue the "precertificate", which has a special poison extension that means clients will reject it, then they log it to CT logs, and then issue the real certificate, which contains a special extension containing proof the precertificate was logged. The real certificate usually gets logged too (but it doesn't have to be).

Yes, that sounds weird.

(Let's Encrypt doesn't double count for rate limiting purposes.)

crt.sh lists both, so there are usually duplicates.

To take this example:

You can see that each certificate has the same serial number and dates and such. One says "Precertificate" at the top and one says "Leaf certificate". Near the bottom of the pages, they each link to each other. So they're a certificate and its corresponding precertificate.

The Let's Debug link earlier filters out the certificate/precertificate duplication and highlights real duplicates.

Edit: Let's Debug displays rate limiting estimates based on Let's Encrypt's default rate limits. mit.edu probably has higher rate limits, but Let's Debug doesn't know that.

Edit: My previous edit said "Let's Encrypt" instead of "Let's Debug", sorry.

2 Likes
11

Hello JP,

were you able to learn anything regarding the issues we’re seeing with
?.MIT.EDU names for Squarespace-hosted sites? The first tier help desk
there are driving our users (and us) nuts with fiddly changes to the DNS
records, none of which appear to be fixing the SSL cert issue :frowning:

12

Hi @hoffmann,

I can offer a quick update on this independent of Squarespace. After investigating more on our side I found that the mit.edu domain rate limit override surfaced a bug in our CA software that caused Squarespace's requests to be rate limited more often than should have happened. I've already fixed the bug for next week's release. I expect that this problem will go away once the fix is in production (ETA: Feb 28th) and with luck @jple might be able to get it fixed even sooner with a different change.

Thanks for your patience while we get this fixed!

3 Likes
13

This shows how valuable it is for people who are having unexpected problems to get in touch! Thanks, @hoffmann!

3 Likes
14

Hello, I’m part of the group that’s trying to get a website for my group via Squarespace. Thank you all for the help with resolving this issue!

Are there were any updates on a fix from Squarespace before the 28th @jple? We’ve been having issues for quite a while and it’s becoming critical for us to have our website up as soon as possible.

1 Like
15

Hi @stellay,

Our SRE team confirmed they would be able to make a policy change later today that should resolve this problem ahead of the production deploy on the 28th. I will update this thread when I know the change has been made later today.

Thanks!

2 Likes
16

Awesome, thanks so much!

Just to clarify, is there anything that I need to do after the bug gets fixed in order to make our website secure? It’s DNS settings are configured in Squarespace, but the SSL certificate shows up as an error. If so, what should I do?

17

If the only problems were related to the bug that caused Squarespace to hit rate limits for mit.edu domains there won't be anything you need to change on your side.

It's possible the rate limit error was hiding other problems but if that's the case it should be possible for you and Squarespace support to resolve them with the usual support process.

Hope that helps!

18

Hi @stellay! I heard back from the Squarespace team today and am working to resolve it there as well, so either way by the 28th you should be up and running!

Thanks so much for your patience and for alerting us to this - we wouldn’t have known about either course of action if this thread hadn’t been raised. Much appreciated.

Best,
JP

1 Like
19

@stellay @hoffman The rate limiting problem between Squarespace and mit.edu domains should now be resolved. Thanks!

3 Likes
20

Mentioning @hoffmann with two Ns. :slight_smile:

(Nice work, @cpu!)

2 Likes