Let's Encrypt servers making daily HTTP challenge requests even after the order has been finalized - but only when a CNAME is involved

Help
1

Hi. I don't need help as such; I'm just trying to find a channel to reach out to report a suspected issue.

I have a process for requesting certificates from Let's Encrypt. If the domain name I request the certificate for resolves directly to an IP address through an A record, then everything behaves as expected. If the domain name I request the certificate for resolves via a CNAME, then I get strange behaviour.

Whether or not there is a CNAME involved is the only difference I can find on my side between expected behaviour and unexpected behaviour.

In both cases, the certificate is issued successfully. The strange behaviour is that the Let's Encrypt servers will start performing daily HTTP requests for unrecognised tokens against the domain when a CNAME is involved.

I know that the daily unexpected request are coming from Let's Encrypt's servers, because the source IP addresses overlap with expected requests from legitimate certificate renewals.

I know that the HTTP requests aren't in response to POST-as-GET challenge requests coming from my environment: the process that requests certificates is currently being run manually, and outbound access is restricted in that environment such that all HTTPS requests go through a forward proxy - and the forward proxy's logs only show HTTPS connections being opened when I manually trigger the process.

I know (or at least strongly suspect) that the strange behaviour is triggered due to a legitimate request for a certificate from my environment, because these daily requests only start after a legitimate request from that environment - the web server logs are silent before then.

The environment is based on Microsoft servers and technology. (Don't judge me.) Here's an extract from the IIS web server log showing requests that I'm not expecting:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2023-12-01 06:31:11 10.0.7.4 GET /.well-known/acme-challenge/DfTGoUNzcptng4hZKsHDnohkZEghVDU8UIQ0_UyCb0M - 80 - 18.222.199.136 Mozilla/5.0+(compatible;+Let's+Encrypt+validation+server;++https://www.letsencrypt.org) - 404 0 2 3330
2023-12-01 06:31:11 10.0.7.4 GET /.well-known/acme-challenge/DfTGoUNzcptng4hZKsHDnohkZEghVDU8UIQ0_UyCb0M - 80 - 23.178.112.201 Mozilla/5.0+(compatible;+Let's+Encrypt+validation+server;++https://www.letsencrypt.org) - 404 0 2 2957
2023-12-01 06:31:11 10.0.7.4 GET /.well-known/acme-challenge/DfTGoUNzcptng4hZKsHDnohkZEghVDU8UIQ0_UyCb0M - 80 - 52.36.2.37 Mozilla/5.0+(compatible;+Let's+Encrypt+validation+server;++https://www.letsencrypt.org) - 404 0 64 3132

I've been receiving similar unexpected requests every morning since the certificate was originally requested for that environment in July. Here I'm renewing that certificate this afternoon:

#Fields: s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
10.0.7.4 GET /.well-known/acme-challenge/zQF0FYZvQ1rCJYEy-tNYdBrDH45DyjqJxoJis4eRW3k - 80 - 18.222.199.136 Mozilla/5.0+(compatible;+Let's+Encrypt+validation+server;++https://www.letsencrypt.org) - 200 0 0 4328

As you can see, the client IP address is the same for this expected request, so I'm confident that the earlier unexpected requests are also coming from Let's Encrypt. Similarly, yesterday morning I received requests for another token I've never seen:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2023-11-30 06:59:13 10.0.7.4 GET /.well-known/acme-challenge/5GgU5RtVTKLlhTa5vQC4dJWUfYNkpavUncBXk6_05KM - 80 - 18.117.107.2 Mozilla/5.0+(compatible;+Let's+Encrypt+validation+server;++https://www.letsencrypt.org) - 404 0 2 3493
2023-11-30 06:59:13 10.0.7.4 GET /.well-known/acme-challenge/5GgU5RtVTKLlhTa5vQC4dJWUfYNkpavUncBXk6_05KM - 80 - 23.178.112.202 Mozilla/5.0+(compatible;+Let's+Encrypt+validation+server;++https://www.letsencrypt.org) - 404 0 2 3155
2023-11-30 06:59:13 10.0.7.4 GET /.well-known/acme-challenge/5GgU5RtVTKLlhTa5vQC4dJWUfYNkpavUncBXk6_05KM - 80 - 54.244.15.119 Mozilla/5.0+(compatible;+Let's+Encrypt+validation+server;++https://www.letsencrypt.org) - 404 0 64 3684

This is happening for every site I host - but only if there's a CNAME in place. Sites that don't have a CNAME get the just the expected HTTP challenge requests; nothing comes in daily for those.

Any thoughts on why Let's Encrypt appears to be trying to satisfy challenges every day, even after the certificate has been finalized and downloaded - but only when a CNAME is in place?

2 Likes
2

My best guess is that there's someone out there trying to make a request to get a certificate for your domain name, which isn't being sent from your network through your outbound proxy. Maybe someone who used to own the name, or someone who doesn't know how to spell their own domain name, or just someone generally being confused. A lot of clients will keep trying to validate names forever, even long after the name is no longer hosted by the server that the client is running on.

Can you give a sense of the scale of what you're seeing? Are you talking about a couple names, or dozens, or thousands? If the difference is whether a CNAME is in place, might that be correlated to something else, like a prior system that used to handle those names?

5 Likes
3

Hi. Thanks for sharing your thoughts. It doesn't feel right, though - it feels like too much of a coincidence.

There's around 100 sites; about 30 of them are accessed through a CNAME. Only those 30 show daily challenge requests from Let's Encrypt.

The CNAME is always set up specifically to access our service, so nothing could have known about it in advance, and the root domain is always a well-established domain owned by our customer.

The root domains are varied - it's not our domain the requests are coming in to; it's always the CNAME. And it starts the day after I request a certificate. It goes like this:

  1. Alice becomes a customer of ours.
  2. I set up alice.bob.com for customer Alice on my bob.com domain. (Hypothetical domain name.)
  3. I acquire and install a Let's Encrypt certificate for alice.bob.com. (It's a fresh cert, not a bob.com wildcard.)
  4. Sometime later, the customer gets around to adding a CNAME record to their DNS. e.g. bob.alice.com -> alice.bob.com.
  5. I add a CAA record for alice.bob.com, and use that and the CNAME to acquire and install a separate certificate for bob.alice.com.
  6. That instance of our service is now accessible through both the old alice.bob.com and the new bob.alice.com. This is the only environment that's ever been told about bob.alice.com.
  7. The day after I acquire bob.alice.com, I start getting daily, morning challenge requests from Let's Encrypt to bob.alice.com:80 - not before.

If I'm the only person seeing this, then clearly it must be down to something I'm doing. I can't for the life of me think what it is, though.

1 Like
5

Please give more detail about that CNAME.

Please give more detail about that CAA record.

That does sound like there is some automated system [in your control] that is trying to get a cert for all your [new] names.

Can you test creating a new [unused] name?
Try only part of the steps each day - until you see which step creates the new requests [on the following day].

5 Likes
6

Also maybe more details on your ACME client and how you're configuring it might be helpful.

5 Likes
7

What if you wait a couple days after the CNAME is added and doing the above.

Do you still see HTTP challenges arrive after the CNAME is added? Or only after you acquired the cert based on it?

5 Likes
8

Thanks for all the comments. In order:

The CNAME is just a standard CNAME: bob CNAME record added to alice.com DNS that points to alice.bob.com, as per the hypothetical example.

The CAA record is:

0 issue letsencrypt.org

I don't think it can be an automated system in my control that's trying to acquire the certificates. We use Azure, but everything's VM-based. We're also very small, and there's only a handful of VMs. There are only two things that know about each bob.alice.com domain when it's set up: our customer's DNS and IIS running in a single Windows VM; I think it's unlikely that 30 customers using a variety of technologies would all randomly do something we hadn't asked for - we even tell them "just add the CNAME - do nothing else"; and I don't believe it can be our web server VM, because all outbound traffic is locked down, and the only way it can reach out to Let's Encrypt to ask for a certificate is through the forward proxy, and logs confirm it's only doing that at expected times. Our DNS is hosted in Azure, but it doesn't know about the CNAME - it only knows about bob.com - so I don't see how it could be possibly be some wayward Azure service.

Our ACME client is a bit of C# code. There are no 3rd party libraries involved; it just performs HTTP requests against the https://acme-v02.api.letsencrypt.org/directory API endpoint. It only runs when I tell it to run (as confirmed by the forward proxy logs). It does the same thing for every domain name, regardless of whether that's a CNAME or not: fetch the account; place the order; fetch the authorizations; save the dns-01 challenge token; tell Let's Encrypt that the challenge is ready; wait for a ready order status; finalize the order; download the certificate.

The steps 1 - 7 (as per my previous post) don't all happen in the same day; there's usually a day or two between each step. I sometimes add the CAA when I ask for the CNAME, and there can be a few days between the customer adding the CNAME and me doing anything with it. All the analysis I've done points to requesting the CNAME certificate as the trigger for the daily challenge requests.

I will set up a new instance next week, as suggested, using a test domain as the customer. This should confirm that I still get the same behaviour for a new site and that the HTTP requests only start coming in after I acquire the CNAME certificate. That will take a good few days, if I wait a day or two to confirm that the requests don't start beforehand.

In the meantime, if you have a similar CNAME set-up, perhaps you could check your web server logs to confirm that you're not seeing daily challenge requests too?

I'm happy to accept it's something I'm doing, but posting here isn't my first course of action - I've been looking into this for a few days - and it would be good to hear someone say, "I'm acquiring separate certs for the CNAME and the domain the CNAME points to, and it's not happening to me." :slight_smile:

9

You don't need to add CAA records for every subdomain.
OR do you?
[I can't imagine why you would]

Note: the base domain CAA record is used when no exact match is found [for the subdomain]

3 Likes
10

I found a single @ CAA record insufficient when a CNAME is involved. (The single @ CAA record is fine for acquiring certificates for any subdomain, but not for some other domain that is pointing at the subdomain through a CNAME - I believe that the CAA record of the root domain that's outside of my control would take effect otherwise.) However, I don't want to take this off-topic; it's the unexpected HTTP challenge requests that are driving me slowly mad, and the certificates are being successfully issued...

EDIT: So if I'm trying to acquire a certificate for bob.alice.com, which is a CNAME that points to alice.bob.com, then if Let's Encrypt fails to find a CAA record for bob.alice.com (which redirects to look for a CAA record at alice.bob.com), then it will move up alice.com, not bob.com. If alice.com is my customer, then they might have their own CAA record for another provider, which would cause Let's Encrypt to refuse to issue the bob.alice.com certificate. Therefore my @ CAA record for bob.com is insufficient to ensure Let's Encrypt will issue for bob.alice.com. Hence the need for CAA records for each subdomain in this scenario.

This is just my guess at what's happening based on personal experience; there may well be a better way.

Anyway - is anyone else seeing daily challenge requests when a CNAME is involved? :slight_smile:

11

The Let's Encrypt ACME Servers do not issue HTTP challenges without being requested by an ACME Client. They just cannot. And would be a complete waste of resources if attempted without a request as the challenge would always fail.

But, it wouldn't have to be your ACME Client making the request. I could use an ACME Client and request a cert for your domain (CNAME'd or otherwise). Of course, the challenge would fail because the expected token would not be returned by your server. As you see as the 404 error (Not Found) in your logs. It would be silly of me to do this but doesn't mean someone isn't being silly.

So, some other actor could be initiating these requests.

It could even be something like a CDN, Load Balancer, or URL redirect or inspection service trying to obtain a cert on your behalf. Why it would affect just the CNAME domain is a mystery. I'm just describing ACME requesting systems that people sometimes don't well understand.

2023-11-30 06:59:13 10.0.7.4 GET /.well-known/acme-challenge/5GgU5RtVTKLlhTa5vQC4dJWUfYNkpavUncBXk6_05KM - 80 - 54.244.15.119 Mozilla/5.0+(compatible;+Let's+Encrypt+validation+server;++https://www.letsencrypt.org) - 404 0 64 3684

3 Likes
12

I've just had a flash of inspiration. We use an email sending service. So when I said there were only two things that new about the CNAME domain, I lied - there are three. It has to be something to do with that.

Thanks to everyone who read and contributed. I strongly suspect that email sending service will be the culprit. Or at least now I have some other path to investigate. I will confirm next week either way.

3 Likes
13

I couldn't not know.

It's the email sending service. When a customer adds a CNAME for us, we email from that domain through a 3rd party. That 3rd party also provide web hosting; we don't use that hosting, but adding the domain as an email sending domain is enough for it to think we might want to set up a website. And they use Let's Encrypt to provide certificates. There's a configuration option to tell this service that a website will not be required, which will stop it trying to acquire certificates on our behalf. Mystery solved.

Thanks to everyone for helping me work through the issue. I only feel slightly silly.

4 Likes
14

Well, that service should feel more silly, for trying to get a certificate for domain names that they aren't hosting. :slight_smile:

Glad that we could help you figure it out!

5 Likes
15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.