My server is constantly getting acme-challenge requests

I spent some time today adding subdomains and fixing up my Nginx config. I was having issues and for a while was throttled by letsencrypt. But all my certs are issued now and everything's fine.

Except, I'm getting an acme-challenge request every 10 seconds:

35.233.151.123 - - [29/May/2023:07:02:09 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"
35.233.151.123 - - [29/May/2023:07:02:19 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"
35.233.151.123 - - [29/May/2023:07:02:29 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"
35.233.151.123 - - [29/May/2023:07:02:39 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"
35.233.151.123 - - [29/May/2023:07:02:49 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"
35.233.151.123 - - [29/May/2023:07:02:59 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"
35.233.151.123 - - [29/May/2023:07:03:09 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"
35.233.151.123 - - [29/May/2023:07:03:19 +0000] "GET /.well-known/acme-challenge/pYufSq0C2_6H9KmdqASotClGP8J2EZEUyagTb3CaMmY HTTP/1.1" 404 134 "-" "cert-manager-challenges/v1.8.0 (linux/amd64) cert-manager/e466a521bc5455def8c224599c6edcd37e86410c"
...

I recently reinstalled certbot, and I don't think its own timer has run yet:

# systemctl list-timers
NEXT                        LEFT          LAST                        PASSED       UNIT                           ACTIVATES                       
Mon 2023-05-29 04:36:00 PDT 4h 22min left n/a                         n/a          snap.certbot.renew.timer       snap.certbot.renew.service

Will this eventually stop?


Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: camsync.io

I ran this command: certbot renew

It produced this output: success

My web server is (include version): Nginx 1.18.0

The operating system my web server runs on is (include version): Ubuntu 22.04

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.6.0

It's not Certbot. From the log files you can see the client is cert-manager, which is an ACME client for Kubernetes written in Go (GitHub - cert-manager/cert-manager: Automatically provision and manage TLS certificates in Kubernetes).

5 Likes

There are two things that stand out to me:

  • the IP address [is not from Cloudflare]:
Name:    123.151.233.35.bc.googleusercontent.com
Address:  35.233.151.123
  • The never-ending 10 second interval requests would not come from LE production systems.
    [even the staging system would choke it after a while]

So, it sounds like someone setup a manual test [in a loop] and forgot about it.

From the clue:

we see that it is running cert-manager for Linux on an AMD system.
[maybe it rings a bell with you]

Everything eventually stops [even the sun].

5 Likes

It's possible this "someone" is not known to you. Someone could be using the wrong IP address on their own system and failing because the requests get sent to your server.

This happens in shared services often when the IP addresses get reused.

4 Likes

cert-manager is relentless in its pursuit of automatically acquiring certificates for domains referenced in ingress manifests. You can disable that activity by updating your ingress manifest(s) to remove the ClusterIssuer reference.

5 Likes

If only there was a way to know which FQDN that was trying to validate...

2 Likes

The webserver should be perfectly able to log the value of the HTTP Hosts header :wink:

3 Likes

Do you mean the Host header? That's api.camsync.io.

Am I completely off-base looking at this:

# dig -x 35.233.151.123 (source of the requests)
;; ANSWER SECTION:
123.151.233.35.in-addr.arpa. 120 IN	PTR	123.151.233.35.bc.googleusercontent.com.

letsencrypt.org:

# dig a letsencrypt.org
;; ANSWER SECTION:
letsencrypt.org.	7	IN	A	35.199.181.187
letsencrypt.org.	7	IN	A	35.247.66.204

# dig -x 35.199.181.187
;; ANSWER SECTION:
187.181.199.35.in-addr.arpa. 120 IN	PTR	187.181.199.35.bc.googleusercontent.com.

# dig -x 35.247.66.204
;; ANSWER SECTION:
204.66.247.35.in-addr.arpa. 120	IN	PTR	204.66.247.35.bc.googleusercontent.com.

I don't know if 35.233.151.123 belongs to letsencrypt, but the above suggests it's possible, no?

Well, then something using cert-manager (Kubernetes) to get a certificate for api.camsync.io.

I don't believe so. The Let's Encrypt website just is also hosted by Google as is the site from where the cert-manager requests are coming from (which is Google Cloud).

The Let's Encrypt ACME API has a different IP address and would use multiple vantage points from around the world, resulting in (currently I believe) 3 different IP addresses.

This looks more like some pre-check by cert-manager itself.

Also, Let's Encrypt obviously doesn't use cert-manager to do the checking.

6 Likes

That IP and useragent both do not belong to Let's Encrypt. That acme challenge URL is also not a Let's Encrypt one.

I think that's cert-manager "self-check", before attempting issuance (with another ACME CA). If this is your domain, you'll have to figure out where there's a kubernetes cluster running cert-manager on Google Compute Engine that's configured to use your domain name.

7 Likes

Do you use that FQDN?
If so, you could move to some other name and remove it from DNS.
If not, then I'd just remove it from DNS.

3 Likes

Thank you for confirming that. I don't use anything with Google cloud or compute. I only just created the DNS entry for that yesterday when I was trying to add it to my setup (which led to my certbot issues). So I wondered if maybe I had accidentally kicked off a process during my multiple attempts to get the cert that was just running indefinitely.

So, I'm not sure why another ACME CA would be running that request. The oldest entry in my access logs is around when I started this work yesterday (29/May/2023:02:56:49 +0000), so it's hard for me to think it's not related.

Is that obvious? Looking at the cert-manager website, Let's Encrypt is right there.

I suppose it's possible Github pages is is trying to set up the cert. It shouldn't be, since I unpublished it yesterday (I used to have camsync.io hosted there).

However, I've just checked it again, and it's published again??? What? Although I've removed the custom domain, it's showing as live.

I've unpublished it again, but maybe Github uses Google & Kubernetes for this? I'll follow up with them.

1 Like

Yes, it is obvious.

You are looking at it in the wrong direction.
LE never use cert-manager.
cert-manager can use LE [and other CAs too].

5 Likes

Okay, then hopefully it's GH Pages trying to get a cert. I've opened a ticket with them. We'll see if they give me a useful answer.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.