ACME staging server fails with IPv6 and HTTPS redirect: “Timeout during connect”

I’m not sure where to report that.

I was trying to obtain a certificate for calmexpress.com, which points to 51.210.109.224 and 2001:41d0:304:200::6461. HTTP redirects to HTTPS with an at-the-time invalid certificate.

I can access my site from a Vultr VPS in the US over IPv6, as tested with curl -vk https://calmexpress.com/

IPv4 is also accessible, as tested with curl -vk https://calmexpress.com/ --resolve calmexpress.com:443:51.210.109.224

Yet, certbot certonly ... repeatedly failed with:

51.210.109.224: Fetching https://calmexpress.com/.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc: Timeout during connect (likely firewall problem)

The error message spits out the server’s IPv4, but my understanding of the log is that IPv4 hasn’t been tried over HTTPS:

2023-10-04 14:29:22,534:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/8672621384 HTTP/1.1" 200 1839
2023-10-04 14:29:22,535:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Wed, 04 Oct 2023 14:29:22 GMT
Content-Type: application/json
Content-Length: 1839
Connection: keep-alive
Boulder-Requester: 120583614
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 63n7bjQMihjT6ClP9PNrMZ5-terc1RrausTQ3h1raChWGPsauzc
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "calmexpress.com"
  },
  "status": "invalid",
  "expires": "2023-10-11T14:28:58Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "51.210.109.224: Fetching https://calmexpress.com/.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc: Timeout during connect (likely firewall problem)",
        "status": 400
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/8672621384/GC0hTw",
      "token": "67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc",
      "validationRecord": [
        {
          "url": "http://calmexpress.com/.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc",
          "hostname": "calmexpress.com",
          "port": "80",
          "addressesResolved": [
            "51.210.109.224",
            "2001:41d0:304:200::6461"
          ],
          "addressUsed": "2001:41d0:304:200::6461"
        },
        {
          "url": "http://calmexpress.com/.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc",
          "hostname": "calmexpress.com",
          "port": "80",
          "addressesResolved": [
            "51.210.109.224",
            "2001:41d0:304:200::6461"
          ],
          "addressUsed": "51.210.109.224"
        },
        {
          "url": "https://calmexpress.com/.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc",
          "hostname": "calmexpress.com",
          "port": "443",
          "addressesResolved": [
            "51.210.109.224",
            "2001:41d0:304:200::6461"
          ],
          "addressUsed": "2001:41d0:304:200::6461"
        }
      ],
      "validated": "2023-10-04T14:28:59Z"
    }
  ]
}

OS: Debian 12.1
Certbot version: certbot 2.1.0
Web server: nginx/1.22.1

I ran this command:

certbot certonly \
--webroot \
--webroot-path "/srv/acme-challenge" \
--domain "calmexpress.com" \
--non-interactive \
--agree-tos \
--register-unsafely-without-email \
--no-eff-email \
--key-type "ecdsa" \
--preferred-chain "ISRG Root X1" \
--test-cert

Using the non-staging server (without --test-cert), it worked correctly:

2023-10-04 15:14:19,706:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/270675300756 HTTP/1.1" 200 1177
2023-10-04 15:14:19,706:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Wed, 04 Oct 2023 15:14:19 GMT
Content-Type: application/json
Content-Length: 1177
Connection: keep-alive
Boulder-Requester: 1341444596
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 3hclikJOquZJR-NbHTelw4YtYVWgd5hJE4pOX6nuNC_BqOuQz1A
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "calmexpress.com"
  },
  "status": "valid",
  "expires": "2023-11-03T15:14:18Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "valid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/270675300756/xJsCHA",
      "token": "WOIZsJfv24EJytI5SgPNoVq7pU8E47BV_JobvtWv32A",
      "validationRecord": [
        {
          "url": "http://calmexpress.com/.well-known/acme-challenge/WOIZsJfv24EJytI5SgPNoVq7pU8E47BV_JobvtWv32A",
          "hostname": "calmexpress.com",
          "port": "80",
          "addressesResolved": [
            "51.210.109.224",
            "2001:41d0:304:200::6461"
          ],
          "addressUsed": "2001:41d0:304:200::6461"
        },
        {
          "url": "https://calmexpress.com/.well-known/acme-challenge/WOIZsJfv24EJytI5SgPNoVq7pU8E47BV_JobvtWv32A",
          "hostname": "calmexpress.com",
          "port": "443",
          "addressesResolved": [
            "51.210.109.224",
            "2001:41d0:304:200::6461"
          ],
          "addressUsed": "2001:41d0:304:200::6461"
        }
      ],
      "validated": "2023-10-04T15:14:15Z"
    }
  ]
}

This is more appropriate for Help so I moved it there. You would have been shown a form to provide extra info but you have already provided all the key info so thanks for that.

That can be confusing. Let's Encrypt will try IPv6 first but for clean timeouts it will then try IPv4. The IPv4 address is then reported as the one failing. If IPv6 fails for other reasons there is no IPv4 retry and the IPv6 address appears in the error message.

I can readily reproduce your problem using the Let's Debug test site. It's own test HTTP challenges to your site work as expected (get a 404). But, the test it does with the Let's Encrypt staging system times out.

Given this works using production Let's Encrypt this is most likely a firewall. LE makes multiple HTTP requests for each cert request and often from different IP addresses. And, it even caches successful challenges so may rely just on that for subsequent requests.

Do you have anything that blocks things that look like bots, or from different geographic areas, or even specific IPv6/IPv4 addresses?

4 Likes

Nope.

Regarding potential caching issue: I had IPv6 unconfigured on the server previously, despite having set a DNS entry for it, and tried staging and non-staging unsuccessfully. The logs I provided are from after IPv6 being correctly set.

The caching is for 30 days and is per account and domain name. It doesn't matter if a successful challenge was by IPv4 or IPv6. See more here

I mention it only as it sometimes affects behavior of repeated requests

In this case staging is consistently failing. The IPv6/4 retry won't be visible in the Certbot logs.

One thing is your production certs only include a single domain name. In the (distant) past your cert included both names and this is more typical. Personally, I would focus on getting this correct first. Each domain returns the correct cert but usually you want just one nginx server block with both names if they both serve the same content. If you want help with that please show result of this

sudo certbot certificates

3 Likes

I’m good with separate certs, thanks. The 2022 entry was a different owner.

I’m wondering what’s the course of action to have this reported/resolved if staging is shown to keep failing for this configuration in the coming days though.

1 Like

hmm...
It sure looks that way.
Perhaps:

  • Geolocation blocking?
  • Fail2BAN
3 Likes

Nothing of the sort. The only firewall configuration is to discard any entry that’s not established or not port 80/443/ssh or ping/pingv6.

Only two validation servers were listed in your [now deleted] output.
LE uses three [or more] validation servers.

It is likely that some succeeded and at least one has failed.
And that failure is the cause or the problem.

3 Likes

Sorry, in my last message I was only looking at HTTPS logs.

Using Let’s Debug and watching my server logs, I notice that the HTTPS request isn’t there on IPv4 (last line of HTTP logs).

HTTP:

2600:3c03::f03c:91ff:fea2:6a56 - - [04/Oct/2023:16:54:53 +0000] "GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)"
172.104.24.29 - - [04/Oct/2023:16:54:53 +0000] "GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)"
2600:1f16:13c:c402:5ca6:e4eb:640e:f229 - - [04/Oct/2023:16:54:54 +0000] "GET /.well-known/acme-challenge/Xzgm_CdYfb8eNUorqw05uAJJ5dbK8m1mz83aDdUkIrI HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:1f14:a8b:502:68df:77c6:e96d:7c8 - - [04/Oct/2023:16:54:54 +0000] "GET /.well-known/acme-challenge/Xzgm_CdYfb8eNUorqw05uAJJ5dbK8m1mz83aDdUkIrI HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [04/Oct/2023:16:55:04 +0000] "GET /.well-known/acme-challenge/Xzgm_CdYfb8eNUorqw05uAJJ5dbK8m1mz83aDdUkIrI HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

HTTPS:

2600:3c03::f03c:91ff:fea2:6a56 - - [04/Oct/2023:16:54:53 +0000] "GET / HTTP/1.1" 200 2089 "-" "Go-http-client/1.1"
2600:3c03::f03c:91ff:fea2:6a56 - - [04/Oct/2023:16:54:53 +0000] "GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1" 404 125 "http://calmexpress.com/.well-known/acme-challenge/letsdebug-test" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)"
172.104.24.29 - - [04/Oct/2023:16:54:53 +0000] "GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1" 404 125 "http://calmexpress.com/.well-known/acme-challenge/letsdebug-test" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)"
2600:1f16:13c:c402:5ca6:e4eb:640e:f229 - - [04/Oct/2023:16:54:54 +0000] "GET /.well-known/acme-challenge/Xzgm_CdYfb8eNUorqw05uAJJ5dbK8m1mz83aDdUkIrI HTTP/1.1" 404 125 "http://calmexpress.com/.well-known/acme-challenge/Xzgm_CdYfb8eNUorqw05uAJJ5dbK8m1mz83aDdUkIrI" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:1f14:a8b:502:68df:77c6:e96d:7c8 - - [04/Oct/2023:16:54:54 +0000] "GET /.well-known/acme-challenge/Xzgm_CdYfb8eNUorqw05uAJJ5dbK8m1mz83aDdUkIrI HTTP/1.1" 404 125 "http://calmexpress.com/.well-known/acme-challenge/Xzgm_CdYfb8eNUorqw05uAJJ5dbK8m1mz83aDdUkIrI" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
1 Like

Looking at the logs from the successful non-staging challenges (both on www. and on the bare domain) I see that three IPv6 addresses were used, zero IPv4, which might be the determining factor.

But I don’t have any kind of pesky firewall, and it’s accessible as can be seen with curl -vk https://calmexpress.com/ --resolve calmexpress.com:443:51.210.109.224 > /dev/null.

The last entry in HTTP is not seen again in HTTPS.
It received a 301 redirection.

3 Likes

Also interesting is that 3rd HTTP Challenge from Let's Encrypt (the last in the HTTP snip above) came in on IPv4. The first two arrived from IPv6

And @calmexpress, just to be clear, Let's Debug sends its own requests to probe for problems. It will use IPv4 and IPv6 explicitly so that gives different info than what happens with Let's Encrypt staging requests.

Let's Debug has below IP and usually uses its own user-agent

nslookup letsdebug.net
Address: 172.104.24.29
Address: 2600:3c03::f03c:91ff:fea2:6a56
3 Likes

Same pattern with the failed staging challenge I posted in my initial message: The third request, which is IPv4, doesn’t land on HTTPS.

HTTP:

2600:1f16:13c:c402:5ca6:e4eb:640e:f229 - - [04/Oct/2023:14:28:59 +0000] "GET /.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:1f14:a8b:502:68df:77c6:e96d:7c8 - - [04/Oct/2023:14:29:00 +0000] "GET /.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 - - [04/Oct/2023:14:29:10 +0000] "GET /.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Maybe interestingly it happens 10 seconds after the IPv6 ones (14:29:10 vs 14:29:00 & 14:28:59).

HTTPS:

2600:1f16:13c:c402:5ca6:e4eb:640e:f229 - - [04/Oct/2023:14:29:00 +0000] "GET /.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc HTTP/1.1" 200 87 "http://calmexpress.com/.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:1f14:a8b:502:68df:77c6:e96d:7c8 - - [04/Oct/2023:14:29:00 +0000] "GET /.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc HTTP/1.1" 200 87 "http://calmexpress.com/.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

cURL simulation:
curl -vL4 http://calmexpress.com/.well-known/acme-challenge/67cnv5_bye63Ob9-uavx2crSeQVaXpp3b545g4aCimc

1 Like

That delay is likely the timeout of the original IPv6 attempt. The Let's Encrypt server retries that timeout with IPv4 which does reach your HTTP log.

You redirect that HTTP request to HTTPS and the process repeats. But, we don't see either IPv6 or IPv4 request arrive in your HTTPS logs. Update: I later learned the process repeats for redirects but without any IPv4 fallback if IPv6 times out (see docs).

You could improve your port 80 server block(s) to not redirect the HTTP Challenges. Something along below lines. You seem skilled enough to adapt to your system.

As long as either IPv4 or 6 arrive to HTTP it will get replied to without the extra round-trip to HTTPS. You still might get failures due to whatever underlying problem is causing this. But, it will remove half the traffic so odds will increase.

In any case, it is good practice to not redirect Let's Encrypt HTTP Challenges even though it is supported.

server {
    listen 80;
    listen [::]:80; 
    server_name calmexpress.com www.calmexpress.com;

    location /.well-known/acme-challenge/ {
        root /srv/acme-challenge;       # or as you prefer
    }
    location / {
       return 301 https://$host$request_uri;
    }
}
3 Likes

Here is more info although mostly for benefit of other volunteers.

The 2 IP addresses that work for IPv6 both originate from secondary AWS centers.

The 1 IP that succeeds only as IPv4 is at the primary Flexential center.

So, from at least this small log sample it looks like an IPv6 routing problem between Flexential and OVH

I am puzzled though why the redirected HTTP request to HTTPS does not arrive even using IPv4. Not enough info to make good guess - yet :slight_smile:

3 Likes

Wouldn’t a good guess be that something’s wrong with the HTTP-01 ACME challenge process when it uses IPv4/Flexential when using a path that isn’t mainstream (HTTP to HTTPS redirect)?

No, not really. HTTP->HTTPS redirects are common even though not optimal.

And, Flexential is the primary center which must succeed to grant certs. At over 300 million certs issued per day it is not likely a routine comms problem in or near Flexential. Also, IPv4 is more common than IPv6.

You got a cert from production so had to pass Flexential validation at least once. And, in fact twice because you have separate certs for your 2 domain names.

I suggested avoiding the redirect as we saw IPv4 reach you just fine - at least once. Reducing the traffic increases the odds of success. Agree that something seems wrong in a comms config somewhere but it more likely is nearer OVH than originating at Flexential center.

Do you have a good working relationship with OVH network support? Because they could check the logs at their network edges to see what traffic arrives destined for you. That would help isolate where in the path this is going wrong. We won't necessarily know the originating IP at Let's Encrypt (as they rotate frequently) but they know your destination IP's and could filter for that.

4 Likes

Do you have older nginx access logs from your successful production certs?

It would be interesting to see the IP addresses and HTTP->HTTPS flows from those.

4 Likes

Confirming: 2600:3000:2710:200::82 validated calmexpress.com, 2600:3000:1511:200::86 validated www.calmexpress.com.

I’m just using a cheap VPS, I don’t feel like bothering them at this point.

It’s my first time using this HTTP->HTTPS flow. I might do some other tests later though. I’m thinking of trying IPv4 only and IPv6 only from the same VPS, and maybe try different VPS providers/locations.

1 Like

IPv4-only OVH on staging (ipv4only.calmexpress.com): success with no Flexential IP

3.149.251.108 - - [05/Oct/2023:08:01:51 +0000] "GET /.well-known/acme-challenge/nMrW_ntcy5RJ9cOPYRnXnjfcZ8yiX4QEWEksjkXdL-E HTTP/1.1" 200 87 "http://ipv4only.calmexpress.com/.well-known/acme-challenge/nMrW_ntcy5RJ9cOPYRnXnjfcZ8yiX4QEWEksjkXdL-E" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.86.153.167 - - [05/Oct/2023:08:01:52 +0000] "GET /.well-known/acme-challenge/nMrW_ntcy5RJ9cOPYRnXnjfcZ8yiX4QEWEksjkXdL-E HTTP/1.1" 200 87 "http://ipv4only.calmexpress.com/.well-known/acme-challenge/nMrW_ntcy5RJ9cOPYRnXnjfcZ8yiX4QEWEksjkXdL-E" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
23.178.112.106 - - [05/Oct/2023:08:01:55 +0000] "GET /.well-known/acme-challenge/nMrW_ntcy5RJ9cOPYRnXnjfcZ8yiX4QEWEksjkXdL-E HTTP/1.1" 200 87 "http://ipv4only.calmexpress.com/.well-known/acme-challenge/nMrW_ntcy5RJ9cOPYRnXnjfcZ8yiX4QEWEksjkXdL-E" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

IPv6-only OVH on staging (ipv6only.calmexpress.com): fail, with only two connections logged even on HTTP.

HTTP:

2600:1f16:13c:c400:f1fd:b882:812b:ad8 - - [05/Oct/2023:08:02:27 +0000] "GET /.well-known/acme-challenge/eDE_Zqq4c8jhqD6_jNtNhR7AFdBPfl44ZKCT-FlAGTM HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:1f14:a8b:500:793:e1a3:6023:ecd2 - - [05/Oct/2023:08:02:27 +0000] "GET /.well-known/acme-challenge/eDE_Zqq4c8jhqD6_jNtNhR7AFdBPfl44ZKCT-FlAGTM HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

HTTPS:

2600:1f16:13c:c400:f1fd:b882:812b:ad8 - - [05/Oct/2023:08:02:27 +0000] "GET /.well-known/acme-challenge/eDE_Zqq4c8jhqD6_jNtNhR7AFdBPfl44ZKCT-FlAGTM HTTP/1.1" 200 87 "http://ipv6only.calmexpress.com/.well-known/acme-challenge/eDE_Zqq4c8jhqD6_jNtNhR7AFdBPfl44ZKCT-FlAGTM" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:1f14:a8b:500:793:e1a3:6023:ecd2 - - [05/Oct/2023:08:02:27 +0000] "GET /.well-known/acme-challenge/eDE_Zqq4c8jhqD6_jNtNhR7AFdBPfl44ZKCT-FlAGTM HTTP/1.1" 200 87 "http://ipv6only.calmexpress.com/.well-known/acme-challenge/eDE_Zqq4c8jhqD6_jNtNhR7AFdBPfl44ZKCT-FlAGTM" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Repeated in a Let’s Debug: Let's Debug

IPv4 and IPv6 OVH on staging (4and6-1.calmexpress.com): success with three IPv6 connections, including one from Flexential

2600:1f16:13c:c400:f1fd:b882:812b:ad8 - - [05/Oct/2023:08:03:30 +0000] "GET /.well-known/acme-challenge/8c7Mfr6x2qF6wtP8fSnmm11EZAfiDptTNuwuEyEAmvQ HTTP/1.1" 200 87 "http://4and6-1.calmexpress.com/.well-known/acme-challenge/8c7Mfr6x2qF6wtP8fSnmm11EZAfiDptTNuwuEyEAmvQ" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:1f14:a8b:500:793:e1a3:6023:ecd2 - - [05/Oct/2023:08:03:30 +0000] "GET /.well-known/acme-challenge/8c7Mfr6x2qF6wtP8fSnmm11EZAfiDptTNuwuEyEAmvQ HTTP/1.1" 200 87 "http://4and6-1.calmexpress.com/.well-known/acme-challenge/8c7Mfr6x2qF6wtP8fSnmm11EZAfiDptTNuwuEyEAmvQ" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::21 - - [05/Oct/2023:08:03:31 +0000] "GET /.well-known/acme-challenge/8c7Mfr6x2qF6wtP8fSnmm11EZAfiDptTNuwuEyEAmvQ HTTP/1.1" 200 87 "http://4and6-1.calmexpress.com/.well-known/acme-challenge/8c7Mfr6x2qF6wtP8fSnmm11EZAfiDptTNuwuEyEAmvQ" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

:man_shrugging:t2:

2 Likes