Staging rate limitng when fetching issuer certificate

Hello

I’m testing writing hooks for the dehydrated Let’s Encrypt client and using the staging server to test. However the client gets stuck fetching the staging issuer certificate from http://cert.stg-int-x1.letsencrypt.org/ like I am being blocked after the certificate is issued and it fetches the root certiifcate too many times while testing. This is client agnostic though as I can’t curl the URL from my server either but can after around an hour.

When the error happens, curl just stalls until the connection times out.

Thanks

Interesting, not something I’ve seen. Can you obtain a full debug log from dehydrated ?

Will do when I next reproduce the issue!

Alright.

Doing more testing with the staging server and it can’t retrieve the issuer certificate, here is the output from dehydrated.

I don’t think dehydrated has any other verbose/logging functionality?

Processing $DOMAIN
 + Signing domains...
 + Creating new directory /opt/dehydrated/certs/$DOMAIN ...
 + Generating private key...
 + Generating signing request...
 + Requesting challenge for $DOMAIN...
 + Responding to challenge for $DOMAIN...
 + Challenge is valid!
 + Requesting certificate...
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...

And when I check with ps -ef | grep curl

root     15548 15268  0 18:19 pts/3    00:00:00 curl -4 -s -w %{http_code} -o /tmp/dehydrated-0T7fSz http://cert.stg-int-x1.letsencrypt.org/

I understand that host is behind Akamai’s CDN. When I’m able to curl the URL manually from my host I can curl it a few times and then it just sits there and the TCP connection times out, like I am being rate limited. I can PM you the external public IP of this host as well and do any DNS lookups, etc.

Thanks

Can you reproduce the problem using curl -vvv and paste the logs here?

Sure but I have a theory on why this happens that is related to the Akamai CDN and the way my server is setup.

I’m running dehydrated from my Linux load balancer in a VM environment, now the networking. It’s handled so basicly all incoming/outgoing traffic is through GRE tunnel in a remote datacenter to avoid having to pay for a static IP (have a custom daemon to manage the connection) and also DDoS mitigation. however I currently just have the DNS proxied to home router which runs it’s own DNS resolver, but you see the Akamai CDN IP is actually at my home ISP, and I suspect that they are rate limiting connections from people who aren’t their customers.

; <<>> DiG 9.11.0-P2 <<>> cert.stg-int-x1.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 10229
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cert.stg-int-x1.letsencrypt.org. IN	A

;; ANSWER SECTION:
cert.stg-int-x1.letsencrypt.org. 600 IN	CNAME	cert.stg-int-x1.letsencrypt.org.edgesuite.net.
cert.stg-int-x1.letsencrypt.org.edgesuite.net. 21600 IN	CNAME a1759.dscd.akamai.net.
a1759.dscd.akamai.net.	20	IN	A	209.121.139.177
a1759.dscd.akamai.net.	20	IN	A	209.121.139.178

;; Query time: 407 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sat Mar 04 00:14:36 PST 2017
;; MSG SIZE  rcvd: 183

jtl-macbookpro-ethernet:~ jtl$ geoip 209.121.139.177
Resolving IP: 209.121.139.177
{
  "ip": "209.121.139.177",
  "hostname": "No Hostname",
  "city": "Cambridge",
  "region": "Massachusetts",
  "country": "US",
  "loc": "42.3626,-71.0843",
  "org": "AS852 TELUS Communications Inc.",
  "postal": "02142"
}

But if we try a different DNS server

jtl-macbookpro-ethernet:~ jtl$ dig @4.2.2.1 cert.stg-int-x1.letsencrypt.org

; <<>> DiG 9.11.0-P2 <<>> @4.2.2.1 cert.stg-int-x1.letsencrypt.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46386
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 8192
;; QUESTION SECTION:
;cert.stg-int-x1.letsencrypt.org. IN	A

;; ANSWER SECTION:
cert.stg-int-x1.letsencrypt.org. 600 IN	CNAME	cert.stg-int-x1.letsencrypt.org.edgesuite.net.
cert.stg-int-x1.letsencrypt.org.edgesuite.net. 21600 IN	CNAME a1759.dscd.akamai.net.
a1759.dscd.akamai.net.	20	IN	A	23.215.102.82
a1759.dscd.akamai.net.	20	IN	A	23.215.102.88

;; Query time: 341 msec
;; SERVER: 4.2.2.1#53(4.2.2.1)
;; WHEN: Sat Mar 04 00:20:26 PST 2017
;; MSG SIZE  rcvd: 183

jtl-macbookpro-ethernet:~ jtl$ geoip 23.215.102.82
Resolving IP: 23.215.102.82
{
  "ip": "23.215.102.82",
  "hostname": "a23-215-102-82.deploy.static.akamaitechnologies.com",
  "city": "Cambridge",
  "region": "Massachusetts",
  "country": "US",
  "loc": "42.3626,-71.0843",
  "org": "AS20940 Akamai International B.V.",
  "postal": "02142"
}

So It’s an interesting erratum involving Akamai’s CDN and not really related to LetsEncrypt service.

OK you wanted to see curl -vvv with this issue in action?

root@lb-lan:~# curl -vvv 'http://cert.stg-int-x1.letsencrypt.org/ ' |/dev/null
bash: /dev/null: Permission denied
* Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 209.121.139.177...
* Connected to cert.stg-int-x1.letsencrypt.org (209.121.139.177) port 80 (#0)
> GET /  HTTP/1.1
> User-Agent: curl/7.38.0
> Host: cert.stg-int-x1.letsencrypt.org
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0

At which point it just stalls, curl eventually errors with exit code 56.

I will try using other DNS for my load balancer not died to my main home ISP which should solve it, but thought I’d document this for the rare edge case.

Thanks

1 Like

Thanks for sharing! I agree that this is an issue with Akamai’s CDN rather than Let’s Encrypt specifically, but we’re always interested to hear about issues people have reaching Akamai.

It sounds like your tunneling setup is a bit unusual and might be a contributing factor. If you run the curl commands from the machine on the publicly-facing end of your tunnel, do you get the same results?

Yup.

The server at the other end of the tunnel is just running standard Debian 8.x with SNAT and DNAT rules through iptables to pass traffic to the other tunnel endpoint on one of it’s public IP’s.

DNS resolution for this server is just standard 8.8.8.8

Also I messaged you the public IP of my server.

Here’s the output when the issue appears, after many requests to the Akamai server.

I can basicly curl the same URL on my home connection (TELUS) many times and not seemingly get blocked/rate limited.

root@lb-wan:~# curl -vvv -o /dev/null 'http://cert.stg-int-x1.letsencrypt.org/'
* Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 184.28.188.178...
* Connected to cert.stg-int-x1.letsencrypt.org (184.28.188.178) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.38.0
> Host: cert.stg-int-x1.letsencrypt.org
> Accept: */*
>
  0     0    0     0    0     0      0      0 --:--:--  0:00:36 --:--:--     0* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer
root@lb-wan:~#

Obviously because this affects the ability of my server to connect to any Akamai site it would be great to get this fixed.

I’ll bring this to the attention of my provider later as well.

Did some more testing, same as always.

As a temporary solution I’ve altered my VM hosts routing table so traffic for those two Akamai IP’s gets routed through my home connection and not the tunnel, which appears to be getting blocked by Akamai.

Currently writing a ticket to my tunnels hosting provider as well.

You have any contacts at Akamai who can look into this?

Another question: does your tunnel’s hosting provider have local DNS? If not, can you change DNS to use your home ISP’s resolver, or use a --resolve flag to curl to force resolution to the same IP address you get from home?

The problem with using 8.8.8.8 as a resolver is that CDNs will generally use DNS-based Global Software Load Balancing based on the IP address of the recursive resolver. That means that the IP addresses returned by queries to 8.8.8.8 are shared by a lot of people and may be overloaded, or may hit local limits faster.

Right.

I’ve also contacted my hosting provider and they are in contact with Akamai.

I tested with my home ISP’s DNS servers and got some new IP’s.

; <<>> DiG 9.11.0-P2 <<>> @75.153.176.1 cert.stg-int-x1.letsencrypt.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62452
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cert.stg-int-x1.letsencrypt.org. IN	A

;; ANSWER SECTION:
cert.stg-int-x1.letsencrypt.org. 600 IN	CNAME	cert.stg-int-x1.letsencrypt.org.edgesuite.net.
cert.stg-int-x1.letsencrypt.org.edgesuite.net. 21600 IN	CNAME a1759.dscd.akamai.net.
a1759.dscd.akamai.net.	20	IN	A	104.80.89.19
a1759.dscd.akamai.net.	20	IN	A	104.80.89.42

;; Query time: 26 msec
;; SERVER: 75.153.176.1#53(75.153.176.1)
;; WHEN: Tue Mar 07 19:58:16 PST 2017
;; MSG SIZE  rcvd: 183

When used with my tunnel interestingly enough I don’t seem to get rate limited :slight_smile:

I will update if the situation changes.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

I’ve resolved the issue.

When troubleshooting an unrelated issue I noticed I couldn’t access certain sites through the GRE tunnel, such as Yahoo services, speedtest.net and of course Akamai.

The fix was getting my provider to disable some of the DDoS mitigation firewall rules (which probably explains why it starts working after an hour) and I had to enable MSS clamping on my load balancer/router by adding this to my iptables rules.

-t mangle -A POSTROUTING -o gre1 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

I also reduced the MTU of the GRE tunnel for good measure at both ends, although this might not be strictly needed (from 1476 to 1418)

I’ve also moved hosts since this issue started and haven’t seen it since.

1 Like