Wildcard sertificates dosn't issueed have timeout (acme.sh)

gator · September 24, 2019, 9:04am

Yes it is still running!

gator · September 24, 2019, 9:07am

problem begins at

2019-09-24 00:45:02 (MSK)

_az · September 24, 2019, 9:07am

Wow, you weren’t kidding.

curl 7.19.7 on CentOS 6 literally hangs if you run that command.

[root@d485198a77dc /]# sh -c 'curl -L -v --dump-header /root/.acme.sh/http.header -g --user-agent "acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)" -X HEAD -H "Content-Type: application/jose+json" -H -H -H -H -H --data https://acme-v02.api.letsencrypt.org/acme/new-nonce'
* About to connect() to acme-v02.api.letsencrypt.org port 443 (#0)
*   Trying 172.65.32.248... connected
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=acme-v01.api.letsencrypt.org
*       start date: Sep 13 17:57:16 2019 GMT
*       expire date: Dec 12 17:57:16 2019 GMT
*       common name: acme-v01.api.letsencrypt.org
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> HEAD /acme/new-nonce HTTP/1.1
> User-Agent: acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)
> Host: acme-v02.api.letsencrypt.org
> Accept: */*
> Content-Type: application/jose+json
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 24 Sep 2019 09:06:44 GMT
< Connection: keep-alive
< Cache-Control: public, max-age=0, no-cache
< Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
< Replay-Nonce: 00016RUI6R2f5C6woaEjXqPQbqMw3i53QV9qxM_VhdBNBz8
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
* no chunk, no close, no size. Assume close to signal end
<

_az · September 24, 2019, 9:08am

Wait, WTF. Why is the response code HTTP 200?

mnordhoff · September 24, 2019, 9:09am

HEAD is 200 and GET is 204.

_az · September 24, 2019, 9:09am

So that explains it, right? curl is expecting a body because it’s 200 and wasn’t told to handle it otherwise (though I guess -X HEAD should really tell it).

Certainly curl behaves differently between 7.19.7 and e.g. 7.64.0 where it works.

mnordhoff · September 24, 2019, 9:12am

My curl man page says, about -X HEAD:

This option only changes the actual word used in the HTTP request, it does not alter the way curl behaves. So for example if you want to make a proper HEAD request, using -X HEAD will not suffice. You need to use the -I, --head option.

_az · September 24, 2019, 9:17am

Yeah, but check out this magic trick:

[root@d485198a77dc /]# echo '104.110.202.18 acme-v02.api.letsencrypt.org' >> /etc/hosts
[root@d485198a77dc /]# curl -L -vvv --dump-header /root/.acme.sh/http.header -g --user-agent "acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)" -X HEAD -H "Content-Type: application/jose+json" -H -H -H -H -H --data https://acme-v02.api.letsencrypt.org/acme/new-nonce
* About to connect() to acme-v02.api.letsencrypt.org port 443 (#0)
*   Trying 104.110.202.18... connected
* Connected to acme-v02.api.letsencrypt.org (104.110.202.18) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=acme-v02.api.letsencrypt.org
*       start date: Jul 19 04:46:54 2019 GMT
*       expire date: Oct 17 04:46:54 2019 GMT
*       common name: acme-v02.api.letsencrypt.org
*       issuer: CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US
> HEAD /acme/new-nonce HTTP/1.1
> User-Agent: acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)
> Host: acme-v02.api.letsencrypt.org
> Accept: */*
> Content-Type: application/jose+json
>
< HTTP/1.1 200 OK
< Server: nginx
< Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
< Replay-Nonce: 0102CyZ80JVdmYTcGeNtp57ksCqsaZMhxkk9TxaamltlcUY
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Content-Length: 0
< Expires: Tue, 24 Sep 2019 09:19:41 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Tue, 24 Sep 2019 09:19:41 GMT
< Connection: keep-alive
<
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
* Closing connection #0
[root@d485198a77dc /]#

Not hung on the Akamai CDN.

@Neilpang something about the new CDN has broken your way of building HEAD request.

@lestaff too. tl;dr: curl 7.19.7 (CentOS 6) HEAD request used to work, no longer works under Cloudflare and results in a hang.

mnordhoff · September 24, 2019, 9:20am

I’m guessing Akamai instantly murders the connection, and the new setup allows the client to hang on for some extremely large amount of time.

gator · September 24, 2019, 9:22am

now i will try to compile fresh curl version …
here https://curl.haxx.se/download.html

and give you report

_az · September 24, 2019, 9:27am

Oh man. I think it's the missing Content-Length: 0 in the Cloudflare response.

When it hangs, it complains:

mnordhoff · September 24, 2019, 9:28am

That makes sense. If you’re using GET semantics, a 200 OK with no Content-Length header means to wait for the response…

gator · September 24, 2019, 9:30am

i compile fresh curl version

[root@serv3 ~]# /opt/curl766/bin/curl -V
curl 7.66.0 (x86_64-pc-linux-gnu) libcurl/7.66.0 OpenSSL/1.0.1e-fips zlib/1.2.3
Release-Date: 2019-09-11
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp
Features: AsynchDNS HTTPS-proxy IPv6 Largefile libz NTLM NTLM_WB SSL UnixSockets
[root@serv3 ~]#
[root@serv3 ~]#
[root@serv3 ~]# /opt/curl766/bin/curl -L --include --dump-header /root/.acme.sh/http.header -g --user-agent “acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)” -X HEAD -H “Content-Type: application/jose+json” https://acme-v02.api.letsencrypt.org/acme/new-nonce
Warning: Setting custom HTTP method to HEAD with -X/–request may not work the
Warning: way you want. Consider using -I/–head instead.
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 24 Sep 2019 09:28:08 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 0002Wj75ACEflSW5BXSiUVR-7hG5EM35YPFNl4WLAkHMJ7Q
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

…
waiting waiting waiting waiting waiting waiting waiting waiting waiting …
^C

[root@serv3 ~]# ping acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248: icmp_seq=1 ttl=60 time=11.4 ms
^C
— ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics —
1 packets transmitted, 1 received, 0% packet loss, time 553ms
rtt min/avg/max/mdev = 11.414/11.414/11.414/0.000 ms

_az · September 24, 2019, 9:31am

Interesting. I wonder why it works in 7.19.7 but not the current version. (Edit: did you build with nghttp2? I think you need to build with it for the fix)

I think we probably? found the real issue. And it’s one that either Let’s Encrypt/Cloudflare or the author of acme.sh has to fix.

In the meantime, you can try this temporary fix:

echo '104.110.202.18 acme-v02.api.letsencrypt.org' >> /etc/hosts

but don’t rely on it.

gator · September 24, 2019, 9:32am

[root@serv3 ~]# /opt/curl766/bin/curl -L --include --dump-header /root/.acme.sh/http.header -g --user-agent “acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)” --head -H “Content-Type: application/jose+json” https://acme-v02.api.letsencrypt.org/acme/new-nonce
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 24 Sep 2019 09:31:09 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 0001FgztXh2dk9XCtwKJtmauV2BqCDgDhrIs9aEUwAUhHgk
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

[root@serv3 ~]# curl -L --include --dump-header /root/.acme.sh/http.header -g --user-agent “acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)” --head -H “Content-Type: application/jose+json” https://acme-v02.api.letsencrypt.org/acme/new-nonce
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 24 Sep 2019 09:31:15 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 0002XmBsNTOtJuKJ1jEotLNX5aXuzCBW6B2_wpBgAVALHrE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

[root@serv3 ~]#

with --head
work fine!

mnordhoff · September 24, 2019, 9:32am

Seems to be a related acme.sh pull request open:

CyberCr33p · September 24, 2019, 9:33am

I have the same issue which started today:

Yesterday I issue few certificates without issue.

My curl version is 7.66.

gator · September 24, 2019, 9:36am

do you really sure,
that we must to do it

where you get this IP 104.110.202.18 ?
can we trust it?

yes i build fresh curl with nghttp2 support!

_az · September 24, 2019, 9:38am

That’s just the old IP before the new CDN. If it was malicious, it wouldn’t have a valid certificate for the Let’s Encrypt API domain.

But an even better fix is probably to just patch your acme.sh with the tiny fix from the PR that @mnordhoff linked.

gator · September 24, 2019, 9:40am

   if _post "" "$nonceurl" "" "HEAD" "$__request_conent_type"; then

   if _post "" "$nonceurl" "" "HEAD --head" "$__request_conent_type"; then

(from here https://github.com/Neilpang/acme.sh/pull/2499/commits/cba97c58d5a6b93fc2ec1d415efa602332e530d1 )
right now i will try …