Wildcard sertificates dosn't issueed have timeout (acme.sh)

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: *.hostland.ru

I ran this command:
acme.sh/acme.sh --issue --force -d ‘*.hostland.ru’ --dns --yes-I-know-dns-manual-mode-enough-go-ahead-please --debug --output-insecure

It produced this output:
[Tue Sep 24 11:08:15 MSK 2019] Lets find script dir.
[Tue Sep 24 11:08:15 MSK 2019] SCRIPT=’/home/michael/acme.sh/acme.sh’
[Tue Sep 24 11:08:15 MSK 2019] _script=’/home/michael/acme.sh/acme.sh’
[Tue Sep 24 11:08:15 MSK 2019] _script_home=’/home/michael/acme.sh’
[Tue Sep 24 11:08:15 MSK 2019] Using default home:/root/.acme.sh
[Tue Sep 24 11:08:15 MSK 2019] Using config home:/root/.acme.sh


v2.8.3
[Tue Sep 24 11:08:15 MSK 2019] Running cmd: issue
[Tue Sep 24 11:08:15 MSK 2019] _main_domain=’.hostland.ru’
[Tue Sep 24 11:08:15 MSK 2019] _alt_domains=‘no’
[Tue Sep 24 11:08:15 MSK 2019] Using config home:/root/.acme.sh
[Tue Sep 24 11:08:15 MSK 2019] ACME_DIRECTORY=‘https://acme-v02.api.letsencrypt.org/directory’
[Tue Sep 24 11:08:15 MSK 2019] DOMAIN_PATH=’/root/.acme.sh/.hostland.ru’
[Tue Sep 24 11:08:15 MSK 2019] Using ACME_DIRECTORY: https://acme-v02.api.letsencrypt.org/directory
[Tue Sep 24 11:08:15 MSK 2019] _init api for server: https://acme-v02.api.letsencrypt.org/directory
[Tue Sep 24 11:08:15 MSK 2019] GET
[Tue Sep 24 11:08:15 MSK 2019] url=‘https://acme-v02.api.letsencrypt.org/directory’
[Tue Sep 24 11:08:15 MSK 2019] timeout=
[Tue Sep 24 11:08:15 MSK 2019] _CURL=‘curl -L --silent --dump-header /root/.acme.sh/http.header -g ’
[Tue Sep 24 11:08:16 MSK 2019] ret=‘0’
[Tue Sep 24 11:08:16 MSK 2019] ACME_KEY_CHANGE=‘https://acme-v02.api.letsencrypt.org/acme/key-change’
[Tue Sep 24 11:08:16 MSK 2019] ACME_NEW_AUTHZ
[Tue Sep 24 11:08:16 MSK 2019] ACME_NEW_ORDER=‘https://acme-v02.api.letsencrypt.org/acme/new-order’
[Tue Sep 24 11:08:16 MSK 2019] ACME_NEW_ACCOUNT=‘https://acme-v02.api.letsencrypt.org/acme/new-acct’
[Tue Sep 24 11:08:16 MSK 2019] ACME_REVOKE_CERT=‘https://acme-v02.api.letsencrypt.org/acme/revoke-cert’
[Tue Sep 24 11:08:16 MSK 2019] ACME_AGREEMENT=‘https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf’
[Tue Sep 24 11:08:16 MSK 2019] ACME_NEW_NONCE=‘https://acme-v02.api.letsencrypt.org/acme/new-nonce’
[Tue Sep 24 11:08:16 MSK 2019] ACME_VERSION=‘2’
[Tue Sep 24 11:08:16 MSK 2019] Le_NextRenewTime
[Tue Sep 24 11:08:16 MSK 2019] _on_before_issue
[Tue Sep 24 11:08:16 MSK 2019] _chk_main_domain=’.hostland.ru’
[Tue Sep 24 11:08:16 MSK 2019] _chk_alt_domains
[Tue Sep 24 11:08:16 MSK 2019] Le_LocalAddress
[Tue Sep 24 11:08:16 MSK 2019] d=’.hostland.ru’
[Tue Sep 24 11:08:16 MSK 2019] Check for domain=’.hostland.ru’
[Tue Sep 24 11:08:16 MSK 2019] _currentRoot=‘dns’
[Tue Sep 24 11:08:16 MSK 2019] d
[Tue Sep 24 11:08:16 MSK 2019] _saved_account_key_hash is not changed, skip register account.
[Tue Sep 24 11:08:16 MSK 2019] Read key length:
[Tue Sep 24 11:08:16 MSK 2019] _createcsr
[Tue Sep 24 11:08:16 MSK 2019] Single domain=’.hostland.ru’
[Tue Sep 24 11:08:16 MSK 2019] Getting domain auth token for each domain
[Tue Sep 24 11:08:16 MSK 2019] d
[Tue Sep 24 11:08:16 MSK 2019] url=‘https://acme-v02.api.letsencrypt.org/acme/new-order’
[Tue Sep 24 11:08:16 MSK 2019] payload=’{“identifiers”: [{“type”:“dns”,“value”:"*.hostland.ru"}]}’
[Tue Sep 24 11:08:16 MSK 2019] RSA key
[Tue Sep 24 11:08:16 MSK 2019] HEAD
[Tue Sep 24 11:08:16 MSK 2019] _post_url=‘https://acme-v02.api.letsencrypt.org/acme/new-nonce’
[Tue Sep 24 11:08:16 MSK 2019] _CURL='curl -L --silent --dump-header /root/.acme.sh/http.header -g ’

[root@serv3 ~]# ps aux | grep curl
root 2449 0.0 0.0 188160 10348 pts/2 S+ 11:08 0:00 curl -L --silent --dump-header /root/.acme.sh/http.header -g --user-agent acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh) -X HEAD -H Content-Type: application/jose+json -H -H -H -H -H --data https://acme-v02.api.letsencrypt.org/acme/new-nonce
root 7280 0.0 0.0 103324 1988 pts/3 S+ 11:13 0:00 grep curl
[root@serv3 ~]# strace -p 2449
Process 2449 attached - interrupt to quit
restart_syscall(<… resuming interrupted call …>) = 0
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 0) = 0 (Timeout)
poll([{fd=3, events=POLLIN|POLLPRI|POLLRDNORM|POLLRDBAND}], 1, 1000^C <unfinished …>

My web server is (include version):
nginx

The operating system my web server runs on is (include version):
[root@serv3 ~]# uname -a
Linux serv3.hostland.ru 4.9.130-11.el6.x86_64 #1 SMP Tue Oct 2 17:19:17 MSK 2018 x86_64 x86_64 x86_64 GNU/Linux

My hosting provider, if applicable, is:
Hostland LTD Russia

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
[root@serv3 ~]# curl --version
curl 7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Protocols: tftp ftp telnet dict ldap ldaps http file https ftps scp sftp
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz

2

If you try to run “curl -v https://acme-v02.api.letsencrypt.org/directory” in another shell, what happens?

3

no problem

[root@serv3 ~]# curl -v https://acme-v02.api.letsencrypt.org/directory

GET /directory HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.27.1 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
Host: acme-v02.api.letsencrypt.org
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 24 Sep 2019 08:27:17 GMT
< Content-Type: application/json
< Content-Length: 658
< Connection: keep-alive
< Cache-Control: public, max-age=0, no-cache
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
<
{
"a2o6fAzH3gY": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"

But if i do:
[root@serv3 ~]# wget https://acme-v02.api.letsencrypt.org/acme/new-nonce
--2019-09-24 11:27:57-- https://acme-v02.api.letsencrypt.org/acme/new-nonce
Resolving acme-v02.api.letsencrypt.org... 172.65.32.248, 2606:4700:60:0:f53d:5624:85c7:3a2c
Connecting to acme-v02.api.letsencrypt.org|172.65.32.248|:443... connected.
HTTP request sent, awaiting response... 204 No Content
Length: unspecified
Saving to: “new-nonce.4”

[      <=>                                                                                                                                        ] 0           --.-K/s

(internal process - no data received)

4

Is the new-nonce request actually sitting there waiting and doing nothing?

Returning 204 No Content and an empty response is correct – the nonce is included in an HTTP header, not a body.

5

Could you also show what peer addresses ss -tnp reports for the PID associated with the hung curl process?

Oh, and whether a curl in a terminal also hangs for curl --head .../new-nonce.

6

right! new-nonce request actually waiting and doing nothing …
waiting waiting waiting waiting waiting …

7

That’s bizarre. It works for me:

$ wget https://acme-v02.api.letsencrypt.org/acme/new-nonce
--2019-09-24 08:32:09--  https://acme-v02.api.letsencrypt.org/acme/new-nonce
Resolving acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)... 2606:4700:60:0:f53d:5624:85c7:3a2c, 172.65.32.248
Connecting to acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)|2606:4700:60:0:f53d:5624:85c7:3a2c|:443... connected.
HTTP request sent, awaiting response... 204 No Content
2019-09-24 08:32:09 (0.00 B/s) - ‘new-nonce’ saved [0]

Totally instant.

8

[root@serv3 ~]# ss -tnp
…
ESTAB 0 0 185.26.122.3:55380 172.65.32.248:443 users:((“curl”,23118,3))
…

and crazy:

[root@serv3 ~]# curl --include https://acme-v02.api.letsencrypt.org/acme/new-nonce
HTTP/1.1 204 No Content
Server: nginx
Date: Tue, 24 Sep 2019 08:40:08 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 0001cmXsNiih1TD0kjetuPdDU4qzQUp3roBjhj9FYFZhXCQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

[root@serv3 ~]#
[root@serv3 ~]#
[root@serv3 ~]# curl --head --include https://acme-v02.api.letsencrypt.org/acme/new-nonce
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 24 Sep 2019 08:40:13 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 00023rjGei2BRkf1hB3WdFYoPHmMZm25QVElH6ZipBhNcrg
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

so it perfectly work!

9

but i still have

[root@serv3 ~]# wget https://acme-v02.api.letsencrypt.org/acme/new-nonce
--2019-09-24 11:41:02-- https://acme-v02.api.letsencrypt.org/acme/new-nonce
Resolving acme-v02.api.letsencrypt.org... 172.65.32.248, 2606:4700:60:0:f53d:5624:85c7:3a2c
Connecting to acme-v02.api.letsencrypt.org|172.65.32.248|:443... connected.
HTTP request sent, awaiting response... 204 No Content
Length: unspecified
Saving to: “new-nonce.6”

[         <=>                                                                                                                                                           ] 0           --.-K/s

and waiting waiting waiting ...

10

So the request hanging thing might just be an HTTP 204 bug from old versions of wget.

I can’t reproduce it on Ubuntu 19.04, but I can reproduce it on Debian Wheezy:

root@ae0e82840ffd:/# wget --no-check-certificate https://acme-v02.api.letsencrypt.org/acme/new-nonce
--2019-09-24 08:41:12--  https://acme-v02.api.letsencrypt.org/acme/new-nonce
Resolving acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)... 172.65.32.248, 2606:4700:60:0:f53d:5624:85c7:3a2c
Connecting to acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)|172.65.32.248|:443... connected.
WARNING: The certificate of `acme-v02.api.letsencrypt.org' is not trusted.
WARNING: The certificate of `acme-v02.api.letsencrypt.org' hasn't got a known issuer.
HTTP request sent, awaiting response... 204 No Content
Length: unspecified
Saving to: `new-nonce'

    [                                                              <=>                                                                                                          ] 0           --.-K/s

Version:

root@ae0e82840ffd:/# wget -V
GNU Wget 1.13.4 built on linux-gnu.

I think it might be a red herring … but who knows.

1 Like
11

Check. I can reproduce it on older versions of Ubuntu. It was fixed some time between 14.04 and 16.04.

1 Like
12

a have a lot of production machines

[root@serv3 ~]# wget -V
GNU Wget 1.12 built on linux-gnu.
(doesnt work)

  1. another server
    [root@serv5 include]# wget --version
    GNU Wget 1.14 built on linux-gnu.
    (dosnt work too)

please remember: wget connection - this is for example!
by default script work with a curl!

[root@serv5 include]# wget --version
GNU Wget 1.14 built on linux-gnu.

+digest +https +ipv6 +iri +large-file +nls +ntlm +opie +ssl/openssl

Wgetrc:
/etc/wgetrc (system)
Locale: /usr/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
-DLOCALEDIR="/usr/share/locale" -I. -I…/lib -I…/lib -O2 -g -pipe
-Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong
–param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic
Link: gcc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions
-fstack-protector-strong --param=ssp-buffer-size=4
-grecord-gcc-switches -m64 -mtune=generic -lssl -lcrypto
/usr/lib64/libssl.so /usr/lib64/libcrypto.so /usr/lib64/libz.so
-ldl -lz -lz -lidn -luuid -lpcre ftp-opie.o openssl.o http-ntlm.o
…/lib/libgnu.a

Copyright © 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
http://www.gnu.org/licenses/gpl.html.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic hniksic@xemacs.org.
Please send bug reports and questions to bug-wget@gnu.org.
[root@serv5 include]# wget https://acme-v02.api.letsencrypt.org/acme/new-nonce
–2019-09-24 11:49:45-- https://acme-v02.api.letsencrypt.org/acme/new-nonce
Resolving acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)… 172.65.32.248, 2606:4700:60:0:f53d:5624:85c7:3a2c
Connecting to acme-v02.api.letsencrypt.org (acme-v02.api.letsencrypt.org)|172.65.32.248|:443… connected.
HTTP request sent, awaiting response… 204 No Content
Length: unspecified
Saving to: ‘new-nonce.3’

[     <=>                                                                                                                                                               ] 0           --.-K/s
13

everythink was work 4 hours ago!
but now - it doesnt work on several our servers

14

Yes, indeed. The only thing we know for certain is that the wget problem has nothing to do with the real problem.

Still have to determine why curl is hanging when acme.sh runs it …

What if you run the exact(ish) command that acme.sh runs?

sh -c 'curl -L -v --dump-header /root/.acme.sh/http.header -g --user-agent "acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)" -X HEAD -H "Content-Type: application/jose+json" -H -H -H -H -H --data https://acme-v02.api.letsencrypt.org/acme/new-nonce'

Edit: fixed the command a bit

15

Apparently wget fixed this issue on 2014-04-22.

https://lists.gnu.org/archive/html/bug-wget/2014-04/threads.html#00052

Assuming the tarball modification times on their FTP site are correct, it probably made it into 1.16 in late 2014.

16

this is good news!
but i have still a problem with curl that have timeout connection!

17

i try this one …

[root@serv5 gena]# curl -L --include --dump-header /root/.acme.sh/http.header -g --user-agent “acme.sh/2.8.3 (https://github.com/Neilpang/acme.sh)” -X HEAD -H “Content-Type: application/jose+json” https://acme-v02.api.letsencrypt.org/acme/new-nonce
HTTP/1.1 200 OK
Server: nginx
Date: Tue, 24 Sep 2019 08:58:00 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
Replay-Nonce: 00026GF_lPjbMd-e3XOgR-ubroquNux6OpjI0UkQTaKiYsI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

and waiting waiting waiting …
and waiting waiting waiting …
and waiting waiting waiting …

18

Wait, so it’s still running?

curl is meant to stop after it receives response headers in an HTTP 204 response.

19

It really worked 4 hours ago? The big CDN change happened hours before that.

20

my admins report to me this problem, i am not sure, the exactly time when it happen