Stuck processing

jim-s · June 18, 2021, 3:55am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
stanworth.cloud

I ran this command:
acmeuser@stanworth-cloud:~$ acme.sh --issue -d stanworth.cloud --keylength 4096 -w /var/www/letsencrypt --key-file /etc/letsencrypt/rsa-certs/privkey.pem --ca-file /etc/letsencrypt/rsa-certs/chain.pem --cert-file /etc/letsencrypt/rsa-certs/cert.pem --fullchain-file /etc/letsencrypt/rsa-certs/fullchain.pem --reloadcmd "sudo /bin/systemctl reload nginx.service"

It produced this output:
[Fri Jun 18 11:28:51 CST 2021] Using CA: https://acme.zerossl.com/v2/DV90
[Fri Jun 18 11:28:51 CST 2021] Creating domain key
[Fri Jun 18 11:28:52 CST 2021] The domain key is here: /home/acmeuser/.acme.sh/stanworth.cloud/stanworth.cloud.key
[Fri Jun 18 11:28:52 CST 2021] Single domain='stanworth.cloud'
[Fri Jun 18 11:28:52 CST 2021] Getting domain auth token for each domain
[Fri Jun 18 11:28:55 CST 2021] Getting webroot for domain='stanworth.cloud'
[Fri Jun 18 11:28:55 CST 2021] Verifying: stanworth.cloud
[Fri Jun 18 11:28:59 CST 2021] Processing
[Fri Jun 18 11:29:02 CST 2021] Processing
[Fri Jun 18 11:29:05 CST 2021] Processing
[Fri Jun 18 11:29:08 CST 2021] Processing
[Fri Jun 18 11:29:11 CST 2021] Processing
[Fri Jun 18 11:29:14 CST 2021] Processing
[Fri Jun 18 11:29:17 CST 2021] Processing

My web server is (include version):
nginx version: nginx/1.19.10

The operating system my web server runs on is (include version):
Ubuntu 18.04.5 LTS
My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
N/A

The certificates normally renew fine (I think the automated renewal is 15 days?). Occassionally it fails and I have to manually call the certificate. This has worked until now - when I cannot renew. I see there is a permissions error in the middle of the debug (below). I have already set:

usermod -a -G www-data acmeuser

While there is an apparent chwon error the challenge ownership is updated (but not consistently). I seperated the lines to show the relevant challenge

root@stanworth-cloud:/var/www/letsencrypt/.well-known/acme-challenge# ls -ls
total 32
4 -rwxrwxr-x 1 www-data www-data 87 Jun 15 18:34 2HMdEzwVXmqd2zLm6gLnyA2qnBAM6_gtMDNI1QihoF4
4 -rw-rw-r-- 1 acmeuser acmeuser 87 Jun 18 11:28 69d4RSJwa-1AXBWho6bPlw-nOqekbZjdeQga7jKXv7Y
4 -rwxrwxr-x 1 www-data www-data 87 Jun 15 10:33 9N8cULEP7X8CQWk1tWsIrHUlWzgBcitPCQzEDHIvXQY
4 -rwxrwxr-x 1 www-data www-data 87 Jun 14 15:59 -AOALmm-A5wHCWOCZqbGMalhc2nPXl1uusUHCSuWSQE

4 -rwxrwxr-x 1 www-data www-data 87 Jun 14 16:00 jwZqKQUHYPV9TBNt0Z3vJNtGs9xPvVUpyzu81qLqecw

4 -rwxrwxr-x 1 www-data www-data 87 Jun 14 15:58 NcAfLF8xaNgp6ePwFQDGEAnQkOIpTywJgA6z1XDFoy4
4 -rwxrwxr-x 1 www-data www-data 87 Jun 17 19:33 nR8z4pL_vsVp3YvhSXR54IkaE-WhZEcXuaiobT9N8yg
4 -rw-rw-r-- 1 acmeuser acmeuser 87 Jun 18 11:40 QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg

I'm clearly missing something here - any help would be really appreciated!

For further reference - this is the debug file
root@stanworth-cloud:~# su - acmeuser
acmeuser@stanworth-cloud:~$ acme.sh --issue -d stanworth.cloud --keylength 4096 -w /var/www/letsencrypt --key-file /etc/letsencrypt/rsa-certs/privkey.pem --ca-file /etc/letsencrypt/rsa-certs/chain.pem --cert-file /etc/letsencrypt/rsa-certs/cert.pem --fullchain-file /etc/letsencrypt/rsa-certs/fullchain.pem --reloadcmd "sudo /bin/systemctl reload nginx.service" --debug
[Fri Jun 18 11:40:18 CST 2021] Lets find script dir.
[Fri Jun 18 11:40:18 CST 2021] SCRIPT='/home/acmeuser/.acme.sh/acme.sh'
[Fri Jun 18 11:40:18 CST 2021] _script='/home/acmeuser/.acme.sh/acme.sh'
[Fri Jun 18 11:40:18 CST 2021] _script_home='/home/acmeuser/.acme.sh'
[Fri Jun 18 11:40:18 CST 2021] Using config home:/home/acmeuser/.acme.sh

v3.0.0
[Fri Jun 18 11:40:18 CST 2021] Running cmd: issue
[Fri Jun 18 11:40:18 CST 2021] _main_domain='stanworth.cloud'
[Fri Jun 18 11:40:18 CST 2021] _alt_domains='no'
[Fri Jun 18 11:40:18 CST 2021] Using config home:/home/acmeuser/.acme.sh
[Fri Jun 18 11:40:18 CST 2021] default_acme_server
[Fri Jun 18 11:40:18 CST 2021] ACME_DIRECTORY='https://acme.zerossl.com/v2/DV90'
[Fri Jun 18 11:40:18 CST 2021] DOMAIN_PATH='/home/acmeuser/.acme.sh/stanworth.cloud'
[Fri Jun 18 11:40:18 CST 2021] Using ACME_DIRECTORY: https://acme.zerossl.com/v2/DV90
[Fri Jun 18 11:40:18 CST 2021] _init api for server: https://acme.zerossl.com/v2/DV90
[Fri Jun 18 11:40:18 CST 2021] GET
[Fri Jun 18 11:40:18 CST 2021] url='https://acme.zerossl.com/v2/DV90'
[Fri Jun 18 11:40:18 CST 2021] timeout=
[Fri Jun 18 11:40:18 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g '
[Fri Jun 18 11:40:19 CST 2021] ret='0'
[Fri Jun 18 11:40:19 CST 2021] ACME_KEY_CHANGE='https://acme.zerossl.com/v2/DV90/keyChange'
[Fri Jun 18 11:40:19 CST 2021] ACME_NEW_AUTHZ
[Fri Jun 18 11:40:19 CST 2021] ACME_NEW_ORDER='https://acme.zerossl.com/v2/DV90/newOrder'
[Fri Jun 18 11:40:19 CST 2021] ACME_NEW_ACCOUNT='https://acme.zerossl.com/v2/DV90/newAccount'
[Fri Jun 18 11:40:19 CST 2021] ACME_REVOKE_CERT='https://acme.zerossl.com/v2/DV90/revokeCert'
[Fri Jun 18 11:40:19 CST 2021] ACME_AGREEMENT='https://secure.trust-provider.com/repository/docs/Legacy/20201020_Certificate_Subscriber_Agreement_v_2_4_click.pdf'
[Fri Jun 18 11:40:19 CST 2021] ACME_NEW_NONCE='https://acme.zerossl.com/v2/DV90/newNonce'
[Fri Jun 18 11:40:19 CST 2021] Le_NextRenewTime
[Fri Jun 18 11:40:19 CST 2021] Using CA: https://acme.zerossl.com/v2/DV90
[Fri Jun 18 11:40:19 CST 2021] _on_before_issue
[Fri Jun 18 11:40:19 CST 2021] _chk_main_domain='stanworth.cloud'
[Fri Jun 18 11:40:19 CST 2021] _chk_alt_domains
[Fri Jun 18 11:40:19 CST 2021] Le_LocalAddress
[Fri Jun 18 11:40:19 CST 2021] d='stanworth.cloud'
[Fri Jun 18 11:40:19 CST 2021] Check for domain='stanworth.cloud'
[Fri Jun 18 11:40:19 CST 2021] _currentRoot='/var/www/letsencrypt'
[Fri Jun 18 11:40:19 CST 2021] d
[Fri Jun 18 11:40:19 CST 2021] _saved_account_key_hash is not changed, skip register account.
[Fri Jun 18 11:40:19 CST 2021] Read key length:4096
[Fri Jun 18 11:40:19 CST 2021] _createcsr
[Fri Jun 18 11:40:19 CST 2021] Single domain='stanworth.cloud'
[Fri Jun 18 11:40:19 CST 2021] Getting domain auth token for each domain
[Fri Jun 18 11:40:19 CST 2021] d
[Fri Jun 18 11:40:19 CST 2021] url='https://acme.zerossl.com/v2/DV90/newOrder'
[Fri Jun 18 11:40:19 CST 2021] payload='{"identifiers": [{"type":"dns","value":"stanworth.cloud"}]}'
[Fri Jun 18 11:40:20 CST 2021] RSA key
[Fri Jun 18 11:40:20 CST 2021] HEAD
[Fri Jun 18 11:40:20 CST 2021] _post_url='https://acme.zerossl.com/v2/DV90/newNonce'
[Fri Jun 18 11:40:20 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g -I '
[Fri Jun 18 11:40:20 CST 2021] _ret='0'
[Fri Jun 18 11:40:21 CST 2021] POST
[Fri Jun 18 11:40:21 CST 2021] _post_url='https://acme.zerossl.com/v2/DV90/newOrder'
[Fri Jun 18 11:40:21 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g '
[Fri Jun 18 11:40:22 CST 2021] _ret='0'
[Fri Jun 18 11:40:22 CST 2021] code='201'
[Fri Jun 18 11:40:22 CST 2021] Le_LinkOrder='https://acme.zerossl.com/v2/DV90/order/V9XYpAC3U-Q4T80D0Hmj2A'
[Fri Jun 18 11:40:22 CST 2021] Le_OrderFinalize='https://acme.zerossl.com/v2/DV90/order/V9XYpAC3U-Q4T80D0Hmj2A/finalize'
[Fri Jun 18 11:40:22 CST 2021] url='https://acme.zerossl.com/v2/DV90/authz/NcWCVA4M8q8By_a74-sq4Q'
[Fri Jun 18 11:40:22 CST 2021] payload
[Fri Jun 18 11:40:22 CST 2021] POST
[Fri Jun 18 11:40:22 CST 2021] _post_url='https://acme.zerossl.com/v2/DV90/authz/NcWCVA4M8q8By_a74-sq4Q'
[Fri Jun 18 11:40:22 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g '
[Fri Jun 18 11:40:23 CST 2021] _ret='0'
[Fri Jun 18 11:40:23 CST 2021] code='200'
[Fri Jun 18 11:40:23 CST 2021] d='stanworth.cloud'
[Fri Jun 18 11:40:23 CST 2021] Getting webroot for domain='stanworth.cloud'
[Fri Jun 18 11:40:23 CST 2021] _w='/var/www/letsencrypt'
[Fri Jun 18 11:40:23 CST 2021] _currentRoot='/var/www/letsencrypt'
[Fri Jun 18 11:40:23 CST 2021] entry='"type":"http-01","url":"https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ","status":"pending","token":"QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg"'
[Fri Jun 18 11:40:23 CST 2021] token='QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg'
[Fri Jun 18 11:40:23 CST 2021] uri='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:40:23 CST 2021] keyauthorization='QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg.cilAIA7Dt9W3Znwlxa7_RFGNHe_cvqVV0ucWQUk_AS4'
[Fri Jun 18 11:40:23 CST 2021] dvlist='stanworth.cloud#QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg.cilAIA7Dt9W3Znwlxa7_RFGNHe_cvqVV0ucWQUk_AS4#https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ#http-01#/var/www/letsencrypt'
[Fri Jun 18 11:40:23 CST 2021] d
[Fri Jun 18 11:40:23 CST 2021] vlist='stanworth.cloud#QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg.cilAIA7Dt9W3Znwlxa7_RFGNHe_cvqVV0ucWQUk_AS4#https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ#http-01#/var/www/letsencrypt,'
[Fri Jun 18 11:40:23 CST 2021] d='stanworth.cloud'
[Fri Jun 18 11:40:23 CST 2021] ok, let's start to verify
[Fri Jun 18 11:40:23 CST 2021] Verifying: stanworth.cloud
[Fri Jun 18 11:40:23 CST 2021] d='stanworth.cloud'
[Fri Jun 18 11:40:23 CST 2021] keyauthorization='QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg.cilAIA7Dt9W3Znwlxa7_RFGNHe_cvqVV0ucWQUk_AS4'
[Fri Jun 18 11:40:23 CST 2021] uri='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:40:23 CST 2021] _currentRoot='/var/www/letsencrypt'
[Fri Jun 18 11:40:23 CST 2021] wellknown_path='/var/www/letsencrypt/.well-known/acme-challenge'
[Fri Jun 18 11:40:23 CST 2021] writing token:QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg to /var/www/letsencrypt/.well-known/acme-challenge/QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg
[Fri Jun 18 11:40:23 CST 2021] Changing owner/group of .well-known to www-data:www-data
[Fri Jun 18 11:40:23 CST 2021] chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/jwZqKQUHYPV9TBNt0Z3vJNtGs9xPvVUpyzu81qLqecw': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/2HMdEzwVXmqd2zLm6gLnyA2qnBAM6_gtMDNI1QihoF4': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/nR8z4pL_vsVp3YvhSXR54IkaE-WhZEcXuaiobT9N8yg': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/NcAfLF8xaNgp6ePwFQDGEAnQkOIpTywJgA6z1XDFoy4': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/-AOALmm-A5wHCWOCZqbGMalhc2nPXl1uusUHCSuWSQE': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/69d4RSJwa-1AXBWho6bPlw-nOqekbZjdeQga7jKXv7Y': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/9N8cULEP7X8CQWk1tWsIrHUlWzgBcitPCQzEDHIvXQY': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known': Operation not permitted
[Fri Jun 18 11:40:23 CST 2021] url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:40:23 CST 2021] payload='{}'
[Fri Jun 18 11:40:23 CST 2021] POST
[Fri Jun 18 11:40:23 CST 2021] _post_url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:40:23 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g '
[Fri Jun 18 11:40:24 CST 2021] _ret='0'
[Fri Jun 18 11:40:24 CST 2021] code='200'
[Fri Jun 18 11:40:24 CST 2021] trigger validation code: 200
[Fri Jun 18 11:40:24 CST 2021] sleep 2 secs to verify
[Fri Jun 18 11:40:26 CST 2021] checking
[Fri Jun 18 11:40:26 CST 2021] url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:40:26 CST 2021] payload
[Fri Jun 18 11:40:26 CST 2021] POST
[Fri Jun 18 11:40:26 CST 2021] _post_url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:40:26 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g '
[Fri Jun 18 11:40:27 CST 2021] _ret='0'
[Fri Jun 18 11:40:27 CST 2021] code='200'
[Fri Jun 18 11:40:27 CST 2021] Processing
[Fri Jun 18 11:40:27 CST 2021] sleep 2 secs to verify
[Fri Jun 18 11:40:29 CST 2021] checking
[Fri Jun 18 11:40:29 CST 2021] url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:40:29 CST 2021] payload
[Fri Jun 18 11:40:29 CST 2021] POST
[Fri Jun 18 11:40:29 CST 2021] _post_url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:40:29 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g '
[Fri Jun 18 11:40:30 CST 2021] _ret='0'
[Fri Jun 18 11:40:30 CST 2021] code='200'
[Fri Jun 18 11:40:30 CST 2021] Processing

[---cut here to save space. iterates for n times----]

[Fri Jun 18 11:41:50 CST 2021] Processing
[Fri Jun 18 11:41:50 CST 2021] sleep 2 secs to verify
[Fri Jun 18 11:41:52 CST 2021] checking
[Fri Jun 18 11:41:52 CST 2021] url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:41:52 CST 2021] payload
[Fri Jun 18 11:41:52 CST 2021] POST
[Fri Jun 18 11:41:52 CST 2021] _post_url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:41:52 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g '
[Fri Jun 18 11:41:53 CST 2021] _ret='0'
[Fri Jun 18 11:41:53 CST 2021] code='200'
[Fri Jun 18 11:41:53 CST 2021] Processing
[Fri Jun 18 11:41:53 CST 2021] stanworth.cloud:Timeout
[Fri Jun 18 11:41:53 CST 2021] Debugging, skip removing: /var/www/letsencrypt/.well-known/acme-challenge/QQPl6A-dwCctrrhvXKE6TfwjuLhPs1VnvgOcWHFQRxg
[Fri Jun 18 11:41:53 CST 2021] pid
[Fri Jun 18 11:41:53 CST 2021] No need to restore nginx, skip.
[Fri Jun 18 11:41:53 CST 2021] _clearupdns
[Fri Jun 18 11:41:53 CST 2021] dns_entries
[Fri Jun 18 11:41:53 CST 2021] skip dns.
[Fri Jun 18 11:41:53 CST 2021] _on_issue_err
[Fri Jun 18 11:41:53 CST 2021] Please add '--debug' or '--log' to check more details.
[Fri Jun 18 11:41:53 CST 2021] See: How to debug acme.sh · acmesh-official/acme.sh Wiki · GitHub
[Fri Jun 18 11:41:53 CST 2021] url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:41:53 CST 2021] payload='{}'
[Fri Jun 18 11:41:54 CST 2021] POST
[Fri Jun 18 11:41:54 CST 2021] _post_url='https://acme.zerossl.com/v2/DV90/chall/bSpExaECwn62yDcnB8xmuQ'
[Fri Jun 18 11:41:54 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header -L -g '
[Fri Jun 18 11:41:54 CST 2021] _ret='0'
[Fri Jun 18 11:41:54 CST 2021] code='200'
[Fri Jun 18 11:41:55 CST 2021] Diagnosis versions:
openssl:openssl
OpenSSL 1.1.1j 16 Feb 2021
apache:
apache doesn't exist.
nginx:
nginx version: nginx/1.19.10
built by gcc 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)
built with OpenSSL 1.1.1 11 Sep 2018 (running with OpenSSL 1.1.1j 16 Feb 2021)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp --http-scgi-temp-path=/var/cache/nginx/scgi_temp --user=nginx --group=nginx --with-compat --with-file-aio --with-threads --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --with-http_v2_module --with-mail --with-mail_ssl_module --with-stream --with-stream_realip_module --with-stream_ssl_module --with-stream_ssl_preread_module --with-cc-opt='-g -O2 -fdebug-prefix-map=/data/builder/debuild/nginx-1.19.10/debian/debuild-base/nginx-1.19.10=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'
socat:
socat by Gerhard Rieger and contributors - see www.dest-unreach.org
socat version 1.7.3.2 on Apr 4 2018 10:06:49
running on Linux version #148-Ubuntu SMP Sat May 8 02:33:43 UTC 2021, release 4.15.0-144-generic, machine x86_64
features:
#define WITH_STDIO 1
#define WITH_FDNUM 1
#define WITH_FILE 1
#define WITH_CREAT 1
#define WITH_GOPEN 1
#define WITH_TERMIOS 1
#define WITH_PIPE 1
#define WITH_UNIX 1
#define WITH_ABSTRACT_UNIXSOCKET 1
#define WITH_IP4 1
#define WITH_IP6 1
#define WITH_RAWIP 1
#define WITH_GENERICSOCKET 1
#define WITH_INTERFACE 1
#define WITH_TCP 1
#define WITH_UDP 1
#define WITH_SCTP 1
#define WITH_LISTEN 1
#define WITH_SOCKS4 1
#define WITH_SOCKS4A 1
#define WITH_PROXY 1
#define WITH_SYSTEM 1
#define WITH_EXEC 1
#undef WITH_READLINE
#define WITH_TUN 1
#define WITH_PTY 1
#define WITH_OPENSSL 1
#undef WITH_FIPS
#define WITH_LIBWRAP 1
#define WITH_SYCLS 1
#define WITH_FILAN 1
#define WITH_RETRY 1
#define WITH_MSGLEVEL 0 /debug/

_az · June 18, 2021, 5:27am

The ZeroSSL (Sectigo) ACME server tends to retry the challenges. It looks like it uses some kind of backoff algorithm, starting at once per minute:

91.199.212.132 - - [18/Jun/2021:15:22:43 +1000] "GET /.well-known/acme-challenge/p3phdf7KRhTnwkK9orBqndPl5YIyQDjF-BE16e_5fpM HTTP/1.1" 404 118 "-" "acme.zerossl.com/v2/DV90" "-"
91.199.212.132 - - [18/Jun/2021:15:23:43 +1000] "GET /.well-known/acme-challenge/p3phdf7KRhTnwkK9orBqndPl5YIyQDjF-BE16e_5fpM HTTP/1.1" 404 118 "-" "acme.zerossl.com/v2/DV90" "-"
91.199.212.132 - - [18/Jun/2021:15:25:42 +1000] "GET /.well-known/acme-challenge/p3phdf7KRhTnwkK9orBqndPl5YIyQDjF-BE16e_5fpM HTTP/1.1" 404 118 "-" "acme.zerossl.com/v2/DV90" "-"

This is different to what we're used to with Let's Encrypt, which immediately fails with an error message, and does not try again.

So when the challenge can't be fulfilled, the ZeroSSL one looks like it's stuck; it's still trying.

But, at the end of the day, the problem is that the ACME server wasn't able to complete the challenge. So as a user, the thing to do is search for the typical reasons why the challenge might not be working.

In this case, it looks like the problem is that port 80 on the server isn't open.

jim-s · June 18, 2021, 5:29am

Thanks. I've changed the CA back to letsencrypt

acmeuser@stanworth-cloud:~$ acme.sh --set-default-ca --server letsenctrypt                                               [Fri Jun 18 13:21:39 CST 2021] Changed default CA to: letsenctrypt

But now it looks like the API is misformed? Output follows from debug:

typeacmeuser@stanworth-cloud:~$ acme.sh --issue -d stanworth.cloud --keylength 4096 -w /var/www/letsencrypt --key-file /etc/letsencrypt/rsa-certs/privkey.pem --ca-file /etc/letsencrypt/rsa-certs/chain.pem --cert-file /etc/letsencrypt/rsa-certs/cert.pem --fullchain-file /etc/letsencrypt/rsa-certs/fullchain.pem --debug
[Fri Jun 18 13:24:48 CST 2021] Lets find script dir.
[Fri Jun 18 13:24:48 CST 2021] _SCRIPT_='/home/acmeuser/.acme.sh/acme.sh'
[Fri Jun 18 13:24:48 CST 2021] _script='/home/acmeuser/.acme.sh/acme.sh'
[Fri Jun 18 13:24:48 CST 2021] _script_home='/home/acmeuser/.acme.sh'
[Fri Jun 18 13:24:48 CST 2021] Using config home:/home/acmeuser/.acme.sh
https://github.com/acmesh-official/acme.sh
v3.0.0
[Fri Jun 18 13:24:48 CST 2021] Running cmd: issue
[Fri Jun 18 13:24:48 CST 2021] _main_domain='stanworth.cloud'
[Fri Jun 18 13:24:48 CST 2021] _alt_domains='no'
[Fri Jun 18 13:24:48 CST 2021] Using config home:/home/acmeuser/.acme.sh
[Fri Jun 18 13:24:48 CST 2021] default_acme_server='letsenctrypt'
[Fri Jun 18 13:24:48 CST 2021] ACME_DIRECTORY='letsenctrypt'
[Fri Jun 18 13:24:48 CST 2021] DOMAIN_PATH='/home/acmeuser/.acme.sh/stanworth.cloud'
[Fri Jun 18 13:24:48 CST 2021] Using ACME_DIRECTORY: letsenctrypt
[Fri Jun 18 13:24:48 CST 2021] _init api for server: letsenctrypt
[Fri Jun 18 13:24:48 CST 2021] GET
[Fri Jun 18 13:24:48 CST 2021] url='letsenctrypt'
[Fri Jun 18 13:24:48 CST 2021] timeout=
[Fri Jun 18 13:24:48 CST 2021] _CURL='curl --silent --dump-header /home/acmeuser/.acme.sh/http.header  -L  -g '
[Fri Jun 18 13:24:48 CST 2021] Please refer to https://curl.haxx.se/libcurl/c/libcurl-errors.html for error code: 6
[Fri Jun 18 13:24:48 CST 2021] ret='6'
[Fri Jun 18 13:24:48 CST 2021] Can not init api for: letsenctrypt.
acmeuser@stanworth-cloud:~$

Grrrr not sure how to solve this

_az · June 18, 2021, 5:30am

You have misspelled letsencrypt here.

But this will fail too. ZeroSSL should work fine if you get port 80 open.

jim-s · June 28, 2021, 8:40am

I've started fresh (I needed to updgrade my build anyway). Thanks - _az I'll be watching my typing
I'm still getting the following error:

[Mon 28 Jun 2021 04:13:26 PM CST] Changing owner/group of .well-known to www-data:www-data
[Mon 28 Jun 2021 04:13:26 PM CST] chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/x9aTqsdbnK5ypAFeYZ2yJbYNObT75NF3ZpF-R3uyYx8': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/yEyLoRq1mLQQu4wKpH6Lptt1kWKs86atrQmKHwb35w0': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge/IyQyU5SBzZYRSQbdHWFLzaYkkBmzknembubvUBaSIsE': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known/acme-challenge': Operation not permitted
chown: changing ownership of '/var/www/letsencrypt/.well-known': Operation not permitted

The problem looks similar to:
https://community.letsencrypt.org/t/error-429-and-operation-not-permited/142941

I've tried writing a test file as acme

echo "this is a test file" > /var/www/letsencrypt/.well-known/acme-challenge/test-acme

This works but when trying to change permissions I get the following:

acmeuser@stanworth-cloud:/var/www/letsencrypt/.well-known/acme-challenge$ chown www-data:www-data test-acme
chown: changing ownership of 'test-acme': Operation not permitted

As far as I know acme user is set correctly:

usermod -a -G www-data acmeuser

the http.conf looks like this:

upstream php-handler {
server unix:/run/php/php8.0-fpm.sock;
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name stanworth.cloud;

location ^~ /.well-known/acme-challenge {
default_type text/plain;
root /var/www/letsencrypt;
}
location / {
return 301 https://$host$request_uri;
}
}

Any input is really appreciated

rg305 · June 28, 2021, 9:06am

Why use /var/www/letsencrypt/ for challenge files?

_az · June 28, 2021, 9:16am

Honestly, I really really think that the "chown" error is a red herring. Even if acme.sh can't chown the directory, it's not necessarily a problem.

Your stanworth.cloud domain is still blocking port 80. That will cause the certificate order to fail guaranteed, unlike the "chown error" which might not matter at all.

I think you should attack the more significant problem first, and then if you still have trouble, we can talk about the "chown" error.

jim-s · June 28, 2021, 10:51am

I've run a lsof -i -P -n
which shows:


nginx      936        www-data    9u  IPv6  27816      0t0  TCP *:443 (LISTEN)
nginx      937        www-data    6u  IPv4  27813      0t0  TCP *:80 (LISTEN)
nginx      937        www-data    7u  IPv6  27814      0t0  TCP *:80 (LISTEN)
nginx      937        www-data    8u  IPv4  27815      0t0  TCP *:443 (LISTEN)
nginx      937        www-data    9u  IPv6  27816      0t0  TCP *:443 (LISTEN)

or more clearly:


root@stanworth-cloud:/home/james# nmap -sT 192.168.2.85
Starting Nmap 7.91 ( https://nmap.org ) at 2021-06-28 19:02 CST
Nmap scan report for 192.168.2.85
Host is up (0.00015s latency).
Not shown: 996 closed ports
PORT     STATE SERVICE
80/tcp   open  http
111/tcp  open  rpcbind
443/tcp  open  https

I'm trying to understand here why you feel it is blocked (:80) -- it seems I'm missing something here?

rg305 · June 28, 2021, 4:02pm

Possibly, this is an internal IP (not the one used over the Internet):

[naturally you are already inside the firewall and can access all ports directly]

_az · June 28, 2021, 9:23pm

Yes, exactly.

Although you might be able to access your nginx server from inside your local network, what matters is that the Certificate Authority should be able to access it as well.

If you pop your domain into an external testing tool like letsdebug.net, you'll see that it's not possible to connect to your server from the internet.

You might have to do some port forwarding on a router, or open up these ports on any firewall that might be between your server and the internet.

jim-s · June 29, 2021, 1:59am

_az and rg305 thanks so much for getting at this.
Yes 80 was closed on the router. I set the forward a while back and then disabled it becuase I thought it was not needed. I'd forgotten it was disabled.
Nice when a problem is an easy fix. Awsome to be back on track.

system · July 29, 2021, 2:00am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Some challenges have failed Help	2	599	May 17, 2023
Invalid response from acme-challenge Help	3	597	November 21, 2019
Not able to create fullchain.pem file Help	11	2731	February 13, 2019
Cannot Issue Cert for one domain Help	9	3022	February 20, 2018
"status": "invalid", Help	3	140	August 13, 2024

Stuck processing

Related topics