My domain is:
www.galionlibrary.org et al (see command below)
I ran this command (most recently; previously was attempting with certbot):
sudo /usr/local/bin/le.pl --key /root/letsencrypt/keys/account_key.pem --csr /root/letsencrypt/keys/gpl-domains.csr --csr-key /root/letsencrypt/keys/gpl-domains.key --crt /root/letsencrypt/certs/gpl-domains.crt --domains "www.galionlibrary.org,galionlibrary.org,www.galionlibrary.net,galionlibrary.net,www.galionlibrary.com,galionlibrary.com,cgi.galion.lib.oh.us,www.galion.lib.oh.us" --renew 30 --path /var/www/html/.well-known/acme-challenge --email [redacted here] --generate-missing --debug
It produced this output [the 600-second sleep is a customization that I introduced during debugging; it made no difference to the outcome]:
2024/04/22 12:10:51 [ Crypt::LE client v0.39 started. ]
2024/04/22 12:10:51 Loading an account key from /root/letsencrypt/keys/account_key.pem
2024/04/22 12:10:51 Account key loaded.
2024/04/22 12:10:51 Loading a CSR from /root/letsencrypt/keys/gpl-domains.csr
2024/04/22 12:10:51 Loaded domain names from CSR: www.galionlibrary.org, galionlibrary.org, www.galionlibrary.net, galio
nlibrary.net, www.galionlibrary.com, galionlibrary.com, cgi.galion.lib.oh.us, www.galion.lib.oh.us
2024/04/22 12:10:51 CSR loaded.
2024/04/22 12:10:51 CSR key loaded
2024/04/22 12:10:51 Checking certificate for expiration (website connection).
2024/04/22 12:10:51 Checking www.galionlibrary.org
2024/04/22 12:10:51 Expiration threshold set at 30 days, the certificate expires in 2 days - will be renewing.
2024/04/22 12:10:51 Account email has been set to 'jonadab@galionlibrary.org'
2024/04/22 12:10:51 Connecting to https://acme-staging-v02.api.letsencrypt.org/directory
2024/04/22 12:10:51 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce
2024/04/22 12:10:52 Directory loaded successfully.
2024/04/22 12:10:52 Registering the account key
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/new-acct
2024/04/22 12:10:52 Key is already registered, reg path: https://acme-staging-v02.api.letsencrypt.org/acme/acct/14535287
4.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/acct/145352874
2024/04/22 12:10:52 Account ID: 145352874
2024/04/22 12:10:52 Registration success: TOS change status - 0, new registration flag - 0.
2024/04/22 12:10:52 The key is already registered. ID: 145352874
2024/04/22 12:10:52 TOS has NOT been changed, no need to accept again.
2024/04/22 12:10:52 Current contact details: jonadab@galionlibrary.org
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/new-order
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/finalize/145352874/16084154694
2024/04/22 12:10:52 Could not finalize an order.
2024/04/22 12:10:52 Requesting challenge.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12105133054
2024/04/22 12:10:52 Received challenges for cgi.galion.lib.oh.us.
2024/04/22 12:10:52 Requesting challenge.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12105133064
2024/04/22 12:10:52 Received challenges for galionlibrary.com.
2024/04/22 12:10:52 Requesting challenge.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12105133074
2024/04/22 12:10:52 Received challenges for galionlibrary.net.
2024/04/22 12:10:52 Requesting challenge.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12105133084
2024/04/22 12:10:52 Received challenges for galionlibrary.org.
2024/04/22 12:10:52 Requesting challenge.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12105133094
2024/04/22 12:10:52 Received challenges for www.galion.lib.oh.us.
2024/04/22 12:10:52 Requesting challenge.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12105133104
2024/04/22 12:10:52 Received challenges for www.galionlibrary.com.
2024/04/22 12:10:52 Requesting challenge.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12105133114
2024/04/22 12:10:52 Received challenges for www.galionlibrary.net.
2024/04/22 12:10:52 Requesting challenge.
2024/04/22 12:10:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12105133124
2024/04/22 12:10:52 Received challenges for www.galionlibrary.org.
2024/04/22 12:10:52 Requested challenges for 8 domain(s).
2024/04/22 12:10:52 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/kiQaNqx-SO8UL-xZogWhns
4XOL8lIW_-980spcQWPYM' for domain 'www.galionlibrary.org'. Sleeping 600 seconds.
2024/04/22 12:20:52 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/MxYecldKr40koF11QZ-sA0
OteEwCSQAPtzqO8Q_Z1Y4' for domain 'galionlibrary.org'. Sleeping 600 seconds.
2024/04/22 12:30:52 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/1VPeZr51xd8BfgIbdGZf3E
kRi5WPY_t72-lT4mJ79j4' for domain 'www.galionlibrary.net'. Sleeping 600 seconds.
2024/04/22 12:40:52 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/epU9ruhu_xeoq1GUTpfNOk
bZBnW3QGlC22Qf9Ufa_sE' for domain 'galionlibrary.net'. Sleeping 600 seconds.
2024/04/22 12:50:52 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/KRx-CLOfth2jkt0XxMmJqQ
K2sgU8j0G3S-P5_fLfWIU' for domain 'www.galionlibrary.com'. Sleeping 600 seconds.
2024/04/22 13:00:52 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/j2s-UgksJo3nHjLG8IBpJD
Tjo9SD0aTj0Y_FHtw_Fp0' for domain 'galionlibrary.com'. Sleeping 600 seconds.
2024/04/22 13:10:52 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/c89ABfRbQA_zGfDidBZq3h
yTUzhMjlxnGkEx1D9Fk70' for domain 'cgi.galion.lib.oh.us'. Sleeping 600 seconds.
2024/04/22 13:20:52 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/EvT95PKGqOhUDC5Ld0pwQd
-BLxfPXVrRw-VA-VAuw40' for domain 'www.galion.lib.oh.us'. Sleeping 600 seconds.
2024/04/22 13:30:52 Accepted challenges for 8 domain(s).
2024/04/22 13:30:52 Connecting to https://acme-staging-v02.api.letsencrypt.org/directory
2024/04/22 13:30:53 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce
2024/04/22 13:30:53 Directory loaded successfully.
2024/04/22 13:30:53 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133124/vpyN4g
2024/04/22 13:30:53 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133124/vpyN4g
2024/04/22 13:30:55 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133124/vpyN4g
2024/04/22 13:30:55 Domain verification results for 'www.galionlibrary.org': error. During secondary validation: 66.213.
116.5: Invalid response from http://www.galionlibrary.org/.well-known/acme-challenge/kiQaNqx-SO8UL-xZogWhns4XOL8lIW_-980
spcQWPYM: 403
2024/04/22 13:30:55 You can now delete the '/var/www/html/.well-known/acme-challenge/kiQaNqx-SO8UL-xZogWhns4XOL8lIW_-980
spcQWPYM' file.
2024/04/22 13:30:55 Domain www.galionlibrary.org has failed verification (status code 200).
2024/04/22 13:30:55 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133084/t2Bs6w
2024/04/22 13:30:55 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133084/t2Bs6w
2024/04/22 13:30:57 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133084/t2Bs6w
2024/04/22 13:30:57 Domain verification results for 'galionlibrary.org': error. During secondary validation: 66.213.116.
5: Invalid response from http://galionlibrary.org/.well-known/acme-challenge/MxYecldKr40koF11QZ-sA0OteEwCSQAPtzqO8Q_Z1Y4
: 403
2024/04/22 13:30:57 You can now delete the '/var/www/html/.well-known/acme-challenge/MxYecldKr40koF11QZ-sA0OteEwCSQAPtzq
O8Q_Z1Y4' file.
2024/04/22 13:30:57 Domain galionlibrary.org has failed verification (status code 200).
2024/04/22 13:30:57 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133114/HwqDNw
2024/04/22 13:30:57 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133114/HwqDNw
2024/04/22 13:30:59 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133114/HwqDNw
2024/04/22 13:30:59 Domain verification results for 'www.galionlibrary.net': error. During secondary validation: 66.213.
116.5: Invalid response from http://www.galionlibrary.net/.well-known/acme-challenge/1VPeZr51xd8BfgIbdGZf3EkRi5WPY_t72-l
T4mJ79j4: 403
2024/04/22 13:30:59 You can now delete the '/var/www/html/.well-known/acme-challenge/1VPeZr51xd8BfgIbdGZf3EkRi5WPY_t72-l
T4mJ79j4' file.
2024/04/22 13:30:59 Domain www.galionlibrary.net has failed verification (status code 200).
2024/04/22 13:30:59 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133074/tbyQKQ
2024/04/22 13:30:59 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133074/tbyQKQ
2024/04/22 13:31:01 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133074/tbyQKQ
2024/04/22 13:31:01 Domain verification results for 'galionlibrary.net': error. During secondary validation: 66.213.116.
5: Invalid response from http://galionlibrary.net/.well-known/acme-challenge/epU9ruhu_xeoq1GUTpfNOkbZBnW3QGlC22Qf9Ufa_sE
: 403
2024/04/22 13:31:01 You can now delete the '/var/www/html/.well-known/acme-challenge/epU9ruhu_xeoq1GUTpfNOkbZBnW3QGlC22Q
f9Ufa_sE' file.
2024/04/22 13:31:01 Domain galionlibrary.net has failed verification (status code 200).
2024/04/22 13:31:01 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133104/yyR3QA
2024/04/22 13:31:02 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133104/yyR3QA
2024/04/22 13:31:04 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133104/yyR3QA
2024/04/22 13:31:04 Domain verification results for 'www.galionlibrary.com': error. During secondary validation: 66.213.
116.5: Invalid response from http://www.galionlibrary.com/.well-known/acme-challenge/KRx-CLOfth2jkt0XxMmJqQK2sgU8j0G3S-P
5_fLfWIU: 403
2024/04/22 13:31:04 You can now delete the '/var/www/html/.well-known/acme-challenge/KRx-CLOfth2jkt0XxMmJqQK2sgU8j0G3S-P
5_fLfWIU' file.
2024/04/22 13:31:04 Domain www.galionlibrary.com has failed verification (status code 200).
2024/04/22 13:31:04 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133064/Qg88UA
2024/04/22 13:31:04 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133064/Qg88UA
2024/04/22 13:31:06 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133064/Qg88UA
2024/04/22 13:31:06 Domain verification results for 'galionlibrary.com': error. During secondary validation: 66.213.116.
5: Invalid response from http://galionlibrary.com/.well-known/acme-challenge/j2s-UgksJo3nHjLG8IBpJDTjo9SD0aTj0Y_FHtw_Fp0
: 403
2024/04/22 13:31:06 You can now delete the '/var/www/html/.well-known/acme-challenge/j2s-UgksJo3nHjLG8IBpJDTjo9SD0aTj0Y_
FHtw_Fp0' file.
2024/04/22 13:31:06 Domain galionlibrary.com has failed verification (status code 200).
2024/04/22 13:31:06 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133054/skZGnw
2024/04/22 13:31:06 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133054/skZGnw
2024/04/22 13:31:08 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133054/skZGnw
2024/04/22 13:31:08 Domain verification results for 'cgi.galion.lib.oh.us': error. During secondary validation: 66.213.1
16.5: Invalid response from http://cgi.galion.lib.oh.us/.well-known/acme-challenge/c89ABfRbQA_zGfDidBZq3hyTUzhMjlxnGkEx1
D9Fk70: 403
2024/04/22 13:31:08 You can now delete the '/var/www/html/.well-known/acme-challenge/c89ABfRbQA_zGfDidBZq3hyTUzhMjlxnGkE
x1D9Fk70' file.
2024/04/22 13:31:08 Domain cgi.galion.lib.oh.us has failed verification (status code 200).
2024/04/22 13:31:08 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133094/WnZN-g
2024/04/22 13:31:09 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133094/WnZN-g
2024/04/22 13:31:11 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12105133094/WnZN-g
2024/04/22 13:31:11 Domain verification results for 'www.galion.lib.oh.us': error. During secondary validation: 66.213.1
16.5: Invalid response from http://www.galion.lib.oh.us/.well-known/acme-challenge/EvT95PKGqOhUDC5Ld0pwQd-BLxfPXVrRw-VA-
VAuw40: 403
2024/04/22 13:31:11 You can now delete the '/var/www/html/.well-known/acme-challenge/EvT95PKGqOhUDC5Ld0pwQd-BLxfPXVrRw-V
A-VAuw40' file.
2024/04/22 13:31:11 Domain www.galion.lib.oh.us has failed verification (status code 200).
2024/04/22 13:31:11 All verifications failed
2024/04/22 13:31:11 All verifications failed
I have left the challenge files in place for now. (Can anyone someone outside the US empirically verify that they're accessible internationally? Everything I know says they should be, but yet we have this error. All the systems I have access to, in order to test, are physically located in America.)
My web server is (include version):
Apache 2.4.59-1~deb11u1
The operating system my web server runs on is (include version):
Devuan chimaera, as up-to-date as chimaera can be. I can update to daedalus if it has a meaningful chance of resolving the issue, but I do not think this is relevant.
My hosting provider, if applicable, is:
Galion Public Library
Upstream ISP is the OPLIN.
I can login to a root shell on my machine (yes or no, or I don't know):
Yes.
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
Does Emacs count?
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Crypt::LE version 0.39
Also used certbot 1.12.0 with very similar results. When I ran into trouble I went looking for a Perl-based solution because it's easier for me to debug (e.g., I was able to absolutely confirm that the challenge files are being written where I think they're being written, with the permissions I think they should have, etc.; I didn't know how to do that with Python.) But I am pretty sure both programs are running into the same issue.
Web server's public IP address is 66.213.116.5
If we are doing any geoblocking, I am not aware of it (and I really really ought to be). I have thought about geoblocking incoming traffic on port 25 (specifically, to not receive any mail from APNIC space, as it's consistently all spam), but I haven't actually implemented that, and I have never had any reason to even consider geoblocking on ports 80 or 443. Admittedly, if OPLIN were geoblocking, I probably would not have noticed it until now. (How can I check that?) But I would not expect firewall-level geoblocking to result in a 403 response in any case, unless it's at the client's end of the connection. Apache, as far as I am aware, is not even capable of geoblocking. (Perhaps with a third-party module? But I haven't installed anything like that.)
I found one old forum thread that suggested incorrect AAAA records could result in this problem, but there shouldn't be (and as far as I know aren't) any AAAA records for any of our domains, as we have never used IPv6 for anything, and I don't think our connectivity provider (OPLIN) even supports IPv6. Also, we're getting the same error on all of our domains, and they don't all use the same authoritative DNS provider. (The .lib.oh.us domains have authoritative DNS provided by the state of Ohio, and the others have it from Network Solutions.)
