I'm pretty sure that's just OPLIN not answering ICMP requests on their network. Probably a red herring.
I'm pretty sure ICMP can't be directed at specific TCP ports [like 80 and 443].
2 Likes
From the traceroute manpages:
-T, --tcp
Use TCP SYN for probes
-p port, --port=port
For UDP tracing, specifies the destination port base traceroute will use (the destination port number will be incremented by each probe).
For ICMP tracing, specifies the initial ICMP sequence value (incremented by each probe too).
For TCP and others specifies just the (constant) destination port to connect. When using the tcptraceroute wrapper, -p specifies the source port.
2 Likes
I was referring to the traceroute results. I think the traceroute results stopping when they hit the OPLIN network, is a red herring. traceroute has always returned nothing for that part of the network, for decades.
My appologies!
I somehow left out the most important part of that post:
traceroute -T -p 80 www.galionlibrary.org
Agreed: ICMP responses/lack thereof isn't much of a real clue.
2 Likes
Oh, hmm.
On a system where it works, I get this:
jonadab@ostrich:~$ sudo traceroute -T -p 80 www.galionlibrary.org
traceroute to www.galionlibrary.org (66.213.116.5), 30 hops max, 60 byte packets
1 * * *
2 syn-142-254-158-145.inf.spectrum.com (142.254.158.145) 15.824 ms 15.819 ms 15.815 ms
3 * * *
4 lag-42.mcr11clmcohib.netops.charter.com (24.33.161.156) 17.424 ms 17.419 ms 17.410 ms
5 lag-27.rcr01clevohek.netops.charter.com (65.29.1.38) 23.745 ms 23.736 ms 23.731 ms
6 lag-27.vinnva0510w-bcr00.netops.charter.com (66.109.6.66) 31.624 ms lag-416.vinnva0510w-bcr00.netops.charter.com (66.109.6.164) 22.959 ms 22.659 ms
7 * * lag-11.asbnva1611w-bcr00.netops.charter.com (66.109.6.30) 28.274 ms
8 lag-310.pr2.dca10.netops.charter.com (209.18.43.59) 26.474 ms lag-0.pr2.dca10.netops.charter.com (66.109.5.117) 38.678 ms lag-310.pr2.dca10.netops.charter.com (209.18.43.59) 25.672 ms
9 eqix-dc2.ohiostateuniv-oarnet.com (206.126.237.96) 26.222 ms 22.454 ms 24.771 ms
10 schrd-r5-et-2-1-0s100.core.oar.net (199.218.20.129) 36.586 ms 36.691 ms 40.143 ms
11 clmbs-r7-et-0-3-0s100.core.oar.net (199.218.20.82) 36.727 ms 41.567 ms 37.426 ms
12 clmbs-r5-et-4-0-0s100.core.oar.net (199.218.20.29) 31.434 ms 36.733 ms 31.371 ms
13 199.218.243.1 (199.218.243.1) 31.585 ms 31.283 ms 36.361 ms
14 * * *
15 cgi.galion.lib.oh.us (66.213.116.5) 37.040 ms 35.396 ms 34.729 ms
16 cgi.galion.lib.oh.us (66.213.116.5) 37.523 ms 36.246 ms 40.463 ms
Update: without the -T -p 80, it leaves off after 199.218.243.1, same as orangepizza reports. This is normal; it's been doing that for decades, and it never stopped anyone from accesssing our website.
Similar to my posted results: Can't renew with certbot or Crypt::LE, 403 during secondary validation, what? - #18 by rg305
Try also on port 443:
traceroute -T -p 443 www.galionlibrary.org
2 Likes
Now we just need a second failure point of view.
[to confirm where the blocker is located]
2 Likes
I get the same "blocked" error using HTTPS as for HTTP.
The cert used for HTTPS is below. Looks like Arista 
openssl s_client -connect www.galionlibrary.org:443
subject=CN = arista.example.com
issuer=C = US, ST = California, L = Santa Clara, O = Arista, OU = Security, CN = edge.arista.com
notBefore=Jan 9 21:32:10 2024 GMT
notAfter=Apr 9 21:32:10 2026 GMT
from this:
curl -Ik https://www.galionlibrary.org
HTTP/1.1 403 Forbidden
Content-Length: 312
Content-Type: text/html
Connection: Close
2 Likes
Ugh. I am really starting to hate that thing.
I'll give the web server a direct connection to the outside world and see if that solves it. This will take time, though (because the firewall and the web server can't both have the same public IP at the same time). A certain amount of caution is require to minimize downtime.
Which is fine. The old cert doesn't expire until Wednesday.
2 Likes
I would imagine that, in order to click the edit pen and see what was there before, you'd have to be the person who wrote the thing originally.
Not exactly - rg305
OIC.
Ok, so the web server is no longer behind our firewall.
And that was necessary, apparently, but not sufficient: the error message is now changed...
2024/04/22 16:14:23 [ Crypt::LE client v0.39 started. ]
2024/04/22 16:14:23 Loading an account key from /root/letsencrypt/keys/account_key.pem
2024/04/22 16:14:23 Account key loaded.
2024/04/22 16:14:23 Loading a CSR from /root/letsencrypt/keys/gpl-domains.csr
2024/04/22 16:14:23 Loaded domain names from CSR: www.galionlibrary.org, galionlibrary.org, www.galionlibrary.net, galionlibrary.net, www.galionlibrary.com, galionlibrary.com, cgi.galion.lib.oh.us, w
ww.galion.lib.oh.us
2024/04/22 16:14:23 CSR loaded.
2024/04/22 16:14:23 CSR key loaded
2024/04/22 16:14:23 Checking certificate for expiration (website connection).
2024/04/22 16:14:23 Checking www.galionlibrary.org
2024/04/22 16:14:23 Expiration threshold set at 30 days, the certificate expires in 2 days - will be renewing.
2024/04/22 16:14:23 Account email has been set to 'jonadab@galionlibrary.org'
2024/04/22 16:14:23 Connecting to https://acme-staging-v02.api.letsencrypt.org/directory
2024/04/22 16:14:23 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce
2024/04/22 16:14:23 Directory loaded successfully.
2024/04/22 16:14:23 Registering the account key
2024/04/22 16:14:23 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/new-acct
2024/04/22 16:14:23 Key is already registered, reg path: https://acme-staging-v02.api.letsencrypt.org/acme/acct/145352874.
2024/04/22 16:14:23 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/acct/145352874
2024/04/22 16:14:23 Account ID: 145352874
2024/04/22 16:14:23 Registration success: TOS change status - 0, new registration flag - 0.
2024/04/22 16:14:23 The key is already registered. ID: 145352874
2024/04/22 16:14:23 TOS has NOT been changed, no need to accept again.
2024/04/22 16:14:23 Current contact details: jonadab@galionlibrary.org
2024/04/22 16:14:23 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/new-order
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/finalize/145352874/16087703054
2024/04/22 16:14:24 Could not finalize an order.
2024/04/22 16:14:24 Requesting challenge.
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12107268444
2024/04/22 16:14:24 Received challenges for cgi.galion.lib.oh.us.
2024/04/22 16:14:24 Requesting challenge.
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12107268454
2024/04/22 16:14:24 Received challenges for galionlibrary.com.
2024/04/22 16:14:24 Requesting challenge.
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12107268464
2024/04/22 16:14:24 Received challenges for galionlibrary.net.
2024/04/22 16:14:24 Requesting challenge.
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12107268474
2024/04/22 16:14:24 Received challenges for galionlibrary.org.
2024/04/22 16:14:24 Requesting challenge.
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12107268484
2024/04/22 16:14:24 Received challenges for www.galion.lib.oh.us.
2024/04/22 16:14:24 Requesting challenge.
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12107268494
2024/04/22 16:14:24 Received challenges for www.galionlibrary.com.
2024/04/22 16:14:24 Requesting challenge.
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12107268504
2024/04/22 16:14:24 Received challenges for www.galionlibrary.net.
2024/04/22 16:14:24 Requesting challenge.
2024/04/22 16:14:24 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/12107268514
2024/04/22 16:14:25 Received challenges for www.galionlibrary.org.
2024/04/22 16:14:25 Requested challenges for 8 domain(s).
2024/04/22 16:14:25 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/Q7RQZ1NTFAv71LWMWmX5Ruj2898mKmhkEzfPWOk-RCw' for domain 'www.galionlibrary.org'. Sleeping 30 seconds.
2024/04/22 16:14:55 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/jf10hXKK-w25_s6HL8oYDjHR13zpmgJGcYxl0kmgFow' for domain 'galionlibrary.org'. Sleeping 30 seconds.
2024/04/22 16:15:25 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/ofGcjrIe_TwYJafSi4GSybOgdpP4W0yaNIGPATHOffw' for domain 'www.galionlibrary.net'. Sleeping 30 seconds.
2024/04/22 16:15:55 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/yzoJWoYObh09k1LSllrGiMitZBQMFBYYMJD_J3QmQVU' for domain 'galionlibrary.net'. Sleeping 30 seconds.
2024/04/22 16:16:25 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/QfDAU21qjW6Kx1LcNfOCuHI7YuVglZWxcu2OAXfXOxA' for domain 'www.galionlibrary.com'. Sleeping 30 seconds.
2024/04/22 16:16:55 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/Vstgd71R_YBUyB8lpKdsNkoNV0imMWuVRzmLNlg9m7M' for domain 'galionlibrary.com'. Sleeping 30 seconds.
2024/04/22 16:17:25 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/KZ3Z3Shi-gdfda3nQK1EIzhAx44KRvkIUiKQ_P5kedk' for domain 'cgi.galion.lib.oh.us'. Sleeping 30 seconds.
2024/04/22 16:17:55 Successfully saved a challenge file '/var/www/html/.well-known/acme-challenge/HgAltBnDAiuFaA_9JZ93YgyZXHmPZOj9IweyXTiGPvw' for domain 'www.galion.lib.oh.us'. Sleeping 30 seconds.
2024/04/22 16:18:25 Accepted challenges for 8 domain(s).
2024/04/22 16:18:25 Connecting to https://acme-staging-v02.api.letsencrypt.org/directory
2024/04/22 16:18:25 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce
2024/04/22 16:18:25 Directory loaded successfully.
2024/04/22 16:18:25 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268514/pVZhEw
2024/04/22 16:18:25 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268514/pVZhEw
2024/04/22 16:18:27 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268514/pVZhEw
2024/04/22 16:18:29 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268514/pVZhEw
2024/04/22 16:18:31 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268514/pVZhEw
2024/04/22 16:18:33 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268514/pVZhEw
2024/04/22 16:18:35 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268514/pVZhEw
2024/04/22 16:18:35 Domain verification results for 'www.galionlibrary.org': error. 66.213.116.5: Fetching http://www.galionlibrary.org/.well-known/acme-challenge/Q7RQZ1NTFAv71LWMWmX5Ruj2898mKmhkEzfPWOk-RCw: Timeout during connect (likely firewall problem)
2024/04/22 16:18:35 You can now delete the '/var/www/html/.well-known/acme-challenge/Q7RQZ1NTFAv71LWMWmX5Ruj2898mKmhkEzfPWOk-RCw' file.
2024/04/22 16:18:35 Domain www.galionlibrary.org has failed verification (status code 200).
2024/04/22 16:18:35 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268474/TImeUg
2024/04/22 16:18:35 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268474/TImeUg
2024/04/22 16:18:37 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268474/TImeUg
2024/04/22 16:18:40 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268474/TImeUg
2024/04/22 16:18:42 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268474/TImeUg
2024/04/22 16:18:44 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268474/TImeUg
2024/04/22 16:18:46 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268474/TImeUg
2024/04/22 16:18:46 Domain verification results for 'galionlibrary.org': error. 66.213.116.5: Fetching http://galionlibrary.org/.well-known/acme-challenge/jf10hXKK-w25_s6HL8oYDjHR13zpmgJGcYxl0kmgFow: Timeout during connect (likely firewall problem)
2024/04/22 16:18:46 You can now delete the '/var/www/html/.well-known/acme-challenge/jf10hXKK-w25_s6HL8oYDjHR13zpmgJGcYxl0kmgFow' file.
2024/04/22 16:18:46 Domain galionlibrary.org has failed verification (status code 200).
2024/04/22 16:18:46 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268504/87bVjg
2024/04/22 16:18:46 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268504/87bVjg
2024/04/22 16:18:48 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268504/87bVjg
2024/04/22 16:18:50 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268504/87bVjg
2024/04/22 16:18:53 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268504/87bVjg
2024/04/22 16:18:55 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268504/87bVjg
2024/04/22 16:18:57 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268504/87bVjg
2024/04/22 16:18:57 Domain verification results for 'www.galionlibrary.net': error. 66.213.116.5: Fetching http://www.galionlibrary.net/.well-known/acme-challenge/ofGcjrIe_TwYJafSi4GSybOgdpP4W0yaNIGPATHOffw: Timeout during connect (likely firewall problem)
2024/04/22 16:18:57 You can now delete the '/var/www/html/.well-known/acme-challenge/ofGcjrIe_TwYJafSi4GSybOgdpP4W0yaNIGPATHOffw' file.
2024/04/22 16:18:57 Domain www.galionlibrary.net has failed verification (status code 200).
2024/04/22 16:18:57 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268464/d4e67w
2024/04/22 16:18:57 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268464/d4e67w
2024/04/22 16:18:59 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268464/d4e67w
2024/04/22 16:19:01 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268464/d4e67w
2024/04/22 16:19:03 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268464/d4e67w
2024/04/22 16:19:05 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268464/d4e67w
2024/04/22 16:19:07 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268464/d4e67w
2024/04/22 16:19:08 Domain verification results for 'galionlibrary.net': error. 66.213.116.5: Fetching http://galionlibrary.net/.well-known/acme-challenge/yzoJWoYObh09k1LSllrGiMitZBQMFBYYMJD_J3QmQVU: Timeout during connect (likely firewall problem)
2024/04/22 16:19:08 You can now delete the '/var/www/html/.well-known/acme-challenge/yzoJWoYObh09k1LSllrGiMitZBQMFBYYMJD_J3QmQVU' file.
2024/04/22 16:19:08 Domain galionlibrary.net has failed verification (status code 200).
2024/04/22 16:19:08 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268494/AA5mkA
2024/04/22 16:19:08 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268494/AA5mkA
2024/04/22 16:19:10 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268494/AA5mkA
2024/04/22 16:19:12 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268494/AA5mkA
2024/04/22 16:19:14 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268494/AA5mkA
2024/04/22 16:19:16 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268494/AA5mkA
2024/04/22 16:19:18 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268494/AA5mkA
2024/04/22 16:19:18 Domain verification results for 'www.galionlibrary.com': error. 66.213.116.5: Fetching http://www.galionlibrary.com/.well-known/acme-challenge/QfDAU21qjW6Kx1LcNfOCuHI7YuVglZWxcu2O
AXfXOxA: Timeout during connect (likely firewall problem)
2024/04/22 16:19:18 You can now delete the '/var/www/html/.well-known/acme-challenge/QfDAU21qjW6Kx1LcNfOCuHI7YuVglZWxcu2OAXfXOxA' file.
2024/04/22 16:19:18 Domain www.galionlibrary.com has failed verification (status code 200).
2024/04/22 16:19:18 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268454/ZBOAxg
2024/04/22 16:19:18 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268454/ZBOAxg
2024/04/22 16:19:20 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268454/ZBOAxg
2024/04/22 16:19:23 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268454/ZBOAxg
2024/04/22 16:19:25 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268454/ZBOAxg
2024/04/22 16:19:27 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268454/ZBOAxg
2024/04/22 16:19:29 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268454/ZBOAxg
2024/04/22 16:19:29 Domain verification results for 'galionlibrary.com': error. 66.213.116.5: Fetching http://galionlibrary.com/.well-known/acme-challenge/Vstgd71R_YBUyB8lpKdsNkoNV0imMWuVRzmLNlg9m7M:
Timeout during connect (likely firewall problem)
2024/04/22 16:19:29 You can now delete the '/var/www/html/.well-known/acme-challenge/Vstgd71R_YBUyB8lpKdsNkoNV0imMWuVRzmLNlg9m7M' file.
2024/04/22 16:19:29 Domain galionlibrary.com has failed verification (status code 200).
2024/04/22 16:19:29 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268444/M-Modw
2024/04/22 16:19:29 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268444/M-Modw
2024/04/22 16:19:31 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268444/M-Modw
2024/04/22 16:19:33 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268444/M-Modw
2024/04/22 16:19:35 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268444/M-Modw
2024/04/22 16:19:38 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268444/M-Modw
2024/04/22 16:19:40 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268444/M-Modw
2024/04/22 16:19:40 Domain verification results for 'cgi.galion.lib.oh.us': error. 66.213.116.5: Fetching http://cgi.galion.lib.oh.us/.well-known/acme-challenge/KZ3Z3Shi-gdfda3nQK1EIzhAx44KRvkIUiKQ_P
5kedk: Timeout during connect (likely firewall problem)
2024/04/22 16:19:40 You can now delete the '/var/www/html/.well-known/acme-challenge/KZ3Z3Shi-gdfda3nQK1EIzhAx44KRvkIUiKQ_P5kedk' file.
2024/04/22 16:19:40 Domain cgi.galion.lib.oh.us has failed verification (status code 200).
2024/04/22 16:19:40 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268484/iJPfkg
2024/04/22 16:19:40 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268484/iJPfkg
2024/04/22 16:19:42 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268484/iJPfkg
2024/04/22 16:19:44 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268484/iJPfkg
2024/04/22 16:19:46 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268484/iJPfkg
2024/04/22 16:19:48 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268484/iJPfkg
2024/04/22 16:19:50 Connecting to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/12107268484/iJPfkg
2024/04/22 16:19:51 Domain verification results for 'www.galion.lib.oh.us': error. 66.213.116.5: Fetching http://www.galion.lib.oh.us/.well-known/acme-challenge/HgAltBnDAiuFaA_9JZ93YgyZXHmPZOj9IweyXT
iGPvw: Timeout during connect (likely firewall problem)
2024/04/22 16:19:51 You can now delete the '/var/www/html/.well-known/acme-challenge/HgAltBnDAiuFaA_9JZ93YgyZXHmPZOj9IweyXTiGPvw' file.
2024/04/22 16:19:51 Domain www.galion.lib.oh.us has failed verification (status code 200).
2024/04/22 16:19:51 All verifications failed
2024/04/22 16:19:51 All verifications failed
Mon_2024-Apr-22_16:19
root@cogitation:~/letsencrypt#
1 Like
What say the firewall logs?
2 Likes
There is no longer a firewall in the path. Well, not on our end. The web server now has 66.213.116.5 as its own, on eth2, and talks directly to the Juniper router (66.213.116.1) provided by the OPLIN. The Arista firewall (66.213.116.2 among others) no longer has 66.213.116.5 bound to its interface.
Mon_2024-Apr-22_16:45
root@cogitation:~# ifconfig ; route -n ; iptables -S
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.0.55.10 netmask 255.255.255.0 broadcast 10.0.55.255
inet6 fe80::d63d:7eff:fe55:7621 prefixlen 64 scopeid 0x20
ether d4:3d:7e:55:76:21 txqueuelen 1000 (Ethernet)
RX packets 3308661 bytes 1547085105 (1.4 GiB)
RX errors 0 dropped 4 overruns 0 frame 0
TX packets 3927299 bytes 2603481914 (2.4 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 66.213.116.5 netmask 255.0.0.0 broadcast 66.213.116.31
inet6 fe80::20a:cdff:fe20:cacc prefixlen 64 scopeid 0x20
ether 00:0a:cd:20:ca:cc txqueuelen 1000 (Ethernet)
RX packets 84325 bytes 12337070 (11.7 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 58056 bytes 22660543 (21.6 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10
loop txqueuelen 1000 (Local Loopback)
RX packets 2846 bytes 515257 (503.1 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2846 bytes 515257 (503.1 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 66.213.116.1 0.0.0.0 UG 2 0 0 eth2
10.0.0.0 10.0.55.1 255.0.0.0 UG 1 0 0 eth0
10.0.55.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
66.0.0.0 0.0.0.0 255.0.0.0 U 0 0 0 eth2
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
Mon_2024-Apr-22_16:45
root@cogitation:~# ping -c 3 66.213.116.1
PING 66.213.116.1 (66.213.116.1) 56(84) bytes of data.
64 bytes from 66.213.116.1: icmp_seq=1 ttl=64 time=0.276 ms
64 bytes from 66.213.116.1: icmp_seq=2 ttl=64 time=0.505 ms
64 bytes from 66.213.116.1: icmp_seq=3 ttl=64 time=0.303 ms
--- 66.213.116.1 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2043ms
rtt min/avg/max/mdev = 0.276/0.361/0.505/0.102 ms
Mon_2024-Apr-22_16:46
root@cogitation:~# traceroute 174.105.114.155
traceroute to 174.105.114.155 (174.105.114.155), 30 hops max, 60 byte packets
1 66.213.116.1 (66.213.116.1) 0.436 ms 0.401 ms 0.378 ms
2 10.213.11.61 (10.213.11.61) 2.737 ms 2.722 ms 2.865 ms
3 clmbs-r5-ae2s440.core.oar.net (199.218.243.0) 2.806 ms 2.836 ms 2.876 ms
4 clmbn-r4-et-4-3-0s100.core.oar.net (199.218.20.34) 3.438 ms 3.427 ms 3.403 ms
5 chcge-r5-et-1-1-0s100.bb.oar.net (199.218.20.74) 10.541 ms 10.531 ms 10.509 ms
6 eqix-ch2.timewarnerny.com (208.115.136.70) 21.588 ms 10.581 ms 10.551 ms
7 lag-111.chcgildt87w-bcr00.netops.charter.com (66.109.5.224) 10.729 ms * *
8 lag-41.chctilwc00w-bcr00.netops.charter.com (66.109.0.229) 11.102 ms lag-31.chctilwc00w-bcr00.netops.charter.com (66.109.10.83) 11.545 ms 11.025 ms
9 lag-3.rcr01clmkohpe.netops.charter.com (66.109.6.55) 20.204 ms 20.353 ms lag-1.rcr01clmkohpe.netops.charter.com (66.109.6.69) 20.395 ms
10 lag-2.mcr11clmkohpe.netops.charter.com (65.29.17.197) 20.530 ms 20.523 ms 20.499 ms
11 lag-1.wwtsoh0201h.netops.charter.com (24.33.161.153) 34.836 ms 34.783 ms 33.277 ms
12 agg1.wwtsoh0203m.midohio.rr.com (65.25.145.34) 22.829 ms 22.810 ms 22.724 ms
...
1 Like
traceroute -T -p 443 www.galionlibrary.org
traceroute to www.galionlibrary.org (66.213.116.5), 30 hops max, 60 byte packets
...
13 schrd-r5-et-2-0-4s100.core.oar.net (199.218.39.241) 37.891 ms clmbn-r4-et-2-1-1s100.core.oar.net (199.218.20.105) 39.122 ms 38.932 ms
14 clmbs-r5-et-3-0-0s100.core.oar.net (199.218.20.33) 39.127 ms 37.924 ms 38.610 ms
15 199.218.243.1 (199.218.243.1) 39.446 ms 39.069 ms clmbs-r5-et-4-0-0s100.core.oar.net (199.218.20.29) 39.316 ms
16 10.213.11.62 (10.213.11.62) 43.263 ms 199.218.243.1 (199.218.243.1) 38.650 ms 10.213.11.62 (10.213.11.62) 40.912 ms
17 10.213.11.62 (10.213.11.62) 40.875 ms 41.257 ms cgi.galion.lib.oh.us (66.213.116.5) 40.729 ms
It is now one hop shorter - that confirms your change..
But it never blocked me; So, I'm not much help with troubleshooting this part.
I can only say that...
It remains high on the list that something remains that is partial/particular about the source IPs.
3 Likes
I don't have any more diagnosis to add than what @rg305 said. It isn't just Rudy that can now see you I am also not blocked anymore. One of Let's Debug two tests is not blocked anymore either. It does an HTTP request from its own server and gets a proper response. The second test with Let's Encrypt Staging system gets timeout same as you. You have to click Verbose output to see this like here:
If this were a new problem in a new thread I would say 99% that it is an IP based firewall. But, given your previous posts I am puzzled.
4 Likes
I'm no longer puzzled.
Co-in-ci-dence
- They recently added a new firewall [which has been proven partly to blame]
- LE recently changed its' Multi-Perspective Validation [which now includes two new countries]
See: Let's Encrypt is adding two new remote perspectives for domain validation - API Announcements - Let's Encrypt Community Support (letsencrypt.org)
I say:
We were dealing with two problems - that are very similar and just happen to overlap each other in this 60 day renewal period.
3 Likes
The current problem looks like a firewall blocking the Primary center. But the Arista one seemed to allow the Primary and only blocked the Secondary. And, their server wasn't blocking the Primary else we shouldn't have seen the Secondary failure before. Or maybe those sequences are not as clear cut as I imagine.
IOW, how could one less piece of equipment now fail on Primary?
That's what is confusing me.
4 Likes
And now that's also confusing me...
But I still think it's more than "one" single problem.
This last problem might be related to the move to the "router" [which may have some sort of ACL/IPS that has gone unnoticed]. So, it may have added to the problem as much as it removed - LOL
4 Likes
The router was there all along. The path used to be this:
web server (10.0.55.10) <==> Firewall (66.213.116.x) <==> Juniper Router (66.213.116.1) <==> OPLIN
Now the path is this:
web server (66.213.116.5) <==> Juniper Router (66.213.116.1) <==> OPLIN
This is, incidentally, the same configuration we previously had before the new firewall was installed, and renewal was working, with this setup, until relatively recently. (Yes, I'm aware there have been perspective changes on LE's end.)
So I'm not clear on how anything new could be going wrong that was going right before. More likely whatever is going wrong now, either didn't have a chance to go wrong before (because the firewall was getting in the way and preventing it from happening at all), or else the other problem was masking it somehow, preventing us from seeing it.
For the moment, I'm thinking of doing DNS-based validation mostly-manually for the NetSol-registered domains, for the time being, to kick the deadline down the road a bit. I don't think anything is actually using https on the .lib.oh.us domains at present. (Editing the zone file for that, requires communicating with someone in Columbus, but it's less important right now in any case.)
3 Likes