Only the US and apparently also Canada on your side of the world. Please use proper (international) formatting like most of the rest of the world 
3 Likes
I do. I have not changed or updated entries for a long time so they should still be the same as before.
2 Likes
Do you implement some sort of IPS or Fail2BAN system?
3 Likes
So when I do a DNS lookup from various servers I get a valid response. I also have a valid A record setup for crm.ilines.net on my server. I do not have an AAAA as I do not use IPV6.
I do not have an IPS or Fail2BAN setup. I do run an ISP so our Edge firewall does have some automatic IP blocking that has been in place for years also I subscribe to IP block lists.
Any IP addresses I need to look for to see if blocked for some reason?
1 Like
Here are the results of a DNS lookup from MXtoolbox and one from DNSchecker. Both seem to look fine.
1 Like
There is no wrong way of writing it.
IMHO, the only "wrong way" is by not accepting any way that is not yours.
Case in point:
Some people write left to right and some write right to left - is there a wrong way to write?
No; I don't think so.
4 Likes
This is still an international Community, I recommend to adhere the most widely used format(s) whenever possible to avoid any confusion. crt.sh also shows the dates in yyyy-mm-dd on their search page. 
Nothing wrong with writing the other way around, it's just absolutely bonkers to mess up the order. Logic dictates you're either going from "largest" to "smallest", i.e. year -> month -> day. Turning around the days and months is just.. Plain silly.
A very good question. Usually SERVFAIL errors are due to DNSSEC errors, but DNSViz cannot find any and UnboundTest also returns an A record without any issue.
3 Likes
Ok. I see your point and will try to remember date format going forward.
Anyway why am I getting these errors, or why does let's encrypt claim DNS errors when other DNS lookup sites show no issues?
1 Like
As far as I can tell looking at the Let's encrypt logs on the server this issue began on 2022-08-12 06:42:36 and has continued since.
It seems to have succeeded at 2022-08-12 06:42:02
2022-08-12 06:42:01,518:DEBUG:certbot.main:certbot version: 0.35.1
2022-08-12 06:42:01,518:DEBUG:certbot.main:Arguments: ['--config', '/data/ssl/certbot.ini']
2022-08-12 06:42:01,518:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-08-12 06:42:01,606:DEBUG:certbot.log:Root logging level set at 30
2022-08-12 06:42:01,606:INFO:certbot.log:Saving debug log to /data/log/ucrm/letsencrypt/letsencrypt.log
2022-08-12 06:42:01,618:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2022-08-12 06:42:01,623:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2022-08-12 06:42:01,978:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2022-08-12 06:42:01,979:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 12 Aug 2022 06:42:01 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
{
"QDd59GIe6jA": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2022-08-12 06:42:01,980:DEBUG:acme.client:Requesting fresh nonce
2022-08-12 06:42:01,980:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2022-08-12 06:42:02,033:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2022-08-12 06:42:02,034:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 12 Aug 2022 06:42:02 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: 00019wdS0QGAGHzOLPUTe6BxNZHiv6JWfkRHNPxog3GfYBE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
2022-08-12 06:42:02,034:DEBUG:acme.client:Storing nonce: 00019wdS0QGAGHzOLPUTe6BxNZHiv6JWfkRHNPxog3GfYBE
2022-08-12 06:42:02,035:DEBUG:acme.client:JWS payload:
b'{\n "contact": [\n "mailto:ilines_support@lebanon-utilities.com"\n ],\n "onlyReturnExisting": true\n}'
2022-08-12 06:42:02,041:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-acct:
{
"protected": "eyJhbGciOiAiUlMyNTYiLCAiandrIjogeyJuIjogIndCVHJRdmtZbnRUbmJxbXhiQm1tN0lRdGVJeFduV3FSN2lSbDJFSF94VjVCTjZKX1dYVExkTG1jeUdvdnhQRWMxTVd0ZUhzTkRTaG1BSFdWS0xHQXZzYW5LM2lXd1g1OERMVlpSVFJPWHZobG5zYkNLYW9HVzM4YTlQalRBWm9TcEFEbWR0QVI3a0ZxbWd0TDE5Rzk5dTYxaXpSWEM5RlBqZXJNNFJ5bm51Y1JrZ3k5cGxYX0M4T29qazZXZ3pqNDZrMUFVQ0t2bDhldWxaeGREU3h0ZU5JTUMwY3o5cTRubkdVNHVuX1EyMmtFcVp0a2pDU1prc3JWaG5LNDFIZFdEMUVUZnAtUTNKM2JaUC14RDltZkVCZzQ2LUdYYVZrSUJwMm03LXptRUtFV2pJamo0WG95cmUxbVpTQnlQYXpMRHI4alNBdF9TWVdxTERJNGNYbW1kdyIsICJlIjogIkFRQUIiLCAia3R5IjogIlJTQSJ9LCAibm9uY2UiOiAiMDAwMTl3ZFMwUUdBR0h6T0xQVVRlNkJ4TlpIaXY2Sldma1JITlB4b2czR2ZZQkUiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1hY2N0In0",
"signature": "SP0qTILSQJJGzXjCXdFBX6wvanbrcu3KyaHYRICDbWi45aTHVoOUdyZXyrqilRZtH6nPAlB8Rq8eZNOi3Z5oAvBGcQ4pXrQhgl96xCLnBNceBdf-w9Im7Bh5G6vp2Zq8KHIWytwubFI6PeZ2c2k3VnCuovbD2VCKADtM3L6xUZgM8nuW5OW4YAFfMr27z0vpYleRNX-qNdNh2GUK42gb1HsD67AGpp-02fdoK41JQoaFRPW_74xV6Q0A9WXG0j0qrb6kt5xxPS7VLd8zj4XUfe0NMKG0pFp9rhr0Ea4MeVYi5sRUFFKqSNL7RmfOa3CDo7QqHjIlUzMdvBjLIjAOHA",
"payload": "ewogICJjb250YWN0IjogWwogICAgIm1haWx0bzppbGluZXNfc3VwcG9ydEBsZWJhbm9uLXV0aWxpdGllcy5jb20iCiAgXSwKICAib25seVJldHVybkV4aXN0aW5nIjogdHJ1ZQp9"
}
2022-08-12 06:42:02,099:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-acct HTTP/1.1" 200 569
2022-08-12 06:42:02,100:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 12 Aug 2022 06:42:02 GMT
Content-Type: application/json
Content-Length: 569
Connection: keep-alive
Boulder-Requester: 163960310
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Location: https://acme-v02.api.letsencrypt.org/acme/acct/163960310
Replay-Nonce: 0002nmYV4Eishl2DLzjdCBCIJL_GHzmdIIvSy4MtnE7OvO8
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
{
"key": {
"kty": "RSA",
"n": "wBTrQvkYntTnbqmxbBmm7IQteIxWnWqR7iRl2EH_xV5BN6J_WXTLdLmcyGovxPEc1MWteHsNDShmAHWVKLGAvsanK3iWwX58DLVZRTROXvhlnsbCKaoGW38a9PjTAZoSpADmdtAR7kFqmgtL19G99u61izRXC9FPjerM4RynnucRkgy9plX_C8Oojk6Wgzj46k1AUCKvl8eulZxdDSxteNIMC0cz9q4nnGU4un_Q22kEqZtkjCSZksrVhnK41HdWD1ETfp-Q3J3bZP-xD9mfEBg46-GXaVkIBp2m7-zmEKEWjIjj4Xoyre1mZSByPazLDr8jSAt_SYWqLDI4cXmmdw",
"e": "AQAB"
},
"contact": [
"mailto:ilines_support@lebanon-utilities.com"
],
"initialIp": "66.103.104.23",
"createdAt": "2021-08-16T18:42:04Z",
"status": "valid"
}
2022-08-12 06:42:02,101:DEBUG:acme.client:Storing nonce: 0002nmYV4Eishl2DLzjdCBCIJL_GHzmdIIvSy4MtnE7OvO8
2022-08-12 06:42:02,101:DEBUG:acme.client:JWS payload:
b'{\n "contact": [\n "mailto:ilines_support@lebanon-utilities.com"\n ],\n "resource": "reg"\n}'
2022-08-12 06:42:02,104:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/acct/163960310:
{
"protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvMTYzOTYwMzEwIiwgIm5vbmNlIjogIjAwMDJubVlWNEVpc2hsMkRMempkQ0JDSUpMX0dIem1kSUl2U3k0TXRuRTdPdk84IiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzE2Mzk2MDMxMCJ9",
"signature": "Q2AS06f0uP3bu16fqN4XB5CQMAXp6GAlVJZgooR7B9cIjHmWDWMP0U-i1BN3-AAsPvfaf4O885s8ywBphFRYzPsLrjSW5Ny1vr06oA-WwSY9OU5YmRARG93g31F4_1CJ5kbS8LSpB-uIR04bg2AJbqqn6WP8fRw0Sh3Kb58NoAN4yLLlABpHS9tra8BE10VxeERujiTAjzUG6Acg6ydXmLHGZ6XmbA9ocC1F8I_5QSpdkNRwnU7AhppytRsXEkHtnfHJHdsInmv_NkryATapJ5-CEkjlcvn7creF9r-BiriTqdH1-AMQPpFF1XD5KquspiDYsPna17TGi9dB_m5J9w",
"payload": "ewogICJjb250YWN0IjogWwogICAgIm1haWx0bzppbGluZXNfc3VwcG9ydEBsZWJhbm9uLXV0aWxpdGllcy5jb20iCiAgXSwKICAicmVzb3VyY2UiOiAicmVnIgp9"
}
2022-08-12 06:42:02,168:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/acct/163960310 HTTP/1.1" 200 569
2022-08-12 06:42:02,169:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 12 Aug 2022 06:42:02 GMT
Content-Type: application/json
Content-Length: 569
Connection: keep-alive
Boulder-Requester: 163960310
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index", https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf;rel="terms-of-service"
Replay-Nonce: 0002W8Y5X1ZiOdWbyv5de6b2l-V9dmM1wF3T-eFMwOgx1Gg
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
{
"key": {
"kty": "RSA",
"n": "wBTrQvkYntTnbqmxbBmm7IQteIxWnWqR7iRl2EH_xV5BN6J_WXTLdLmcyGovxPEc1MWteHsNDShmAHWVKLGAvsanK3iWwX58DLVZRTROXvhlnsbCKaoGW38a9PjTAZoSpADmdtAR7kFqmgtL19G99u61izRXC9FPjerM4RynnucRkgy9plX_C8Oojk6Wgzj46k1AUCKvl8eulZxdDSxteNIMC0cz9q4nnGU4un_Q22kEqZtkjCSZksrVhnK41HdWD1ETfp-Q3J3bZP-xD9mfEBg46-GXaVkIBp2m7-zmEKEWjIjj4Xoyre1mZSByPazLDr8jSAt_SYWqLDI4cXmmdw",
"e": "AQAB"
},
"contact": [
"mailto:ilines_support@lebanon-utilities.com"
],
"initialIp": "66.103.104.23",
"createdAt": "2021-08-16T18:42:04Z",
"status": "valid"
}
2022-08-12 06:42:02,169:DEBUG:acme.client:Storing nonce: 0002W8Y5X1ZiOdWbyv5de6b2l-V9dmM1wF3T-eFMwOgx1Gg
2022-08-12 06:42:02,171:DEBUG:certbot.reporter:Reporting to user: Your e-mail address was updated to ilines_support@lebanon-utilities.com.
1 Like
It "feels" like a firewall blocking certain IP addresses. It is also affecting HTTP requests.
Example ... using the Let's Debug test site detail info, you can see its own HTTP to your server got through (fails with expected 404). But, when it has the Let's Encrypt staging server try your site it times out connecting to it. rg305 test saw it timing out to your DNS but this test shows the HTTP request failing.
I can repeatedly get your site with HTTP just fine and the DNS too. So, it seems specific to Let's Encrypt servers so likely IP
I even tried requests using the same user-agent as LE servers and those worked fine.
The unboundtest.com and dnsviz.net we often use for looking at DNS both see your site fine too.
5 Likes
No, the last time it worked was Jun12 2022. See the crt.sh log here
The log you show is not complete. There are many more requests involved in getting a cert.
So, something since Jun12 changed. Not that this info is that helpful
5 Likes
Help me understand. When I checked the Let's Debug page it references the below site.
https://crm.ilines.net:443/.well-known/acme-challenge/letsdebug-test
However that site does not exist if I put it into a browser.
Also I have no rules anywhere in my network or firewalls that would block HTTP. I just have IP blocks that are based off of either IP block lists or from possible attacks. What are the source IPs used by Let's Encrypt so I can check to see if they are in my lists.
Also I did turn off my firewall Drop rules for "hackers" and it still failed.
1 Like
Well, the site exists but that page does not. So, if you get your server's "Not Found" page that means the connection to your site worked. It's fine that the page wasn't there. We only want to check connectivity to your server.
Contrast that to the test using the Let's Encrypt staging server just below that. It says the connect to your site times out. Very different failure.
Let's Encrypt IP addresses are not published and subject to change even hour-to-hour
5 Likes
Odd then. Any automatic IP blocks expire between 2 hours to 30 days depending on what triggered the block. I checked my DNS servers and I do not have IP blocking enabled on them and their block lists are empty.
So If I turned off all my IP blocks and still get this then what should I explore next?
1 Like
Is there any network gear between your servers and the ISP? Do you control the network gear from your devices out to the actual ISP?
Could one of those have some sort of IP based firewall?
As an example, all year we have been fighting problems with firewalls from vendor Palo Alto Networks. They added a new setting in their firewall which blocked the specific format of ACME URL challenges. How they do this even varies between various of their models. Has been challenging at times.
I don't see that your pattern matches what we saw there. Yours is different. I mention this only as an example of something that changed that was not obvious to their customer (or us!).
5 Likes
I just found a firewall rule on my private servers firewall that would only allow TCP access to uCRM from IP addresses in America, Canada, and Mexico. Once I removed the Source IP list and allowed any IP the test passed and I believe the cert succeded.
My guess is either my list of IPs is out of date for those countries or Let's Encrypt uses addresses from other continents. Not sure when I put that rule in place but I do not remember making any changes this year.
3 Likes
Yes, currently there is one server farm in Europe that does validations. Might change any time. They do global validation.
4 Likes
Some people even use vertical writing.
3 Likes
Ok. So my firewall rule was one rule checking multiple ports. I have now changed the rule and moved ports 80 and 443 to a separate rule with no blocks. The other ports on this server are back to using IP blocking by region.
4 Likes
Sounds good. I see a new cert too
Note that Let's Encrypt caches successful challenges for 30 days (per domain per your account). Just beware that any renewals attempted early (so in the next 30 days) would succeed because of this cache even if you modified your firewall to block.
If you add --dry-run to your certbot command that forces fresh challenges using the staging system. You can do that to test firewall changes to make sure it works without getting production certs.
5 Likes