Unable to get a cert

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: corp.networkingtechnology.org

I ran this command:certbot -v

It produced this output:certbot -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
An unexpected error occurred:
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-
v02.api.letsencrypt.org', port=443): Max retries exceeded with url:
/directory (Caused by
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object
at 0x7fd2191ba6a0>: Failed to establish a new connection: [Errno -2]
Name or service not known',))
Ask for help or search for solutions at
https://community.letsencrypt.org. See the logfile
/var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more
details.

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

I'm picking up my previous thread about "have hackers blocked Let's encrypt.

I created a rule on my firewall to allow all traffic on Ports 80 and 443 to my mail server hermes.corp.networking technology.org (pegasus and alma-86 as well). (My MX record on Networkingtechnology.org points to this server). It had no effect
I UNblocked ALL traffic from ALL AWS addresses. It had NO effect.
I disabled the firewall and get this:
certbot -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
An unexpected error occurred:
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='acme-
v02.api.letsencrypt.org', port=443): Max retries exceeded with url:
/directory (Caused by
NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object
at 0x7fd2191ba6a0>: Failed to establish a new connection: [Errno -2]
Name or service not known',))
Ask for help or search for solutions at
https://community.letsencrypt.org. See the logfile
/var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more
details.
So it isn't the firewall causing the problem

2

The problem is not inbound; It outbound:

Which may actually be caused by a DNS issue:

5 Likes
3

I'm not sure if my age is causing my English to fail me or whether there are preconceived ideas or is it always blame the user and never our system.

Let me try once more. I installed let's encrypt two (2) years ago. I installed the OPNSense firewall BC (before Covid). For all this time, certificates have been renewed and I've never had a problem.

I've been getting (literally) hundreds of attacks by a bot or bots tryng to brute force guess passwords of all the mail users.

I installed Fail2Ban and at the end of the day. I took all the offenders from Fail2Ban and transferred them to the OPNsense firewall. This has been going on for several weeks. Prior to that I only put IPs on the firewall of persistent 404 offenders. Digital Ocean, Amazon and googleusercontent being the worst.

When I noticed that the certificates weren't being renewed. My first thought wass "have I blocked the acme-challenge servers, OR have they found a way to block my server."

When you implied that the entire LetsEncrypt acme_challenge is run by Amazon, I checked how many blocked amazon IP addesses I have. It's a lot (I don't block subnets other than pests on Digital Ocean).

To troubleshoot,

  1. I UN-blocked all the AWS IP's
  2. I created a WAN firewall rule allowing all incoming traffic to my three servers on ports 80 and 443. By default NO traffic is blocked outgoing and I haven't changed that.
  3. The only thing that's changed on any of the three servers is installing updates. Othert han that, I don't do anything 'clever'. 2 websites and a mail server with postfix and Dovecot.
  4. As a last attempt. I disabled the firewall completely and still nothing has changed.

dig acme-v02.api.letsencrypt.org

; <<>> DiG 9.11.36-RedHat-9.11.36-11.el8_9.1 <<>> acme-
v02.api.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: FORMERR, id: 16823
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
; COOKIE: 521eb6b59732065e (echoed)
;; QUESTION SECTION:
;acme-v02.api.letsencrypt.org. IN A

;; Query time: 1 msec
;; SERVER: 192.168.0.240#53(192.168.0.240)
;; WHEN: Tue May 14 17:05:49 CEST 2024
;; MSG SIZE rcvd: 69

I'm TRYING, seems with difficulty, to make you understand this is not a new installation, It's not that it's a new installation of Let's Encrypt. I don't try to do clever things on either the firewall or the servers. They work fine. This happened out of the blue.

Here's PROOF this is something new and there is NO reason for it.

Found the following certs:
Certificate Name: books.corp.networkingtechnology.org
Serial Number: 423d9362a3b617e942567a2f04bf4e3c78a
Key Type: RSA
Domains: books.corp.networkingtechnology.org
Expiry Date: 2024-05-25 19:15:42+00:00 (VALID: 11 days)
Certificate Path: /etc/letsencrypt/live/books.corp.networkingtechnology.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/books.corp.networkingtechnology.org/privkey.pem
Certificate Name: corp.networkingtechnology.org
Serial Number: 323945d07ac67f226b6356d97377a6dafdf
Key Type: RSA
Domains: corp.networkingtechnology.org
Expiry Date: 2024-05-24 15:09:03+00:00 (VALID: 10 days)
Certificate Path: /etc/letsencrypt/live/corp.networkingtechnology.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/corp.networkingtechnology.org/privkey.pem
Certificate Name: support.corp.networkingtechnology.org
Serial Number: 318d192adffa63cf6a54fa310c223c17949
Key Type: RSA
Domains: support.corp.networkingtechnology.org
Expiry Date: 2024-05-24 15:09:08+00:00 (VALID: 10 days)
Certificate Path: /etc/letsencrypt/live/support.corp.networkingtechnology.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/support.corp.networkingtechnology.org/privkey.pem
Certificate Name: writers.corp.networkingtechnology.org
Serial Number: 41bd7842f01956014c8c4997451e67ef333
Key Type: RSA
Domains: writers.corp.networkingtechnology.org
Expiry Date: 2024-05-24 15:09:13+00:00 (VALID: 10 days)
Certificate Path: /etc/letsencrypt/live/writers.corp.networkingtechnology.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/writers.corp.networkingtechnology.org/privkey.pem

You should be able to see from the age of the certificates how long they have been without renewal. The above was from Sunday 12th

4

We don't need to review your history or other problems.

The current problem is that your system's DNS resolver is not working. The dig command you show fails with FORMERR

You need to have that working so you can make outbound requests from your server using a domain name. Like when you request a cert you need to make API requests to the Let's Encrypt server(s).

Try other examples of dig to prove to yourself this is a general problem on your system and not unique to Let's Encrypt.

dig +noall +answer google.com
google.com.             119     IN      A       172.253.62.102
google.com.             119     IN      A       172.253.62.113
(others omitted for brevity)

# You should see something like this 
dig +noall +answer acme-v02.api.letsencrypt.org
acme-v02.api.letsencrypt.org. 300 IN    CNAME   prod.api.letsencrypt.org.
prod.api.letsencrypt.org. 65    IN      CNAME   ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com. 222 IN A 172.65.32.248
4 Likes
5

I don't know this "FORMERR" status code, but it looks like the IP address

does not want to resolve this hostname for your server.

That's what Rudy was pointing at: your server has trouble resolving the ACME servers hostname, the first step before even trying to connect to the ACME server.

2 Likes
6

I'm surprised that no one has highlighted this critical warning yet. That suggests that the nameserver at 192.168.0.240 is not a recursive resolver which is clearly going to be problematic.

4 Likes
7

It doesn't want to resolve anyway from the looks of it, recursive or not :stuck_out_tongue:

That said, maybe it just needs to have it's recursive resolving feature turned "on" or something like that :man_shrugging:t2: I'm not sure what the error code in the "status" field becomes when a fully functional DNS server refuses to do a recursive lookup when requested. :arrow_right: My own BIND uses the "REFUSED" status. :slight_smile:

1 Like
8

What I'm trying to get to grips with is why suddenly out of the blue, it stopped working. I've done NOTHING, nada, zilch to anything because I'm an editor and trying to finish someone's book. I don't have either the time or the inclination to 'fiddle' with things when they are wortking. "If it ain't broken, don't fix it."

How am I supposed to troublsheet something which is outside of my influence. WFT do I even start to solve the problem.

It seems no one there has a solution. It's all very well telling me what isn't working when its got to be something I CAN'T FIX.

The solution is out there, but it isn't here on my system

9

As other replies have pointed out: when you run DNS lookups, they attempt to use 192.168.0.240 as a resolver, and that is failing.

That is a private IP address, so it's somewhere on your local network. What is that device? Is that your server itself, a firewall, or something else?

Once that's identified, you will need to look at the configuration of the DNS service that's running on that device. That will be the next step to see why your DNS lookups are failing.

5 Likes
10

One method of (first step of) debugging might be to set your systems DNS server temporarily to a known working recursively resolving DNS server such as 1.1.1.1 (Cloudflare).

That said, James' suggested approach in the end is more worthwhile. The above is meant to show you it's actually something you can controle/fix and not something on the end of Let's Encrypt.

Now I'm hoping there isn't a firewall blocking DNS requests to 1.1.1.1 :rofl:

2 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.