Certbot unable to issue certificate

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
stage2.liveexperience.io

I ran this command:
sudo certbot certonly --standalone

It produced this output:
Timeout during connect (likely firewall problem)

My web server is (include version):
Node 17.9.1

The operating system my web server runs on is (include version):
Ubuntu 18.04

My hosting provider, if applicable, is:
Azure

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Certbot 1.31.0

Here is the context of the issue: my company has an Azure virtual machine that we use as a reverse proxy to send requests to their designated locations. This application also attaches the SSL certificate to those requests as well. Currently, we use cloudflare which serves to DNS proxy requests to our reverse proxy. Our reverse proxy is a Node application. That application uses a deprecated certificate module called "greenlock-express" to renew the certificates. Thankfully, the module is still working, but due to irrelevant circumstances, some of the certificates were lost and replaced by old ones. Unfortunately, it looks like our automatic renewal system won't renew them because it's waiting for the date that it should have had to renew them had they not been lost. So, after trying literally dozens of different solutions, I think that the best way to do this would just be to temporarily shut down our Node server and run the Certbot server to renew the certificates we lost. Unfortunately, I'm getting this weird time-out error when I do so. I know that port 80 is open and works because our Node application is able to renew the certificates, I just can't figure out why that works and Certbot doesn't.

I stopped running the node server application. I will instead spin up the Certbot server and let it run indefinitely with the "--debug-challenge" option so that anyone who helps me can get some insight.

I REALLY appreciate any help! I've spent about 30 hours banging my head on this issue.

1 Like
2

Hello @ThomasWithYondo, welcome to the Let's Encrypt community. :slightly_smiling_face:

Here is a list of issued certificates crt.sh | stage2.liveexperience.io, the most recent being 2022-10-16

image
image1060×335 19.2 KB

Best Practice - Keep Port 80 Open

Testing and debugging are best done using the Staging Environment as the Rate Limits are much higher. Rate Limits are per week (rolling).

And to assist with debugging there is a great place to start is Let's Debug,

1 Like
3

I can't reach the certbot standalone server. Is it still running? A port scan shows this

nmap -p22,80,443 stage2.liveexperience.io  -Pn
PORT    STATE    SERVICE
22/tcp  filtered ssh
80/tcp  filtered http
443/tcp filtered https

and an http request does this (not suprisingly)

curl -I -m10  stage2.liveexperience.io
curl: (28) Connection timed out after 10001 milliseconds

I also tried from a site which checks from several locations around the world. And, none of those could see your domain with http. I was going to guess a geographic based firewall until this. So, some other firewall ?

3 Likes
4

Is this your IP?

Name:     gateway.zizzvo.com
Address:  52.186.107.119
Aliases:  stage2.liveexperience.io
          warrickco.zizzvo.com

I didn't quite understand:

3 Likes
5

Hey thanks for telling me about the stage environment and rate limits. And as for those certificates, I was able to get them issued using the cloudflare plugin. Unfortunately, this won't work for some of our production sites as we don't have the DNS server. It's a client whose DNS server has a CNAME that points to ours and so the certbot cloudflare plugin wouldn't work. I apologize if any of my terminology is wrong, I am not good with this stuff. I am simply a programmer :slight_smile:

That's why I am now testing with the other certbot command which should just issue an acme challenge. The problem is that it's not working.

I tried Let's Debug, and I will send a picture of the results:

9C4nlYd3dq
9C4nlYd3dq1040×753 48.5 KB

1 Like
6

Honestly I'm at a loss for how a lot of this stuff works. We have a Cloudflare DNS record that points to that IP address (52.186.107.119). The IP address is an Ubuntu VM that has a node server running which serves as a reverse proxy for a lot of the services we offer.

I appreciate the help man!
If you have any more questions I'd love to answer them

7

It's still running. I just checked. That's the weird part though, our Node server on this same VM is able to get requests and respond to them through port 80. Could the firewall be allowing node but blocking Certbot? How could I check that?

8

Maybe try stopping certbot and restarting node server. Let us check if that's still working more broadly.

3 Likes
9

Okay I just did that. You will see that the certificate is fine but that's irrelevant. I was able to obtain that certificate using the DNS option but that won't work for what I'm trying to do on production. I just need it to work with the certbot auto-generated server where it makes a challenge and sends it through port 80.

10

Yeah, that's interesting. I wasn't actually expecting for an HTTP request to succeed :slight_smile:

But, I'm a little puzzled. An HTTP request doesn't have a "Server" response header yet it redirects to HTTPS which says it is IIS. A Server header is not required it just often is present. And, I wasn't expecting to see IIS. Is this what I should see? (some headers omitted)

curl -I -m10  http://stage2.liveexperience.io
HTTP/1.1 301 Moved Permanently
request-context: appId=cid-v1:5682c9f7-8c89-41fd-a426-50da15834e89
Location: https://stage2.liveexperience.io/
Date: Tue, 18 Oct 2022 03:25:50 GMT

curl -I -m10  https://stage2.liveexperience.io
HTTP/1.1 200 OK
request-context: appId=cid-v1:8d94f9bf-eaba-4b3f-afde-2cd1e72325bf
date: Tue, 18 Oct 2022 03:26:18 GMT
server: Microsoft-IIS/10.0
set-cookie: ARRAffinity=296dd4c3a6fbb25a5adf9b2bbfc203287029091f4802561944f163e13e623f43;Path=/;HttpOnly;Secure;Domain=yondo-efe-stage.azurewebsites.net
x-miniprofiler-ids: ["0de9c76b-0089-4e81-b090-9de423be8a18"]
x-aspnet-version: 4.0.30319
x-powered-by: ASP.NET
3 Likes
12

Yes, this is because the Node server that we use to attach the SSL certificates to requests is actually a reverse proxy as a second function. It proxied the request to a .net server and then attached the SSL certificate to that response.

13

Is Certbot running on a Windows machine ?

2 Likes
14

No, I'm trying to run the certbot in place of our Node application. They are both running on Ubuntu 18.04.

The Node application will typically be able to renew certs, but it can't in this case because of some weird circumstances where we basically lost the last certificates that it generated. So, I'm trying to use certbot instead.

15

To clarify, it can renew some certificates, just not the ones that we lost. It's waiting for this renew date that was for the ones we lost. I've tinkered with it a bunch and I can't get it to work after like 10+ hours so that's why I resolved to use Certbot.

Also, the documentation on the Node module we are using is terrible and it's no longer maintained either.

16

OK. Thanks. I have a better understanding now but sadly don't have any good suggestions. I am baffled why certbot standalone wouldn't work.

3 Likes
17

Do you think a firewall could be allowing Node.js through but blocking Certbot? How could I check if that's the issue?

18

I've run:
sudo ufw status
and it says "Status: inactive"

19

Does your procedure to start node also do something with any firewall?

I don't know of any firewall that would block inbound traffic depending on what app was bound to port 80. Maybe @rg305 will know he has more experience. Or, another volunteer

And, you're sure certbot started OK? No error messages ?

EDIT: Is Ubuntu running on Azure too? Is there any inbound rule in Azure that could be affecting this?

3 Likes
20

Oh I forgot to mention we use pm2 to Daemonize the node server.

And certbot started Ok.

I checked for inbound rules and I don't think any of them should do this.

21

Sorry. I am as baffled as you. Hopefully someone else will have insight.

My only other idea (before signing off for night) would be to try something like acme.sh instead of certbot. I have no good reason to suspect it would work better. But, when I get stuck I like to poke things and sometimes I learn a key clue.

Best of luck to you.

3 Likes