The server crashed and attempted to restart from scratch

We have been utilising certificates for more than two years now. However, due to a server crash, we lost the HAPROXY configuration and all related information. Currently, our web servers are active but running on different servers. The certificates we had are set to expire on November 8th, 2023. I have attached the log file below for your reference. Please advise me on how to generate new certificates we can use with HAPROXY. Thank you for your assistance in obtaining the new certificate. Your help is greatly appreciated.

My domains: gabey.com.au

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.7.4

Some information purposely redacted

1 Like

You should first check your A and AAAA records in your DNS. Using Let's Debug test site it can reach that domain with IPv6 (AAAA) but it reaches an nginx server.

Using the IPv4 A address is gets a connection refused. And, this is probably correct since you are using standalone which is only listening on port 80 when it is running.

Let's Encrypt favors an IPv6 address. In this case I think the AAAA record just needs changing to point to the right server (best option). Or, should be removed if you don't support IPv6.

Note the detailed Let's Debug info will always show "connection refused" or similar if using standalone and is not running. The real problem was the success with IPv6

5 Likes

Thank you for the prompt reply. I will communicate this with the DNS hosting and come back.
In the meantime, last night, I solved a DNS issue with the DNS hosting provider. I have attached a screenshot of the AAAA records.

[DNS Checker - DNS Check Propagation Tool](DNS Checker - DNS Check Propagation Tool)

I was surprised to see the label Nginx because I used HAPROXY, which functioned perfectly before the crash. The only issue with DNS was that the records were pointing to the mail gateway, but we have fixed it.

1 Like

A better way to check the DNS records is to use https://unboundtest.com which looks up DNS like Let's Encrypt. Or, just do it with the authoritive servers like they do.

You can check your servers public IP many ways but one is to use these commands

curl -4 https://ifconfig.io
curl -6 https://ifconfig.io

Using the current DNS IP addresses I get the below from my own test server. As noted, a failure to connect is expected if you don't have anything listening on port 80. Note the "Server: nginx" response header with the IPv6 request

curl -i4 -m8 http://gabeyshare.com.au
curl: (7) Failed to connect to gabeyshare.com.au port 80 after 482 ms: Connection refused

curl -i6 -m8 http://gabeyshare.com.au
HTTP/1.1 403 Forbidden
Server: nginx
Date: Sat, 04 Nov 2023 21:33:39 GMT

(data response edited for brevity)
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<p>You don't have permission to access this resource.</p>
<p>Additionally, a 403 Forbidden
error was encountered while trying to use an ErrorDocument to handle the request.</p>

DNS with dig and one of your authoritive DNS servers

dig +noall +answer AAAA gabeyshare.com.au @ns1.syrahost.com
gabeyshare.com.au.      14400   IN      AAAA    2405:3f00:a222:bbbb:bba1:20:ffff:ffff

dig +noall +answer A    gabeyshare.com.au @ns1.syrahost.com
gabeyshare.com.au.      14400   IN      A       49.176.228.55
4 Likes

Yes, it will fail since I had to bring the server down and kill Port 80. Also, HAPROXY is down at the moment. Otherwise, will complain about the ports used by web servers. I am trying to delete the IPV6; it will take 4 hours to propagate. So I have to wait.

yes, I spoke to them and will delete the records and try again.

Normally not that long. Let's Encrypt looks directly at the authoritative DNS servers and is not affected by TTL propagation.

Sometimes DNS auth servers take a long time to sync between themselves but this is rare. Usually it is only seconds or a few minutes.

Try unbountest or dig to the auth servers to see when it is gone from the auth servers. That said, I just looked and the AAAA record is still there.

4 Likes

Thank you , I will update

1 Like

It seems those IPs are not even from the same ISP.

3 Likes

Thank you for your guidance in resolving this issue. I followed your suggestion and obtained new certificates. Consequently, I plan to update the AAAA records soon and modify the edge devices to enable the use of IPV6.

To assist others who may encounter the same problem, let me share what we did to address the issue: we deleted all the AAAA records from the DNS hosting.

To locate the DNSSEC authentication chain, I used Ubuntu or online tools such as DNSViz, a DNSViz | A DNS visualisation tool. I made sure that the A records authentication chain was valid and also checked whether the AAAA records disappeared after deletion. Please note that the time taken for this process may vary depending on your location.
I used the Ubuntu command line to ensure the removal of AAAA records. This prevents Letsencrypt from picking up authentication issues.

To ensure the job is done correctly, it is essential to stop all web servers and reverse proxy services so that port 80 is free. You can do this by running the command "sudo service apache2 stop". Once the servers are stopped, check that port 80 is available by executing "netstat -na | grep ':80.*LISTEN'". Finally, I stopped the reverse proxy on Ubuntu.

In my case, I used "sudo certbot certonly --standalone --preferred-challenges http --http-01-port 80 -d [type your domain name here]"
to generate new certificates successfully.

Whenever certbot is used in --standalone mode, HTTP is the only challenge type allowed.
So, you can remove "--preferred-challenges http" - it's redundant.
And assigning HTTP to port 80 is also redundant - port 80 is the default port for HTTP.
So, you can also remove "--http-01-port 80".

So, that boils down to:
"sudo certbot certonly --standalone --preferred-challenges http --http-01-port 80 -d [type your domain name here]"
OR
"sudo certbot certonly --standalone -d [type your domain name here]"

3 Likes

Great! I can automate this process now. It's simpler than I thought. Thanks for the feedback!

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.