Renewing certificate problem

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:corp.networkingtchnology.org

I ran this command:cert bot - v renew

It produced this output: [root@hermes ~]# certbot -v Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator apache, Installer apache Which names would you like to activate HTTPS for? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1: hermes.corp.networkingtechnology.org - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Select the appropriate numbers separated by commas and/or spaces, or leave input blank to select all options shown (Enter 'c' to cancel): 1 Certificate is due for renewal, auto-renewing... Renewing an existing certificate for hermes.corp.networkingtechnology.org Performing the following challenges: http-01 challenge for hermes.corp.networkingtechnology.org Waiting for verification... Challenge failed for domain hermes.corp.networkingtechnology.org http-01 challenge for hermes.corp.networkingtechnology.org Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems: Domain: hermes.corp.networkingtechnology.org Type: unauthorized Detail: 79.132.230.58: Invalid response from https://hermes.corp.networkingtechnology.org/?url=/.well-known/acme-challenge/qpJB3MV22PtyeeIHlY1FcN7PVQI3QdW-pA8b8IYzWfY : "\n<html lang="en-US" class="no- js">\n \n\n <meta charset="UTF-8" />\n <meta http- equiv="X-UA-Compatible" " Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet. Cleaning up challenges Some challenges have failed. Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details. [root@hermes ~]#

My web server is (include version): hermes.corp.networkingtechnology.org

The operating system my web server runs on is (include version):
Alma Linux v8.9
My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):1.22.0

I've been sitting for months, unable to reach any of my servers on Port 443. I've finally solved the problem, but I am still unable to renew certificates. See output above

There's a problem with how the web service is forwarding the ACME challenge request.

Here is a sample request:

curl -Ii http://hermes.corp.networkingtechnology.org/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 301 Moved Permanently
Location: https://hermes.corp.networkingtechnology.org/.well-known/acme-challenge/Test_File-1234
Date: Fri, 27 Sep 2024 10:40:21 GMT
Server: OPNsense

That went OK.
The redirection merely adds an "S" to the HTTP request.

Now, here is the response to the HTTPS request:

curl -Iik https://hermes.corp.networkingtechnology.org/.well-known/acme-challenge/Test_File-1234
HTTP/2 302
set-cookie: PHPSESSID=e4b15be3edb6400a0f3704bccfa6147c; path=/
set-cookie: PHPSESSID=e4b15be3edb6400a0f3704bccfa6147c; path=/; secure; HttpOnly
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: no-store, no-cache, must-revalidate
pragma: no-cache
location: /?url=/.well-known/acme-challenge/Test_File-1234   <<<<<<<<<<<<<<
content-type: text/html; charset=UTF-8
date: Fri, 27 Sep 2024 10:40:26 GMT
server: OPNsense

We see that it now re-redirects and has added "/?url=" into the path.
That is unexpected and problematic.

I would recommend that you handle the ACME challenge requests in HTTP [no redirection].
That would allow you to retain whatever implements that ["/?url="] in HTTPS [without change].

4 Likes

Thanks for the insight, but can you walk me through doing that. I'm getting a bit long in the tooth as I close in on 84 and there are so many pages and pages of stuff in OPNSense, and asking them for help is like asking a brick wall. Could it be on my server in the setup?

1 Like

I don't know what web server you are using.
It shows:
Server: OPNsense
[I don't know how to "configure" that]

3 Likes

OPNSense was forked from pfSense which means there is a reasonable chance that it may also use the same ACME client. Due to heavily restricting connections to my firewalls, I exclusively use DNS-01 challenges with my pfSense systems. I have no advice for making an HTTP-01 challenge work with a redirect on that platform.

4 Likes

At last, after all this time, by pure accident I found the cause of the problem. Over 6 months ago, we had a power failure. The UPS battery was kaput, so everything went down.
When the power came back, everything worked fine, but then weeks after we found this problem with Let's Encrypt no longer working.
I've been through every page on the firewall and I can't think of anything I hadn't checked, but same result.
Yesterday the problem escalated and one of the virtual machines had NO open ports. I had this bright idea - install a new network card It was only then I found it.
We use VMWare 6.0 and the vsphere 6.0 client. When I clicked on edit. I suddenly found that the version of the VMware virtual machine had changed from 8 to 11. For this version, you need the vsphere web client, which doesn't work as Flash no longer exists.
The only way to set the machine back to 8 will be to reinstall it and move everything to it.
In the meantime, we installed the DNS version of Let's Encrypt, and I will need to move all the certificates in /etc/letsencrypt to a new v8 machine.
I tested this by installing a new identical Alma 8.10 Linux server and tried to copy the /etc/letsencrypt folder to it. but I' can't find a way to do this and every method I've attempted doesn't work.

Can someone tell me how to move the certificates to a new installed identical virtual machine?

What OPNsense offers to install as an "ACME Client" is indeed acme.sh. It also lets you install Caddy, which will act as its own ACME client (and that's (part of) how I'm using it on one of my OPNsense firewalls).

4 Likes

Oh dear. How do I make myself clear? It has NOTHING to do with OPNSEnse. It's VMWARE. ALL the Linux servers after the power failure 'upgraded' the virtual machines to v11 from the original v8.

In the absence of being able to use a vsphere web client, I can't manage the virtual servers. VMware's answer is to upgrade to a higher version, but to do that would mean having to pay for 4 new Proliant servers. and pay to upgrade VMware.
Hasn't anyone out there realised that the cost of living is going sky high?
As I see it the ONLY solution is to reinstall every virtual Linux machine and try to copy whatever I can to new ones.
That includes having to reinstall OPNSense, create new VMs for every Linux server and try to copy whatever I can rescue to a new VM.
I seem to be wasting my time trying to get a simple answer.
CAN I MOVE THE LET'S ENCRYPT CERTIFICATES TO A NEW SERVER? If so how?

Can we just forget the word OPNSense which seems to have become an obsession. The problem is VMWARE not OPNSense.

This is so frustrating

It may be simpler to just get new certs on the new VMs.

But to answer your question: Yes; It is possible to copy/move all the certificates from one server to another.
How?
That depends on the client.
For certbot [Linux]: Copy the entire subdirectory "/etc/letsencrypt/" [without modification - keeping all the same rights and links]

4 Likes

I thought of that and tried to copy them but it failed. It copies everything except privkey.pem and for some reason it refuses to copy that file. What I TRIED to do was setup a clean new server. Copt all the '/etc/letsencrypt files to it. Install certbot and the dns-dynu plugin, do a dry-run and if it worked, bring done the server as a backup in case the server with the certs crashes.
bring it up every month for updates etc.
It refuses to copy that one file.

Did you copy the files as root?

Please provide the actual commands used and the literal output. Without details we are left to a guessing game or using crystal balls (but we all know the latter don't exist).

3 Likes

Yes I used Filezilla and set the server to allow me to login as root. I tried again after rebooting the Proliant and the vms and this time it copied it. BUT...
Certbot renew --dry-run still gives me an error:
Renewal configuration file /etc/letsencrypt/renewal/corp.networkingtechnology.org.conf is broken.
The error was: target /etc/letsencrypt/archive/corp.networkingtechnology.org/privkey1.pem of symlink /etc/letsencrypt/live/corp.networkingtechnology.org/privkey.pem does not exist
Skipping.
I tried:
sudo ln -sf /etc/letsencrypt/archive/corp.networkingtechnology.org/privkey1.pem /etc/letsencrypt/live/corp.networkingtechnology.org/privkey.pem
and I tried login in as root and doing the same command but same result

I'm not that familiar with FileZilla, but usually one would use e.g. rsync with the --archive option so that symlinks, ownership and permissions are all kept.

Or store the entire directory in a tar file e.g.

3 Likes

I'll try it with rsync

Please show:
ls -l /etc/letsencrypt/archive/corp.networkingtechnology.org/privkey1.pem
ls -l /etc/letsencrypt/live/corp.networkingtechnology.org/privkey.pem

3 Likes

After many attempts, I finally copied the certbot, dynu plugin and the certificates to a new server. I ran certbot renew --dry-run and AT LAST. It worked! I managed to run FileZilla logged in as root on both servers and it worked.

I now have a certificate for corp.networkingtechnology.org.

I have another question, but let's say this is SOLVED and I'll open another thread for the other problem

Thanks to everyone who tried to help.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.