Thanks for your reply. I have posted the chains in my reply with a more complete description of the problem.
That option should work but requires Certbot v1.12 or later
It was introduced just before that version but had some bugs until 1.12
Did you update your Certbot before trying it? You should definitely see a "fullchain.pem" file that contains 3 certs (at least through Jun5 as Jun6 the alt chain is being dropped).
Hi Mike.
Yes I did.
I got a response saying renewal succesfull.
What did the ../live/(domain)/fullchain.pem file look like? Did it have 2 or 3 certs?
Can you show the entire command and result messages?
And, did you check the Certbot version? Sometimes people end up with two versions could you have been running the older one without knowing it? Check:
certbot --version
Yes. I just re-ran the command and it generated errors this time.
here is the full output of the command.
root@***:~/compose# certbot renew --force-renewal --preferred-chain "DST Root CA X3"
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Processing /etc/letsencrypt/renewal/admin-staging.hubcontrols.com.conf
Renewing an existing certificate for admin-staging.hubcontrols.com
Processing /etc/letsencrypt/renewal/api2.hubcontrols.com.conf
Renewing an existing certificate for api2.hubcontrols.com
Reloading nginx server after certificate renewal
Failed to renew certificate api2.hubcontrols.com with error: nginx restart failed:
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] still could not bind()
Processing /etc/letsencrypt/renewal/api8.hubcontrols.com-0001.conf
Renewing an existing certificate for api8.hubcontrols.com
Reloading nginx server after certificate renewal
Failed to renew certificate api8.hubcontrols.com-0001 with error: nginx restart failed:
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] still could not bind()
Processing /etc/letsencrypt/renewal/api8.hubcontrols.com.conf
Renewal configuration file /etc/letsencrypt/renewal/api8.hubcontrols.com.conf is broken.
The error was: renewal config file {} is missing a required file reference
Skipping.
Processing /etc/letsencrypt/renewal/monaghan.hubcontrols.com.conf
Renewing an existing certificate for monaghan.hubcontrols.com
The following renewals succeeded:
/etc/letsencrypt/live/admin-staging.hubcontrols.com/fullchain.pem (success)
/etc/letsencrypt/live/monaghan.hubcontrols.com/fullchain.pem (success)
The following renewals failed:
/etc/letsencrypt/live/api2.hubcontrols.com/fullchain.pem (failure)
/etc/letsencrypt/live/api8.hubcontrols.com-0001/fullchain.pem (failure)
Additionally, the following renewal configurations were invalid:
/etc/letsencrypt/renewal/api8.hubcontrols.com.conf (parsefail)
Hook 'post-hook' reported error code 129
Hook 'post-hook' ran with error output:
pkill: killing pid 8302 failed: Permission denied
pkill: killing pid 8381 failed: Permission denied
pkill: killing pid 8382 failed: Permission denied
pkill: killing pid 8383 failed: Permission denied
pkill: killing pid 8384 failed: Permission denied
Hangup
2 renew failure(s), 1 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
certbot --version
certbot 2.10.0
Several issues here. Let's start with admin-staging which renewed successfully
In this file:
/etc/letsencrypt/live/admin-staging.hubcontrols.com/fullchain.pem
How many certs are there?
If not 3, can you show contents of this
/etc/letsencrypt/renewal/admin-staging.hubcontrols.com.conf
Btw he mentioned nginx is on docker so --nginx wouldn't work, he need webroot or standalone
I'm only going to rephrase what others said before, hopefully in a more clear manner as all the voices above were answering different bits of your problem and confusion may have occured.
You have a large problem. As of right now, the best case scenario is that your devices will all go offline in May 2025. There is no way around that.
There is a "special technique" to abuse a security flaw in early Android systems that LetsEncrypt and the greater TLS community helped users implement for greater reach. The simple explanation is that early versions of Android OS do not validate the end-date of trusted certificates. LetsEncrypt had DST cross-sign an extended version of their intermediate certificate with the expired root. This would allow older clients to access the internet, while most newer clients had a "short circuit" logic implemented by TLS authors to effectively ignore it and short-circuit their logic to a valid certificate with the same private key. That workaround is currently being phased out, as technical constraints will no longer allow it to work within the next few months.
All the effort you are putting into getting the DST root to work with LetsEncrypt above is a short-term fix. You will not be able to renew this after June 5th. The workaround will no longer work after September 30th. The longest effective workaround will be renewing on June 5th to maintain 90 days of coverage until September 3rd.
That leaves the standard long-term support for your devices as the CyberTrust root - which expires in May 2025. By September 3rd, you will need to switch Certificate Authorities from LetsEncrypt to someone who can sell you a certificate chaining up to that CyberTrust root to maintain compatibility from September 2024 through May 2025.
All this work is just a band-aid to buy you time to figure out a solution. Right now, all of your devices will be offline in May 2025.
You have 3 general options:
- If the OS/tech allows, you may be able to hire one of the above CAs to sign an intermediate with their expired root certificate.
- You figure out a way to update your remote devices
- You categorize the remote devices as EOL, and either do a trade-in/repair for customers, end all support, or figure out a way to structure a bankruptcy.
The options above are all very costly and time intensive. They're also - for the most part - outside the scope of dev/ops or engineering/reliability that I assume is your role. This is something your COO and CEO need to be discussing.
Yes there are three certs in /etc/letsencrypt/live/admin-staging.hubcontrols.com/fullchain.pem
Hi jvanasco.
Thank you for your reply.
We plan on replacing all the devices in the field by May 2025 if not before. If we could get the devices working till then it would be all we need.
Blockquote
There is a "special technique" to abuse a security flaw in early Android systems that LetsEncrypt and the greater TLS community helped users implement for greater reach. The simple explanation is that early versions of Android OS do not validate the end-date of trusted certificates. LetsEncrypt had DST cross-sign an extended version of their intermediate certificate with the expired root. This would allow older clients to access the internet, while most newer clients had a "short circuit" logic implemented by TLS authors to effectively ignore it and short-circuit their logic to a valid certificate with the same private key. That workaround is currently being phased out, as technical costraints will no longer allow it to work within the next few months.
Blockquote
Can you tell me more about this technique.
Well it it worked until fed without isrg root their device doesn't care about root's expire date too, so may be able to ask identrust or cacert to craft custom certificate: it's super expensive though
It was explained in full in the post that you quoted. There is no more to tell.
Ok Thanks.
Okay good. That proves the --preferred-chain worked. To further convince yourself you could copy/paste the last cert in that file into a decoder to see it is the DST cert. But, I am confident it is.
Let's try doing just api2. What does this show?
sudo certbot renew --dry-run --cert-name api2.hubcontrols.com
The --dry-run will not affect existing certs (and won't run all hooks either). But, just want to walk through and debug each cert renewal as those 3 showed different results.
of I found something interesting about cacert.org
your iot device doesn't care root's expire date like android,
and from FAQ/Class3Resign - CAcert Wiki cacert (which isn't publiccally turst) resigned class 3 root with new validity date until 2031 so you can get a cert signed with cacert until 2031 with same pubkey
you'd still need some triming(like remove unneeded class1 to class 3 intermediate) but it'd work until 2031 I guess
I'm not sure this flys in CA/B but they are not in public root store so whatever they wants I guess
This is the output of sudo certbot renew --dry-run --cert-name api2.hubcontrols.com
certbot renew --dry-run --cert-name api2.hubcontrols.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Processing /etc/letsencrypt/renewal/api2.hubcontrols.com.conf
Simulating renewal of an existing certificate for api2.hubcontrols.com
Encountered exception during recovery: certbot.errors.MisconfigurationError: nginx restart failed:
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] still could not bind()
Failed to renew certificate api2.hubcontrols.com with error: nginx restart failed:
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
nginx: [emerg] still could not bind()
All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/api2.hubcontrols.com/fullchain.pem (failure)
Hook 'post-hook' reported error code 129
Hook 'post-hook' ran with error output:
pkill: killing pid 8302 failed: Permission denied
pkill: killing pid 8381 failed: Permission denied
pkill: killing pid 8382 failed: Permission denied
pkill: killing pid 8383 failed: Permission denied
pkill: killing pid 8384 failed: Permission denied
Hangup
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
As a note here nginx is running in a container while certbot is running on the vm itself. Im not sure if that is causing some of the error messages.
It is. What authentication method was used for admin-staging ?
You could show the renewal conf files for each that are in
/etc/letsencrypt/renewal
I suspect you used something like --webroot for admin-staging.
But, how could this ever have worked before? The --nginx plugin if run in the host can only control nginx servers also running in the host.
I'll check and see if there is something in the docker configs that would allow the nginx server running in the container to access the lets encrypt files and il check if there is any thing in the renewal conf files and post them here. Thanks again. I'm afk for a while now