I have used certbot for over a year on this EC2 instance and added many subdomains. Today it doesn't work, for some reason I cannot understand.
Note: I have two aws.osmium.app server blocks in nginx, one of which is "ignored", is that a problem?
Thank you!
My domain is: qr.osmium.app
I ran this command: sudo certbot --nginx -d qr.osmium.app
It produced this output:
You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/aws.osmium.app.conf)
It contains these names:
You requested these names for the new certificate: qr.osmium.app.
My web server is (include version): nginx version: nginx/1.18.0 (Ubuntu)
The operating system my web server runs on is (include version): ubuntu pro 22.04 LTS
My hosting provider, if applicable, is: AWS EC2
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.6.0
And, what do you mean by this? Do you have two server blocks with the exact same domain name for the same listen criteria? Because that is not a valid nginx config
Or, maybe because your DNS for these two domain names point to different EC2 instances? Please clarify - thanks.
~$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
verifying the signature of the certificate located at /etc/letsencrypt/live/aws.osmium.app/cert.pem has failed. Details:
Traceback (most recent call last):
File "/snap/certbot/3024/lib/python3.8/site-packages/certbot/crypto_util.py", line 308, in verify_renewable_cert_sig
verify_signed_payload(pk, cert.signature, cert.tbs_certificate_bytes,
File "/snap/certbot/3024/lib/python3.8/site-packages/certbot/crypto_util.py", line 333, in verify_signed_payload
public_key.verify(
File "/snap/certbot/3024/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 571, in verify
_rsa_sig_verify(
File "/snap/certbot/3024/lib/python3.8/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line 322, in _rsa_sig_verify
raise InvalidSignature
cryptography.exceptions.InvalidSignature
Renewal configuration file /etc/letsencrypt/renewal/aws.osmium.app.conf produced an unexpected error: verifying the signature of the certificate located at /etc/letsencrypt/live/aws.osmium.app/cert.pem has failed. Details: . Skipping.
This seems to be the problem? Or no, you are right, the IP addresses should be the same! I'll double check this and write back soon.
Yes, I have to confess that I have two server blocks with aws.osmium.app - it happened at setup and nginx -t warns about it but says it's OK. I didn't want to take the risk to mess with those serverblocks as we're running some production loads, so decided to let it be.
Yes, my mistake, the IP address was from an old record. Now I have corrected that, but the error persists. I was too quick to paste the output of sudo certbot certificate, here is the rest:
You corrected the IPv4 address for qr so that's good. But, I see you have an IPv6 only for the aws subdomain and not any of your others. This requires some care in your nginx config to work right so is something you should review carefully.
Hmmm. What did you do to that certificate?
Can you show this
sudo ls -lR /etc/letsencrypt/{live,archive}/aws.osmium.com
Looks like something happened to the cert13.pem file. Note its date is different than the privkey and fullchain it should be matched with.
This explains the error Certbot has for this domain.
You have several problems to sort but I won't have more time until later today. Maybe someone else will pickup
Your duplicate domain names in nginx
The error shown in your first post that Certbot says aws subdomain part of qr request
That nginx sends an aws.osmium.app cert for requests to qr domain
I'm very humbled by your willingness to help. Much appreciated.
Before I copy the 12 as 13, what's the danger? We've got a fleet of wind turbines running on this subdomain. It's working fine on the server, I can get all the pages in my browsers.
OK, thank you, I copied over the cert12.pem to the cert13.pem. Website still works, pfew
now I get another error:
$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/aws.osmium.app.conf produced an unexpected error: fullchain does not match cert + chain for aws.osmium.app!. Skipping.
O M G, What's with all the [unnecessary] validations??? LOL
I guess you will have to replace the 13 with the actual real contents.
See previous post [by @MikeMcQ]
In case you missed it: https://crt.sh/?d=9777347192
That sorted the error in sudo certbot certificates and let me run sudo certbot --nginx -d qr.osmium.app without the previous problem.
I now only need sort out the double serverblocks.
Differences:
The second one has
server {
if ($host = aws.osmium.app) {
return 301 https://$host$request_uri;
} # managed by Certbot
server_name aws.osmium.app; # managed by Certbot
return 404; # managed by Certbot
at the beginning and the first one just server_name aws.osmium.app;
also, the first one has
listen [::]:443 ssl ipv6only=on; # managed by Certbot
whereas the second one has
listen [::]:443 ssl; # managed by Certbot
I've added some error handling to the first server block, which is working on the website, so my hunch is that it is the first one which is active and that I should delete the second one? (I've never really understood the reason for the return 404; # managed by Certbot line.)
That "if ($host..." redirect is correctly used in a server block for port 80 to redirect to HTTPS (443). Your block2 is for port 443 already so this redirect would come back to itself causing a loop.
The if combined with the return 404 are only helpful when that server block is also your default. That means you either explicitly set it as such or it is the first server block nginx sees which becomes the default.
These make sure only valid SNI requests for that domain name are redirected to HTTPS (when properly in an HTTP server block). Often bots and IP scanners are not well-behaved so this keeps some load off your server to handle the second request. It also doesn't "leak" info about your domain name if the scanner was just using an IP address (although this is somewhat obscure).
Having these in other server blocks is not (usually) helpful as nginx will only choose non-default server blocks for legit SNI requests matching the server_name. I say usually because people can do weird things with listen clauses where this can get messy.
UPDATE:
Also, you should remove the AAAA record from the DNS for aws.osmium.app until you are able to get it working. And, you should try to do that.
But, connections on IPv6 are not working right now so the faulty AAAA can cause problems.
