Nginx not finding fullchain.pem

meltedeyes · December 30, 2020, 4:33pm

My web server is (include version):
nginx version: nginx/1.14.2

The operating system my web server runs on is (include version):
"Raspbian GNU/Linux 10 (buster)"
** Linux 5.4.79-v7l+ armv7l**

My hosting provider, if applicable, is:
self hosted with static public IP on Raspberry 4

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.31.0

After a couple rough starts, I got the following:
root@oerlikon:/var/www/html# certbot --nginx

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
No names were found in your configuration files. Please enter in your domain
name(s) (comma and/or space separated) (Enter 'c' to cancel): joshcampana.com dreadnought-brewing.com dreadnought-welding.com dreadnoughtfabrication.com dreadnoughtmetalworks.com dreadnoughtwelding.com dreadnoughtpc.com hotmessexpresscreations.com weldingshit.com
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/joshcampana.com.conf)

What would you like to do?

1: Attempt to reinstall this existing certificate
2: Renew & replace the cert (limit ~5 per 7 days)

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Keeping the existing certificate
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default
Deploying Certificate to VirtualHost /etc/nginx/sites-enabled/default

Please choose whether or not to redirect HTTP traffic to HTTPS, removing HTTP access.

1: No redirect - Make no further changes to the webserver configuration.
2: Redirect - Make all requests redirect to secure HTTPS access. Choose this for
new sites, or if you're confident your site works on HTTPS. You can undo this
change by editing your web server's configuration.

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default
Redirecting all traffic on port 80 to ssl in /etc/nginx/sites-enabled/default

Congratulations! You have successfully enabled https://joshcampana.com,
https://dreadnought-brewing.com, https://dreadnought-welding.com,
https://dreadnoughtfabrication.com, https://dreadnoughtmetalworks.com,
https://dreadnoughtwelding.com, https://dreadnoughtpc.com,
https://hotmessexpresscreations.com, and https://weldingshit.com

You should test your configuration at:
SSL Server Test: joshcampana.com (Powered by Qualys SSL Labs)
SSL Server Test (Powered by Qualys SSL Labs)
SSL Server Test (Powered by Qualys SSL Labs)
SSL Server Test (Powered by Qualys SSL Labs)
SSL Server Test: dreadnoughtmetalworks.com (Powered by Qualys SSL Labs)
SSL Server Test (Powered by Qualys SSL Labs)
SSL Server Test: dreadnoughtpc.com (Powered by Qualys SSL Labs)
SSL Server Test (Powered by Qualys SSL Labs)
SSL Server Test (Powered by Qualys SSL Labs)

IMPORTANT NOTES:

Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/joshcampana.com/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/joshcampana.com/privkey.pem
Your cert will expire on 2021-03-29. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the "certonly" option. To non-interactively renew all of
your certificates, run "certbot renew"

If you like Certbot, please consider supporting our work by:

Donating to ISRG / Let's Encrypt: Donate - Let's Encrypt
Donating to EFF: Support EFF's Work on Let's Encrypt | Electronic Frontier Foundation

however nginx -t is not happy

root@oerlikon:/etc/letsencrypt/live# nginx -t
nginx: [emerg] SSL_CTX_load_verify_locations("/etc/letsencrypt/live//fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such fil e or directory:fopen('/etc/letsencrypt/live//fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file error:0B084002:x509 certif icate routines:X509_load_cert_crl_file:system lib)
nginx: configuration file /etc/nginx/nginx.conf test failed

root@oerlikon:/etc/letsencrypt/live# ls -al joshcampana.com/
total 12
drwxr-xr-x 2 root root 4096 Dec 29 18:12 .
drwx------ 3 root root 4096 Dec 29 18:45 ..
-rw-r--r-- 1 root root 692 Dec 29 18:12 README
lrwxrwxrwx 1 root root 39 Dec 29 18:12 cert.pem -> ../../archive/joshcampana.com/cert1.pem
lrwxrwxrwx 1 root root 40 Dec 29 18:12 chain.pem -> ../../archive/joshcampana.com/chain1.pem
lrwxrwxrwx 1 root root 44 Dec 29 18:12 fullchain.pem -> ../../archive/joshcampana.com/fullchain1.pem
lrwxrwxrwx 1 root root 42 Dec 29 18:12 privkey.pem -> ../../archive/joshcampana.com/privkey1.pem

I do a bit of research and chown to www-data. same error. then I remove the sym links and copy the files over, now I get a similar, but different error:

root@oerlikon:/etc/letsencrypt/live# nginx -t
nginx: [emerg] BIO_new_file("/etc/letsencrypt/live/joshcampana.com/fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/joshcampana.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

====================

root@oerlikon:/etc/letsencrypt/live/joshcampana.com# ls -al
total 24
drwxr-xr-x 2 www-data root 4096 Dec 29 19:23 .
drwx------ 4 root root 4096 Dec 29 19:23 ..
-rw-r--r-- 1 www-data root 2110 Dec 29 19:23 cert1.pem
-rw-r--r-- 1 www-data root 1586 Dec 29 19:23 chain1.pem
-rw-r--r-- 1 www-data root 3696 Dec 29 19:23 fullchain1.pem
-rw------- 1 www-data root 1704 Dec 29 19:23 privkey1.pem

Lines that I believe are relevant from nginx/sites-available/default:

listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/joshcampana.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/joshcampana.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

I just noticed that nginx is looking for fullchain.pem, but the file is called fullchain1.pem. I renamed it to fullchain.pem (so now same as the symlink) and I get the SSL_CTX error again along with the instead of my domain.

Osiris · December 30, 2020, 6:15pm

You just broke your certbot.

meltedeyes · December 30, 2020, 6:48pm

Sure did, but hey backups. a quick cp and I'm back to where I started. but whether or not certbot can renew isn't the issue I am having. just laying out some of the troubleshooting that I've done.

rg305 · December 30, 2020, 7:21pm

I see two consecutive slashes
[luckily my eyes haven't yet melted - lol]

meltedeyes · December 30, 2020, 7:55pm

Hmmm interesting. odd thing is I figured out the issue, and it corrected the double slash.

I ended up going through nginx.conf and commenting out stuff. lo and behold nginx -t succeeded. so then I went and started uncommenting everything that certbot had added one line at a time and testing.

This line was the culprit:

include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot

I went and checked that file, sure enough, the path had . switched that out with my domain and now everything works.

Oh, and yeah, I restored the symlinks so certbot isn't broken.

rg305 · December 30, 2020, 7:57pm

Please show (your current version of) this file:

I have a feeling something may have been changed in there...

Mine (0.31.0) shows this:

# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.

ssl_session_cache shared:le_nginx_SSL:1m;
ssl_session_timeout 1440m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;

ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AE
S256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-
RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDH
E-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS";

meltedeyes · December 30, 2020, 8:23pm

You were correct, there was an error in this file.

ssl_trusted_certificate /etc/letsencrypt/live/joshcampana.com/fullchain.pem;

was originally

ssl_trusted_certificate /etc/letsencrypt/live/<default>/fullchain.pem;

I mentioned that I fixed it in a roundabout way in a previous comment. But just in case I have something else wrong, here it is:

root@oerlikon:/etc/letsencrypt# cat options-ssl-nginx.conf
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.

ssl_session_cache shared:le_nginx_SSL:1m;
ssl_session_timeout 10m;

ssl_session_tickets off;
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /etc/letsencrypt/live/joshcampana.com/fullchain.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers on;

ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS";

thanks for the help.

rg305 · December 30, 2020, 8:51pm

Exactly why do you need this again?

The only reference I could find on docs.nginx.com was this:

But it didn't make it clear why.
I mean it sounds like you would use it when trying to trust self-signed certs or certs that aren't found in the trust store.
But that isn't the case with LE certs and chains.
So I'm confused on what benefit it provides.

Osiris · December 30, 2020, 9:07pm

While according to the current docs, for OCSP to work only a valid intermediate is required in ssl_certificate:

For the OCSP stapling to work, the certificate of the server certificate issuer should be known. If the ssl_certificate file does not contain intermediate certificates, the certificate of the server certificate issuer should be present in the ssl_trusted_certificate file.

However, I think to remember that some time ago an extra ssl option was required for OCSP to work. So might be from that.

meltedeyes · January 10, 2021, 3:21pm

Hell if I know. I'm just a dumb welder following a tutorial (i think it was on linuxbabe.com), not a sysadmin getting paid for this

rg305 · January 10, 2021, 6:01pm

I think it is safe to remove that line.
I've used OSCP and never had to do anything like that.

rg305 · January 13, 2021, 12:48pm

Did you finally get things all sorted out?

system · February 12, 2021, 12:48pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Error SSL Not found fullchain.pem Help	16	52	October 7, 2024
Letsencrypt doesn't create fullchain.pem \| Your time & help appreciated :+1: Help	17	1163	July 20, 2024
Nginx exiting because he can't find privkey and fullchain, but they exist Help	7	376	March 5, 2023
`server block not found` even though I am 99% sure it should be Help	13	790	March 15, 2023
Certbot not discovering domains on nginx Help	5	4379	May 20, 2020

Nginx not finding fullchain.pem

Related topics