Today I run into an issue while updating the certificates of a server ( Red Hat Enterprise Linux 9.5 (Plow) ), when executing /usr/bin/certbot renew
I get this strange message
Error while running nginx -c /etc/nginx/nginx.conf -t.
nginx: [alert] OPENSSL_init_ssl() failed (SSL: error:12800067:DSO support routines::could not load the shared library:filename(/snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so): /snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so: cannot open shared object file: No such file or directory error:12800067:DSO support routines::could not load the shared library error:07880025:common libcrypto routines::reason(37):name=fips error:0700006D:configuration file routines::module initialization error:module=providers, value=provider_sect retcode=-1 )
Inside /snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/ I just have "legacy.so" and find /snap/certbot/4325/usr/lib/x86_64-linux-gnu/ -name "*fips*" gives nothing.
Running "nginx -c /etc/nginx/nginx.conf -t " directly from the console displays no errors.
I've tried to reinstall certbot with snap and install the latest edge version, but nothing worked.
That is very strange. And, I don't have any suggestions. If no other volunteer here offers help you might try the EFF's github for Certbot: Issues · certbot/certbot · GitHub
This looks to me like a possible packaging problem. And Certbot v3 had a couple migration issues although not identical to this.
A work-around would be to change from using the --nginx option to --webroot. The --webroot option does not need to parse your nginx so does not issue the nginx -t command.
There is also a reconfigure command in current versions of Certbot. Reconfigure is a pretty easy way to switch methods. You should also use a --deploy-hook with --webroot to reload nginx after getting a fresh cert. Let us know if you want help with that.
Meanwhile I investigate "reconfigure" (got the same error), I've succesfully managed to renew the certificate by just simply moving the .conf files outside nginx, restarting nginx, and then renew worked.
After that I moved the .conf files for nginx again to their place and restarted.
The error popped up while invoking nginx -c, so it's not a Certbot error but an Nginx error. nginx -c just checks the config file, it doesn't invoke certbot.
It looks like your nginx was somehow built against the Certbot OpenSSL libraries, and that (perhaps) you started with a fips distribution and then turned it off. That could have happened within Snap or on the main OS.
Do you remember how nginx was installed and if you toggled any fips support?
Edit:
Just to clarify the above:
Certbot invokes Nginx, not the other way around:
The issue is likely happening here, when Certbot runs a config_test:
The raised exception is not within Certbot code, it's within Nginx and due to nginx referencing a shared library of a fips implementation of OpenSSL that was taken off the machine sometime after being compiled.
This is weird. A first I thought it might be related to this:
But that's in the certbot install and should be sandboxed away from nginx and what was raising that exception.
I'm trying to figure out ways this could happen, and I do think it's likely from certbot's distribution and more likely from some chnge on the system.
The Certbot --nginx plugin issues such a command. I agree with your comment but the error only occurred running that command during a Certbot renew (presumably with --nginx option). Running that same command on the command line produced no error.
You may well be right about the mixed up openssl packages. That's outside my range
You posted that just as I was typing the above clarifications. The error is being raised within Nginx code in a subprocess, which should be picking up the system nginx, so there's no certbot code involved.
Perhaps this could be debugged a bit by running in the line above that run_scrpt invocation:
that should show which nginx Certbot is picking up and help figurout out what is going on. perhaps a version got installed into the snap sandbox somehow and that is shadowing the system nginx ?
Do you remember how nginx was installed and if you toggled any fips support?
I'm sorry I can't remember, but I'm 99% sure I installed it with yum from the RHEL 9.5 repos. Regarding FIPS, no, I can't remember too.
I'l try to run some of the suggested commands to see if I can reproduce the error. I've another mirror machine that must have the same configuration too.
nginx: [alert] OPENSSL_init_ssl() failed (SSL: error:12800067:DSO support routines::could not load the shared library:filename(/snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so): /snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so: cannot open shared object file: No such file or directory error:12800067:DSO support routines::could not load the shared library error:07880025:common libcrypto routines::reason(37):name=fips error:0700006D:configuration file routines::module initialization error:module=providers, value=provider_sect retcode=-1 )
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/sites-enabled/REDACTED-api-auth.conf:38
Last metadata expiration check: 0:21:51 ago on Sat 18 Jan 2025 09:03:21 AM CET.
Installed Packages
Name : nginx
Epoch : 2
Version : 1.20.1
Release : 20.el9
Architecture : x86_64
Size : 148 k
Source : nginx-1.20.1-20.el9.src.rpm
Repository : @System
From repo : rhel-9-appstream-rhui-rpms
Summary : A high performance web server and reverse proxy server
URL : https://nginx.org
License : BSD
Description : Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 and
: IMAP protocols, with a strong focus on high concurrency, performance and low
: memory usage.
Yes, the --webroot authenticator option during reconfigure is needed to replace the --nginx authenticator. Trying reconfigure with all needed options should work-around this problem.
I've already noted the packaging issue is outside my range. I'll let @jvanasco continue with that Or someone else with better knowledge about those internals.
I think this is likely a Certbot Issue now, and I would post the above information there.
They took out FIPS detection a few weeks ago; but IIRC should be happening in Python code or rust bindings. This looks to be caused by Certbot somehow picking up an nginx binary that was compiled against a fips version of OpenSSL within the sandboxed snap environment.
Short of trying to reinstall snapd and nginx to see if that fixes things, I'm not sure how to address this or what is the root cause
Not sure if this is the same issue, but it's similar enough that it might help steer in the right direction...
I noticed a similar error when renewing certificates with DNS validation, where I use some custom scripts for managing DNS zones (adding / removing acme-challenge records).
And when these scripts ran via certbot hooks, they reported this error:
After some debugging I found out that the issue was caused by certobot setting and exporting an environment variable OPENSSL_FORCE_FIPS_MODE="0", which in turn caused Bind commands (such as rndc and named-checkconf) to fail with the above error.
My solution was to simply unset this variable in my scripts.
Whatever caused this happened after January 6th as renewals worked then, but I wasn't able to find out exactly what it was.
That landed in the
3.1.0 release about 2 weeks ago.
Snapd updates pacakges 4x daily by default, and although they slowly phase updates out, that fix should have landed on the OP's machine a few weeks ago.
@driade The last thing I can think of is to share your certbot --version and if it's somehow less than 3.1.0 forcing snapd to update certbot. Your system should have updated to that 2 weeks ago though.
@servekon That's not applicable here, as that wouldn't use the snap method of installing Certbot, but using the distro's own package manager (which is not recommended by the Certbot team).