Certbot snap Error while renewing (OPENSSL_init_ssl)

Hi, good morning.

Today I run into an issue while updating the certificates of a server ( Red Hat Enterprise Linux 9.5 (Plow) ), when executing /usr/bin/certbot renew

I get this strange message


Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: [alert] OPENSSL_init_ssl() failed (SSL: error:12800067:DSO support routines::could not load the shared library:filename(/snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so): /snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so: cannot open shared object file: No such file or directory error:12800067:DSO support routines::could not load the shared library error:07880025:common libcrypto routines::reason(37):name=fips error:0700006D:configuration file routines::module initialization error:module=providers, value=provider_sect retcode=-1 )


Inside /snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/ I just have "legacy.so" and find /snap/certbot/4325/usr/lib/x86_64-linux-gnu/ -name "*fips*" gives nothing.

Running "nginx -c /etc/nginx/nginx.conf -t " directly from the console displays no errors.

I've tried to reinstall certbot with snap and install the latest edge version, but nothing worked.

Thanks in advance for your help.

That is very strange. And, I don't have any suggestions. If no other volunteer here offers help you might try the EFF's github for Certbot: Issues · certbot/certbot · GitHub

This looks to me like a possible packaging problem. And Certbot v3 had a couple migration issues although not identical to this.

A work-around would be to change from using the --nginx option to --webroot. The --webroot option does not need to parse your nginx so does not issue the nginx -t command.

There is also a reconfigure command in current versions of Certbot. Reconfigure is a pretty easy way to switch methods. You should also use a --deploy-hook with --webroot to reload nginx after getting a fresh cert. Let us know if you want help with that.

2 Likes

Hi MikeMcQ, thank you very much.

Meanwhile I investigate "reconfigure" (got the same error), I've succesfully managed to renew the certificate by just simply moving the .conf files outside nginx, restarting nginx, and then renew worked.

After that I moved the .conf files for nginx again to their place and restarted.

You got the "same" error using reconfigure to change to --webroot option? Because that is even stranger than your first post :slight_smile:

I don't quite understand what you did but you can test renewal using

sudo certbot renew --dry-run

It will not affect your production certs. But will test renewal.

1 Like

The error popped up while invoking nginx -c, so it's not a Certbot error but an Nginx error. nginx -c just checks the config file, it doesn't invoke certbot.

It looks like your nginx was somehow built against the Certbot OpenSSL libraries, and that (perhaps) you started with a fips distribution and then turned it off. That could have happened within Snap or on the main OS.

Do you remember how nginx was installed and if you toggled any fips support?

Edit:

Just to clarify the above:

Certbot invokes Nginx, not the other way around:

The issue is likely happening here, when Certbot runs a config_test:

The raised exception is not within Certbot code, it's within Nginx and due to nginx referencing a shared library of a fips implementation of OpenSSL that was taken off the machine sometime after being compiled.

This is weird. A first I thought it might be related to this:

But that's in the certbot install and should be sandboxed away from nginx and what was raising that exception.

I'm trying to figure out ways this could happen, and I do think it's likely from certbot's distribution and more likely from some chnge on the system.

3 Likes

The Certbot --nginx plugin issues such a command. I agree with your comment but the error only occurred running that command during a Certbot renew (presumably with --nginx option). Running that same command on the command line produced no error.

You may well be right about the mixed up openssl packages. That's outside my range :slight_smile:

1 Like

You posted that just as I was typing the above clarifications. The error is being raised within Nginx code in a subprocess, which should be picking up the system nginx, so there's no certbot code involved.

Perhaps this could be debugged a bit by running in the line above that run_scrpt invocation:

stdout, stderr = util.run_script(["which", "nginx", ])
print ("stdout:", stdout)

that should show which nginx Certbot is picking up and help figurout out what is going on. perhaps a version got installed into the snap sandbox somehow and that is shadowing the system nginx ?

1 Like

Hi! I think I made a mistake here and ran reconfigure with --webroot-path and without --webroot firs, so it used nginx plugin and hence the error.

1 Like

Hi @jvanasco , thank you

Do you remember how nginx was installed and if you toggled any fips support?

I'm sorry I can't remember, but I'm 99% sure I installed it with yum from the RHEL 9.5 repos. Regarding FIPS, no, I can't remember too.

I'l try to run some of the suggested commands to see if I can reproduce the error. I've another mirror machine that must have the same configuration too.

Here's the output of the command from another mirror machine

/usr/bin/certbot renew --post-hook 'service nginx reload' --force-renewal

nginx: [alert] OPENSSL_init_ssl() failed (SSL: error:12800067:DSO support routines::could not load the shared library:filename(/snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so): /snap/certbot/4325/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so: cannot open shared object file: No such file or directory error:12800067:DSO support routines::could not load the shared library error:07880025:common libcrypto routines::reason(37):name=fips error:0700006D:configuration file routines::module initialization error:module=providers, value=provider_sect retcode=-1 )
nginx: [warn] the "ssl" directive is deprecated, use the "listen ... ssl" directive instead in /etc/nginx/sites-enabled/REDACTED-api-auth.conf:38

uname -a

Linux ip-172-31-45-171.eu-west-1.compute.internal 5.14.0-427.13.1.el9_4.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 10 10:29:16 EDT 2024 x86_64 x86_64 x86_64 GNU/Linux

sudo yum info nginx

Last metadata expiration check: 0:21:51 ago on Sat 18 Jan 2025 09:03:21 AM CET.
Installed Packages
Name : nginx
Epoch : 2
Version : 1.20.1
Release : 20.el9
Architecture : x86_64
Size : 148 k
Source : nginx-1.20.1-20.el9.src.rpm
Repository : @System
From repo : rhel-9-appstream-rhui-rpms
Summary : A high performance web server and reverse proxy server
URL : https://nginx.org
License : BSD
Description : Nginx is a web server and a reverse proxy server for HTTP, SMTP, POP3 and
: IMAP protocols, with a strong focus on high concurrency, performance and low
: memory usage.

sudo fips-mode-setup --check

Installation of FIPS modules is not completed.

FIPS mode is disabled.

Yes, the --webroot authenticator option during reconfigure is needed to replace the --nginx authenticator. Trying reconfigure with all needed options should work-around this problem.

I've already noted the packaging issue is outside my range. I'll let @jvanasco continue with that :slight_smile: Or someone else with better knowledge about those internals.

2 Likes

I think this is likely a Certbot Issue now, and I would post the above information there.

They took out FIPS detection a few weeks ago; but IIRC should be happening in Python code or rust bindings. This looks to be caused by Certbot somehow picking up an nginx binary that was compiled against a fips version of OpenSSL within the sandboxed snap environment.

Short of trying to reinstall snapd and nginx to see if that fixes things, I'm not sure how to address this or what is the root cause :man_shrugging:

3 Likes

Hi @jvanasco For the record, reinstalling snapd/nginx didn't work.

Not sure if this is the same issue, but it's similar enough that it might help steer in the right direction...

I noticed a similar error when renewing certificates with DNS validation, where I use some custom scripts for managing DNS zones (adding / removing acme-challenge records).

And when these scripts ran via certbot hooks, they reported this error:

tls.c:90:tls_initialize(): fatal error: RUNTIME_CHECK(OPENSSL_init_ssl(OPENSSL_INIT_ENGINE_ALL_BUILTIN | OPENSSL_INIT_LOAD_CONFIG, NULL) == 1) failed
Aborted (core dumped)

After some debugging I found out that the issue was caused by certobot setting and exporting an environment variable OPENSSL_FORCE_FIPS_MODE="0", which in turn caused Bind commands (such as rndc and named-checkconf) to fail with the above error.

My solution was to simply unset this variable in my scripts.

Whatever caused this happened after January 6th as renewals worked then, but I wasn't able to find out exactly what it was.

Hope this helps a bit...

1 Like

I think you may have the opposite problem - the OP's system is looking for fips that isn't there.

Your problem was caused by this PR that I linked above snap: disable FIPS detection by wgreenberg · Pull Request #10067 · certbot/certbot · GitHub

That landed in the
3.1.0
release about 2 weeks ago.

Snapd updates pacakges 4x daily by default, and although they slowly phase updates out, that fix should have landed on the OP's machine a few weeks ago.

@driade The last thing I can think of is to share your certbot --version and if it's somehow less than 3.1.0 forcing snapd to update certbot. Your system should have updated to that 2 weeks ago though.

2 Likes

Hi @jvanasco it's 3.1.0

Thanks!

2 Likes

On AlmaLinux 9 was for me helpful:
yum install python3-certbot-nginx

@servekon That's not applicable here, as that wouldn't use the snap method of installing Certbot, but using the distro's own package manager (which is not recommended by the Certbot team).

1 Like