Supposed Nginx misconfiguration stops certbot from working

SebasFashionmusthave · February 5, 2025, 12:37pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: themusthaves.co.uk

I ran this command: certbot --nginx -d themusthaves.co.uk

It produced this output: Saving debug log to /var/log/letsencrypt/letsencrypt.log
Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: configuration file /etc/nginx/nginx.conf test failed

The nginx plugin is not working; there may be problems with your existing configuration.
The error was: MisconfigurationError('Error while running nginx -c /etc/nginx/nginx.conf -t.\n\nnginx: configuration file /etc/nginx/nginx.conf test failed\n')

My web server is (include version): nginx version: nginx/1.25.4

The operating system my web server runs on is (include version): PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"

My hosting provider, if applicable, is: ANS (ans.co.uk)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 3.1.0

For about a year certbot worked fine on several servers we are hosting. Now from one moment to the next, about 4 weeks ago, certbot has stopped on all four machines. The four machines all have the same installations of Alma Linux, Nginx etc.

The thing is, when I run 'nginx -t' or 'service nginx reload' it works fine. I am quite sure there are no problems with the Nginx configuration.

Yet when I try to renew certbot certificates things fail. Things also fail when the daily cron runs to update the certificates. Certbot is installed with the appropriate nginx package and with snap.

Does anyone have an idea what the actual problem here can be?

Kind regards,

Sebastiaan

MikeMcQ · February 5, 2025, 12:58pm

Not yet but let's gather some more info. Would you show output of these

sudo nginx -c /etc/nginx/nginx.conf -t
sudo systemctl status -l --no-pager nginx

SebasFashionmusthave · February 5, 2025, 1:23pm

I have added an attachment here so you can see I am not making things up.

MikeMcQ · February 5, 2025, 1:38pm

What about this one? I was trying to confirm your nginx install matched the Certbot default selection.

I was also wanting to check if it was a new permissions problem by asking to see result w/sudo

SebasFashionmusthave · February 5, 2025, 1:51pm

I added the output from the first command in the screenshot as well. To verify I will post the output again:

[root@i-d3757544 httpdocs]# sudo nginx -c /etc/nginx/nginx.conf -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

MikeMcQ · February 5, 2025, 2:00pm

Oh, sorry, missed it. And, does Certbot run as root ?

Was there any further error text? Maybe in the Certbot log? Searching for similar errors I usually see an explanation after the Misconfiguration line. Like this thread: I need help to install and config ssl certificate on my app

SebasFashionmusthave · February 5, 2025, 2:13pm

I wished I could upload a txt file but it seems that is not possible. I added output of 'ps aux' as attachment. In this I am assuming that snapd is actually the user responsible to run certbot. Snapd is running as root.

Hereby the output of the /var/log/letsencrypt/letsencrypt.log for today:

2025-02-05 12:31:05,328:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2025-02-05 12:31:05,447:DEBUG:certbot._internal.main:certbot version: 3.1.0
2025-02-05 12:31:05,447:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/4325/bin/certbot
2025-02-05 12:31:05,447:DEBUG:certbot._internal.main:Arguments: ['--nginx', '-d', 'themusthaves.co.uk', '--preconfigured-renewal']
2025-02-05 12:31:05,447:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2025-02-05 12:31:05,467:DEBUG:certbot._internal.log:Root logging level set at 30
2025-02-05 12:31:05,467:DEBUG:certbot._internal.plugins.selection:Requested authenticator nginx and installer nginx
2025-02-05 12:31:05,475:ERROR:certbot.util:Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: configuration file /etc/nginx/nginx.conf test failed

2025-02-05 12:31:05,475:DEBUG:certbot._internal.plugins.disco:Misconfigured PluginEntryPoint#nginx: Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: configuration file /etc/nginx/nginx.conf test failed
Traceback (most recent call last):
File "/snap/certbot/4325/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 1008, in config_test
util.run_script([self.conf('ctl'), "-c", self.nginx_conf, "-t"])
File "/snap/certbot/4325/lib/python3.12/site-packages/certbot/util.py", line 199, in run_script
raise errors.SubprocessError(msg)
certbot.errors.SubprocessError: Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: configuration file /etc/nginx/nginx.conf test failed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/snap/certbot/4325/lib/python3.12/site-packages/certbot/_internal/plugins/disco.py", line 112, in prepare
self._initialized.prepare()
File "/snap/certbot/4325/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 204, in prepare
self.config_test()
File "/snap/certbot/4325/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 1010, in config_test
raise errors.MisconfigurationError(str(err))
certbot.errors.MisconfigurationError: Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: configuration file /etc/nginx/nginx.conf test failed

2025-02-05 12:31:05,476:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: Authenticator, Installer, Plugin
Entry point: EntryPoint(name='nginx', value='certbot_nginx._internal.configurator:NginxConfigurator', group='certbot.plugins')
Initialized: <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7fcc6493e570>
Prep: Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: configuration file /etc/nginx/nginx.conf test failed

MikeMcQ · February 5, 2025, 2:23pm

Ah, that's a good clue. Is it possible you have a mix of snap and non-stop nginx plugin?

Did you change from non-snap to snap about a month ago when this started failing?

That's about as far as my knowledge of Certbot packaging goes. If that's not enough for you to figure out what's wrong you'll have to wait for another volunteer. Maybe @Osiris ?

Osiris · February 5, 2025, 5:42pm

Not sure what's going on. The code on that line 1008 isn't that difficult:

github.com/certbot/certbot

certbot-nginx/certbot_nginx/_internal/configurator.py

dcd52b071


      
              """
              nginx_restart(self.conf('ctl'), self.nginx_conf, self.conf('sleep-seconds'))
          
          def config_test(self) -> None:
              """Check the configuration of Nginx for errors.
          
              :raises .errors.MisconfigurationError: If config_test fails
          
              """
              try:
                  util.run_script([self.conf('ctl'), "-c", self.nginx_conf, "-t"])
              except errors.SubprocessError as err:
                  raise errors.MisconfigurationError(str(err))
          
          def _nginx_version(self) -> str:
              """Return results of nginx -V
          
              :returns: version text
              :rtype: str
          
              :raises .PluginError:

Did it work with previous versions of Certbot? Perhaps check if you have older versions of Certbot laying around with sudo snap list certbot and perhaps revert to an older version with sudo snap revert certbot --revision $revisionnumberofoldercertbot.

MikeMcQ · February 5, 2025, 5:53pm

Do you know where this comes from?

Osiris · February 5, 2025, 5:59pm

Yeah, that's just because the MisconfigurationError from the code above propagates up the 'chain' of function calls and ultimately the code that tried to include the nginx plugin into the Certbot main application catches that MisconfigurationError in the code

github.com/certbot/certbot

certbot/certbot/_internal/plugins/disco.py

dcd52b071


      
          if self._prepared is None:
              try:
                  self._initialized.prepare()
              except errors.MisconfigurationError as error:
                  logger.debug("Misconfigured %r: %s", self, error, exc_info=True)

and logs that error. That "PluginEntryPoint" is I guess something to ignore, it just says it was the nginx plugin that malfunctioned due to some kind of misconfiguration.

Too bad the output of util.run_script isn't included in the log.. As in, WHY did that command fail?

Verement · February 20, 2025, 4:04am

I have started getting this error as well. Server has been working fine for over a year and on the last renewal attempt it stared failing.

Certbot says:

Error while running nginx -c /etc/nginx/nginx.conf -t.

nginx: configuration file /etc/nginx/nginx.conf test failed

The nginx plugin is not working; there may be problems with your existing configuration.
The error was: MisconfigurationError('Error while running nginx -c /etc/nginx/nginx.conf -t.\n\nnginx: configuration file /etc/nginx/nginx.conf test failed\n')

Running nginx -c /etc/nginx/nginx/conf -t returns:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

The site is up and running just fine over port 80 at http://sevendooms.rpgcampaign.tools .

Running systemctl status nginx shows nginx running with no issue.

Certbot is certbot 3.2.0
nginx is 2:1.27.4-1.el9.ngx
OS is AlmaLinux release 9.3 (Shamrock Pampas Cat)

There is one file in /etc/nginx/conf.d named sevendooms.conf with the following:

server {
  server_name   sevendooms.rpgcampaign.tools;

  client_max_body_size  300M;

  listen 80;

  location / {
    proxy_set_header   Host $host;
    proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header   X-Forwarded-Proto $scheme;

    proxy_set_header  Upgrade $http_upgrade;
    proxy_set_header  Connection "Upgrade";

    proxy_pass        http://127.0.0.1:30001;
  }
}

MikeMcQ · February 20, 2025, 4:20am

@Verement Normally we like each person to start their own thread. In your case though this looks eerily similar.

We didn't have any good suggestions to the first person and no other volunteer offered insights either.

I suggest posting your problem at the EFF's github for Certbot. Their dev team will more readily see this problem there. EFF github here: GitHub · Where software is built Link to this thread and note that you and another person using Alma Linux were working fine until recent update to Certbot version.

You could switch to using --webroot method rather than --nginx. Almost certainly will avoid this problem. A couple "tricks" would make that transition easier so let us know if you want advice for that.

SebasFashionmusthave · February 20, 2025, 8:18am

I have to say though, I have 5 web servers and on two of those web servers certbot still works fine. On three of them, certbot has failed simultaneously.

All these five servers are the same when I look at the software on them, like their Alma Linux, OpenSSL, Nginx versions and everything.

The only noticeable difference really is that the three web servers where certbot failed are connected through a shared hard drive, and these three web servers are connected to another server which hosts their databases. The shared hard drive is for the website files and code, which each of those servers also still has a local hard drive for the overall software like Nginx, Redis etc.

The two web servers where certbot still runs are hosting everything on their own local hard drive.

Do you have any sort of similar setup?

jvanasco · February 20, 2025, 5:25pm

I think this may be the same, or a variant, of this issue:

github.com/certbot/certbot

Certbot halts while renewing because can't start Nginx: `/snap/certbot/4412/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so`: No such file or directory error

opened 12:11PM - 13 Feb 25 UTC

Vanav

## My operating system is (include version) CentOS 9.5 (Oracle Linux), Certbot 3….2.0 (snap), Nginx 1.26.2 (nginx.org repository). Certbot is installed using Snap as [the recommended way](https://eff-certbot.readthedocs.io/en/latest/install.html). ## I ran this command and it produced this output `certbot renew`: > 2025-02-13 13:26:13,598:DEBUG:certbot._internal.plugins.selection:Requested authenticator nginx and installer nginx Certbot halts on this line. Nginx `error.log`: > 2025/02/13 13:26:13 [alert] 138461#138461: OPENSSL_init_ssl() failed (SSL: error:12800067:DSO support routines::could not load the shared library:filename(/snap/certbot/4412/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so): /snap/certbot/4412/usr/lib/x86_64-linux-gnu/ossl-modules/fips.so: cannot open shared object file: No such file or directory error:12800067:DSO support routines::could not load the shared library error:07880025:common libcrypto routines::reason(37):name=fips error:0700006D:configuration file routines::module initialization error:module=providers, value=provider_sect retcode=-1 ) Nginx halts on this line with 100% one core CPU usage. ## Certbot's behavior and environment Nginx halts with 100% one core CPU usage, and Certbot halts too waiting for Nginx. Contents of directories: ``` # ls -al /snap/certbot/current/usr/lib/x86_64-linux-gnu/ossl-modules -rw-r--r-- 1 root root 133584 Feb 11 23:18 legacy.so # ls -al /usr/lib64/ossl-modules -rwxr-xr-x 1 root root 1337160 Nov 13 01:52 fips.so -rwxr-xr-x 1 root root 136336 Oct 30 16:38 legacy.so ``` FIPS status on OS: ``` # fips-mode-setup --check Installation of FIPS modules is not completed. FIPS mode is disabled. # cat /proc/sys/crypto/fips_enabled 0 ``` ## Workaround `/etc/fstab`: ``` /usr/lib64/ossl-modules /snap/certbot/current/usr/lib/x86_64-linux-gnu/ossl-modules none bind 0 0 ``` Not a stable workaround, because when Certbot autoupdates, `fips.so` will be missing again. ## Discussion This issue reported by multiple users: https://community.letsencrypt.org/t/certbot-snap-error-while-renewing-openssl-init-ssl/232145 Maybe related and more discussion: #10044.

which was previously discussed here: Certbot snap Error while renewing (OPENSSL_init_ssl)

In all situations, it's this configtest that is randomly failing on some platforms.

bmw · February 26, 2025, 1:40am

for those hitting this, if you run certbot and get this error, does anything interesting show up in nginx's error log afterwards? i believe that is at /var/log/nginx/error.log or another file in that directory on most systems

SebasFashionmusthave · March 18, 2025, 8:14am

As sort of a conclusion for myself for this topic: I have tried to re-install certbot but that didn't have any effect. I have tried to check for errors, but the whole problem is that Nginx or other software on the servers are not giving errors. In fact, I had to reload Nginx multiple times in the meantime for other reasons. That all went without problems.

Now luckily, for me Mike's suggestion has given me an acceptable workaround which is to use the --webroot option. So if anyone else is facing unexplained errors in Nginx, you probably are used to a command like:

cerbot --nginx -d domainname.com -d www.domainname.com

And you will have to replace it with:

certbot --webroot -d domainname.com -d www.domainname.com

And then you will have to submit whatever is the root directory from your domain, which in my case is something like /var/www/vhosts/domainname.com/httpdocs

MikeMcQ · March 18, 2025, 1:24pm

Glad the work-around worked out. I have a couple suggestions ... these are the "tricks" I hinted at in my earlier post

A complete command for --webroot is ideally like:

sudo certbot certonly --webroot -w PATH -d example.com -d www.example.com --deploy-hook CMD

Where:
PATH is the root folder in the server block for this domain
CMD is the command used to reload nginx. Example: "sudo systemctl reload nginx"

The deploy-hook will run that command each time it gets a fresh cert. An nginx reload is needed for nginx to pickup the new cert.

@SebasFashionmusthave You might want to use the sudo certbot reconfigure command to add a deploy-hook. Or re-run the command I just showed with your path and domains. Or, just use a different way to regularly reload nginx (like a daily cron).

system · April 17, 2025, 1:25pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
The nginx plugin is not working; there may be problems with your existing configuration. The error was: MisconfigurationError Help	14	508	October 13, 2023
Certbot is not working Help	1	426	July 23, 2020
Possible NGINX misconfiguration Help	2	71	May 8, 2025
I cant install certbot ssl certificate on vps Help	11	98	March 25, 2025
Failed to Authenticate Help	21	1654	February 9, 2023

Supposed Nginx misconfiguration stops certbot from working

Related topics