Certbot throws segmentation fault

Certbot throws a segmentation fault with "certbot --apache" or "certbot --apache -d 'domain'" but not with "certbot renew". It ran flawlessly a few weeks ago, because I setup the server from scratch. No problem adding domains, nothing. Of curse system updates were done and the system has the latest updates installed. Then suddenly the problem occured.

OS: Ubuntu 22.04.2 LTS
Tried with
Certbot installed via apt: 1.21.0
Certbot installed via snap: 2.3.0

Below output of commands for each version (1.21.0 and 2.3.0 farther below):

certbot 1.21.0:

certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Segmentation fault (core dumped)

/var/log/letsencrypt/letsencrypt.log:

2023-02-28 19:42:40,802:DEBUG:certbot._internal.main:certbot version: 1.21.0
2023-02-28 19:42:40,803:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2023-02-28 19:42:40,803:DEBUG:certbot._internal.main:Arguments:
2023-02-28 19:42:40,803:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,Plug inEntryPoint#webroot)
2023-02-28 19:42:40,817:DEBUG:certbot._internal.log:Root logging level set at 30
2023-02-28 19:42:40,818:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2023-02-28 19:42:41,141:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.52
2023-02-28 19:42:48,479:DEBUG:certbot._internal.main:certbot version: 1.21.0
2023-02-28 19:42:48,480:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2023-02-28 19:42:48,480:DEBUG:certbot._internal.main:Arguments: ['--apache']
2023-02-28 19:42:48,480:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,Plug inEntryPoint#webroot)
2023-02-28 19:42:48,487:DEBUG:certbot._internal.log:Root logging level set at 30
2023-02-28 19:42:48,487:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2023-02-28 19:42:48,904:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.52
2023-02-28 19:50:14,984:DEBUG:certbot._internal.main:certbot version: 1.21.0
2023-02-28 19:50:14,984:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2023-02-28 19:50:14,984:DEBUG:certbot._internal.main:Arguments: ['--apache']
2023-02-28 19:50:14,984:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,Plug inEntryPoint#webroot)
2023-02-28 19:50:14,991:DEBUG:certbot._internal.log:Root logging level set at 30
2023-02-28 19:50:14,992:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2023-02-28 19:50:15,237:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.52
2023-02-28 19:55:03,249:DEBUG:certbot._internal.main:certbot version: 1.21.0
2023-02-28 19:55:03,249:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2023-02-28 19:55:03,249:DEBUG:certbot._internal.main:Arguments: ['--apache']
2023-02-28 19:55:03,250:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,Plug inEntryPoint#webroot)
2023-02-28 19:55:03,258:DEBUG:certbot._internal.log:Root logging level set at 30
2023-02-28 19:55:03,259:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2023-02-28 19:55:03,549:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.52

certbot 2.3.0
certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Segmentation fault (core dumped)

/var/log/letsencrypt/letsencrypt.log:

2023-02-28 19:57:07,080:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2023-02-28 19:57:07,348:DEBUG:certbot._internal.main:certbot version: 2.3.0
2023-02-28 19:57:07,348:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/2772/bin/certbot
2023-02-28 19:57:07,348:DEBUG:certbot._internal.main:Arguments: ['-v', '--preconfigured-renewal']
2023-02-28 19:57:07,349:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-02-28 19:57:07,356:DEBUG:certbot._internal.log:Root logging level set at 20
2023-02-28 19:57:07,356:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2023-02-28 19:57:07,587:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.52
2023-02-28 19:57:25,362:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2023-02-28 19:57:25,619:DEBUG:certbot._internal.main:certbot version: 2.3.0
2023-02-28 19:57:25,619:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/2772/bin/certbot
2023-02-28 19:57:25,619:DEBUG:certbot._internal.main:Arguments: ['--apache', '--preconfigured-renewal']
2023-02-28 19:57:25,619:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-02-28 19:57:25,627:DEBUG:certbot._internal.log:Root logging level set at 30
2023-02-28 19:57:25,627:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2023-02-28 19:57:25,882:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.52

syslog:

Feb 28 20:07:19 reverseproxy01 systemd[1]: Started snap.certbot.certbot.989dd26b-ddbc-4d45-9e8f-ddef15cf4a0f.scope.
Feb 28 20:07:21 reverseproxy01 kernel: [10042.873834] python3[11754]: segfault at 7ffe9991dff8 ip 00007efc382c5929 sp 00007ffe9991e000 error 6 in libaugeas.so.0.24.2[7efc38299000+44000]
Feb 28 20:07:21 reverseproxy01 kernel: [10042.873849] Code: 00 31 c9 ba f9 00 00 00 48 8d 35 df ec 01 00 31 c0 e8 ab f2 ff ff eb bc 66 0f 1f 84 00 00 00 00 00 41 57 45 89 c7 41 56 41 55 <41> 54 49 89 f4 55 4c 89 cd 53 48 89 fb 48 83 ec 28 89 54 24 04 64
Feb 28 20:07:21 reverseproxy01 systemd[1]: snap.certbot.certbot.989dd26b-ddbc-4d45-9e8f-ddef15cf4a0f.scope: Deactivated successfully.
Feb 28 20:07:21 reverseproxy01 systemd[1]: snap.certbot.certbot.989dd26b-ddbc-4d45-9e8f-ddef15cf4a0f.scope: Consumed 1.811s CPU time.

You should search for an existing ticket against Certbot's issue tracker, or file a new one. GitHub - certbot/certbot: Certbot is EFF's tool to obtain certs from Let's Encrypt and (optionally) auto-enable HTTPS on your server. It can also act as a client for any other CA that uses the ACME protocol.

Based on your description, the issue is most likely in the Apache plugin and is most likely related to the parsing of your Apache configuration files. The parsing errors are usually caused by having some unsupported plugin or syntax invoked within the configuration file.

You should be able to use Certbot in standalone mode to obtain new certificates.

2 Likes

Technically python3 had the segfault
Would be nice to add the python version to the reported information, and possibly even the pip3 list output.

1 Like

I would very much recommend the webroot plugin instead of the standalone!

Not sure if it's Python as Bruce suggested or libaugeas.so, the library Certbot uses for parsing the Apache configuration files. Can you perhaps update augeas? Might be called libaugeas on your system.

2 Likes

it looks like a crash in Augeas, which is the native library that Certbot uses to parse your Apache configuration. Two possibilities:

  • You have stumbled across a bug in Augeas which is triggered by a certain piece of your Apache configuration.
  • A system update has broken something in Augeas

Augeas hasn't been updated in Ubuntu since December 2021, so I think the latter case is more likely: the crash is by something specific in your Apache configuration.

To investigate this, we'd probably need to see a full copy of your Apache configuration files. Sorry, I know that's annoying.

3 Likes

Hm, but OP is running Ubuntu 22.0.4 LTS, a.k.a. "Jammy". The segfault says it's in the file libaugeas.so.0.24.2. But when comparing the file lists for the jammy and focal (20.04 LTS), we can see:

Jammy packing version 1.13.0 (Ubuntu – Details of package libaugeas0 in jammy and Ubuntu – File list of package libaugeas0/jammy/amd64):

(…)
/usr/lib/x86_64-linux-gnu/libaugeas.so.0.25.0
(…)

Focal packing version 1.12.0 (Ubuntu – Details of package libaugeas0 in focal and Ubuntu – File list of package libaugeas0/focal/amd64):

(…)
/usr/lib/x86_64-linux-gnu/libaugeas.so.0.24.2
(…)

So OP seems to have a version from focal installed (20.04) instead of the one for jammy (22.04)? Perhaps it can be upgraded from 1.12.0 to 1.13.0 still?

2 Likes

Augeas comes from the snap, which is based on core20. I should have linked the changelog for focal, which is updated even less recently than for jammy!

2 Likes

Hm, OK. Then if OPs system Augeas is up to date to 1.13.0 (but I haven't seen proof of that yet) I guess it doesn't matter which version of Augeas is used :slight_smile:

2 Likes

I managed to reproduce this on Ubuntu 22.04 (sort of, the EIP points to a different instruction, but it's close by) by having a very large Apache configuration, which is a known issue:

[ 1151.924549] python3[3161]: segfault at 7ffee210cff8 ip 00007fe17538595e sp 00007ffee210d000 error 6 in libaugeas.so.0.24.2[7fe175359000+44000]
[ 1151.924564] Code: 89 fb 48 83 ec 28 89 54 24 04 64 48 8b 04 25 28 00 00 00 48 89 44 24 18 31 c0 48 8b 07 48 8b 70 20 48 8b 78 08 48 89 44 24 08 <e8> 4d ff ff ff 8b 48 0c 85 c9 0f 84 62 01 00 00 48 8b 70 10 8d 51

It would be helpful to know how big your Apache configuration is:

# find /etc/apache2 -type f -print0 | wc -l --files0-from=- | tail -n1
296385 total

Your crash seems to be at the start of this function in Augeas, which unfortunately doesn't give me any hints at all without seeing the configuration. But it is the same function that crashes when the configuration is too large.

5 Likes

Thanks to all for your input. It was a large configuration.

There is this file with 131606 lines: /etc/apache2/conf-available/ip-blacklist.conf

Disabling this configuration and it worked:

a2disconf ip-blacklist
systemctl reload apache2

certbot --apache

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Which names would you like to activate HTTPS for?
We recommend selecting either all domains, or all domains in a VirtualHost/server block.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[... text removed ...]
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 

Would be nice if there was some form of error message or maybe even a fix that will prevent a crash on large configs.

3 Likes

Why in the world would you block IPs that way????
[that doesn't scale well - as you've noticed]

Try using --webroot instead.

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.