Certbot renew segfaults my apache2

My domain is: sunnydale.russenmafia.at

I ran this command: “/usr/bin/certbot renew”
and after i hit the rate limit
“/usr/bin/certbot renew --staging --break-my-certs”

It produced this output:

/usr/bin/certbot renew --staging --break-my-certs

Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/sunnydale.russenmafia.at.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for sunnydale.russenmafia.at
Waiting for verification…
Cleaning up challenges
Attempting to renew cert (sunnydale.russenmafia.at) from /etc/letsencrypt/renewal/sunnydale.russenmafia.at.conf produced an unexpected error: Failed authorization procedure. sunnydale.russenmafia.at (tls-sni-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Timeout after connect (your server may be slow or overloaded). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/sunnydale.russenmafia.at/fullchain.pem (failure)


All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/sunnydale.russenmafia.at/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: sunnydale.russenmafia.at
    Type: connection
    Detail: Timeout after connect (your server may be slow or
    overloaded)

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

My web server is (include version):
Server version: Apache/2.4.33 (Debian)
Server built: 2018-05-28T17:29:02

The operating system my web server runs on is (include version):
debian buster/sid

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

when certbot is on this line:
“tls-sni-01 challenge for sunnydale.russenmafia.at”

all hell breaks lose in my apache error log. lots of segfaults …
[Thu Jun 28 14:20:19.403724 2018] [mpm_prefork:notice] [pid 24352] AH00171: Graceful restart requested, doing restart
[Thu Jun 28 14:20:19.438262 2018] [ssl:warn] [pid 24352] AH01906: e93d5dd467e19245a47dd573950cb21b.f668b4a43afa88719f4eff557103363e.acme.inval
id:443:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Thu Jun 28 14:20:19.438643 2018] [mpm_prefork:notice] [pid 24352] AH00163: Apache/2.4.33 (Debian) mpm-itk/2.4.7-04 OpenSSL/1.1.0h mod_perl/2.
0.10 Perl/v5.26.2 configured – resuming normal operations
[Thu Jun 28 14:20:19.438648 2018] [core:notice] [pid 24352] AH00094: Command line: ‘/usr/sbin/apache2’
[Thu Jun 28 14:20:19.440318 2018] [core:notice] [pid 24352] AH00052: child pid 29045 exit signal Segmentation fault (11)
[Thu Jun 28 14:20:19.440328 2018] [core:error] [pid 24352] AH00546: no record of generation 0 of exiting child 29045
[Thu Jun 28 14:20:19.440331 2018] [core:notice] [pid 24352] AH00052: child pid 29046 exit signal Segmentation fault (11)
[Thu Jun 28 14:20:19.440332 2018] [core:error] [pid 24352] AH00546: no record of generation 0 of exiting child 29046
[Thu Jun 28 14:20:19.440335 2018] [core:notice] [pid 24352] AH00052: child pid 29047 exit signal Segmentation fault (11)
[Thu Jun 28 14:20:19.440336 2018] [core:error] [pid 24352] AH00546: no record of generation 0 of exiting child 29047
[Thu Jun 28 14:20:20.442505 2018] [core:notice] [pid 24352] AH00052: child pid 29048 exit signal Segmentation fault (11)
[Thu Jun 28 14:20:20.442573 2018] [core:error] [pid 24352] AH00546: no record of generation 0 of exiting child 29048
[Thu Jun 28 14:20:20.442593 2018] [core:notice] [pid 24352] AH00052: child pid 29049 exit signal Segmentation fault (11)
[Thu Jun 28 14:20:20.442601 2018] [core:error] [pid 24352] AH00546: no record of generation 0 of exiting child 29049
[Thu Jun 28 14:20:21.443722 2018] [core:notice] [pid 24352] AH00052: child pid 29126 exit signal Segmentation fault (11)

after that, my apache2 is in defunct state, not reacting anymore. i have to restart the whole service.

and i have absolutely no clue why it segfaults.

Can't act too surprised then :stuck_out_tongue: .

Can you configure CoreDumpDirectory so it's possible to figure out whether this is crash in OpenSSL or Apache?

tried that, but it won’t write any dumps
configured it to /tmp (which is 1777)
even put “ulimit -c unlimited” into /etc/init.d/apache2 in the do_start function, just to be sure

apache2 uses [PrivateTmp] in Buster, you’d need to use some other location. Check selinux too.

sestatus

SELinux status: disabled

grep Pri /etc/systemd/system/apache2.service

PrivateTmp=false

and then i even tried to set it to a directory with 700 permissions for user www-data. still no cores written.
even though i see that the “/usr/sbin/apache2 -k start” + it’s segfaulting childs are changing directory to the configured dumpdir. (with strace -f)

That’s unfortunate, maybe the Certbot devs will have some idea (@bmw). I wasn’t able to reproduce a crash with TLS-SNI under sid with:

Server version: Apache/2.4.33 (Debian)
Server built:   2018-05-28T17:29:02

OpenSSL 1.1.0h  27 Mar 2018

certbot 0.25.0

so it doesn’t look easily reproducible either :frowning: .

Maybe systemd has the core dump?

sudo coredumpctl list

just got systemd to capture coredumps, here’s the gdb output of one:

coredumpctl gdb 20261

       PID: 20261 (/usr/sbin/apach)
       UID: 0 (root)
       GID: 0 (root)
    Signal: 11 (SEGV)
 Timestamp: Thu 2018-06-28 19:47:53 CEST (4min 18s ago)

Command Line: /usr/sbin/apache2 -k start
Executable: /usr/sbin/apache2
Control Group: /system.slice/apache2.service
Unit: apache2.service
Slice: system.slice
Boot ID: fb5bb58db2c4417db6cce49bb7b04435
Machine ID: 6eb9f0854f630f342494ccf20000000a
Hostname: sunnyserver
Storage: /var/lib/systemd/coredump/core.\x2fusr\x2fsbin\x2fapach.0.fb5bb58db2c4417db6cce49bb7b04435.20261.1530208073000000.lz4
Message: Process 20261 (/usr/sbin/apach) of user 0 dumped core.

            Stack trace of thread 20261:
            #0  0x00007fa235131677 n/a (libcap-ng.so.0)
            #1  0x00007fa2429e2a25 n/a (mod_mpm_prefork.so)
            #2  0x00007fa2429e3a0e n/a (mod_mpm_prefork.so)
            #3  0x0000561918c4cb7e ap_run_mpm (apache2)
            #4  0x0000561918c4546b main (apache2)
            #5  0x00007fa247386a87 __libc_start_main (libc.so.6)
            #6  0x0000561918c4556a _start (apache2)

GNU gdb (Debian 7.12-6+b2) 7.12.0.20161007-git
Copyright © 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type “show copying”
and “show warranty” for details.
This GDB was configured as “x86_64-linux-gnu”.
Type “show configuration” for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type “help”.
Type “apropos word” to search for commands related to “word”…
Reading symbols from /usr/sbin/apache2…(no debugging symbols found)…done.

warning: core file may not match specified executable file.
[New LWP 20261]
[Thread debugging using libthread_db enabled]
Using host libthread_db library “/lib/x86_64-linux-gnu/libthread_db.so.1”.
Core was generated by `/usr/sbin/apache2 -k start’.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fa235131677 in ?? () from /lib/x86_64-linux-gnu/libcap-ng.so.0

(gdb) bt
#0 0x00007fa235131677 in ?? () from /lib/x86_64-linux-gnu/libcap-ng.so.0
#1 0x00007fa24742962e in __libc_fork () at …/sysdeps/nptl/fork.c:204
#2 0x00007fa2429e2a25 in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#3 0x00007fa2429e3a0e in ?? () from /usr/lib/apache2/modules/mod_mpm_prefork.so
#4 0x0000561918c4cb7e in ap_run_mpm ()
#5 0x0000561918c4546b in main ()
(gdb)

Does apachectl graceful, with no involvement from Certbot, also produce the crashes?

yes.
apachectl graceful
and
apachectl restart

both induce the crash, and only a /etc/init.d/apache2 restart fixes it
guess i’ll have to open a ticket on the debian bugtracker :slight_smile:

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.