Renewing Certificate fails. Timeout after connect (your server may be slow or overloaded)

Hello all,
I read a lot about my issue and tried a lot but nothing helped. I first recognized problems with my apache, which was not available. The log was flooded with the following errors. Over google I landed here and found out the certbot was causing this.

[Wed Sep 12 00:00:15.502607 2018] [core:notice] [pid 17700] AH00052: child pid 18391 exit signal Segmentation fault (11)
[Wed Sep 12 00:00:15.502662 2018] [core:error] [pid 17700] AH00546: no record of generation 0 of exiting child 18391

According to the mxtoolbox my DNS is fine, and my server seems to be reachable over port 443 and port 80.
It is getting late and I think I will go to sleep. Maybe you can give me a hint where this may come from?

My domain is: gnaarl.de

I ran this command: certbot renew

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gnaarl.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for gnaarl.de
tls-sni-01 challenge for mail.gnaarl.de
tls-sni-01 challenge for www.gnaarl.de
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (gnaarl.de) from /etc/letsencrypt/renewal/gnaarl.de.conf produced an unexpected error: Failed authorization procedure. gnaarl.de (tls-sni-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Timeout after connect (your server may be slow or overloaded), mail.gnaarl.de (tls-sni-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Timeout after connect (your server may be slow or overloaded), www.gnaarl.de (tls-sni-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Timeout after connect (your server may be slow or overloaded). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gnaarl.de/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gnaarl.de/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: gnaarl.de
   Type:   connection
   Detail: Timeout after connect (your server may be slow or
   overloaded)

   Domain: mail.gnaarl.de
   Type:   connection
   Detail: Timeout after connect (your server may be slow or
   overloaded)

   Domain: www.gnaarl.de
   Type:   connection
   Detail: Timeout after connect (your server may be slow or
   overloaded)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version): Apache 2.4

The operating system my web server runs on is (include version): debian buster/sid

My hosting provider, if applicable, is: self hosted

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

Thanks in advance to everyone reading this!

Hi,

I’m not 100% certain what’s the issue… But my guess is TLS-SNI-01 made some change to your virtual host that is overloading your server…
Since TLS-SNI-01 is a depreciated challenge type for new issuance & sometimes cause some issue on existing domain renewals, could you please try to run the command with --preferred-challenge http-01 ?

The full command: sudo certbot renew --preferred-challenge http-01

Thank you

The message differs a bit but again the apache crashes and the cert is not renewed.
Maybe edit the options-ssl-apache.conf ?

 certbot renew --preferred-challenge http-01
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gnaarl.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for gnaarl.de
http-01 challenge for mail.gnaarl.de
http-01 challenge for www.gnaarl.de
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (gnaarl.de) from /etc/letsencrypt/renewal/gnaarl.de.conf produced an unexpected error: Failed                                                                                                                        authorization procedure. www.gnaarl.de (http-01): urn:ietf:params:acme:error:connection :: The server could not conne                                                                                                                       ct to the client to verify the domain :: Fetching http://www.gnaarl.de/.well-known/acme-challenge/51SI4gdplE1VyCxSAAW9                                                                                                                       ZXqUNBEDGsMEmghU72HH3YU: Timeout after connect (your server may be slow or overloaded), gnaarl.de (http-01): urn:ietf:                                                                                                                       params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://gna                                                                                                                       arl.de/.well-known/acme-challenge/yZgLR1QnrT42fB_iyNiAiqlzYanHwx_Y_swGnEc21ak: Timeout after connect (your server may                                                                                                                        be slow or overloaded), mail.gnaarl.de (http-01): urn:ietf:params:acme:error:connection :: The server could not connec                                                                                                                       t to the client to verify the domain :: Fetching http://mail.gnaarl.de/.well-known/acme-challenge/2nnPuHaUwhSHwZh7uDgB                                                                                                                       23gc02vcKplXA9aAWu2MQYY: Timeout after connect (your server may be slow or overloaded). Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gnaarl.de/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gnaarl.de/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.gnaarl.de
   Type:   connection
   Detail: Fetching
   http://www.gnaarl.de/.well-known/acme-challenge/51SI4gdplE1VyCxSAAW9ZXqUNBEDGsMEmghU72HH3YU:
   Timeout after connect (your server may be slow or overloaded)

   Domain: gnaarl.de
   Type:   connection
   Detail: Fetching
   http://gnaarl.de/.well-known/acme-challenge/yZgLR1QnrT42fB_iyNiAiqlzYanHwx_Y_swGnEc21ak:
   Timeout after connect (your server may be slow or overloaded)

   Domain: mail.gnaarl.de
   Type:   connection
   Detail: Fetching
   http://mail.gnaarl.de/.well-known/acme-challenge/2nnPuHaUwhSHwZh7uDgB23gc02vcKplXA9aAWu2MQYY:
   Timeout after connect (your server may be slow or overloaded)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

Hi,

Do you happen to have any firewall is iptables that blocks IPs?

it might be the time to call @lestaff … (There’s literally nothing I could think of that caused this issue…)

Thank you

Hi @Gnaarl

your website is completely down. No reaction, timeout. Tested with FireFox from Berlin.

Website down - no http - 01 - challenge is possible. Look there:

emm…

The site was definitely up when i checked it 1 days ago.

Now your website is up.

https://letsdebug.net/www.gnaarl.de/4869

There is all green / ok, so test it again.

Hi there,
the site was down cause I tested the renewal again and forgot to restart it again.

[Thu Sep 13 20:00:55.419435 2018] [core:error] [pid 12187] AH00546: no record of generation 0 of exiting child 21835
[Thu Sep 13 20:00:55.419506 2018] [mpm_prefork:notice] [pid 12187] AH00169: caught SIGTERM, shutting down
[Thu Sep 13 20:00:55.562057 2018] [mpm_prefork:notice] [pid 21841] AH00163: Apache/2.4.34 (Debian) OpenSSL/1.1.0h configured -- re
suming normal operations
[Thu Sep 13 20:00:55.562153 2018] [core:notice] [pid 21841] AH00094: Command line: '/usr/sbin/apache2'
[Thu Sep 13 20:11:27.668784 2018] [mpm_prefork:notice] [pid 21841] AH00171: Graceful restart requested, doing restart
[Thu Sep 13 20:11:27.808022 2018] [mpm_prefork:notice] [pid 21841] AH00163: Apache/2.4.34 (Debian) OpenSSL/1.1.0h configured -- re
suming normal operations
[Thu Sep 13 20:11:27.808045 2018] [core:notice] [pid 21841] AH00094: Command line: '/usr/sbin/apache2'
[Thu Sep 13 20:11:27.814069 2018] [core:notice] [pid 21841] AH00052: child pid 22017 exit signal Segmentation fault (11)
[Thu Sep 13 20:11:27.814110 2018] [core:error] [pid 21841] AH00546: no record of generation 0 of exiting child 22017
[Thu Sep 13 20:11:27.814122 2018] [core:notice] [pid 21841] AH00052: child pid 22018 exit signal Segmentation fault (11)
[Thu Sep 13 20:11:27.814128 2018] [core:error] [pid 21841] AH00546: no record of generation 0 of exiting child 22018
[Thu Sep 13 20:11:27.814134 2018] [core:notice] [pid 21841] AH00052: child pid 22019 exit signal Segmentation fault (11)

Is there any critical information inside the letsencrypt.log I should not post?

And I could not find that the certbot needs anything besides port 80 or 443 right?

Chain INPUT (policy DROP)
target prot opt source destination
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp – anywhere anywhere
ACCEPT tcp – anywhere anywhere tcp dpt:ssh
ACCEPT tcp – anywhere anywhere tcp dpt:http
ACCEPT tcp – anywhere anywhere tcp dpt:https
[…]

Chain OUTPUT (policy DROP)
target prot opt source destination
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere state RELATED,ESTABLISHED
ACCEPT icmp – anywhere anywhere
ACCEPT tcp – anywhere anywhere tcp dpt:http
ACCEPT tcp – anywhere anywhere tcp dpt:https
[…]

First: Use the http-01 - challenge, not the tls-sni-01 - challenge.

Second: Is this

the problem? Is this a Window with a virtual Linux? If you have (using http-01) this error

Timeout after connect (your server may be slow or overloaded)

it looks like a local block. Rewrite rules? Fetching

http://www.gnaarl.de/.well-known/acme-challenge/1234

sends a http-status 404, this is ok. But I don't see a timeout. Do you use third-party-tools?

Thanks for the fast answers :slight_smile:

First: I did run http-01 - has thrown the same error as before.

Second: Maybe I did not understand the hosting question right. Netcup ist hosting my vServer.

Has been some time since I looked into my apache config. I have to search for my rewrite rules.

hmm… found only the following in an .htaccess file in my owncloud-dir…
With a fat “#### DO NOT CHANGE ANYTHING ABOVE THIS LINE ####”

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteRule .* - [env=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
  RewriteRule ^\.well-known/host-meta /public.php?service=host-meta [QSA,L]
  RewriteRule ^\.well-known/host-meta\.json /public.php?service=host-meta-json [QSA,L]
  RewriteRule ^\.well-known/carddav /remote.php/dav/ [R=301,L]
  RewriteRule ^\.well-known/caldav /remote.php/dav/ [R=301,L]
  RewriteRule ^remote/(.*) remote.php [QSA,L]
  RewriteRule ^(?:build|tests|config|lib|3rdparty|templates)/.* - [R=404,L]
  RewriteCond %{REQUEST_URI} !^/.well-known/acme-challenge/.*
  RewriteRule ^(?:\.|autotest|occ|issue|indie|db_|console).* - [R=404,L]
</IfModule>

And I thought I have seen it in the letsencrypt.log:

2018-09-13 20:11:27,487:INFO:certbot.auth_handler:http-01 challenge for www.gnaarl.de
2018-09-13 20:11:27,499:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: gnaarl.de in: /etc/apache2/sites-enabled/000-default.conf
2018-09-13 20:11:27,500:DEBUG:certbot_apache.http_01:writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]
    
2018-09-13 20:11:27,500:DEBUG:certbot_apache.http_01:writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>
    
2018-09-13 20:11:27,534:DEBUG:certbot.reverter:Creating backup of /etc/apache2/sites-enabled/000-default.conf
2018-09-13 20:11:30,713:INFO:certbot.auth_handler:Waiting for verification...
2018-09-13 20:11:30,714:DEBUG:acme.client:JWS payload:

Timeout? Or file not found?

You have an online vServer. There are other users with "self hosted", they have a windows server in their home office with a virtual Linux-machine and Apache.

If a rewrite rule is wrong, that doesn't generate a timeout.

Upps - did you ask your hoster? Netcup supports Letsencrypt. But perhaps there are problems using an own certbot.

I always had the timeout message.

They only offer Letsencrypt for managed servers and webhosting. Since I have an "unmanaged vServer" it does not affect me. Netcup offers it since 2016, I user letsencrypt with the certbot since November 2017 and it has worked since then. Had 4 renewals already with no issues so far and my apache was not changed since December 2017 (besides the updates).

Thinking about that - it has to be some update which is causing the error :cold_sweat:

Try the following:

  • use certonly and --staging - the testsystem
  • use --webroot as authentication with -w and the path to your webroot
  • use only one domain name (to make it simpler)

certbot certonly --staging --webroot -w YourPath -d www.gnaarl.de

Letsdebug

https://letsdebug.net/www.gnaarl.de/4879?debug=y

gets a 404 using the staging system.

If it is not a firewall (or another blocking system), then it may be a problem with redirects / rewrite rules.

1 Like

That one worked

~# certbot certonly --staging --webroot -w /var/www/ -d gnaarl.de
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for gnaarl.de
Using the webroot path /var/www for all unmatched domains.
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/gnaarl.de-0001/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/gnaarl.de-0001/privkey.pem
   Your cert will expire on 2018-12-13. To obtain a new or tweaked
   version of this certificate in the future, simply run certbot
   again. To non-interactively renew *all* of your certificates, run
   "certbot renew"

I moved all .htaccess files away for testing and commented some rules out. With no effect so far on the renewal. I will take a deeper look today evening - thanks a lot so far!

1 Like

Good to know. It's only a test certificate from Fake LE signed, but you can use the test system (with higher limits) to check, if this part (creating a new certificate) works.

1 Like

It is working… somehow… but not as it is intended to be… I used “break-my-certs” to test the productive side, which somehow brought me a step forward…

The good thing:

certbot renew --force-renewal --preferred-challenge http-01
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gnaarl.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for gnaarl.de
http-01 challenge for mail.gnaarl.de
http-01 challenge for www.gnaarl.de
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/gnaarl.de/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gnaarl.de-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for gnaarl.de
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/gnaarl.de-0001/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/gnaarl.de/fullchain.pem (success)
  /etc/letsencrypt/live/gnaarl.de-0001/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@Junkrat:/etc/letsencrypt/live/gnaarl.de# openssl x509 -inform pem -text -in fullchain.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            fa:d7:f3:d0:0b:00:a2:f5:b3:e3:85:ad:9c:42:49:b1:c4:01
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = Fake LE Intermediate X1
        Validity
            Not Before: Sep 15 05:50:47 2018 GMT
            Not After : Dec 14 05:50:47 2018 GMT
        Subject: CN = gnaarl.de
        Subject Public Key Info:

The weird thing:
I changed some things (commented “AccessFileName .htaccess” out etc.) - it worked but the apache crashed.
By now I changed everything back: It is still working like above, but the apache still crashes…

Is the Test-Environment somehow different from the productive one? Even if it is both testing now, the config seems to be different (sdiff)

# renew_before_expiry = 30 days                                 # renew_before_expiry = 30 days
version = 0.27.0                                                version = 0.27.0
archive_dir = /etc/letsencrypt/archive/gnaarl.de              | archive_dir = /etc/letsencrypt/archive/gnaarl.de-0001
cert = /etc/letsencrypt/live/gnaarl.de/cert.pem               | cert = /etc/letsencrypt/live/gnaarl.de-0001/cert.pem
privkey = /etc/letsencrypt/live/gnaarl.de/privkey.pem         | privkey = /etc/letsencrypt/live/gnaarl.de-0001/privkey.pem
chain = /etc/letsencrypt/live/gnaarl.de/chain.pem             | chain = /etc/letsencrypt/live/gnaarl.de-0001/chain.pem
fullchain = /etc/letsencrypt/live/gnaarl.de/fullchain.pem     | fullchain = /etc/letsencrypt/live/gnaarl.de-0001/fullchain.pe

# Options used in the renewal process                           # Options used in the renewal process
[renewalparams]                                                 [renewalparams]
authenticator = apache                                        <
installer = apache                                            <
account = Bananarama                      						account = Bananarama
pref_challs = http-01,                                        <
server = https://acme-staging-v02.api.letsencrypt.org/directo   server = https://acme-staging-v02.api.letsencrypt.org/directo
                                                              > authenticator = webroot
                                                              > pref_challs = http-01,
                                                              > [[webroot_map]]
                                                              > gnaarl.de = /var/www

And last but not least I dont get why my apache is crashing…

You didn’t create a new productive certificate

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:gnaarl.de;issuer_uid:4428624498008853827&lu=cert_search

shows only the certificate created 13.07.2018.

But: https://gnaarl.de/ is running (so your apache is working) and has a certificate from the test system: Fake LE Intermediate X1 as Issuer, created today.

Add

--server https://acme-v02.api.letsencrypt.org/directory

to your command line to use the productive system.

Hi,
I know I didn't create a new productive certificate, that's why I mentioned that I did “break-my-certs” to use fake certs at my webserver.

The renewal with the fake certificates was succesfull - but like before my apache crashed! I found a case of someone who had the same problem: Certbot renew segfaults my apache2 - wrote him a message and I hope he has some solution.

And that is where the really strange things happen:

root@Junkrat:~# certbot renew --force-renewal --preferred-challenge http-01 --server https://acme-v02.api.letsencrypt.org/directory
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gnaarl.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for gnaarl.de
http-01 challenge for mail.gnaarl.de
http-01 challenge for www.gnaarl.de
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (gnaarl.de) from /etc/letsencrypt/renewal/gnaarl.de.conf produced an unexpected error: Failed authorization procedure. gnaarl.de (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://gnaarl.de/.well-known/acme-challenge/NpybEtOhNou-mVIpTwZDUCIsH9MtO70MG5vi-J94jr0: Timeout after connect (your server may be slow or overloaded), www.gnaarl.de (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://www.gnaarl.de/.well-known/acme-challenge/JH9S-gZF8vQh4VrZkVLpSilFP3_nJEVWAXnL9yz1lwY: Timeout after connect (your server may be slow or overloaded), mail.gnaarl.de (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mail.gnaarl.de/.well-known/acme-challenge/1FktXdlu09nB_RxmCeAx706ZuG7H8NYlBZz274DVB1w: Timeout after connect (your server may be slow or overloaded). Skipping.

The same error as before. I don't get why the same config is renewing my Test-Cert (even if the apache crashes) but can not renew my productive cert (I think because the apache crashes).

Since "apache2ctl graceful" leads to an apache crash - is there a way to let certbot do a full restart instead of a graceful restart?!?

edit: Even an "apache2ctl restart" is causing the segmentation faults and a crash... Only /etc/init.d/apache2 restart seems to do the job...