Renewals stopped working, looking for debugging advice


#1

So, I’ve got a dozen or so ssl certs set up on a single server and have been using and renewing them for a little over a year without much issue. Recently though I haven’t been able to renew the certificates whatsoever, I just keep getting connection refused errors and I cannot figure out what changed that caused this issue. Since I have tons of virtualhosts split across apache config files, I wasn’t able to use the fully automated setup from certbot and so have the certificates manually configured for the web server and am not using the certbot apache plugin. When I need to renew I simply stop the webserver for a moment and then use the --standalone switch when using the renew command. So: service apache2 stop; sleep 2; certbot renew --standalone; sleep 2; service apache2 start. Up until recently this had worked just fine. The only change done to the system in the recent past is that I installed docker.
Basically what I’ve tried so far is:

  • Rebooting the server, issuing the renew manually rather than letting the cronjob do it, trying to adjust the timing of killing the webserver, trying with and without the --standalone switch, etc.
  • Uninstalling and reinstalling certbot
  • Uninstalling docker (I’ve yet to reinstall it)
  • Trying different versions of certbot (I tried 0.10.2 and 0.25.0 which were both available via apt)

My domain is: leopard.hosting (also vote.block.land, pecon.us, and some others)

I ran this command: certbot renew --standalone

It produced this output: The server could not connect to the client to verify the domain :: Connection refused. Skipping.

My web server is (include version): apache 2.4.25-3

The operating system my web server runs on is (include version): Debian 9

My hosting provider, if applicable, is: vultr.com

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

At this point I’d just really appreciate any tips or ideas on how I should proceed trying to figure out what is causing this issue. I’ve still got a month until the first certificates expire, but this is quickly getting very irritating and I’m out of ideas at the moment.


#2

Ensure your system is accessible from the Internet on port 443 and port 80.


#3

It definitely is. The sites are all publicly available and http+https currently works. As I described in the post I kill the webserver before using the standalone renew option to ensure the port is available to be used by certbot.


#4

Please show complete log details.


#5
root@leopard:~# service apache2 stop; sleep 2; certbot renew --standalone; sleep 2; service apache2 start
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/dungeons.block.land.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/pecon.us.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/pecon.us-0001.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/forums.block.land.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/agora.leopard.hosting.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer apache

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/repository.pecon.us.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/serverstatus.block.land.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer apache

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/vote.block.land.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer apache
Renewing an existing certificate
/usr/lib/python3/dist-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
  signer = key.signer(self.padding, self.hash)
Performing the following challenges:
tls-sni-01 challenge for vote.block.land
tls-sni-01 challenge for diamond-integration.com
tls-sni-01 challenge for leopard.hosting
tls-sni-01 challenge for serverstatus.block.land
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (vote.block.land) from /etc/letsencrypt/renewal/vote.block.land.conf produced an unexpected error: Failed authorization procedure. diamond-integration.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Connection refused. Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/diamond-integration.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer apache
Renewing an existing certificate
/usr/lib/python3/dist-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
  signer = key.signer(self.padding, self.hash)
Performing the following challenges:
tls-sni-01 challenge for diamond-integration.com
tls-sni-01 challenge for leopard.hosting
tls-sni-01 challenge for serverstatus.block.land
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (diamond-integration.com) from /etc/letsencrypt/renewal/diamond-integration.com.conf produced an unexpected error: Failed authorization procedure. diamond-integration.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Connection refused. Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/leopard.hosting.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/bossbattles.pecon.us.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/vote.block.land/fullchain.pem (failure)
  /etc/letsencrypt/live/diamond-integration.com/fullchain.pem (failure)

-------------------------------------------------------------------------------

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/dungeons.block.land/fullchain.pem expires on 2018-12-15 (skipped)
  /etc/letsencrypt/live/pecon.us/fullchain.pem expires on 2019-01-19 (skipped)
  /etc/letsencrypt/live/pecon.us-0001/fullchain.pem expires on 2019-01-19 (skipped)
  /etc/letsencrypt/live/forums.block.land/fullchain.pem expires on 2019-02-02 (skipped)
  /etc/letsencrypt/live/agora.leopard.hosting/fullchain.pem expires on 2018-12-24 (skipped)
  /etc/letsencrypt/live/repository.pecon.us/fullchain.pem expires on 2018-12-22 (skipped)
  /etc/letsencrypt/live/serverstatus.block.land/fullchain.pem expires on 2019-01-22 (skipped)
  /etc/letsencrypt/live/leopard.hosting/fullchain.pem expires on 2018-12-22 (skipped)
  /etc/letsencrypt/live/bossbattles.pecon.us/fullchain.pem expires on 2018-12-22 (skipped)
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/vote.block.land/fullchain.pem (failure)
  /etc/letsencrypt/live/diamond-integration.com/fullchain.pem (failure)
-------------------------------------------------------------------------------
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: diamond-integration.com
   Type:   connection
   Detail: Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - The following errors were reported by the server:

   Domain: diamond-integration.com
   Type:   connection
   Detail: Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
root@leopard:~# 

#6

Try:
certbot renew --standalone --preferred-challenges http


#7

I’m really not sure what to make of this. Some kind of issue with the standalone server?

root@leopard:~# service apache2 stop; sleep 2; certbot renew --standalone --preferred-challenges http; sleep 2; service apache2 start
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/dungeons.block.land.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/pecon.us.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/pecon.us-0001.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/forums.block.land.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/agora.leopard.hosting.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer apache

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/repository.pecon.us.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/serverstatus.block.land.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer apache

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/vote.block.land.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer apache
Renewing an existing certificate
/usr/lib/python3/dist-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
  signer = key.signer(self.padding, self.hash)
Performing the following challenges:
http-01 challenge for vote.block.land
http-01 challenge for diamond-integration.com
http-01 challenge for leopard.hosting
http-01 challenge for serverstatus.block.land
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (vote.block.land) from /etc/letsencrypt/renewal/vote.block.land.conf produced an unexpected error: Failed authorization procedure. diamond-integration.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://diamond-integration.com/.well-known/acme-challenge/AwQ64AfswhEokSOMZDRSzaBg8j4TtWOHesMQE1Jfvl8: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p". Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/diamond-integration.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator standalone, Installer apache
Renewing an existing certificate
/usr/lib/python3/dist-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
  signer = key.signer(self.padding, self.hash)
Performing the following challenges:
http-01 challenge for diamond-integration.com
http-01 challenge for leopard.hosting
http-01 challenge for serverstatus.block.land
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (diamond-integration.com) from /etc/letsencrypt/renewal/diamond-integration.com.conf produced an unexpected error: Failed authorization procedure. diamond-integration.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://diamond-integration.com/.well-known/acme-challenge/gZ3yRZNmnpcojRF3A3dMqjX086DUFlpiw-sy45eeg8Q: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p". Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/leopard.hosting.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/bossbattles.pecon.us.conf
-------------------------------------------------------------------------------
Cert not yet due for renewal
Plugins selected: Authenticator standalone, Installer None
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/vote.block.land/fullchain.pem (failure)
  /etc/letsencrypt/live/diamond-integration.com/fullchain.pem (failure)

-------------------------------------------------------------------------------

The following certs are not due for renewal yet:
  /etc/letsencrypt/live/dungeons.block.land/fullchain.pem expires on 2018-12-15 (skipped)
  /etc/letsencrypt/live/pecon.us/fullchain.pem expires on 2019-01-19 (skipped)
  /etc/letsencrypt/live/pecon.us-0001/fullchain.pem expires on 2019-01-19 (skipped)
  /etc/letsencrypt/live/forums.block.land/fullchain.pem expires on 2019-02-02 (skipped)
  /etc/letsencrypt/live/agora.leopard.hosting/fullchain.pem expires on 2018-12-24 (skipped)
  /etc/letsencrypt/live/repository.pecon.us/fullchain.pem expires on 2018-12-22 (skipped)
  /etc/letsencrypt/live/serverstatus.block.land/fullchain.pem expires on 2019-01-22 (skipped)
  /etc/letsencrypt/live/leopard.hosting/fullchain.pem expires on 2018-12-22 (skipped)
  /etc/letsencrypt/live/bossbattles.pecon.us/fullchain.pem expires on 2018-12-22 (skipped)
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/vote.block.land/fullchain.pem (failure)
  /etc/letsencrypt/live/diamond-integration.com/fullchain.pem (failure)
-------------------------------------------------------------------------------
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: diamond-integration.com
   Type:   unauthorized
   Detail: Invalid response from
   http://diamond-integration.com/.well-known/acme-challenge/AwQ64AfswhEokSOMZDRSzaBg8j4TtWOHesMQE1Jfvl8:
   "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: diamond-integration.com
   Type:   unauthorized
   Detail: Invalid response from
   http://diamond-integration.com/.well-known/acme-challenge/gZ3yRZNmnpcojRF3A3dMqjX086DUFlpiw-sy45eeg8Q:
   "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
root@leopard:~#```

#8

diamond-integration.com has a different IP address than the other hostnames in those two certificates. Is it pointing to the same server?

diamond-integration.com.  (insecure)  3600  A     208.91.197.91

vote.block.land.          (insecure)  3600  A     45.32.94.236

leopard.hosting.          (insecure)  300   A     45.32.94.236
leopard.hosting.          (insecure)  300   AAAA  2001:19f0:6001:555:5400:ff:fe51:2e58

serverstatus.block.land.  (insecure)  1800  A     45.32.94.236

#9

Wow, I can’t believe I didn’t notice that sooner. That site must have been down for two weeks and I guess the client didn’t realize it? They transferred registrars recently which must have screwed up that record somehow. I’ll retry in a few hours when the dns propagates.

Thank you very much for noticing that. I never checked that one because the challenge for vote.block.land appeared to happen before it and was also failing.