Webmin Apache2 Certificate Renewal Failure


#1
  • 3vr.club
  • My web server: Apache 2.4.10
  • OS: Debian 8
  • SSH and webshell available
  • I’m using Webmin 1.890

I ran this command: sudo certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/3vr.club-0001.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for 3vr.club
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/3vr.club/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/3vr.club-0001.conf produced an unexpected error: Failed authorization procedure. 3vr.club (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching https://3vr.club.well-known/acme-challenge/xx-token-xx: Error getting validation data. Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/3vr.club.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for 3vr.club
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/3vr.club/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/3vr.club.conf produced an      unexpected error: failed authorization procedure. 3vr.club (http-01): urn:acme:     error:connection :: The server could not connect to the client to verify the dom     ain :: Fetching https://3vr.club.well-known/acme-challenge/xx-token-xx: Error getting validation data. Skipping.
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/3vr.club-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/3vr.club/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: 3vr.club
   Type:   connection
   Detail: Fetching
    'token link' 
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - The following errors were reported by the server:

   Domain: 3vr.club
   Type:   connection
   Detail: Fetching
   'token link' 
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

Note: After 10 or so attempts with different settings (like commenting out the HTTP --> HTTPS redirect rule) the error is no longer “error getting validation data” but this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/3vr.club-0001.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for 3vr.club
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/3vr.club/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/3vr.club-0001.conf produced an unexpected error: failed authorization procedure. 3vr.club (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching 'token link' Connection refused. Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/3vr.club.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for 3vr.club
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/3vr.club/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/3vr.club.conf produced an unexpected error: Failed authorization procedure. 3vr.club (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching 'token link'  Connection refused. Skipping.
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/3vr.club-0001/fullchain.pem (failure)
  /etc/letsencrypt/live/3vr.club/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: 3vr.club
   Type:   connection
   Detail: Fetching
    'token link' 
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.
 - The following errors were reported by the server:

   Domain: 3vr.club
   Type:   connection
   Detail: Fetching
    'token link' 
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

The /var/log/letsencrypt/letsencrypt.log shows this:

Fetching: 'token link' 
Connection refused

To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address. Additionally, please check that your comput$
2018-08-02 06:11:32,751:INFO:certbot.auth_handler:Cleaning up challenges
2018-08-02 06:11:32,751:DEBUG:certbot.plugins.webroot:Removing /var/www/3vr.club/.well-known/acme-challenge/xx-token-xx
2018-08-02 06:11:32,751:INFO:certbot.plugins.webroot:Unable to clean up challenge directory /var/www/3vr.club/.well-known/acme-challenge
2018-08-02 06:11:32,752:DEBUG:certbot.plugins.webroot:Error was: [Errno 39] Directory not empty: '/var/www/3vr.club/.well-known/acme-challenge'
2018-08-02 06:11:32,752:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/3vr.club-0001.conf produced an unexpected error: Failed authorization procedure. 3vr.club
2018-08-02 06:11:32,754:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/certbot/renewal.py", line 413, in handle_renewal_request
    main.obtain_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 626, in obtain_cert
    action, _ = _auth_from_available(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 103, in _auth_from_available
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/renewal.py", line 296, in renew_cert
    new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
  File "/usr/lib/python2.7/dist-packages/certbot/client.py", line 262, in obtain_certificate
    self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 77, in get_authorizations
    self._respond(resp, best_effort)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 134, in _respond
    self._poll_challenges(chall_update, best_effort)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 198, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. 3vr.club (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching  'token link' 

2018-08-02 06:11:32,758:DEBUG:parsedatetime:parse (top of loop): [30 days][]
2018-08-02 06:11:32,758:DEBUG:parsedatetime:weekday False, dateStd False, dateStr False, time False, timeStr False, meridian False
2018-08-02 06:11:32,758:DEBUG:parsedatetime:dayStr False, modifier False, modifier2 False, units True, qunits False
2018-08-02 06:11:32,758:DEBUG:parsedatetime:_evalString(30 days, time.struct_time(tm_year=2018, tm_mon=8, tm_mday=2, tm_hour=6, tm_min=11, tm_sec=32, tm_wday=3, tm_yday=214, tm_isdst=0))
2018-08-02 06:11:32,758:DEBUG:parsedatetime:_buildTime: [30 ][][days]
2018-08-02 06:11:32,758:DEBUG:parsedatetime:units days --> realunit days
2018-08-02 06:11:32,759:DEBUG:parsedatetime:return
2018-08-02 06:11:32,759:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2018-08-03 23:11:53 UTC.
2018-08-02 06:11:32,759:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2018-08-02 06:11:32,778:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer apache
2018-08-02 06:11:33,219:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.configurator:ApacheConfigurator
Initialized: <certbot_apache.configurator.ApacheConfigurator object at 0x7f81cf7a1950>
Prep: True
2018-08-02 06:11:33,226:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f81cf7a1790>
Prep: True
2018-08-02 06:11:33,226:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7f81cf7a1790> and installer <certbot_apache.configurator.ApacheConfi$
2018-08-02 06:11:33,230:DEBUG:certbot.main:Picked account: <Account(xxx)>
2018-08-02 06:11:33,231:DEBUG:root:Sending GET request to https://acme-staging.api.letsencrypt.org/directory.
2018-08-02 06:11:33,232:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
2018-08-02 06:11:33,468:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 704
2018-08-02 06:11:33,469:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 704
Replay-Nonce: xxx
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Thu, 02 Aug 2018 06:11:33 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Thu, 02 Aug 2018 06:11:33 GMT
Connection: keep-alive
{...}

I created the initial certificate by using the webmin configuration --> SSL Encryption --> Let’s Encrypt (tab) [[ Request Cert ]] and then made it more secure by following the advice from SSLlabs to get A+ rating. (related pic) The website worked properly up until now (placeholder), the only issue is that I can’t find out how to renew the certificate without breaking things (if they aren’t already broken)

Apache2 gl.3vr.club.conf file:

<VirtualHost 3vr.club:443>
DocumentRoot /var/www/3vr.club
ServerName gl.3vr.club
SSLEngine on  
SSLCertificateFile /etc/webmin/letsencrypt-cert.pem
SSLCertificateKeyFile /etc/webmin/letsencrypt-key.pem
SSLCertificateChainFile /etc/webmin/letsencrypt-ca.pem
SSLProtocol -ALL +TLSv1.2
SSLHonorCipherOrder on
SSLCipherSuite EECDH+AESGCM:EDH+AESGCM:AES256+EECDH:AES256+EDH
SSLInsecureRenegotiation off
Header always set Strict-Transport-Security "max-age=63072000; includeSubDomains"
<Directory "/var/www/3vr.club">
allow from all
Options None
Require all granted
</Directory>
</VirtualHost>
<VirtualHost 3vr.club:80>
ServerName gl.3vr.club
Redirect  permanent / https://3vr.club
<Directory "/var/www/3vr.club">
allow from all
Options None
Require all granted
</Directory>
</VirtualHost>
Header set Content-Security-Policy "default-src 'self';"
Header always set X-Frame-Options "SAMEORIGIN"
Header always set X-Xss-Protection "1; mode=block"
Header always set X-Content-Type-Options "nosniff"
Header always set Referrer-Policy "no-referrer-when-downgrade"

apachectl -S output:

VirtualHost configuration:
188.138.127.94:443     gl.3vr.club (/etc/apache2/sites-enabled/gl.3vr.club.conf:1)
188.138.127.94:80      gl.3vr.club (/etc/apache2/sites-enabled/gl.3vr.club.conf:19)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/lock/apache2" mechanism=fcntl
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33

I’m particularly worried that I will be locked out completely in 23 hours. I have little experience with SSL certification and I thought I still had a couple days left but apparently there’s a second cert that will expire in 23 hours. Not sure what the consequence of that would be.

Edit: Added information on the initial certificate origin. / Formatting / Added more info.


#2

This is a bad redirect that causes problems.

Try

Redirect permanent / https://3vr.club/

#3

Changed. Absolutely mental that you spotted that so quick!

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/3vr.club-0001.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for 3vr.club
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/3vr.club/.well-known/acme-challenge
Generating key (2048 bits): /etc/letsencrypt/keys/0004_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0004_csr-certbot.pem

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/3vr.club-0001/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/3vr.club.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for 3vr.club
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /var/www/3vr.club/.well-known/acme-challenge
Generating key (2048 bits): /etc/letsencrypt/keys/0005_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0005_csr-certbot.pem

-------------------------------------------------------------------------------
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/3vr.club/fullchain.pem
-------------------------------------------------------------------------------

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/3vr.club-0001/fullchain.pem (success)
  /etc/letsencrypt/live/3vr.club/fullchain.pem (success)

However, given hard refresh of the webpage, it still displays (both in browser and on the webmin config page) that the cert expires in 3 days. Am I missing something or do the certs switch on expiry? (yes, I deleted them from the browser)


#4

Hmm.

certbot certificates

grep -Ri sslcertificatefile /etc/apache2

ls -lah /etc/webmin/letsencrypt-cert.pem

#5

root@gl:~# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Found the following certs:
  Certificate Name: 3vr.club-0001
    Domains: 3vr.club
    Expiry Date: 2018-10-31 07:31:51+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/3vr.club-0001/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/3vr.club-0001/privkey.pem
  Certificate Name: 3vr.club
    Domains: 3vr.club
    Expiry Date: 2018-10-31 07:31:58+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/3vr.club/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/3vr.club/privkey.pem
-------------------------------------------------------------------------------

root@gl:~# grep -Ri sslcertificatefile /etc/apache2

/etc/apache2/apache2.conf:SSLCertificateFile /etc/webmin/letsencrypt-cert.pem
/etc/apache2/sites-enabled/000-default.conf.save:#SSLCertificateFile /etc/apache2/3vr.club/ssl.crt
/etc/apache2/sites-enabled/000-default.conf.save:   SSLCertificateFile /etc/webmin/letsencrypt-cert.pem
/etc/apache2/sites-enabled/gl.3vr.club.conf:SSLCertificateFile /etc/webmin/letsencrypt-cert.pem
/etc/apache2/sites-available/gl.3vr.club.conf:SSLCertificateFile /etc/webmin/letsencrypt-cert.pem

root@gl:~# ls -lah /etc/webmin/letsencrypt-cert.pem
-rw------- 1 root root 2.1K May 7 11:55 /etc/webmin/letsencrypt-cert.pem

My intuition says the ‘-0001’ one is redundant? Not sure how it even got there


#6

So, you see that Certbot saves the certificate data in its own directory:

But you have Apache reading the certificate files from a different location:

If you want Certbot to automatically update Apache, you’ll need to tell Apache to read the certificate and private key from /etc/letsencrypt/live.


#7

On it! Here’s a rainbow cat to thank you for the help so far…

Image

https://c1.staticflickr.com/5/4069/5078809204_c84b6b5c9a_b.jpg

Edit: It works now, but I’m sure I could still improve somewhere. Time for more research :slight_smile:
Thanks again, @_az !


#8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.