[Solved] Failure renewing certs with certbot behind reverse proxy

Help
1

My domain is: biszumbitterenen.de

I ran this command: (issue description see below)

My web server is (include version): HAproxy 1.8.12-r0

The operating system my web server runs on is (include version): Alpine Linux 3.8.1

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

I want to renew any of my domains, e.g. www.biszumbitterenen.de, mail.biszumbitterenen.de.
Every single domain has an DNS A/AAAA record pointing to the same IP.
I have only 1 static IP and therefore I use a reverse proxy (HAProxy) to access any subdomain.

I was not expecting an error with renew considering this setup because creation finished w/o errors.

I use this certbot command for renew:
certbot renew --force-renewal --preferred-challenges http-01 --http-01-port=9785 --deploy-hook '/bin/run-parts --exit-on-error /etc/letsencrypt/deploy-hook.d/' --post-hook '/bin/run-parts --exit-on-error /etc/letsencrypt/post-hook.d/'

This output is displayed:

Processing /etc/letsencrypt/renewal/mail.biszumbitterenen.de.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator standalone, Installer None
Attempting to renew cert (mail.biszumbitterenen.de) from /etc/letsencrypt/renewal/mail.biszumbitterenen.de.conf produced an unexpected error: ('Connection aborted.', BadStatusLine("''",)). Skipping.
/usr/local/bin/renew-le-certs.sh: line 13:  1306 Killed                  certbot renew --force-renewal --preferred-challenges http-01 --http-01-port=9785 --deploy-hook '/bin/run-parts --exit-on-error /etc/letsencrypt/deploy-hook.d/' --post-hook '/bin/run-parts --exit-on-error /etc/letsencrypt/post-hook.d/'

I assume the certbot command for renew is incorrect; however this should work when reverse proxy is running.
Any advice for solving this error is appreciated.

THX

2

Hi @cmonty14

your configuration looks buggy.

You have a lot of active certificates with the same name and the same date.

biszumbitterenen.de - three certificates, created 23.06.2018

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:biszumbitterenen.de&lu=cert_search

mail.biszumbitterenen.de - 4 certificates, one from 23.06.2018, three created today.

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:mail.biszumbitterenen.de&lu=cert_search

So if you run renew, there may be three certificates with the same domain name renewed (instead of one).

And there are other subdomains:

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:true;domain:biszumbitterenen.de&lu=cert_search

And you have always only one domain name per certificate. So you risk to hit the limit - 5 identical domain names in one week.

So check your installation with

certbot certificates

and delete (certbot delete ...) all duplicated certificates not used (first, backup the complete folder).

Then don’t use renew, instead renew every single certificate (-d domainname). And - if this is the problem - remove --deploy-hook and -post-hook, do that manual.

Looks like the “mail” - domain works, but the post-hook crashes.

3

The output of certbot certificates don't report any duplicates:
ct102-haproxy:~# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: mail.biszumbitterenen.de
    Domains: mail.biszumbitterenen.de
    Expiry Date: 2018-12-14 09:35:03+00:00 (VALID: 88 days)
    Certificate Path: /etc/letsencrypt/live/mail.biszumbitterenen.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mail.biszumbitterenen.de/privkey.pem
  Certificate Name: pve.biszumbitterenen.de
    Domains: pve.biszumbitterenen.de
    Expiry Date: 2018-12-14 09:35:12+00:00 (VALID: 88 days)
    Certificate Path: /etc/letsencrypt/live/pve.biszumbitterenen.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/pve.biszumbitterenen.de/privkey.pem
  Certificate Name: www.biszumbitterenen.de
    Domains: www.biszumbitterenen.de
    Expiry Date: 2018-09-21 11:06:32+00:00 (VALID: 5 days)
    Certificate Path: /etc/letsencrypt/live/www.biszumbitterenen.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/www.biszumbitterenen.de/privkey.pem
  Certificate Name: mehl.biszumbitterenen.de
    Domains: mehl.biszumbitterenen.de
    Expiry Date: 2018-09-21 11:05:34+00:00 (VALID: 5 days)
    Certificate Path: /etc/letsencrypt/live/mehl.biszumbitterenen.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mehl.biszumbitterenen.de/privkey.pem
  Certificate Name: cal.biszumbitterenen.de
    Domains: cal.biszumbitterenen.de
    Expiry Date: 2018-09-21 11:05:27+00:00 (VALID: 5 days)
    Certificate Path: /etc/letsencrypt/live/cal.biszumbitterenen.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/cal.biszumbitterenen.de/privkey.pem
  Certificate Name: rainloop.biszumbitterenen.de
    Domains: rainloop.biszumbitterenen.de
    Expiry Date: 2018-12-14 09:39:58+00:00 (VALID: 88 days)
    Certificate Path: /etc/letsencrypt/live/rainloop.biszumbitterenen.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/rainloop.biszumbitterenen.de/privkey.pem
  Certificate Name: biszumbitterenen.de
    Domains: biszumbitterenen.de
    Expiry Date: 2018-09-21 11:03:00+00:00 (VALID: 5 days)
    Certificate Path: /etc/letsencrypt/live/biszumbitterenen.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/biszumbitterenen.de/privkey.pem
  Certificate Name: webmail.biszumbitterenen.de
    Domains: webmail.biszumbitterenen.de
    Expiry Date: 2018-09-21 11:05:56+00:00 (VALID: 5 days)
    Certificate Path: /etc/letsencrypt/live/webmail.biszumbitterenen.de/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/webmail.biszumbitterenen.de/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Then with regards to renew every single certificate this seems to be a little complicated when executing this automatically.
I think it would make sense to use a wildcard domain.

1 Like
4

Your original post lacks the information it would take to diagnose your issue:

  • Contents of the /var/log/letsencrypt/ log file(s) that show what Certbot was doing when it encountered the error
  • The contents of your post/deploy hooks
  • The contents of your /etc/letsencrypt/renewal parameters
  • The contents of renew-le-certs.sh, probably.

The way you are calling Certbot is usually not the way to go. You do not want to call renew yourself and let your system cron do the job. You also only need to pass parameters on the first issuance, and Certbot will save and remember those flags in /etc/letsencrypt/renewal/ for that domain when renewing.

5

This is an option if your dns provider supports an api.

But you use nameservers from speicherzentrum.de

nslookup -type=ns biszumbitterenen.de.

biszumbitterenen.de nameserver = ns6.speicherzentrum.de
biszumbitterenen.de nameserver = ns4.speicherzentrum.de
biszumbitterenen.de nameserver = ns5.speicherzentrum.de

This

https://www.speicherzentrum.de/features_dns.html

looks like they have no own api.

6

@JuergenAuer
Thanks for providing this valuable input.
This means that I cannot use wildcards as of now.

I reproduced the issue (with HAproxy service running).
This is the content of /var/log/letsencrypt/letsencrypt.log:

2018-09-16 12:11:59,365:DEBUG:certbot.main:certbot version: 0.27.1
2018-09-16 12:11:59,365:DEBUG:certbot.main:Arguments: ['--force-renewal', '--http-01-port=9785', '--deploy-hook', '/bin/run-parts --e
2018-09-16 12:11:59,365:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEn
2018-09-16 12:11:59,380:DEBUG:certbot.log:Root logging level set at 20
2018-09-16 12:11:59,380:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-09-16 12:11:59,407:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7f051a3e5c90> and i
2018-09-16 12:11:59,407:DEBUG:certbot.cli:Var http01_port=9785 (set by user).
2018-09-16 12:11:59,407:DEBUG:certbot.cli:Var deploy_hook=/bin/run-parts --exit-on-error /etc/letsencrypt/deploy-hook.d/ (set by user
2018-09-16 12:11:59,407:DEBUG:certbot.cli:Var renew_hook=set(['deploy_hook']) (set by user).
2018-09-16 12:11:59,407:DEBUG:certbot.cli:Var post_hook=/bin/run-parts --exit-on-error /etc/letsencrypt/post-hook.d/ (set by user).
2018-09-16 12:11:59,429:DEBUG:certbot.renewal:Auto-renewal forced with --force-renewal...
2018-09-16 12:11:59,429:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
2018-09-16 12:11:59,528:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7f051a3e5510>
Prep: True
2018-09-16 12:11:59,529:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x
2018-09-16 12:11:59,529:INFO:certbot.plugins.selection:Plugins selected: Authenticator standalone, Installer None
2018-09-16 12:11:59,538:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_serv
2018-09-16 12:11:59,539:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2018-09-16 12:11:59,543:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2018-09-16 12:12:00,002:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2018-09-16 12:12:00,005:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 658
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 16 Sep 2018 10:11:59 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 16 Sep 2018 10:11:59 GMT
Connection: keep-alive

{
  "WcsFzFz7dUg": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2018-09-16 12:12:00,006:INFO:certbot.main:Renewing an existing certificate
2018-09-16 12:12:00,178:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0087_key-certbot.pem
2018-09-16 12:12:00,220:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0087_csr-certbot.pem
2018-09-16 12:12:00,245:DEBUG:acme.client:Requesting fresh nonce
2018-09-16 12:12:00,246:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-order.
2018-09-16 12:12:00,449:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-order HTTP/1.1" 405 0
2018-09-16 12:12:00,451:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 103
Allow: POST
Replay-Nonce: IC1iM6wPCCz-Mqqgkp75f3Q_WLybaG_fhe9wAhtkASQ
Expires: Sun, 16 Sep 2018 10:12:00 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 16 Sep 2018 10:12:00 GMT
Connection: keep-alive


2018-09-16 12:12:00,451:DEBUG:acme.client:Storing nonce: IC1iM6wPCCz-Mqqgkp75f3Q_WLybaG_fhe9wAhtkASQ
2018-09-16 12:12:00,452:DEBUG:acme.client:JWS payload:
{
  "status": "pending",
  "identifiers": [
    {
      "type": "dns",
      "value": "mail.biszumbitterenen.de"
    }
  ],
  "resource": "new-order"
}
2018-09-16 12:12:00,476:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJub25jZSI6ICJJQzFpTTZ3UENDei1NcXFna3A3NWYzUV9XTHliYUdfZmhlOXdBaHRrQVNRIiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZ
  "payload": "ewogICJzdGF0dXMiOiAicGVuZGluZyIsIAogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwgCiAgICAgICJ2YWx1ZSI6ICJ
  "signature": "juqkIn5Mb4ER-EZdVROX6zNlBLcO-rt3A-50H0kBMFenW-NkravcW4zNuGAKHEkfkG2P52WeI69du2yXBs2QcIVW5dX9WqVaz_THFWpsFarCABaGHnwjg
}
2018-09-16 12:12:00,697:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 380
2018-09-16 12:12:00,699:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 380
Boulder-Requester: 26341752
Location: https://acme-v02.api.letsencrypt.org/acme/order/26341752/67304704
Replay-Nonce: CU_F2FvPyXcqM_sEKxdRATRbPSMgG_HAnD7VYfwjsys
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 16 Sep 2018 10:12:00 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 16 Sep 2018 10:12:00 GMT
Connection: keep-alive

{
  "status": "ready",
  "expires": "2018-09-23T10:12:00.506893844Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "mail.biszumbitterenen.de"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz/UnhBQWywVrpCg_8lFOPksneaqHdAII-qpDlMWVg8n-A"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/26341752/67304704"
}
2018-09-16 12:12:00,700:DEBUG:acme.client:Storing nonce: CU_F2FvPyXcqM_sEKxdRATRbPSMgG_HAnD7VYfwjsys
2018-09-16 12:12:00,702:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/acme/authz/UnhBQWywVrpCg_8lFOPk
2018-09-16 12:12:00,902:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /acme/authz/UnhBQWywVrpCg_8lFOPksn
2018-09-16 12:12:00,905:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 1411
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 16 Sep 2018 10:12:00 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 16 Sep 2018 10:12:00 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "mail.biszumbitterenen.de"
  },
  "status": "valid",
  "expires": "2018-10-15T10:25:04Z",
  "challenges": [
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/UnhBQWywVrpCg_8lFOPksneaqHdAII-qpDlMWVg8n-A/7369160884",
      "token": "V2KGLks-Oc-gFnhHO2dOdStTevQnr9w_1YkUtQm7Sws"
    },
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/UnhBQWywVrpCg_8lFOPksneaqHdAII-qpDlMWVg8n-A/7369160891",
      "token": "0I8iFETxAmaWPjKh04_lMOWkkjH29CrpvvFInpAHpGc"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/UnhBQWywVrpCg_8lFOPksneaqHdAII-qpDlMWVg8n-A/7369160893",
      "token": "eTelrtwIq-DmPUlwbo0Wx5q82XmQAfSdrFxvcRv1I48"
    },
    {
      "type": "tls-sni-01",
      "status": "valid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/challenge/UnhBQWywVrpCg_8lFOPksneaqHdAII-qpDlMWVg8n-A/7369160899",
      "token": "tp8upb-MFKdC60R2A8jWehEmlgQDQ8IVTtwxoJeJ4N8",
      "validationRecord": [
        {
          "hostname": "mail.biszumbitterenen.de",
          "port": "443",
          "addressesResolved": [
            "94.79.184.226"
          ],
          "addressUsed": "94.79.184.226"
        }
      ]
    }
  ]
}
2018-09-16 12:12:00,911:INFO:certbot.auth_handler:Performing the following challenges:
2018-09-16 12:12:00,917:INFO:certbot.auth_handler:tls-sni-01 challenge for mail.biszumbitterenen.de
2018-09-16 12:12:00,922:DEBUG:acme.standalone:Failed to bind to :443 using IPv6
2018-09-16 12:12:00,923:DEBUG:acme.standalone:Failed to bind to :443 using IPv4
2018-09-16 12:12:00,925:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 75, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 126, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 234, in perform
    return [self._try_perform_single(achall) for achall in achalls]
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 241, in _try_perform_single
    _handle_perform_error(error)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 239, in _try_perform_single
    return self._perform_single(achall)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 247, in _perform_single
    servers, response = self._perform_tls_sni_01(achall)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 264, in _perform_tls_sni_01
    servers = self.servers.run(port, challenges.TLSSNI01, listenaddr=addr)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 78, in run
    raise errors.StandaloneBindError(error, port)
StandaloneBindError: Problem binding to port 443: Could not bind to IPv4 or IPv6.

2018-09-16 12:12:00,926:DEBUG:certbot.error_handler:Calling registered functions
2018-09-16 12:12:00,926:INFO:certbot.auth_handler:Cleaning up challenges
2018-09-16 12:12:00,930:WARNING:certbot.renewal:Attempting to renew cert (mail.biszumbitterenen.de) from /etc/letsencrypt/renewal/mai
2018-09-16 12:12:00,939:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 430, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1197, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 115, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 305, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 334, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 370, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 75, in handle_authorizations
    resp = self._solve_challenges(aauthzrs)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 126, in _solve_challenges
    resp = self.auth.perform(all_achalls)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 234, in perform
    return [self._try_perform_single(achall) for achall in achalls]
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 241, in _try_perform_single
    _handle_perform_error(error)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 239, in _try_perform_single
    return self._perform_single(achall)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 247, in _perform_single
    servers, response = self._perform_tls_sni_01(achall)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 264, in _perform_tls_sni_01
    servers = self.servers.run(port, challenges.TLSSNI01, listenaddr=addr)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/standalone.py", line 78, in run
    raise errors.StandaloneBindError(error, port)
StandaloneBindError: Problem binding to port 443: Could not bind to IPv4 or IPv6.

2018-09-16 12:12:00,955:DEBUG:certbot.cli:Var http01_port=9785 (set by user).
2018-09-16 12:12:00,959:DEBUG:certbot.cli:Var deploy_hook=/bin/run-parts --exit-on-error /etc/letsencrypt/deploy-hook.d/ (set by user
2018-09-16 12:12:00,967:DEBUG:certbot.cli:Var renew_hook=set(['deploy_hook']) (set by user).
2018-09-16 12:12:00,970:DEBUG:certbot.cli:Var post_hook=/bin/run-parts --exit-on-error /etc/letsencrypt/post-hook.d/ (set by user).
2018-09-16 12:12:01,136:DEBUG:certbot.renewal:Auto-renewal forced with --force-renewal...
2018-09-16 12:12:01,143:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
2018-09-16 12:12:01,209:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7f051a3ea190>
Prep: True
2018-09-16 12:12:01,219:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x
2018-09-16 12:12:01,228:INFO:certbot.plugins.selection:Plugins selected: Authenticator standalone, Installer None
2018-09-16 12:12:01,427:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_serv
2018-09-16 12:12:01,445:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2018-09-16 12:12:01,474:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org

The content of the different /etc/letsencrypt/renewal parameters is similar (if not equal) for every subdomain:

ct102-haproxy:~# more /etc/letsencrypt/renewal/mail.biszumbitterenen.de.conf
# renew_before_expiry = 30 days
version = 0.27.1
archive_dir = /etc/letsencrypt/archive/mail.biszumbitterenen.de
cert = /etc/letsencrypt/live/mail.biszumbitterenen.de/cert.pem
privkey = /etc/letsencrypt/live/mail.biszumbitterenen.de/privkey.pem
chain = /etc/letsencrypt/live/mail.biszumbitterenen.de/chain.pem
fullchain = /etc/letsencrypt/live/mail.biszumbitterenen.de/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = standalone
account = c8af689281a1eb5d92664dfabe350f46
http01_port = 9785
renew_hook = /bin/run-parts --exit-on-error /etc/letsencrypt/deploy-hook.d/
post_hook = /bin/run-parts --exit-on-error /etc/letsencrypt/post-hook.d/
server = https://acme-v02.api.letsencrypt.org/directory

And (finally) the content of the renew script:
ct102-haproxy:~# more /usr/local/bin/renew-le-certs.sh
#!/bin/sh

## Configuration
SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

## Renew/update certificates with certbot
certbot renew \
--force-renewal \
--http-01-port=9785 \
--deploy-hook '/bin/run-parts --exit-on-error /etc/letsencrypt/deploy-hook.d/' \
--post-hook '/bin/run-parts --exit-on-error /etc/letsencrypt/post-hook.d/'
7

That's problematic since it won't work with your haproxy ACL to forward the challenge requests. Try with:

--preferred-challenges http-01

Sometimes Certbot can choose TLS-SNI because it used to be the old default ....

8

Some additional information regarding the configuration of reverse proxy:
I’m using SSL termination on reverse proxy.

9

Nice.
Adding parameter --preferred-challenges http-01 looks like to be the key driver for the solution.
This means domain mail.biszumbitterenen.de was renewed successfully.
There was an error with running deploy-hook command though, but I will address this in another ticket.

However, there’s an issue with domain pve.biszumbitterenen.de.
Certbot seems to hang in step “Cleaning up challenges”.
Please note that this domain is not accessible from outside and there’s no relevant configuration in HAproxy.

This is the latest output in /var/log/letsencrypt/letsencrypt.log:
-----END CERTIFICATE-----

2018-09-16 12:45:26,485:DEBUG:certbot.storage:Writing new private key to /etc/letsencrypt/archive/pve.biszumbitterenen.de/privkey7.pem.
2018-09-16 12:45:26,564:DEBUG:certbot.storage:Writing certificate to /etc/letsencrypt/archive/pve.biszumbitterenen.de/cert7.pem.
2018-09-16 12:45:26,651:DEBUG:certbot.storage:Writing chain to /etc/letsencrypt/archive/pve.biszumbitterenen.de/chain7.pem.
2018-09-16 12:45:26,739:DEBUG:certbot.storage:Writing full chain to /etc/letsencrypt/archive/pve.biszumbitterenen.de/fullchain7.pem.
Cleaning up challenges
/usr/local/bin/renew-le-certs.sh: line 11:  2014 Killed                  certbot renew --force-renewal --preferred-challenges http-01
+ --http-01-port=9785 --deploy-hook '/bin/run-parts --exit-on-error /etc/letsencrypt/deploy-hook.d/' --post-hook '/bin/run-parts --exit-on-error /etc/letsencrypt/post-hook.d/'
/usr/local/bin/renew-le-certs.sh: line 13: --http-01-port=9785: command not found

The renewed cert files have been written correctly to /etc/letsencrypt/archive/pve.biszumbitterenen.de, but after this there’s no progress.

Any advice for this issue?

10

Yep, that worked. But now you have created 4 certificates with this domain name, 3 yesterday, one today.

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:mail.biszumbitterenen.de&lu=cert_search

So if you test this two times again, the second time you fail hitting the 5 certificates per domain name set per week - limit.

Same with pve.biszumbitterenen.de

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:pve.biszumbitterenen.de&lu=cert_search

Perhaps you may use --stage as option, so you use the test system (with own and higher limits).

11

I think this is probably about the way you are calling Certbot, like I mentioned earlier.

Since you are calling

certbot renew --force-renewal --deploy-hook ... --post-hook ...

on every certificate being managed by Certbot, hooks for those domains (like pve...) which are not part of haproxy's purview are unnecessarily being invoked. That's why usually renewal params file drives the hook invocation rather than CLI params.

Your actual error:

Is probably either the result of your script dereferencing an unset hook variable (which could be a result of the above problem), or an errant space after a \, or a different kind of syntax error, can't tell from here.

I'm not sure what's the best way of going about turning your present configuration into a more stable one, but starting from scratch, a "good" config might look like:

  1. Install Certbot
  2. Create deployment hook at /etc/letsencrypt/haproxy-deploy.sh
  3. Create post hook at /etc/letsencrypt/haproxy-post.sh
  4. Issue each certificate as: certbot certonly -a standalone --preferred-challenges http-01 --http-01-port 1234 --deploy-hook ... --post-hook ... -d example.org
  5. Issue any non-haproxy managed certificates whatever way applies to them.
  6. Test certbot renew --dry-run
  7. Allow your system cron/systemd timer (automatically installed) to automatically manage renewals of each certificate individually.
12

OK.
Having duplicate certs is not wanted.
Is there any way of deleting the duplicates?
Or must I revoke the cert and re-create it afterwards?

And how can this be avoided in the future with automatic renewal?
The script checks every week if the certs must be renewed, and then the certbot renew command is triggered.
My understanding is that for every certbot renew another duplicate will be created.

13

Agree… restarting from scratch makes sense.

Could you please advice how to renew each certificate individually?
What is the correct syntax for this for the different cases
a) non-haproxy managed domains
b) haproxy managed domains?

14

That’s kind of my point. Certbot already knows when each certificate needs to be renewed and how.

So you don’t need to do anything other than tell Certbot how to do it the first time (step 4).

If you have a non-haproxy managed certificate, just provide different parameters for that certificate in step 4 (whatever they are, webroot or DNS or whatever you have configured).

15

OK.
Here is my lack of knowledge:
Which parameters should I provide when creating the certificate for non-haproxy managed domains and which for others?

Please advise.

16

I dunno dude, you haven’t told us how you are planning on authenticating the non-haproxy certificates.

You need to demonstrate control over them somehow, and your two choices are a webroot challenge or updating a DNS TXT record. How you do that depends entirely on how your webserver and webapps work.

17

That wouldn't help.

Revoking certificates does not reset rate limits, because the resources used to issue those certificates have already been consumed.

I don't use that command. But User Guide — Certbot 2.7.0.dev0 documentation says:

--http-01-port HTTP01_PORT

with a space, not a =

18

Hm… I think I will create a DNS TXT record (with my DNS provider) for any subdomain.
Hereby I can access the domain later if needed simply by modifying the reverse proxy config.

What do you think? Would this work?

19

The question is whether the DNS provider has an API that Certbot can use to update the TXT record, because the required TXT record’s content will be different for every renewal.

20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.