Automatic cert renewal fails, by hand --dry run successes

Help
#1

My domain is:
dizsrv004p.diz.med.uni-muenchen.de

I ran this command:
systemctl status certbot

It produced this output:
● certbot.service - Certbot
Loaded: loaded (/lib/systemd/system/certbot.service; static; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2019-09-24 11:12:31 CEST; 4h 11min ago
Docs: file:///usr/share/doc/python-certbot-doc/html/index.html
https://letsencrypt.readthedocs.io/en/latest/
Process: 29459 ExecStart=/usr/bin/certbot -q renew (code=exited, status=1/FAILURE)
Main PID: 29459 (code=exited, status=1/FAILURE)
CPU: 788ms

Sep 24 11:12:25 dizsrv004p systemd[1]: Starting Certbot…
Sep 24 11:12:31 dizsrv004p certbot[29459]: Attempting to renew cert (dizsrv004p.diz.med.uni-muenchen.de) from /etc/letsencrypt/renewal/dizsrv004p.diz.med.uni-muenchen.de.conf produced an unexpected error: (“bad handshake: SysCallError(-1, ‘Unexpected EOF’)”,). Skipping.
Sep 24 11:12:31 dizsrv004p certbot[29459]: All renewal attempts failed. The following certs could not be renewed:
Sep 24 11:12:31 dizsrv004p certbot[29459]: /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/fullchain.pem (failure)
Sep 24 11:12:31 dizsrv004p certbot[29459]: 1 renew failure(s), 0 parse failure(s)
Sep 24 11:12:31 dizsrv004p systemd[1]: certbot.service: Main process exited, code=exited, status=1/FAILURE
Sep 24 11:12:31 dizsrv004p systemd[1]: Failed to start Certbot.
Sep 24 11:12:31 dizsrv004p systemd[1]: certbot.service: Unit entered failed state.
Sep 24 11:12:31 dizsrv004p systemd[1]: certbot.service: Failed with result ‘exit-code’.

When i run it “by hand” it will succed:
certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/dizsrv004p.diz.med.uni-muenchen.de.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for api.proskive.diz.med.uni-muenchen.de
http-01 challenge for auth.proskive.diz.med.uni-muenchen.de
http-01 challenge for dizsrv004p.diz.med.uni-muenchen.de
http-01 challenge for proskive.diz.med.uni-muenchen.de
Using default address 80 for authentication.
Waiting for verification…
Cleaning up challenges

new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/fullchain.pem

** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/fullchain.pem (success)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)

My web server is (include version):
nginx version: nginx/1.14.1

The operating system my web server runs on is (include version):
uname -a
Debian 9
Linux dizsrv004p 4.9.0-9-amd64 #1 SMP Debian 4.9.168-1+deb9u5 (2019-08-11) x86_64 GNU/Linux

We have other systems with apache which misbehave the same way.

My hosting provider, if applicable, is:
University of Munich

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 0.28.0

kind regards and thanks for any hints

#2

Hi @Bodo-von-Greif

that

looks like a temporary error.

Wait one day, perhaps next time it works.

#3

What does “curl -v https://acme-v02.api.letsencrypt.org/directory” output?

Edit:

Can you find and post the traceback from /var/log/letsencrypt/letsencrypt.log?

#4

Hi Juergen,

i tried on different VMs, several dozen of times since weeks now. :slight_smile:

#5

CONNECT acme-v02.api.letsencrypt.org:443 HTTP/1.1
Host: acme-v02.api.letsencrypt.org:443
User-Agent: curl/7.52.1
Proxy-Connection: Keep-Alive

< HTTP/1.0 200 Connection Established
< Content-Type: text/html
< Proxy-Connection: Keep-Alive
<

  • Proxy replied OK to CONNECT request
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
    CApath: /etc/ssl/certs
  • TLSv1.2 (OUT), TLS header, Certificate Status (22):
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS change cipher, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Finished (20):
  • SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=acme-v01.api.letsencrypt.org
  • start date: Sep 13 17:57:16 2019 GMT
  • expire date: Dec 12 17:57:16 2019 GMT
  • subjectAltName: host “acme-v02.api.letsencrypt.org” matched cert’s “acme-v02.api.letsencrypt.org”
  • issuer: C=US; O=Let’s Encrypt; CN=Let’s Encrypt Authority X3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x5651d2ac8e80)

GET /directory HTTP/1.1
Host: acme-v02.api.letsencrypt.org
User-Agent: curl/7.52.1
Accept: /

#6

2019-09-25 03:51:50,279:DEBUG:certbot.main:certbot version: 0.28.0
2019-09-25 03:51:50,280:DEBUG:certbot.main:Arguments: [’-q’]
2019-09-25 03:51:50,281:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-09-25 03:51:50,288:DEBUG:certbot.log:Root logging level set at 30
2019-09-25 03:51:50,288:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-09-25 03:51:50,294:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7fcffc4ca160> and installer <certbot.cli._Default object at 0x7fcffc4ca160>
2019-09-25 03:51:50,300:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2019-10-01 10:33:30 UTC.
2019-09-25 03:51:50,300:INFO:certbot.renewal:Cert is due for renewal, auto-renewing…
2019-09-25 03:51:50,300:DEBUG:certbot.plugins.selection:Requested authenticator nginx and installer nginx
2019-09-25 03:51:50,475:DEBUG:certbot.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator
Initialized: <certbot_nginx.configurator.NginxConfigurator object at 0x7fcffc52ac18>
Prep: True
2019-09-25 03:51:50,477:DEBUG:certbot.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator
Initialized: <certbot_nginx.configurator.NginxConfigurator object at 0x7fcffc52ac18>
Prep: True
2019-09-25 03:51:50,477:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_nginx.configurator.NginxConfigurator object at 0x7fcffc52ac18> and installer <certbot_nginx.configurator.NginxConfigurator object at 0x7fcffc52ac18>
2019-09-25 03:51:50,477:INFO:certbot.plugins.selection:Plugins selected: Authenticator nginx, Installer nginx
2019-09-25 03:51:50,479:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(new_authzr_uri=None, terms_of_service=None, uri=‘https://acme-v02.api.letsencrypt.org/acme/acct/59430948’, body=Registration(only_return_existing=None, key=None, terms_of_service_agreed=None, agreement=None, status=None, contact=())), a6e713919c404552a674ac96e1a7e839, Meta(creation_host=‘dizsrv004p.diz.med.uni-muenchen.de’, creation_dt=datetime.datetime(2019, 6, 17, 13, 25, 47, tzinfo=)))>
2019-09-25 03:51:50,480:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-09-25 03:51:50,484:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2019-09-25 03:51:50,525:WARNING:certbot.renewal:Attempting to renew cert (dizsrv004p.diz.med.uni-muenchen.de) from /etc/letsencrypt/renewal/dizsrv004p.diz.med.uni-muenchen.de.conf produced an unexpected error: (“bad handshake: SysCallError(-1, ‘Unexpected EOF’)”,). Skipping.
2019-09-25 03:51:50,528:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py”, line 417, in wrap_socket
cnx.do_handshake()
File “/usr/lib/python3/dist-packages/OpenSSL/SSL.py”, line 1426, in do_handshake
self._raise_ssl_error(self._ssl, result)
File “/usr/lib/python3/dist-packages/OpenSSL/SSL.py”, line 1167, in _raise_ssl_error
raise SysCallError(-1, “Unexpected EOF”)
OpenSSL.SSL.SysCallError: (-1, ‘Unexpected EOF’)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 594, in urlopen
chunked=chunked)
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 350, in _make_request
self._validate_conn(conn)
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 837, in validate_conn
conn.connect()
File “/usr/lib/python3/dist-packages/urllib3/connection.py”, line 323, in connect
ssl_context=context)
File "/usr/lib/python3/dist-packages/urllib3/util/ssl.py", line 324, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File “/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py”, line 424, in wrap_socket
raise ssl.SSLError(‘bad handshake: %r’ % e)
ssl.SSLError: (“bad handshake: SysCallError(-1, ‘Unexpected EOF’)”,)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/requests/adapters.py”, line 423, in send
timeout=timeout
File “/usr/lib/python3/dist-packages/urllib3/connectionpool.py”, line 624, in urlopen
raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: (“bad handshake: SysCallError(-1, ‘Unexpected EOF’)”,)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/certbot/renewal.py”, line 430, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1166, in renew_cert
le_client = _init_le_client(config, auth, installer)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 611, in _init_le_client
return client.Client(config, acc, authenticator, installer, acme=acme)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 248, in init
acme = acme_from_config_key(config, self.account.key, self.account.regr)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 51, in acme_from_config_key
return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 763, in init
directory = messages.Directory.from_json(net.get(server).json())
File “/usr/lib/python3/dist-packages/acme/client.py”, line 1097, in get
self._send_request(‘GET’, url, **kwargs), content_type=content_type)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 1046, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 488, in request
resp = self.send(prep, **send_kwargs)
File “/usr/lib/python3/dist-packages/requests/sessions.py”, line 609, in send
r = adapter.send(request, **kwargs)
File “/usr/lib/python3/dist-packages/requests/adapters.py”, line 497, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError: (“bad handshake: SysCallError(-1, ‘Unexpected EOF’)”,)

2019-09-25 03:51:50,529:ERROR:certbot.renewal:All renewal attempts failed. The following certs could not be renewed:
2019-09-25 03:51:50,529:ERROR:certbot.renewal: /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/fullchain.pem (failure)
2019-09-25 03:51:50,529:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.28.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1340, in main
return config.func(config, plugins)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1247, in renew
renewal.handle_renewal_request(config)
File “/usr/lib/python3/dist-packages/certbot/renewal.py”, line 455, in handle_renewal_request
len(renew_failures), len(parse_failures)))
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)

#7

Another error:

If i do a

certbot certificates

I get:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
OCSP check failed for /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/cert.pem (are we offline?)

/var/log/letsencrypt/letsencrypt.log

2019-09-26 11:01:11,606:INFO:certbot.ocsp:OCSP check failed for /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/cert.pem (are we offline?)
2019-09-26 11:23:25,353:DEBUG:certbot.main:certbot version: 0.28.0
2019-09-26 11:23:25,354:DEBUG:certbot.main:Arguments:
2019-09-26 11:23:25,354:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-09-26 11:23:25,361:DEBUG:certbot.log:Root logging level set at 20
2019-09-26 11:23:25,362:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-09-26 11:23:25,378:DEBUG:certbot.ocsp:Querying OCSP for /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/cert.pem
2019-09-26 11:23:25,378:DEBUG:certbot.ocsp:openssl ocsp -no_nonce -issuer /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/chain.pem -cert /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/cert.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/chain.pem -verify_other /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/chain.pem -trust_other -header Host=ocsp.int-x3.letsencrypt.org
2019-09-26 11:23:25,410:DEBUG:certbot.ocsp:Error while running openssl ocsp -no_nonce -issuer /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/chain.pem -cert /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/cert.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/chain.pem -verify_other /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/chain.pem -trust_other -header Host=ocsp.int-x3.letsencrypt.org.

Error querying OCSP responder

Found the following certs:
Certificate Name: dizsrv004p.diz.med.uni-muenchen.de
Domains: dizsrv004p.diz.med.uni-muenchen.de api.proskive.diz.med.uni-muenchen.de auth.proskive.diz.med.uni-muenchen.de proskive.diz.med.uni-muenchen.de
Expiry Date: 2019-10-01 10:33:30+00:00 (VALID: 5 days)
Certificate Path: /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/fullchain.pem
Private Key Path: /etc/letsencrypt/live/dizsrv004p.diz.med.uni-muenchen.de/privkey.pem

#8

That's not good, but shouldn't block creating a certificate.

If --dry-run works manual, works the renew manual?

PS:

The q argument is bad, that may hide some informations. Try

certbot renew -vvv

instead.

#9

Hi Juergen,

the renew without --dry-run works (on other VMs).

I guess tis because we have to use a firewall proxy despite this subnet should be “open”

The question now is where to place it:

I try add

&& export https=http://myproxy.mydoman.de

in /etc/cron.d/certbot

i will report

best,

Bodo

#10

Just so you know, you can create ‘ENV’ variables in your crons without having to explicitely use ‘export’

Something like

SHELL=/bin/bash
HTTP_PROXY_WHATEVER=proxy-address

      • … certbot renew…

would inherit the HTTP_PROXY_WHATEVER (at least it does work for other stuff of mine). Hoping that can help you figure it out.

You could try renewing by dns, mount the specifc letsencrypt directory somewhere else and forward your web verification there so you could use that other server as a renew server… since the target server is mounted on the renewal (more open) server, then whatever is written is being transfered right away (i mount via sshfs even on a cloudy day)

#11

Hi oldfart,

on debian systems crontab entries are created by certbot but are -not- executed.
I used cron for decades and know how to handle it.
But now its systemd and you have to edit
/lib/systemd/system/certbot.service
and
/lib/systemd/system/certbot.timer
and enable with
systemctl reenable --now certbot.timer
The new child says:
… [/lib/systemd/system/certbot.service:7] Executable path is not absolute, ignoring: export https=https://styx35.med.uni-muenchen.de:8080 && /usr/bin/certbot -q --dry-run renew
… certbot.service: Service lacks both ExecStart= and ExecStop= setting. Refusing.

So there is still the question:

Where to give the proxy information to the automatic systemd certbot renewal?

kind regards,

Bodo

#12

And here is the solution:

edit
/lib/systemd/system/certbot.service
add under [Service]
Environment=“HTTPS_PROXY=http://youproxy.yourdomain:yourportnumber”
and check
/lib/systemd/system/certbot.timer
and enable with
systemctl daemon-reload
fire a test with
systemctl start certbot
check with
watch “systemctl status certbot”

Thanks for all who lead me in the right direction,

Bodo

1 Like
#13

FWIW, that file might get overwritten when Certbot is upgraded.

You can use “sudo systemctl edit certbot” to make permanent changes. (It presents a blank file that gets added onto the original service. So you’d just add the [Service] and Environment lines. I think.)

#14

Why not add a systemd env variable to your service file? Or add a pre start. It really seems like there is lots that can be done here.

I personnaly never use ‘the’ certbot service to renew. Adding a simple daily cron does it for all my servers, that each have about 300ish domains on them.

#15

Hi,

yes this already happened (apt-get update … config is gone)
I will follow your advice

Thanks very much for your reply!

#16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.