Hi. I am experiencing a persistent problem trying to get a certbot to automatically renew certificates when run via cron. However, I can renew certificates fine using the same commands from sudo. Via cron, the immediate error is “Error while running apachectl graceful.”
systemctl status httpd.service complains thusly:
Dec 28 11:55:01 opmiwiki.mghpcc.org CROND[718]: (root) CMD (/usr/bin/certbot renew --quiet)
Dec 28 11:55:03 opmiwiki.mghpcc.org httpd[746]: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
Dec 28 11:55:03 opmiwiki.mghpcc.org httpd[746]: SSLCertificateFile: file '/var/lib/letsencrypt/LsNQY6mcFER3HPFz-0D_z6aIWosQq-KDWjCz2V2hOJY.crt' does not exist or is empty
Dec 28 11:55:03 opmiwiki.mghpcc.org systemd[1]: httpd.service: control process exited, code=exited status=1
Dec 28 11:55:03 opmiwiki.mghpcc.org systemd[1]: Reload failed for The Apache HTTP Server.
I installed certbot 0.9.3 using the EPEL package python-certbot-apache.
CentOS Linux release 7.3.1611 (Core) with SELinux enabled.
Apache (httpd) 2.4.6-45.el7.centos
HTTPS host = opmiwiki.mghpcc.org.
crontab lines:
55 11 * * * root /usr/bin/certbot renew --quiet
55 23 * * * root /usr/bin/certbot renew --quiet
When these cron entries run, they generate the following /var/log/letsencrypt/letsencrypt.log:
2016-12-28 16:55:01,845:DEBUG:certbot.main:Root logging level set at 30
2016-12-28 16:55:01,845:INFO:certbot.main:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2016-12-28 16:55:01,845:DEBUG:certbot.main:certbot version: 0.9.3
2016-12-28 16:55:01,845:DEBUG:certbot.main:Arguments: ['--quiet']
2016-12-28 16:55:01,845:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,PluginEntryPoint#standalone)
2016-12-28 16:55:01,852:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2017-01-23 12:38:00 UTC.
2016-12-28 16:55:01,852:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2016-12-28 16:55:01,853:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2016-12-28 16:55:02,257:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.configurator:ApacheConfigurator
Initialized: <certbot_apache.configurator.ApacheConfigurator object at 0x1ac40d0>
Prep: True
2016-12-28 16:55:02,257:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.configurator:ApacheConfigurator
Initialized: <certbot_apache.configurator.ApacheConfigurator object at 0x1ac40d0>
Prep: True
2016-12-28 16:55:02,258:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.configurator.ApacheConfigurator object at 0x1ac40d0> and installer <certbot_apache.configurator.ApacheConfigurator object at 0x1ac40d0>
2016-12-28 16:55:02,379:DEBUG:certbot.main:Picked account: <Account(6833b58cc8eea8e3697ecb13583d0a46)>
2016-12-28 16:55:02,383:DEBUG:root:Sending GET request to https://acme-v01.api.letsencrypt.org/directory. args: (), kwargs: {}
2016-12-28 16:55:02,386:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2016-12-28 16:55:02,653:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 352
2016-12-28 16:55:02,654:DEBUG:root:Received <Response [200]>. Headers: {'content-length': '352', 'strict-transport-security': 'max-age=604800', 'boulder-request-id': 'Dhaq-T1kkGKQCjgMsM3w5n1elNW3b3Jc79HJAnCDNq8', 'expires': 'Wed, 28 Dec 2016 16:50:13 GMT', 'server': 'nginx', 'connection': 'keep-alive', 'pragma': 'no-cache', 'cache-control': 'max-age=0, no-cache, no-store', 'date': 'Wed, 28 Dec 2016 16:50:13 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'replay-nonce': 'ohTeyksorJQR__tSrrRcewK8dkhFybFdbDEih53HHpk'}. Content: '{\n "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",\n "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",\n "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",\n "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",\n "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"\n}'
2016-12-28 16:55:02,654:DEBUG:acme.client:Received response <Response [200]> (headers: {'content-length': '352', 'strict-transport-security': 'max-age=604800', 'boulder-request-id': 'Dhaq-T1kkGKQCjgMsM3w5n1elNW3b3Jc79HJAnCDNq8', 'expires': 'Wed, 28 Dec 2016 16:50:13 GMT', 'server': 'nginx', 'connection': 'keep-alive', 'pragma': 'no-cache', 'cache-control': 'max-age=0, no-cache, no-store', 'date': 'Wed, 28 Dec 2016 16:50:13 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'replay-nonce': 'ohTeyksorJQR__tSrrRcewK8dkhFybFdbDEih53HHpk'}): '{\n "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",\n "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",\n "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",\n "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",\n "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"\n}'
2016-12-28 16:55:02,654:INFO:certbot.main:Renewing an existing certificate
2016-12-28 16:55:02,654:DEBUG:root:Requesting fresh nonce
2016-12-28 16:55:02,654:DEBUG:root:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz. args: (), kwargs: {}
2016-12-28 16:55:02,738:DEBUG:requests.packages.urllib3.connectionpool:"HEAD /acme/new-authz HTTP/1.1" 405 0
2016-12-28 16:55:02,738:DEBUG:root:Received <Response [405]>. Headers: {'content-length': '91', 'allow': 'POST', 'boulder-request-id': 'dFCr9OqRDdhJwBqg2jLHYPYYMv--Uzo4hWuhlLLz_v8', 'expires': 'Wed, 28 Dec 2016 16:50:13 GMT', 'server': 'nginx', 'connection': 'keep-alive', 'pragma': 'no-cache', 'cache-control': 'max-age=0, no-cache, no-store', 'date': 'Wed, 28 Dec 2016 16:50:13 GMT', 'content-type': 'application/problem+json', 'replay-nonce': 'TrPQSDcBKKe1KrrRU9MLA9aCYP5PO69n2jGC-K3jNxI'}. Content: ''
2016-12-28 16:55:02,738:DEBUG:acme.client:Storing nonce: 'N\xb3\xd0H7\x01(\xa7\xb5*\xba\xd1S\xd3\x0b\x03\xd6\x82`\xfeO;\xafg\xda1\x82\xf8\xad\xe37\x12'
2016-12-28 16:55:02,738:DEBUG:acme.jose.json_util:Omitted empty fields: expires=None, challenges=None, combinations=None, status=None
2016-12-28 16:55:02,738:DEBUG:acme.client:Serialized JSON: {"identifier": {"type": "dns", "value": "opmiwiki.mghpcc.org"}, "resource": "new-authz"}
2016-12-28 16:55:02,739:DEBUG:acme.jose.json_util:Omitted empty fields: x5c=(), crit=(), typ=None, jwk=None, x5u=None, kid=None, alg=None, cty=None, x5tS256=None, jku=None, x5t=None
2016-12-28 16:55:02,741:DEBUG:acme.jose.json_util:Omitted empty fields: x5c=(), crit=(), typ=None, x5u=None, kid=None, cty=None, nonce=None, x5tS256=None, jku=None, x5t=None
2016-12-28 16:55:02,741:DEBUG:root:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz. args: (), kwargs: {'data': '{"header": {"alg": "RS256", "jwk": {"e": "AQAB", "kty": "RSA", "n": "yFIrwFz0oma-7oNFUIU6Xx0oEO9ZGAlIIDwaxfrD20M3wBOW3sP1PkhY-Fhw7DxPm9LP3y5G63GCLaN24BFjHt92Cc625AmeuODkOZhApkLgYEq2t3oW8hgNAMloMtbckLCh6oejD_XCO6GxR9_n1G2ReMUOWj_VbD9OSIEVcGWZnU2UxICXu9fcDcO_EwWPh2j_B6c6iHTaK0bFuz8rR8d53HeMrbcgbtULJECUr94KgJDmSPAqkyBeEFr6m530SBq3N9m3-75mpO-z_MYjt32guFVtREu1eCtFtpeKeuJU-QQH9MAox96ad-KyAKjrvJZzO8iX6JdMPE5xC-Aj-w"}}, "protected": "eyJub25jZSI6ICJUclBRU0RjQktLZTFLcnJSVTlNTEE5YUNZUDVQTzY5bjJqR0MtSzNqTnhJIn0", "payload": "eyJpZGVudGlmaWVyIjogeyJ0eXBlIjogImRucyIsICJ2YWx1ZSI6ICJvcG1pd2lraS5tZ2hwY2Mub3JnIn0sICJyZXNvdXJjZSI6ICJuZXctYXV0aHoifQ", "signature": "XzE2mA1l2s_1uNYX3Cu-T5D0G45JXzDCE5dqcLDvufP4IiODWcEmHxkijaIP763l0-s6nUO8VPn2kj00llSRC3YyNL6Xp-s6320RLpq1_2qiri8wANdPSfqaPrjkEAU3wq-dvtTYDDfGwEgRU8U2HsHKmUGo-BafCYlSjjXoufUBxI7ijNZ953uyknvvXl3h6o3k_rFHhQN-txtq6NVesgUIdXDi1oG3TErpoE5kZuooifCugmlU3a-q93ndhq_mQoAmDZv14qQuEjNbT7CNCjLTiAOBU9OpvHHo9ShUkBpKP5k43DGJuOHDr9ZZ13D-wC-xc3cVBOlpv-peuy9k1g"}'}
2016-12-28 16:55:02,849:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/new-authz HTTP/1.1" 201 1004
2016-12-28 16:55:02,850:DEBUG:root:Received <Response [201]>. Headers: {'content-length': '1004', 'strict-transport-security': 'max-age=604800', 'boulder-request-id': 'lgDUzfE5mcEkSxVioiX-x8mp56MRjvA1e4oMEyne__M', 'boulder-requester': '3583115', 'expires': 'Wed, 28 Dec 2016 16:50:13 GMT', 'server': 'nginx', 'connection': 'keep-alive', 'link': '<https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"', 'location': 'https://acme-v01.api.letsencrypt.org/acme/authz/3OXjell4ejOBk1qufFllZUd8zVzCbDFncn-ghgQWOJU', 'pragma': 'no-cache', 'cache-control': 'max-age=0, no-cache, no-store', 'date': 'Wed, 28 Dec 2016 16:50:13 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'replay-nonce': 'b3tFIXNRBbROozg22wjHd8AN6yHdLkoB6P_J9syvSiY'}. Content: '{\n "identifier": {\n "type": "dns",\n "value": "opmiwiki.mghpcc.org"\n },\n "status": "pending",\n "expires": "2017-01-04T16:50:13.305632065Z",\n "challenges": [\n {\n "type": "tls-sni-01",\n "status": "pending",\n "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3OXjell4ejOBk1qufFllZUd8zVzCbDFncn-ghgQWOJU/452374912",\n "token": "LsNQY6mcFER3HPFz-0D_z6aIWosQq-KDWjCz2V2hOJY"\n },\n {\n "type": "dns-01",\n "status": "pending",\n "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3OXjell4ejOBk1qufFllZUd8zVzCbDFncn-ghgQWOJU/452374913",\n "token": "6waUQjjFL0aBB6Cqn9CO-1R8dkb2mKv9RQZB1gz4IfE"\n },\n {\n "type": "http-01",\n "status": "pending",\n "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3OXjell4ejOBk1qufFllZUd8zVzCbDFncn-ghgQWOJU/452374914",\n "token": "YN7T-Oo4Au2wFHXQBZ0xj53dU70-HbAlyPrI4mkekF4"\n }\n ],\n "combinations": [\n [\n 1\n ],\n [\n 2\n ],\n [\n 0\n ]\n ]\n}'
2016-12-28 16:55:02,850:DEBUG:acme.client:Storing nonce: 'o{E!sQ\x05\xb4N\xa386\xdb\x08\xc7w\xc0\r\xeb!\xdd.J\x01\xe8\xff\xc9\xf6\xcc\xafJ&'
2016-12-28 16:55:02,850:DEBUG:acme.client:Received response <Response [201]> (headers: {'content-length': '1004', 'strict-transport-security': 'max-age=604800', 'boulder-request-id': 'lgDUzfE5mcEkSxVioiX-x8mp56MRjvA1e4oMEyne__M', 'boulder-requester': '3583115', 'expires': 'Wed, 28 Dec 2016 16:50:13 GMT', 'server': 'nginx', 'connection': 'keep-alive', 'link': '<https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"', 'location': 'https://acme-v01.api.letsencrypt.org/acme/authz/3OXjell4ejOBk1qufFllZUd8zVzCbDFncn-ghgQWOJU', 'pragma': 'no-cache', 'cache-control': 'max-age=0, no-cache, no-store', 'date': 'Wed, 28 Dec 2016 16:50:13 GMT', 'x-frame-options': 'DENY', 'content-type': 'application/json', 'replay-nonce': 'b3tFIXNRBbROozg22wjHd8AN6yHdLkoB6P_J9syvSiY'}): '{\n "identifier": {\n "type": "dns",\n "value": "opmiwiki.mghpcc.org"\n },\n "status": "pending",\n "expires": "2017-01-04T16:50:13.305632065Z",\n "challenges": [\n {\n "type": "tls-sni-01",\n "status": "pending",\n "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3OXjell4ejOBk1qufFllZUd8zVzCbDFncn-ghgQWOJU/452374912",\n "token": "LsNQY6mcFER3HPFz-0D_z6aIWosQq-KDWjCz2V2hOJY"\n },\n {\n "type": "dns-01",\n "status": "pending",\n "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3OXjell4ejOBk1qufFllZUd8zVzCbDFncn-ghgQWOJU/452374913",\n "token": "6waUQjjFL0aBB6Cqn9CO-1R8dkb2mKv9RQZB1gz4IfE"\n },\n {\n "type": "http-01",\n "status": "pending",\n "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/3OXjell4ejOBk1qufFllZUd8zVzCbDFncn-ghgQWOJU/452374914",\n "token": "YN7T-Oo4Au2wFHXQBZ0xj53dU70-HbAlyPrI4mkekF4"\n }\n ],\n "combinations": [\n [\n 1\n ],\n [\n 2\n ],\n [\n 0\n ]\n ]\n}'
2016-12-28 16:55:02,851:INFO:certbot.auth_handler:Performing the following challenges:
2016-12-28 16:55:02,851:INFO:certbot.auth_handler:tls-sni-01 challenge for opmiwiki.mghpcc.org
2016-12-28 16:55:02,915:DEBUG:certbot_apache.tls_sni_01:Adding Include /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf to /files/etc/httpd/conf/httpd.conf
2016-12-28 16:55:02,915:DEBUG:certbot_apache.tls_sni_01:writing a config file with text:
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName 9b62a8509384a78bd731a72eafe3ef09.d0471c2cb9e462b55b55c58e5ac72ec2.acme.invalid
UseCanonicalName on
SSLStrictSNIVHostCheck on
LimitRequestBody 1048576
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /var/lib/letsencrypt/LsNQY6mcFER3HPFz-0D_z6aIWosQq-KDWjCz2V2hOJY.crt
SSLCertificateKeyFile /var/lib/letsencrypt/LsNQY6mcFER3HPFz-0D_z6aIWosQq-KDWjCz2V2hOJY.pem
DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
</VirtualHost>
</IfModule>
2016-12-28 16:55:02,931:DEBUG:certbot.reverter:Creating backup of /etc/httpd/conf/httpd.conf
2016-12-28 16:55:03,172:ERROR:certbot.util:Error while running apachectl graceful.
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
2016-12-28 16:55:03,173:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 112, in _solve_challenges
resp = self.auth.perform(self.achalls)
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1755, in perform
self.restart()
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1665, in restart
self._reload()
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1676, in _reload
raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apachectl graceful.
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
2016-12-28 16:55:03,174:DEBUG:certbot.error_handler:Calling registered functions
2016-12-28 16:55:03,174:INFO:certbot.auth_handler:Cleaning up challenges
2016-12-28 16:55:03,511:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/opmiwiki.mghpcc.org.conf produced an unexpected error: Error while running apachectl graceful.
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
. Skipping.
2016-12-28 16:55:03,512:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 348, in renew_all_lineages
main.obtain_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 563, in obtain_cert
action, _ = _auth_from_domains(le_client, config, domains, lineage)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 96, in _auth_from_domains
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 238, in renew_cert
new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File "/usr/lib/python2.7/site-packages/certbot/client.py", line 253, in obtain_certificate
self.config.allow_subset_of_names)
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 74, in get_authorizations
resp = self._solve_challenges()
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 112, in _solve_challenges
resp = self.auth.perform(self.achalls)
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1755, in perform
self.restart()
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1665, in restart
self._reload()
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1676, in _reload
raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apachectl graceful.
Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
2016-12-28 16:55:03,514:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 9, in <module>
load_entry_point('certbot==0.9.3', 'console_scripts', 'certbot')()
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 776, in main
return config.func(config, plugins)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 592, in renew
renewal.renew_all_lineages(config)
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 365, in renew_all_lineages
len(renew_failures), len(parse_failures)))
Error: 1 renew failure(s), 0 parse failure(s)
Two months ago, I ran the exact same command from sudo, and renewal worked. This time, sudo /usr/bin/certbot renew --dry-run succeeded, but I have otherwise left my environment alone for testing.