LetsEncrypt suddenly failing to create temporary web root for verification?

Server
1

Hello,

I have been using LetsEncrypt on my web server for some time now, slowly adding certificates for domains as my old certificates with another CA are expiring.

I recently started having a problem with the process, and I can no longer issue certificates. I do not know what caused these issues, as I hadn’t made any changes that I am aware of, it just stopped working.

Since the issue started, I deleted my letsencrypt folder and recloned it from Git, upgrading to the latest version, however the issue persists.

Environment: CentOS 7, Apache (httpd.x86_64 2.4.6-40.el7.centos)

Steps: I run the letsencrypt-auto script and choose the domain name and its www. subdomain. The client fails with the following error:

Error while running apachectl graceful.

Job for httpd.service invalid.

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: www.[redacted]
    Type: connection
    Detail: Failed to connect to host for DVSNI challenge

    Domain: [redacted]
    Type: connection
    Detail: Failed to connect to host for DVSNI challenge

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

Further debug information: After this happens, httpd process on my server is dead. If I run ‘service httpd status’, I see that a letsencrypt documentroot doesn’t exist. The path specified genuinely does not exist. Looks like maybe LetsEncrypt is failing to create it? But I don’t see any errors indicating that’s happening, and the program is being run as root.

[root@server letsencrypt]# service httpd status
Redirecting to /bin/systemctl status httpd.service
● httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2016-02-29 21:17:00 EST; 37s ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 13913 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)
Process: 13910 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Process: 8075 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)
Main PID: 8075 (code=exited, status=1/FAILURE)
Status: “Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec”

Feb 29 21:17:00 server httpd[13910]: AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
Feb 29 21:17:00 server httpd[13910]: AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
Feb 29 21:17:00 server httpd[13910]: AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
Feb 29 21:17:00 server systemd[1]: Reloaded The Apache HTTP Server.
Feb 29 21:17:00 server systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Feb 29 21:17:00 server kill[13913]: kill: cannot find process ""
Feb 29 21:17:00 server systemd[1]: httpd.service: control process exited, code=exited status=1
Feb 29 21:17:00 server systemd[1]: Unit httpd.service entered failed state.
Feb 29 21:17:00 server systemd[1]: httpd.service failed.
Feb 29 21:17:07 server systemd[1]: Unit httpd.service cannot be reloaded because it is inactive.
[root@server letsencrypt]#

Any insights would be greatly appreciated. :slight_smile:

2

Is this the incorrect forum for letsencrypt-auto client support?

I am still having issues with this and have not determined why yet.

3

Let's Encrypt adds a temporary VirtualHost to your apache configuration for tls-sni-01 verification. The client is supposed to remove that VirtualHost after verification has succeeded or failed. It shouldn't leave any of that configuration behind (that would indicate the client ran into some kind of bug). I would recommend removing that VirtualHost manually if it still exists, as I'm not sure how the client would deal with that.

Aside from that: Are there any SELinux policies or something similar that might prevent the client from creating the webroot?

Could you provide /var/log/letsencrypt/letsencrypt.log after trying ./letsencrypt-auto again, including the flags -vvvv?

4

Hi pfg,

Thanks for the tip, I have gone ahead and produced the log for you.

You can view it here: https://iota.lt/d/fZYTqB328N/YmMdyaPl.txt

I have attempted this with SELinux running in permissive mode and with my iptables rules set completely open. It is not being caused by those protections.

When Apache dies, this is what is logged by it:

[Mon Mar 28 02:51:02.342034 2016] [mpm_prefork:notice] [pid 23314] AH00171: Graceful restart requested, doing restart AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist (98)Address already in use: AH00072: make_sock: could not bind to address [::]:443 (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443 [Mon Mar 28 02:51:02.430901 2016] [mpm_prefork:alert] [pid 23314] no listening sockets available, shutting down [Mon Mar 28 02:51:02.430915 2016] [:emerg] [pid 23314] AH00019: Unable to open logs, exiting

Please let me know if you have any further suggestions. :slight_smile:

Thanks,
Kirk

5

Just to clarify, is your apache configuration in a valid state after the client ran (i.e. you can manually start it again), or do you have to fix the configuration before you’re able to start it at all? If it’s the latter, could you provide your httpd.conf (or anything else you have to modify)?

If you simply run apachectl graceful (just that, no letsencrypt commands) while apache is running, does that work and is apache still running afterwards, or does it fail in a similar way?

6

The configuration is valid before and after the LetsEncrypt run.

The only issue is that running the LetsEncrypt client is causing Apache to crash and need a manual restart.

I can run apachectl graceful on its own without any issues.

7

Hey I’m having the same issues with apache2 on centos7
At first LetsEncrypt installed perfect but when i try to renew the certs the same issue as above occurs
LE does something nasty causing the apache2 to die

Was this bug ever solved ?

8

Nope. Have this issue on Linux Mint Serena/Ubuntu Xenial.

9

same issue here - brand new centos 7 with apache 2 and 5 domains same warnings - any way of fixing this ?

10

So it still works OK for everyone to run apachectl graceful directly as root, yet not from within Certbot?

11

Hi,
I also have this problem on two different CentOS servers. The apachectl graceful command works w/o any problem when started from terminal:

# apachectl graceful
# echo $?
0

That error message comes from cron (via e-mail):

Error while running apachectl graceful.

Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

Attempting to renew cert from /etc/letsencrypt/renewal/xxx.conf produced an unexpected error: Error while running apachectl graceful.

Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
. Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/xxx/fullchain.pem (failure)
1 renew failure(s), 0 parse failure(s)

Here is relevant (i.e. 5 AM when there were attempt to renew the certificate) content of systemctl status httpd.service -l:

kvě 15 05:58:03 xxx httpd[3276]: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
kvě 15 05:58:03 xxx httpd[3276]: SSLCertificateFile: file '/var/lib/letsencrypt/TB6OODfKkRP-uJqt60w5EHIMuITODA__Nmi0V-J0ed0.crt' does not exist or is empty
kvě 15 05:58:03 xxx systemd[1]: httpd.service: control process exited, code=exited status=1
kvě 15 05:58:03 xxx systemd[1]: Reload failed for The Apache HTTP Server.
kvě 15 05:58:04 xxx systemd[1]: Reloaded The Apache HTTP Server.

Directory /var/lib/letsencrypt/ exists and it contains directory backup.

Versions (it is regularly updated production CentOS):

# cat /etc/redhat-release 
CentOS Linux release 7.3.1611 (Core)
# rpm -qa |grep certbot
python2-certbot-0.12.0-4.el7.noarch
python2-certbot-apache-0.12.0-1.el7.noarch
certbot-0.12.0-4.el7.noarch

When I run /usr/bin/certbot renew it works as expected and the certificate is renewed. So for me it looks like problem when running from cron (environment issue?). Please note that we have SELinux enabled.

Please let me know if you want more information or to test something.

Thanks,

Tomas

12

@bmw, do you think you could figure out anything about the problem that @stoupa-cz is having?

13

@stoupa-cz, can you provide one of Certbot’s logs showcasing this problem? By default they are stored in /var/log/letsencrypt. Feel free to redact domains, e-mail, and IP addresses as you wish.

14

Hi,

log from the run is below. I cannot upload it as a “new user”, so sorry for pasting it directly. Only the domain is redacted, the hostnames remain the same (we have 2 DNS names on that particular server).

2017-05-15 03:58:01,731:DEBUG:certbot.main:Root logging level set at 30
2017-05-15 03:58:01,732:INFO:certbot.main:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-05-15 03:58:01,732:DEBUG:certbot.main:certbot version: 0.12.0
2017-05-15 03:58:01,732:DEBUG:certbot.main:Arguments: ['--quiet']
2017-05-15 03:58:01,732:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,PluginEntryPoint#standalone)
2017-05-15 03:58:01,740:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2017-06-12 10:21:00 UTC.
2017-05-15 03:58:01,741:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2017-05-15 03:58:01,756:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2017-05-15 03:58:02,160:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.configurator:ApacheConfigurator
Initialized: <certbot_apache.configurator.ApacheConfigurator object at 0x315c7d0>
Prep: True
2017-05-15 03:58:02,161:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.configurator:ApacheConfigurator
Initialized: <certbot_apache.configurator.ApacheConfigurator object at 0x315c7d0>
Prep: True
2017-05-15 03:58:02,161:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.configurator.ApacheConfigurator object at 0x315c7d0> and installer <certbot_apache.configurator.ApacheConfigurator object at 0x315c7d0>
2017-05-15 03:58:02,313:DEBUG:certbot.main:Picked account: <Account(f470b685abb1fbed474d020598c88590)>
2017-05-15 03:58:02,314:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2017-05-15 03:58:02,317:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2017-05-15 03:58:02,661:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 352
2017-05-15 03:58:02,663:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 352
Boulder-Request-Id: w40O5I5hN8CIhNBxQJuV9N2jNqImgVI6wJfrTc3MYFs
Replay-Nonce: 7j-SHOn5PJh6wjJTMuU6DxZCSQzba8Q6kx5JYlO3vlE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Mon, 15 May 2017 03:58:05 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Mon, 15 May 2017 03:58:05 GMT
Connection: keep-alive

{
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
}
2017-05-15 03:58:02,663:INFO:certbot.main:Renewing an existing certificate
2017-05-15 03:58:02,665:DEBUG:acme.client:Requesting fresh nonce
2017-05-15 03:58:02,665:DEBUG:acme.client:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz.
2017-05-15 03:58:02,864:DEBUG:requests.packages.urllib3.connectionpool:"HEAD /acme/new-authz HTTP/1.1" 405 0
2017-05-15 03:58:02,865:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 91
Allow: POST
Boulder-Request-Id: IZnyL76eI67MqOxDJb-xfZHq7xxxG1vztnYQP1_cdWg
Replay-Nonce: mtt6PjpzcnDel2gIlOMqQewl1j6KDfMBZP4fieGagc4
Expires: Mon, 15 May 2017 03:58:05 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Mon, 15 May 2017 03:58:05 GMT
Connection: keep-alive


2017-05-15 03:58:02,865:DEBUG:acme.client:Storing nonce: mtt6PjpzcnDel2gIlOMqQewl1j6KDfMBZP4fieGagc4
2017-05-15 03:58:02,865:DEBUG:acme.client:JWS payload:
{
  "identifier": {
    "type": "dns", 
    "value": "iotdata.XXX"
  }, 
  "resource": "new-authz"
}
2017-05-15 03:58:02,872:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz:
{
  "header": {
    "alg": "RS256", 
    "jwk": {
      "e": "AQAB", 
      "kty": "RSA", 
      "n": "rlxONXYWuc3hEuVPk-PGopmFJU7zUmJNyvxXbnxWkF3LjMJZZL9SaydN3KUP0ObjUkEGFDwEyOFGPyOWvwqU9xpR9QHf7sjDkmTsmrbKO8QAYRaYua4eUc2H2ippf62aYvqX2oGbPouXPXLsynGt8Rfw-m4j9d4CEG0xrGupTv_cPdeV366Oiq-7Li7lI2nMB4Ts39xAyCNlUQ5Vw3Tuh3N_JCgifhtzzszmfOongmLTmICquZ0jlRLgay8Ym0GhF75yxIwt3hTKPKCv_Zr7xPcaoIzxVOm1G6-xFml3WxRYqZLBj490HpW2BWa-M-wUvoB0A3jiRR8EmTYwYgcw7w"
    }
  }, 
  "protected": "eyJub25jZSI6ICJtdHQ2UGpwemNuRGVsMmdJbE9NcVFld2wxajZLRGZNQlpQNGZpZUdhZ2M0In0", 
  "payload": "ewogICJpZGVudGlmaWVyIjogewogICAgInR5cGUiOiAiZG5zIiwgCiAgICAidmFsdWUiOiAiaW90ZGF0YS5maXQudnV0YnIuY3oiCiAgfSwgCiAgInJlc291cmNlIjogIm5ldy1hdXRoeiIKfQ", 
  "signature": "nBfp31kDVZG-c0H6OlhDzS5hUatgGM8V7vf4eZ05D0a1aYkwe8ILVEaKvYKpt7-uTYwHnKmpt-b8RBUVXu1Qkm51c1IpXwqh2ywfcbgqSC45hKFYIkM9L6EgOyaNSA3eLFVEG-jeYG9JwW7R8q9QwxWiRcMzEJTA6UGevMJH4Ubq8Ben7qHs7K5ckBCMGO6ur762UqGriVLqO_ZB6IItVx2t28pFHPn-qpewQK19a4U5KkBK--L-YFk_yvK-bW1jAIKz4MleCrDECABy5NEeR0MINMTvhBF1VyQS-znXEOPm3Yygx8K_V4mxi-z29oor3DgYEykMtLahRApTejYINQ"
}
2017-05-15 03:58:03,088:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/new-authz HTTP/1.1" 201 1008
2017-05-15 03:58:03,089:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 1008
Boulder-Request-Id: z1uaokW7enAS6aaBPgL35l4D7QReH6I8ytpWKx9JyTo
Boulder-Requester: 10607561
Link: <https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"
Location: https://acme-v01.api.letsencrypt.org/acme/authz/QxPNUyfs8fa4VYyBTK7NKtEsHB1Q4jfzGdLYcIZkASo
Replay-Nonce: FajfGW8PiADCHSn7SMmi1k-NFVNBn-t54Js0aH89qmI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Mon, 15 May 2017 03:58:05 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Mon, 15 May 2017 03:58:05 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "iotdata.XXX"
  },
  "status": "pending",
  "expires": "2017-05-22T03:58:05.640801717Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/QxPNUyfs8fa4VYyBTK7NKtEsHB1Q4jfzGdLYcIZkASo/1176510650",
      "token": "BEIv3Kx5no5_oN9v88nxRxDmdrM2j7R1vGp0eyaXljA"
    },
    {
      "type": "tls-sni-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/QxPNUyfs8fa4VYyBTK7NKtEsHB1Q4jfzGdLYcIZkASo/1176510651",
      "token": "TB6OODfKkRP-uJqt60w5EHIMuITODA__Nmi0V-J0ed0"
    },
    {
      "type": "http-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/QxPNUyfs8fa4VYyBTK7NKtEsHB1Q4jfzGdLYcIZkASo/1176510652",
      "token": "sJn5kk1Pkuc_Z11Hos4J6PyYlcjJUp35HcDtLSk6HMQ"
    }
  ],
  "combinations": [
    [
      2
    ],
    [
      0
    ],
    [
      1
    ]
  ]
}
2017-05-15 03:58:03,089:DEBUG:acme.client:Storing nonce: FajfGW8PiADCHSn7SMmi1k-NFVNBn-t54Js0aH89qmI
2017-05-15 03:58:03,091:DEBUG:acme.client:JWS payload:
{
  "identifier": {
    "type": "dns", 
    "value": "antdev.XXX"
  }, 
  "resource": "new-authz"
}
2017-05-15 03:58:03,097:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz:
{
  "header": {
    "alg": "RS256", 
    "jwk": {
      "e": "AQAB", 
      "kty": "RSA", 
      "n": "rlxONXYWuc3hEuVPk-PGopmFJU7zUmJNyvxXbnxWkF3LjMJZZL9SaydN3KUP0ObjUkEGFDwEyOFGPyOWvwqU9xpR9QHf7sjDkmTsmrbKO8QAYRaYua4eUc2H2ippf62aYvqX2oGbPouXPXLsynGt8Rfw-m4j9d4CEG0xrGupTv_cPdeV366Oiq-7Li7lI2nMB4Ts39xAyCNlUQ5Vw3Tuh3N_JCgifhtzzszmfOongmLTmICquZ0jlRLgay8Ym0GhF75yxIwt3hTKPKCv_Zr7xPcaoIzxVOm1G6-xFml3WxRYqZLBj490HpW2BWa-M-wUvoB0A3jiRR8EmTYwYgcw7w"
    }
  }, 
  "protected": "eyJub25jZSI6ICJGYWpmR1c4UGlBRENIU243U01taTFrLU5GVk5Cbi10NTRKczBhSDg5cW1JIn0", 
  "payload": "ewogICJpZGVudGlmaWVyIjogewogICAgInR5cGUiOiAiZG5zIiwgCiAgICAidmFsdWUiOiAiYW50ZGV2LmZpdC52dXRici5jeiIKICB9LCAKICAicmVzb3VyY2UiOiAibmV3LWF1dGh6Igp9", 
  "signature": "dyxpxsHpUdzmvyywCuEvzxsEn8_r5bNu_oQE1sblBAr4LQwaFA7RDGNHiWdmVl_fIwRmwo-cl_e92mw3W3dtlNxkNXCwsM67aqX3U9_VzDUOaI8eMvYpC7jXP8J4FKNHHXd7Yl6fiLqpf_yrq4-RjFCdZ7fDFjthRCPv_NsnQ1rUqpUdNoK7XvvyaFCtjjIr5HAF1SmyYeNymVEDgI0nrGZ6WjJqyH7Z7cRbLpB_Xek9IMe_dcYHDZTwgBZkgNxnHizs9KofHKG26j0fZ9X-B2phfVYKrLHxMO8CD_ZhhG04yn5xwPrCpbC37k37jXANJRQpwU9ra7TwbUQ_XHMrfg"
}
2017-05-15 03:58:03,336:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/new-authz HTTP/1.1" 201 1007
2017-05-15 03:58:03,337:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 1007
Boulder-Request-Id: -jPymuhYMT0JcUCqkHbQuDIaJf6C9ZdMALQbZ_iNSyY
Boulder-Requester: 10607561
Link: <https://acme-v01.api.letsencrypt.org/acme/new-cert>;rel="next"
Location: https://acme-v01.api.letsencrypt.org/acme/authz/5riyoRV9SSRhPukyrlU7uK9Owd6k6tbw78tUJYgiwjo
Replay-Nonce: TuBeMuWFt1omt0RlA8BmcA4P97uDj_wupwq7DG61eKs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Mon, 15 May 2017 03:58:06 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Mon, 15 May 2017 03:58:06 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "antdev.XXX"
  },
  "status": "pending",
  "expires": "2017-05-22T03:58:05.891114136Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/5riyoRV9SSRhPukyrlU7uK9Owd6k6tbw78tUJYgiwjo/1176510663",
      "token": "H_JnnL6HnjN8LT20RdN9gHOVa8FVJSzGmpjRsvOKCoc"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/5riyoRV9SSRhPukyrlU7uK9Owd6k6tbw78tUJYgiwjo/1176510665",
      "token": "l5PGnI-rYXYVBp5hZpvj2JD2LL48rqbQmMCmCjl3V_E"
    },
    {
      "type": "tls-sni-01",
      "status": "pending",
      "uri": "https://acme-v01.api.letsencrypt.org/acme/challenge/5riyoRV9SSRhPukyrlU7uK9Owd6k6tbw78tUJYgiwjo/1176510666",
      "token": "pO82JVV9WjALd5X6UBqbBu70fKVeJ6BASTTQaSMU4Y8"
    }
  ],
  "combinations": [
    [
      1
    ],
    [
      2
    ],
    [
      0
    ]
  ]
}
2017-05-15 03:58:03,337:DEBUG:acme.client:Storing nonce: TuBeMuWFt1omt0RlA8BmcA4P97uDj_wupwq7DG61eKs
2017-05-15 03:58:03,338:INFO:certbot.auth_handler:Performing the following challenges:
2017-05-15 03:58:03,338:INFO:certbot.auth_handler:tls-sni-01 challenge for iotdata.XXX
2017-05-15 03:58:03,338:INFO:certbot.auth_handler:tls-sni-01 challenge for antdev.XXX
2017-05-15 03:58:03,575:DEBUG:certbot_apache.tls_sni_01:Adding Include /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf to /files/etc/httpd/conf/httpd.conf
2017-05-15 03:58:03,575:DEBUG:certbot_apache.tls_sni_01:writing a config file with text:
 <IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName 149d6ffc82f437ae5b09479b12a5d541.6ba8c946d1905cdd6c174610a1b9956b.acme.invalid
    UseCanonicalName on
    SSLStrictSNIVHostCheck on

    LimitRequestBody 1048576

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /var/lib/letsencrypt/TB6OODfKkRP-uJqt60w5EHIMuITODA__Nmi0V-J0ed0.crt
    SSLCertificateKeyFile /var/lib/letsencrypt/TB6OODfKkRP-uJqt60w5EHIMuITODA__Nmi0V-J0ed0.pem

    DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
</VirtualHost>

<VirtualHost *:443>
    ServerName 01a0105c6ed58c4b1981b73b7be8596c.4754dff74df8c50baefc6317bb90974e.acme.invalid
    UseCanonicalName on
    SSLStrictSNIVHostCheck on

    LimitRequestBody 1048576

    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /var/lib/letsencrypt/pO82JVV9WjALd5X6UBqbBu70fKVeJ6BASTTQaSMU4Y8.crt
    SSLCertificateKeyFile /var/lib/letsencrypt/pO82JVV9WjALd5X6UBqbBu70fKVeJ6BASTTQaSMU4Y8.pem

    DocumentRoot /var/lib/letsencrypt/tls_sni_01_page/
</VirtualHost>

</IfModule>

2017-05-15 03:58:03,691:DEBUG:certbot.reverter:Creating backup of /etc/httpd/conf/httpd.conf
2017-05-15 03:58:03,972:ERROR:certbot.util:Error while running apachectl graceful.

Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.

2017-05-15 03:58:03,975:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 111, in _solve_challenges
    resp = self.auth.perform(self.achalls)
  File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1748, in perform
    self.restart()
  File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1658, in restart
    self._reload()
  File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1669, in _reload
    raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apachectl graceful.

Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.


2017-05-15 03:58:03,975:DEBUG:certbot.error_handler:Calling registered functions
2017-05-15 03:58:03,975:INFO:certbot.auth_handler:Cleaning up challenges
2017-05-15 03:58:04,303:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/iotdata.XXX.conf produced an unexpected error: Error while running apachectl graceful.

Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.
. Skipping.
2017-05-15 03:58:04,304:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 418, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 650, in renew_cert
    _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 87, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 296, in renew_cert
    new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 265, in obtain_certificate
    self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 73, in get_authorizations
    resp = self._solve_challenges()
  File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 111, in _solve_challenges
    resp = self.auth.perform(self.achalls)
  File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1748, in perform
    self.restart()
  File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1658, in restart
    self._reload()
  File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1669, in _reload
    raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apachectl graceful.

Job for httpd.service failed because the control process exited with error code. See "systemctl status httpd.service" and "journalctl -xe" for details.


2017-05-15 03:58:04,307:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 9, in <module>
    load_entry_point('certbot==0.12.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 896, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 702, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 435, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
Error: 1 renew failure(s), 0 parse failure(s)
15

Any chance you’re running SELinux in “enforcing” mode? This would explain why the temporary certificate cannot be read from /var/lib/letsencrypt.

You could switch to “permissive” mode temporarily to verify this (which only logs actions not allowed by your security policy instead of blocking them), and/or use chcon to fix the permissions permanently afterwards (I don’t recall what the full syntax is, sorry!).

16

Yes, SELinux is running in the enforcing mode.

The certificates are fresh now. Should I add “–force” argument to the cron job definition and try again in the permissive mode? Or there is another way to test it?

Thanks.

17

You should be able to use the --dry-run flag in your cronjob to request a certificate from the staging environment without actually storing it on disk. This would avoid rate-limiting issues further down the line if you have to repeat your tests a couple of times.

If it does indeed work in permissive mode, some chcon magic should be able to fix the policy. The SELinux audit logs might contain more details on why access is blocked (or would be in enforcing mode.

18

So I can confirm that this issue is related to SELinux. Renew via cron with --dry-run does work in permissive mode, but it doesn’t work in enforcing mode.

Here is the audit.log produced under permissive mode:

type=USER_ACCT msg=audit(1495093081.524:99884): pid=23891 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:accounting grantors=pam_access,pam_unix,pam_localuser acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_ACQ msg=audit(1495093081.524:99885): pid=23891 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=LOGIN msg=audit(1495093081.525:99886): pid=23891 uid=0 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 old-auid=4294967295 auid=0 old-ses=4294967295 ses=2111 res=1
type=USER_START msg=audit(1495093081.551:99887): pid=23891 uid=0 auid=0 ses=2111 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=CRED_REFR msg=audit(1495093081.552:99888): pid=23891 uid=0 auid=0 ses=2111 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=AVC msg=audit(1495093084.432:99889): avc:  denied  { getattr } for  pid=23923 comm="httpd" path="/var/lib/letsencrypt/b6_hE70f-xDdtcN311G_MlAJhwu_EdiZCzZkXHqf6Iw.crt" dev="md125" ino=548675949 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cron_var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1495093084.432:99889): arch=c000003e syscall=4 success=yes exit=0 a0=7f258458fba8 a1=7fff0488f260 a2=7fff0488f260 a3=43 items=0 ppid=1 pid=23923 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1495093084.482:99890): avc:  denied  { read } for  pid=1109 comm="httpd" name="pmTkEYXqroPmRyvjdR7b-_dpMIC_pjq8ADdIThPpgm8.crt" dev="md125" ino=548675952 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cron_var_lib_t:s0 tclass=file
type=AVC msg=audit(1495093084.482:99890): avc:  denied  { open } for  pid=1109 comm="httpd" path="/var/lib/letsencrypt/pmTkEYXqroPmRyvjdR7b-_dpMIC_pjq8ADdIThPpgm8.crt" dev="md125" ino=548675952 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:cron_var_lib_t:s0 tclass=file
type=SYSCALL msg=audit(1495093084.482:99890): arch=c000003e syscall=2 success=yes exit=12 a0=7ffeed32ce50 a1=80000 a2=0 a3=7efd55666381 items=0 ppid=1 pid=1109 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="httpd" exe="/usr/sbin/httpd" subj=system_u:system_r:httpd_t:s0 key=(null)

These two messages came a bit later, I’m not sure if they are directly related:

type=CRED_DISP msg=audit(1495093096.053:99891): pid=23891 uid=0 auid=0 ses=2111 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:setcred grantors=pam_env,pam_unix acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'
type=USER_END msg=audit(1495093096.059:99892): pid=23891 uid=0 auid=0 ses=2111 subj=system_u:system_r:crond_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct="root" exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success'

What next? I can fix it by myself via audit2allow, but my SELinux knowledge ends here… Is there anybody to create proper upstream patch?

19

If you’re on CentOS 7, we recommend you use the package from EPEL. Instructions on how to do this can be found here. Since this package is created and maintain for your specific system, it’s much more likely to make Certbot work with the SELinux configuration on your system rather than the system independent certbot-auto.

1 Like
20

I have this exact same problem with certbot as @stoupa-cz.

It fails with the same error messages in cron, but if I run it manually it works.
We are both using the certbot from EPEL, so it’s problem with the packaged version.

So the problem is caused by SELinux? I don’t really want to turn it off in our server, so is there some workaround for it? Could I touch the files certbot tries to use as sertificate and cchon them or something?