Renewal Failure Centos 7

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:


I ran this command:
sudo certbot
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org

Which names would you like to activate HTTPS for?

1:
fc64dc65e7001a1d24901ffc11200868.47b29b96e4a5841208f9f5d0e9fd7fcd.acme.invalid
2:
0dbc4ccbe966bdf4957cc1c46f754de7.68fb6b3e34586bdea1eb17cb03c60188.acme.invalid
3:
2a4c9c2e4401ccb9072487defd87fd73.7603522b42ef65515681332d9db90492.acme.invalid
4:
028cc4de8c46dfa152872df651e793cd.aad6b19e04e067406fd0dc6fda7dc8c1.acme.invalid
5:
608211e0eab9d7e025822db4121e508b.be748fcc7356c65098c7f1ff00d4124b.acme.invalid
6:
82275afffa4ea33ddb10fa4a353a95ea.e83ba77afe037792d8c30f19803320b4.acme.invalid
7: defluris.com
8: www.defluris.com
9: monasteryfruitcake.org
10: www.monasteryfruitcake.org
11: rosecitychocolates.com
12: www.rosecitychocolates.com

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter ‘c’ to cancel): 7 8 9 10 11 12
Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for defluris.com
tls-sni-01 challenge for www.defluris.com
tls-sni-01 challenge for monasteryfruitcake.org
tls-sni-01 challenge for www.monasteryfruitcake.org
tls-sni-01 challenge for rosecitychocolates.com
tls-sni-01 challenge for www.rosecitychocolates.com
Cleaning up challenges
Error while running apachectl configtest.

AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00526: Syntax error on line 55 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
SSLCertificateFile: file ‘/var/lib/letsencrypt/nxwfT253XBsnUBB2N0RPU9rTe-AyW6WmMBE-J4ZDFvw.crt’ does not exist or is empty

Encountered exception during recovery
Error while running apachectl configtest.

AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00526: Syntax error on line 55 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
SSLCertificateFile: file ‘/var/lib/letsencrypt/nxwfT253XBsnUBB2N0RPU9rTe-AyW6WmMBE-J4ZDFvw.crt’ does not exist or is empty
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/certbot/error_handler.py”, line 100, in _call_registered
self.funcs-1
File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 284, in _cleanup_challenges
self.auth.cleanup(achalls)
File “/usr/lib/python2.7/site-packages/certbot_apache/configurator.py”, line 1945, in cleanup
self.restart()
File “/usr/lib/python2.7/site-packages/certbot_apache/configurator.py”, line 1833, in restart
self.config_test()
File “/usr/lib/python2.7/site-packages/certbot_apache/configurator.py”, line 1856, in config_test
raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apachectl configtest.

AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00526: Syntax error on line 55 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
SSLCertificateFile: file ‘/var/lib/letsencrypt/nxwfT253XBsnUBB2N0RPU9rTe-AyW6WmMBE-J4ZDFvw.crt’ does not exist or is empty

An unexpected error occurred:
OSError: [Errno 17] File exists: '/var/lib/letsencrypt/nxwfT253XBsnUBB2N0RPU9rTe-AyW6WmMBE-J4ZDFvw.pem’
Please see the logfiles in /var/log/letsencrypt for more details.

My web server is (include version):
CentOS Linux release 7.4.1708
The operating system my web server runs on is (include version):
Apache/2.4.6 (CentOS)
My hosting provider, if applicable, is:
Linode
I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
No

2

Parts of this problem have been solved; however I am still facing this response from the command ‘sudo certbot renew’

Processing /etc/letsencrypt/renewal/defluris.com.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for defluris.com
tls-sni-01 challenge for monasteryfruitcake.org
tls-sni-01 challenge for rosecitychocolates.com
tls-sni-01 challenge for www.defluris.com
tls-sni-01 challenge for www.monasteryfruitcake.org
tls-sni-01 challenge for www.rosecitychocolates.com
Cleaning up challenges
Unable to recover files from /var/lib/letsencrypt/temp_checkpoint
Incomplete or failed recovery for /var/lib/letsencrypt/temp_checkpoint
Encountered exception during recovery
Unable to revert temporary config
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/certbot/error_handler.py”, line 100, in _call_registered
self.funcs-1
File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 284, in _cleanup_challenges
self.auth.cleanup(achalls)
File “/usr/lib/python2.7/site-packages/certbot_apache/configurator.py”, line 1944, in cleanup
self.revert_challenge_config()
File “/usr/lib/python2.7/site-packages/certbot_apache/augeas_configurator.py”, line 194, in revert_challenge_config
self.revert_temporary_config()
File “/usr/lib/python2.7/site-packages/certbot/plugins/common.py”, line 172, in revert_temporary_config
raise errors.PluginError(str(err))
PluginError: Unable to revert temporary config
Attempting to renew cert (defluris.com) from /etc/letsencrypt/renewal/defluris.com.conf produced an unexpected error: Unable to save to file!. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/defluris.com/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/defluris.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

3

@bmw, could you take a look at this one too?

@charlie_0037, which version of Certbot are you running?

4

Also, @joohoi, do you think this could be related somehow to the Augeas problem that another user was having that you just looked at? I don’t think it’s the same circumstances but I think it’s the same Augeas error message.

5

I am running certbot 0.19.0 on centos 7. All packages are updated.

Running apachectl configtest results in:

AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00526: Syntax error on line 44 of /etc/httpd/sites-enabled/defluris.com.conf:
SSLCertificateKeyFile: file ‘/etc/letsencrypt/live/defluris.com/privkey.pem’ does not exist or is empty

The file /etc/letsencrypt/live/defluris.com/privkey.pem exists. I can view it with

cat /etc/letsencrypt/live/defluris.com/privkey.pem http://defluris.com/privkey.pem

but this command fails:

sudo openssl x509 -text -noout -in /etc/letsencrypt/live/defluris.com/privkey.pem http://defluris.com/privkey.pem
140532307412896:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:707:Expecting: TRUSTED CERTIFICATE

This certificate successfully renewed 60 days ago.

6

@charlie_0037 looks like the file is empty.

What comes to mind is SELinux policy denying write access. Could you run certbot renew again while keeping an eye on /var/log/audit/audit.log ?

And / or try the following. Note that this will disable SELinux policy enforcing temporarily:

setenforce 0
certbot renew
setenforce 1

7

Thank-you for your response.

SELinux is disabled on this server. /var/log/audit.log outputs the following during a certbot renew failure:

type=CRED_DISP msg=audit(1510841401.264:43921): pid=25572 uid=0 auid=0 ses=855 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=“root” exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success’
type=USER_END msg=audit(1510841401.271:43922): pid=25572 uid=0 auid=0 ses=855 msg='op=PAM:session_close grantors=pam_loginuid,pam_keyinit,pam_limits,pam_systemd acct=“root” exe="/usr/sbin/crond" hostname=? addr=? terminal=cron res=success’
type=USER_CMD msg=audit(1510841450.091:43923): pid=25600 uid=0 auid=1000 ses=847 msg='cwd="/etc/letsencrypt/archive/defluris.com" cmd=7461696C202D66202F7661722F6C6F672F61756469742F61756469742E6C6F67 terminal=pts/0 res=success’
type=CRED_REFR msg=audit(1510841450.091:43924): pid=25600 uid=0 auid=1000 ses=847 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success’
type=USER_START msg=audit(1510841450.091:43925): pid=25600 uid=0 auid=1000 ses=847 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success’
type=USER_END msg=audit(1510841475.891:43926): pid=25600 uid=0 auid=1000 ses=847 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success’
type=CRED_DISP msg=audit(1510841475.891:43927): pid=25600 uid=0 auid=1000 ses=847 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success’
type=USER_CMD msg=audit(1510841482.631:43928): pid=25621 uid=0 auid=1000 ses=847 msg='cwd="/etc/letsencrypt/archive/defluris.com" cmd=7461696C202D66202F7661722F6C6F672F61756469742F61756469742E6C6F67 terminal=pts/0 res=success’
type=CRED_REFR msg=audit(1510841482.631:43929): pid=25621 uid=0 auid=1000 ses=847 msg='op=PAM:setcred grantors=pam_env,pam_unix acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success’
type=USER_START msg=audit(1510841482.634:43930): pid=25621 uid=0 auid=1000 ses=847 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits acct=“root” exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success’
type=CRYPTO_KEY_USER msg=audit(1510841498.271:43931): pid=25630 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:f5:f1:e3:2d:b8:97:35:b9:ed:36:51:eb:82:ab:e2:a3:cb:20:e7:ff:b8:cf:33:09:4d:0e:36:66:9f:5b:67:a8 direction=? spid=25630 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’
type=CRYPTO_KEY_USER msg=audit(1510841498.271:43932): pid=25630 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:a8:f4:1e:16:58:5e:dd:5a:25:7b:2a:68:49:85:f8:82:52:bc:12:3f:82:79:08:ef:20:bc:2e:76:c2:08:14:d5 direction=? spid=25630 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’
type=CRYPTO_KEY_USER msg=audit(1510841498.271:43933): pid=25630 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:59:d1:49:6a:50:30:ec:60:a0:98:75:82:aa:da:24:31:3e:4e:f4:6a:ef:10:3c:6d:6e:85:f3:e6:75:e1:c2:a7 direction=? spid=25630 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’
type=CRYPTO_SESSION msg=audit(1510841498.488:43934): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg='op=start direction=from-server cipher=aes128-ctr ksize=128 mac=hmac-sha2-256 pfs=diffie-hellman-group-exchange-sha256 spid=25630 suid=74 rport=59719 laddr=45.33.78.85 lport=22 exe="/usr/sbin/sshd" hostname=? addr=169.255.5.138 terminal=? res=success’
type=CRYPTO_SESSION msg=audit(1510841498.488:43935): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg='op=start direction=from-client cipher=aes128-ctr ksize=128 mac=hmac-sha2-256 pfs=diffie-hellman-group-exchange-sha256 spid=25630 suid=74 rport=59719 laddr=45.33.78.85 lport=22 exe="/usr/sbin/sshd" hostname=? addr=169.255.5.138 terminal=? res=success’
type=CRYPTO_KEY_USER msg=audit(1510841510.701:43936): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:59:d1:49:6a:50:30:ec:60:a0:98:75:82:aa:da:24:31:3e:4e:f4:6a:ef:10:3c:6d:6e:85:f3:e6:75:e1:c2:a7 direction=? spid=25630 suid=74 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’
type=CRYPTO_KEY_USER msg=audit(1510841510.701:43937): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=session fp=? direction=both spid=25630 suid=74 rport=59719 laddr=45.33.78.85 lport=22 exe="/usr/sbin/sshd" hostname=? addr=169.255.5.138 terminal=? res=success’
type=USER_ERR msg=audit(1510841510.701:43938): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg='op=PAM:bad_ident grantors=? acct="?" exe="/usr/sbin/sshd" hostname=169.255.5.138 addr=169.255.5.138 terminal=ssh res=failed’
type=CRYPTO_KEY_USER msg=audit(1510841510.704:43939): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:f5:f1:e3:2d:b8:97:35:b9:ed:36:51:eb:82:ab:e2:a3:cb:20:e7:ff:b8:cf:33:09:4d:0e:36:66:9f:5b:67:a8 direction=? spid=25629 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’
type=CRYPTO_KEY_USER msg=audit(1510841510.704:43940): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:a8:f4:1e:16:58:5e:dd:5a:25:7b:2a:68:49:85:f8:82:52:bc:12:3f:82:79:08:ef:20:bc:2e:76:c2:08:14:d5 direction=? spid=25629 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’
type=CRYPTO_KEY_USER msg=audit(1510841510.704:43941): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg='op=destroy kind=server fp=SHA256:59:d1:49:6a:50:30:ec:60:a0:98:75:82:aa:da:24:31:3e:4e:f4:6a:ef:10:3c:6d:6e:85:f3:e6:75:e1:c2:a7 direction=? spid=25629 suid=0 exe="/usr/sbin/sshd" hostname=? addr=? terminal=? res=success’
type=USER_LOGIN msg=audit(1510841510.704:43942): pid=25629 uid=0 auid=4294967295 ses=4294967295 msg=‘op=login acct="(unknown)" exe="/usr/sbin/sshd" hostname=? addr=169.255.5.138 terminal=ssh res=failed’

As an aside, the command apachectl configtest when run as root does not complain about syntax as in:

AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
Syntax OK

When run as the user, it finds a syntax error in the host file:

AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00526: Syntax error on line 44 of /etc/httpd/sites-enabled/defluris.com.conf:
SSLCertificateKeyFile: file ‘/etc/letsencrypt/live/defluris.com/privkey.pem’ does not exist or is empty

This may not mean anything. I have checked file permissions and owners for the letsencrypt installation and all seems correct.

I appreciate the help

8

The SSLCertificateKeyFile error is to be expected when running as unprivileged user, as the permissions to the directory containing the certificate files restrict access for non-root users.

Any chance that there’s something weird going on with your /var partition, or /var/lib/letsencrypt directory in particular?

9

Users and permissions seem correct for /var, /var/lib, and /var/lib/letsencrypt

drwxr-xr-x. 20 root root 4096 Jan 17 2017 var

drwxr-xr-x. 31 root root 4096 Nov 15 10:56 lib

drwxr-xr-x 4 root root 4096 Nov 16 09:16 letsencrypt

/var/lib/letsencrypt

drwxr-xr-x 4 root root 4096 Nov 16 09:16 .
drwxr-xr-x. 31 root root 4096 Nov 15 10:56 …
drwxr-xr-x 2 root root 4096 Nov 15 07:42 backups
-rw-r–r-- 1 root root 1147 Nov 15 11:08 _lqxWpvqq2aRRJGvb5fw8f6MTKtKjvaOQG63G1CdZT8.crt
-r-------- 1 root root 1704 Nov 15 11:08 _lqxWpvqq2aRRJGvb5fw8f6MTKtKjvaOQG63G1CdZT8.pem
-rw-r–r-- 1 root root 1147 Nov 15 11:08 QMiFyo1FQGJNQtUr1CVEiKLksSWRWF0MOYTdi4XBYQ4.crt
-r-------- 1 root root 1708 Nov 15 11:08 QMiFyo1FQGJNQtUr1CVEiKLksSWRWF0MOYTdi4XBYQ4.pem
-rw-r–r-- 1 root root 1147 Nov 15 11:08 spv3X8QfNaFl_73xCG55FPVCmykQnfPsSPBNIWHdoho.crt
-r-------- 1 root root 1704 Nov 15 11:08 spv3X8QfNaFl_73xCG55FPVCmykQnfPsSPBNIWHdoho.pem
-rw-r–r-- 1 root root 1147 Nov 15 11:08 sv2cP5DqvGlVJmgyw91JBfVMVDWgpHhLPnADdP03704.crt
-r-------- 1 root root 1704 Nov 15 11:08 sv2cP5DqvGlVJmgyw91JBfVMVDWgpHhLPnADdP03704.pem
drwxr-xr-x 2 root root 4096 Nov 15 11:08 temp_checkpoint
-rw-r–r-- 1 root root 1147 Nov 15 11:08 xV1bGGq4CWl869V6GGZToEOjKAXH20Glffes4tOE7as.crt
-r-------- 1 root root 1704 Nov 15 11:08 xV1bGGq4CWl869V6GGZToEOjKAXH20Glffes4tOE7as.pem
-rw-r–r-- 1 root root 1147 Nov 15 11:08 zjhpMNyXvkjp14pCN87KIKPcajcrHVSOgSHLEPSsoJA.crt
-r-------- 1 root root 1704 Nov 15 11:08 zjhpMNyXvkjp14pCN87KIKPcajcrHVSOgSHLEPSsoJA.pem

What does the error message 'Unable to recover files from /var/lib/letsencrypt/temp_checkpoint’
mean?

10

What does the error message 'Unable to recover files from /var/lib/letsencrypt/temp_checkpoint’
mean?

Certbot is unable to read the temp_checkpoint for reason or another.

I tried to reproduce this on the exactly same version of CentOS and Apache, but was unable to do so. If you wish to continue debugging the problem, I would suggest adding an auditctl watch on the /var/lib/letsencrypt and /etc/letsencrypt directories to learn what's going on in the filesystem.

To do so, issue the following commands:

sudo auditctl -w /var/lib/letsencrypt -p rwxa -k certbotlib
sudo auditctl -w /etc/letsencrypt -p rwxa -k certbotetc

This will cause auditd to record all file accesses under these directories. After issuing the above commands, please run certbot again. After you are done, retrieve the logs by issuing:

sudo ausearch -k certbotlib
sudo ausearch -k certbotetc

These will produce a lot of log lines, so please either pastebin / gist the lines alongside of Certbot logs for the last run or email the logs to me: joona.hoikkala@eff.org . To remove the watch rules afterwards, issue:

sudo auditctl -W /var/lib/letsencrypt -p rwxa -k certbotlib
sudo auditctl -W /etc/letsencrypt -p rwxa -k certbotetc

note the uppercase W.

11

Ok, I was able to reproduce the problem! What I think is the issue here is /var/lib/letsencrypt/temp_checkpoint/FILEPATHS file being left behind from abnormal exit of Certbot. On the startup, Certbot tries to recover that one, but fails.

What I believe will fix your issue, is removing directory
/var/lib/letsencrypt/temp_checkpoint

and running Certbot again. I will create an issue to our GitHub issuetracker about this.

Something to note if people are reading this thread later on: this is merely the solution for issue in later parts of the thread. The original problem was something else that was fixed by @charlie_0037 .

12

With this command, I removed the directory:

sudo rm -r /var/lib/letsencrypt/temp_checkpoint

Then:

[charlie@magpie ~]$ sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

13

So certbot halts completely? This is a separate issue, which most likely has caused the process to get killed in the first place, resulting into the situation with temp_checkout. Try running Certbot with -vvv parameter to get more verbose output, in order to find out what’s it could be stuck at.

14

My continued thanks for working on this. Perhaps we are getting to the heart of the problem. Here is the output of cerbot renew -vvv:

[charlie@magpie ~]$ sudo certbot renew -vvv
[sudo] password for charlie:
Root logging level set at -10
Saving debug log to /var/log/letsencrypt/letsencrypt.log

15

What about the contents of /var/log/letsencrypt/letsencrypt.log after a halting run?

16

2017-11-19 14:17:04,179:DEBUG:certbot.main:certbot version: 0.19.0
2017-11-19 14:17:04,179:DEBUG:certbot.main:Arguments: []
2017-11-19 14:17:04,179:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2017-11-19 14:17:04,233:DEBUG:certbot.log:Root logging level set at 20
2017-11-19 14:17:04,233:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-11-19 14:17:04,235:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2017-11-19 14:17:04,423:ERROR:certbot.reverter:Unable to recover files from /var/lib/letsencrypt/temp_checkpoint
2017-11-19 14:17:04,424:CRITICAL:certbot.reverter:Incomplete or failed recovery for /var/lib/letsencrypt/temp_checkpoint
2017-11-19 14:17:04,424:DEBUG:certbot.plugins.disco:Other error:(PluginEntryPoint#apache): Unable to revert temporary config
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/certbot/plugins/disco.py”, line 130, in prepare
self._initialized.prepare()
File “/usr/lib/python2.7/site-packages/certbot_apache/configurator.py”, line 164, in prepare
self.init_augeas()
File “/usr/lib/python2.7/site-packages/certbot_apache/augeas_configurator.py”, line 51, in init_augeas
self.recovery_routine()
File “/usr/lib/python2.7/site-packages/certbot_apache/augeas_configurator.py”, line 184, in recovery_routine
super(AugeasConfigurator, self).recovery_routine()
File “/usr/lib/python2.7/site-packages/certbot/plugins/common.py”, line 161, in recovery_routine
raise errors.PluginError(str(err))
PluginError: Unable to revert temporary config
2017-11-19 14:17:04,426:DEBUG:certbot.plugins.selection:Multiple candidate plugins: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x1b17c10>
Prep: True

  • webroot
    Description: Place files in webroot directory
    Interfaces: IAuthenticator, IPlugin
    Entry point: webroot = certbot.plugins.webroot:Authenticator
    Initialized: <certbot.plugins.webroot.Authenticator object at 0x1b179d0>
    Prep: True
    2017-11-19 14:17:17,944:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
    2017-11-19 14:17:17,944:INFO:certbot.main:Could not choose appropriate plugin: authenticator could not be determined or is not installed
    2017-11-19 14:17:17,945:DEBUG:certbot.log:Exiting abnormally:
    Traceback (most recent call last):
    File “/bin/certbot”, line 9, in
    load_entry_point(‘certbot==0.19.0’, ‘console_scripts’, ‘certbot’)()
    File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 861, in main
    return config.func(config, plugins)
    File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 765, in certonly
    installer, auth = plug_sel.choose_configurator_plugins(config, plugins, “certonly”)
    File “/usr/lib/python2.7/site-packages/certbot/plugins/selection.py”, line 201, in choose_configurator_plugins
    diagnose_configurator_problem(“authenticator”, req_auth, plugins)
    File “/usr/lib/python2.7/site-packages/certbot/plugins/selection.py”, line 297, in diagnose_configurator_problem
    raise errors.PluginSelectionError(msg)
    PluginSelectionError: authenticator could not be determined or is not installed
17

This log dump seems to be from a run where there is a broken /var/lib/letsencrypt/temp_checkpoint/FILEPATHS file present. Could you do one more run:

  1. rm -rf /var/lib/letsencrypt/temp_checkpoint
  2. run certbot renew

and paste the /var/log/letsencrypt/letsencrypt.log contents here.

18

Joona,

I am sending the letsencrypt log as an attachment. I set it in the body of the email, but I exceeded the 30000 character limit on emails to lestencrypt. Sorry for the delay

letsencrypt.txt (36.1 KB)

19

Ok, so we’re still seeing problems related to the state left behind by abnormal exit of the Certbot. I’ll add this to the issue.

Now, the next run I would like to see both, Certbot output and /var/log/letsencrypt/letsencrypt.log contents would be:

  1. rm -rf /var/lib/letsencrypt/temp_checkpoint
  2. rm -rf /var/lib/letsencrypt/*.pem
  3. rm -rf /var/lib/letsencrypt/*.crt
  4. Run certbot

I suspect you will hit the original freeze state after that, but that’s the log I’m intrested in-

20

After deleting the files, certbot returned this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log