When renewing: "an unexpected error: Unable to save to file!. Skipping."

[olbender]

Hi,

I have a Dockerized server that uses some Let's Encrypt certificates. Originally I had problems getting the Apache authenticator to work properly and in the end the only thing that did work was a manual DNS authentication.

Now it's time to renew the certificates and I got into problems again (I really would like the automated thing to work).

On all renewals I get:

unexpected error: Unable to save to file!. Skipping.

The log output is:

2017-11-07 18:25:31,216:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/certbot/renewal.py", line 425, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python3.6/site-packages/certbot/main.py", line 743, in renew_cert
_get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python3.6/site-packages/certbot/main.py", line 80, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3.6/site-packages/certbot/renewal.py", line 297, in renew_cert
new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File "/usr/lib/python3.6/site-packages/certbot/client.py", line 318, in obtain_certificate
self.config.allow_subset_of_names)
File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 74, in get_authorizations
resp = self._solve_challenges()
File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 115, in _solve_challenges
resp = self.auth.perform(self.achalls)
File "/usr/lib/python3.6/site-packages/certbot_apache/configurator.py", line 1919, in perform
sni_response = chall_doer.perform()
File "/usr/lib/python3.6/site-packages/certbot_apache/tls_sni_01.py", line 79, in perform
self.configurator.save("Don't lose mod_config changes", True)
File "/usr/lib/python3.6/site-packages/certbot_apache/augeas_configurator.py", line 150, in save
self.aug.save()
File "/usr/lib/python3.6/site-packages/augeas/init.py", line 482, in save
raise IOError("Unable to save to file!")
OSError: Unable to save to file!

The output here is note very informative. It would be extremely helpful to know what file is actually the problem. I tried to hack away in the python code to try the get a path out, but it was not really possible since the augeas lib is a wrapper for a C lib, and I did not really have the energy to start recompiling that to get some reasonable debug messages out.

I also tried to give the root user all write permissions to the /etc/httpd and /etc/letsencrypt folders, but that didn't help. Perhaps a missing folder somewhere?

Please help me!

EDIT: I want to use the tls challenge with Apache.

[schoen]

@bmw, could you take a look at this issue?

[olbender]

Thank you, that would be most welcome.

I could also add that the files /etc/httpd/conf/httpd.conf.xyz are created as expected.

Edit: On more thing, I also hacked getpass.getuser() into the Python script to make sure it was run as root, and it was.

[bmw]

Sorry for the trouble! You’re correct that the library we use for parsing and modifying Apache configuration files doesn’t give us any more information here.

With that said, since you seem feel comfortable making changes to Certbot’s code, you could check the value of save_files after this line. This should be set of all the files in your Apache configuration that Certbot is trying to modify.

Also, I’m curious about your setup here. Are you running Apache and Certbot in the same Docker container?

[olbender]

Thank you for the advice. I did this and it says:

{'/etc/httpd/conf/httpd.conf'}

This is however very strange since this file has the following permissions

-rw-rw-rw- 1 root root 19993 Nov 7 17:27 /etc/httpd/conf/httpd.conf

I can also edit and save the file from within the container using an editor.

Apache and certbot are in the same container, and the config file is mounter as a volume from the host. The container is started with docker-compose with the following:

version: '3.3'

services:
web:
image: server/server-web:latest
ports:
- "80:80"
- "443:443"
volumes:
- ./server_data/etc/httpd/conf/httpd.conf:/etc/httpd/conf/httpd.conf
- ./server_data/etc/httpd/conf/extra/httpd-vhosts.conf:/etc/httpd/conf/extra/httpd-vhosts.conf
- ./server_data/etc/letsencrypt:/etc/letsencrypt
- ./server_data/etc/php/php.ini:/etc/php/php.ini
- ./server_data/etc/webapps:/etc/webapps
- ./server_data/srv/http:/srv/http
- ./server_data/var/log/httpd:/var/log/httpd
- ./server_data/run/mysqld:/run/mysqld
command: "/usr/bin/httpd -DFOREGROUND -ewarn"

Strange...

[bmw]

Strange indeed. @joohoi, do you have any ideas?

[joohoi]

Looks really weird indeed. Unfortunately the actual IOError thrown does not come from python stdlib, but is instead raised by python-augeas in a case when its call to C-library returns a non-zero value so we have really limited visibility on the actual error.

That said I’m not very sure if the Certbot-Apache will be able to handle the renewal in your usecase from inside the container, because it does calls on the apache2ctl (or equivalent on your distribution) to reload the httpd to refresh the configuration. Your container seems to be running httpd directly on the foreground (and most likely not using the apache2ctl specific pid file locations and so forth). It will be unable to restart your httpd in any case, unofortunately. These two could be related, because when Certbot reaches the point where the error you are experiencing is thrown, it has already called apache2ctl.

[olbender]

Aha, that might perhaps be related to the problem then. I could try to change how I start Apache in the container. The overall problem is of course ‘how to make cerbot and apache work inside a container in the best way’, and the solution to this might perhaps be of general interest. I will try to change this and get back.

[joohoi]

On other thread there was an error similar to yours. I’m thinking that the actual error from file write could be related to SELinux policies. The thread and my answer is here: Renewal Failure Centos 7

You could try checking out the audit.log as well.

The main issue that you will be facing when running Apache and Certbot in a same container will be having some process that will actually block, that would still allow Apache restarts.

[olbender]

Yeah, but that I normally fix by having a

tail -f /some/log

in the end of a startup scripts.

Another more Docker-like solution would perhaps be to run certbot tasks in their own non-persistent containers and simply share the pid between containers. I can perhaps try that as well.

Have there been any previous discussions on how to best manage these certificates in an containerized environment?

[joohoi]

Yeah, tail does work!

What I have personally deployed have been:

1. Nginx reverse proxy in front of application containers, with autogenerated configuration and global directory override for .well-known/acme-challenge to a shared volume, and a second shared volume for /etc/letsencrypt. Separate Certbot container that hooks to both of these volumes, and then is able to write the challenge token files as well as the certs to /etc/letsencrypt/…

2. Shared /etc/letsencrypt volume with actual container needing the certificates and separate Certbot container doing DNS validation.

[olbender]

Sorry for the delay on this. I just tried to reconfigure my container, to start a httpd in the background, and I verified the change by restarting apache from a bash started with ‘docker exec’.

That worked fine, but the above problem still persists. Any other ideas?