Renew certificate on read-only fs

Help
1

My domain is: acloud.acawa.be

I ran this command: certbot certificates

It produced this output:

The following error was encountered:
[Errno 30] Read-only file system: '/etc/letsencrypt/.certbot.lock'
Either run as root, or set --config-dir, --work-dir, and --logs-dir to writeable paths.

My web server is (include version): apache2

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is: self hosted

I can login to a root shell on my machine (yes or no, or I don't know): opensuse

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): /

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.9.0

The problem lies on my OS, which I am only able to boot as read-only since a kernel update. I still have to fix this but now the system is running as read-only (only way to boot it for now). That means the /etc directory is not writable for the moment so certbot cannot update my certificates.

Maybe some good news is that the /srv directory is mounted as read-write. This is where my webserver files are hosted. So my Nextcloud installation is running on the Apache webserver.

Now my question is: can I change the --config-dir, --work-dir, and --logs-dir to writeable paths? Like some path in the /srv mount, which is read-write...

Can I do that without a system reboot?
And what are the recommended steps?

Thanks a lot in advance,
kleajmp

2

If you give all those options, yes: obviously it won't able to save at /etc/letsencrypt/live because you know: as you felt on those options its not first class support

6 Likes
3

I'm aware it's not a first class solution, but the certificate needs to be renewed temporarily until i find time to fix the OS (which I am not looking forward to)

So just before I start my journey on this...
What would be the complete steps to complete the temporary solution?

  • set the config-dir to /srv/certbot ?
  • set the work-dir to /srv/certbot ?
  • run the renew script ?
  • change the paths of the .pem certificates in the apache config? (is that necessary?)
4

Yes, Apache will need to use the path to the new cert. It is likely using the path to your certs in /etc/letsencrypt/live/... You can't update those certs but how would you update Apache config to point anywhere else? Isn't its config also in /etc ?

6 Likes
5

Yes it is :frowning_face:
I think I'm screwed.
No other way I guess?

6

try overlay mount over /etc with /srv/etc upper and /etc lower, and /srv/work as workdir? Overlay filesystem - ArchWiki

5 Likes
7

If you can read it, you can back it out.
Then you can start over and put it all back in :slight_smile:

5 Likes
8

Did you run that command or, hopefully, sudo certbot certificates?
Or were you already root?

2 Likes
9

In theory you could copy all the content in /etc/ (while using all the permissions and owners et c., such as with rsync -a) to e.g. /srv/etc/ as mentioned earlier and then use mount -o bind /srv/etc/ /etc/ to mount that other path to /etc/. Not sure if /etc/ needs to be completely empty or that you can mount the other path "over" it, leaving the original content intact. I think the latter, but I'm not sure.

4 Likes
10

yes off course I did that as root :slight_smile:

3 Likes
11

Thank you for this possible way of getting out of my trouble, looks like a promising way out...
I will try this one out!

2 Likes
12

I suppose you mean with "overlay mount" what user Osiris described in more detail? If so I will try to do that, looks like a possible temporarily workaround to the certificate problem...

I'm really scared when I try to reboot the server I will be in an unpredictable long "server down" phase figuring out how to lead grub2 to boot the still working kernel...

I still have no idea why the system only wants to boot in ro mode for now. I also have some btrfs snapshots which I can try to restore... But I'm new to this and I don't want to screw the whole system over, it took me weeks to set this all up last year. It worked perfecty untill the last kernel upgrade.

13

You would be surprised at the number of times someone has forgot. :slight_smile:

2 Likes
14

The reason Btrfs gone read only is it count corruption on your disk io, stop everything now and backup to new disk:

4 Likes
15

well I figured out now I'm booted from a ro snapshot (btrfs), "526"

BOOT_IMAGE=/boot/vmlinuz-6.7.4-1-default root=UUID=20826742-1681-40c3-b76a-1393db22e8c0 rootflags=subvol=/@/.snapshots/526/snapshot splash=silent systemd.show_status=yes quiet mitigations=auto security=apparmor

as this is working well I should be able to restore this snapshot I guess...
but again first time ever I do this and an important server is at stake...
so I ask any experienced users their advice: Can I just do this? Or should I make any more backups first?

16

true :slight_smile:

1 Like
17

if it's really important you'd better backup it regually : raid isn't backup, nither snapshot by itself

4 Likes
18

thankfully I had both snapshots (snapper btrfs) and a full data backup (timeshift ext4).

19

Well good news!
I figured out the booted ro-system was a snapper btrfs snapshot.
I just had to test it well and after restore it with the simple command:
snapper rollback

after I did a manual certbot renew:
certbot certonly --manual -d cloud.domain.com

And it was fixed!
Thanks to all your tips I did a good investigation before I started with this risky project.

3 Likes