Can't renew my certificates

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
minirobota.sk
I ran this command:
sudo certbot renew
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/minirobota.sk.conf

Could not parse file: /etc/letsencrypt/le_http_01_cert_challenge.conf due to Expected {Group:({[] "#" rest of line}) | Group:(Forward: {Group:({[] {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\$\{)|[^{;\s]')]...}) | Combine:({Re:('(\$\{)|[^{};\s'\"]') [Re:('(\$\{)|[^{;\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '} [{ {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\$\{)|[^{;\s]')]...}) | Combine:({Re:('(\$\{)|[^{};\s'\"]') [Re:('(\$\{)|[^{;\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '}}]... []}) Suppress:("{") Group:({[{{Group:({[] "#" rest of line}) | Group:(: ...)} | Group:({{{{[] {{Combine:({{{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")"} [Re:('(\$\{)|[^{;\s]')]...}) | Combine:({Re:('(\$\{)|[^{};\s) | Group:({[] {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\$\{)|[^{;\s]')]...}) | Combine:({Re:('(\$\{)|[^{};\s'\"]') [Re:('(\$\{)|[^{;\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '} [{ {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\$\{)|[^{;\s]')]...}) | Combine:({Re:('(\$\{)|[^{};\s'\"]') [Re:('(\$\{)|[^{;\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '}}]... [] Suppress:(";")})} (at char 0), (line:1, col:1)
Renewing an existing certificate for minirobota.sk

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: minirobota.sk
Type: connection
Detail: Fetching http://minirobota.sk/.well-known/acme-challenge/3jwFVi2g_tidf2HeXJad64Mi8ptHvbLUX3cdnZiF9ik: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Unable to recover files from /var/lib/letsencrypt/temp_checkpoint
Incomplete or failed recovery for /var/lib/letsencrypt/temp_checkpoint
Encountered exception during recovery: certbot.errors.PluginError: Unable to revert temporary config
Failed to renew certificate minirobota.sk with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/minirobota.sk/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
My web server is (include version):
nginx version: nginx/1.20.1
The operating system my web server runs on is (include version):
Ubuntu 20.04.2 LTS
My hosting provider, if applicable, is:
local
I can login to a root shell on my machine (yes or no, or I don't know):
yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.16.0

2 Likes
2

I'm also getting a timeout on your webserver on port 80: is port 80 open in your firewall? HTTPS seems to be responding. Please see:

Also, that error about "Could not parse file: /etc/letsencrypt/le_http_01_cert_challenge.conf" is very weird.. Perhaps some older configuration files weren't cleaned up properly? Could you share the output of sudo nginx -T please?

3 Likes
3

Hi @mincaeuro, and welcome to the LE community forum :slight_smile:

Also, please show the contents of file:

2 Likes
4

the port 80 was blocked on firewall, thats true, but it worked also without that access.
nginx -t result is OK:
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

and the files /etc/letsencrypt/le_http_01_cert_challenge.conf is empty - tried to remove it, but it was recreated and added to *.conf

here is the content of minirobota.sk.conf

# renew_before_expiry = 30 days
version = 1.16.0
archive_dir = /etc/letsencrypt/archive/minirobota.sk
cert = /etc/letsencrypt/live/minirobota.sk/cert.pem
privkey = /etc/letsencrypt/live/minirobota.sk/privkey.pem
chain = /etc/letsencrypt/live/minirobota.sk/chain.pem
fullchain = /etc/letsencrypt/live/minirobota.sk/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = f059ecfc42b56addabe15f68c502dd9c
authenticator = nginx
installer = nginx
server = https://acme-v02.api.letsencrypt.org/directory

UPDATE:
ok so the renew was now successful (probably the port issue).
But how to get rid of those errors ?

Processing /etc/letsencrypt/renewal/minirobota.sk.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Could not parse file: /etc/letsencrypt/le_http_01_cert_challenge.conf due to Expected {Group:({[<SP><TAB><CR><LF>] "#" rest of line}) | Group:(Forward: {Group:({[<SP><TAB><CR><LF>] {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\'\\"]') [Re:('(\\$\\{)|[^{;\\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '} [{<SP><TAB><CR><LF> {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\'\\"]') [Re:('(\\$\\{)|[^{;\\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '}}]... [<SP><TAB><CR><LF>]}) Suppress:("{") Group:({[{{Group:({[<SP><TAB><CR><LF>] "#" rest of line}) | Group:(: ...)} | Group:({{{{[<SP><TAB><CR><LF>] {{Combine:({{{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")"} [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\) | Group:({[<SP><TAB><CR><LF>] {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\'\\"]') [Re:('(\\$\\{)|[^{;\\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '} [{<SP><TAB><CR><LF> {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\'\\"]') [Re:('(\\$\\{)|[^{;\\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '}}]... [<SP><TAB><CR><LF>] Suppress:(";")})}  (at char 0), (line:1, col:1)
Could not parse file: /etc/letsencrypt/le_http_01_cert_challenge.conf due to Expected {Group:({[<SP><TAB><CR><LF>] "#" rest of line}) | Group:(Forward: {Group:({[<SP><TAB><CR><LF>] {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\'\\"]') [Re:('(\\$\\{)|[^{;\\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '} [{<SP><TAB><CR><LF> {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\'\\"]') [Re:('(\\$\\{)|[^{;\\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '}}]... [<SP><TAB><CR><LF>]}) Suppress:("{") Group:({[{{Group:({[<SP><TAB><CR><LF>] "#" rest of line}) | Group:(: ...)} | Group:({{{{[<SP><TAB><CR><LF>] {{Combine:({{{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")"} [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\) | Group:({[<SP><TAB><CR><LF>] {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\'\\"]') [Re:('(\\$\\{)|[^{;\\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '} [{<SP><TAB><CR><LF> {Combine:({{quoted string, starting with " ending with " | quoted string, starting with ' ending with '} ")" [Re:('(\\$\\{)|[^{;\\s]')]...}) | Combine:({Re:('(\\$\\{)|[^{};\\s\'\\"]') [Re:('(\\$\\{)|[^{;\\s]')]...}) | quoted string, starting with " ending with " | quoted string, starting with ' ending with '}}]... [<SP><TAB><CR><LF>] Suppress:(";")})}  (at char 0), (line:1, col:1)
Renewing an existing certificate for minirobota.sk and 2 more domains
Unable to recover files from /var/lib/letsencrypt/temp_checkpoint
Incomplete or failed recovery for /var/lib/letsencrypt/temp_checkpoint
Encountered exception during recovery: certbot.errors.PluginError: Unable to revert temporary config
Reloading nginx server after certificate renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all renewals succeeded:
  /etc/letsencrypt/live/minirobota.sk/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 Likes
5

@_az more possible parser fun?

2 Likes
6

I suspect that there may be some "unwanted" characters within the config file(s).
[probably just before where the le_http_01_cert_challenge file is being included]

2 Likes
7

its actually empty

and funny now its in the conf twice:

user www-data;
worker_processes 4;
pid /run/nginx.pid;

#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;

#pid logs/nginx.pid;

events {
worker_connections 1024;
}

http {
include /etc/letsencrypt/le_http_01_cert_challenge.conf;
include /etc/letsencrypt/le_http_01_cert_challenge.conf;

2 Likes
8

That seems to indicate that the add process is "working" while the removal process is not.
OR some other error is occurring that keeps adding that line on each renewal attempt.
[I can't imagine what your config file will look like a year from now - LOL]

If all else fails, can you try using --webroot authentication instead?

2 Likes
9

@_az

I concur with @schoen's concerns. I summarized for you below.

1 Like
10

When using the http-01 challenge, that could only have been the case due to cached valid authorizations: valid authz are cached for 30 days. Any new authorization using the http-01 starts at port 80 and blocking port 80 would fail the authz.

2 Likes
11

I concur, @Osiris. I'm baffled though why certbot would add the webserver exception if it's using a cached authorization since there's no need to do any challenge validation at that point.

2 Likes
12

Well, I think there's at least 2 different things going on here.

The nginx parsing error is due to us, in Certbot 1.16.0, changing the logging level of that error from debug to warning as part of auditing error output (line 100). What we failed to do is to consider that it would cause this pretty pointless error message to appear because the nginx parser can't handle empty files right now. So we'll have to fix that for next release, but it's relatively harmless and wouldn't break anything.

That's more worrying. I'm not sure what's happening, from reading this thread. If @mincaeuro can upload the /var/log/letsencrypt/ file associated with that error, that'd help.

Even if the authorization is cached, the nginx config is still parsed during the plugin preparation phase. That's why we see the error come up.

The nginx config doesn't get modified during non-dry-run, cached authz renewals (at least, as I check just now).

3 Likes
13

le-log.txt (1.0 MB)
sure uploading log

2 Likes
14

Thanks! From that log, we can see a system error which suggests server's filesystem might have been mounted in read-only mode:

2021-06-06 00:01:41,831:ERROR:certbot._internal.renewal:Failed to renew certificate minirobota.sk with error: Unable to revert temporary config
2021-06-06 00:01:41,854:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/snap/certbot/1200/lib/python3.8/site-packages/certbot/reverter.py", line 232, in _recover_checkpoint
    shutil.copy2(os.path.join(
  File "/snap/certbot/1200/usr/lib/python3.8/shutil.py", line 432, in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
  File "/snap/certbot/1200/usr/lib/python3.8/shutil.py", line 261, in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
OSError: [Errno 30] Read-only file system: '/etc/letsencrypt/options-ssl-nginx.conf'

That could both the file being empty and also the error about being unable to restore the checkpoint.

Certbot needs write access to a number of directories, including /etc/letsencrypt and /var/lib/letsencrypt.

Is /etc definitely mounted as a writable volume on your Ubuntu server?

3 Likes
15

the file is actually sym link to snap mount:

ls -l /etc/letsencrypt/options-ssl-nginx.conf
lrwxrwxrwx 1 root root 108 May 9 10:29 /etc/letsencrypt/options-ssl-nginx.conf -> /snap/certbot/current/lib/python3.8/site-packages/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf

2 Likes
16

Do you recall if that symlink was manually created? It should ordinarily be a normal file.

I can see how that would cause problems with the reverter, since the method it uses to write files, can attempt to write to the existing file (which is read-only inside the snap squashfs), rather than unlinking it and writing a new one.

If the symlink was created by Certbot, I think that is a mistake and we need to make sure it doesn't happen.

3 Likes
17

no, it was created automatically during the installation

2 Likes
18

To be honest, I don't know how that would be possible, but in case it happens to somebody else as well, I've created an issue to track it.

For now, replace the symlink with a copy of the real file:

sudo cp --remove-destination `readlink /etc/letsencrypt/options-ssl-nginx.conf` /etc/letsencrypt/options-ssl-nginx.conf
4 Likes
19

ok, thanks for the info, I've unlinked it and made a real copy of it for now

3 Likes
20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.