Certbot 3.3.0 fails to update certs

Help
1

My domain is: backyardbowl.ca (failing with multiple certifcates though).

I ran this command:

sudo certbot renew

It produced this output:

Failed to renew certificate backyardbowl.ca with error: list index out of range

My web server is (include version):

Ubuntu 20.04.6 LTS (GNU/Linux 5.4.0-204-generic x86_64) (yes I know it is old)

My hosting provider, if applicable, is:

Digital Ocean

I can login to a root shell on my machine (yes or no, or I don't know):

Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

3.3.0

Rerverting to 3.2.0 fixed the issue.

Error in the logs;

2025-03-27 13:14:52,929:DEBUG:acme.client:Storing nonce: 3u7v6Bm_lirtOMayx3q47JBl50Z4NebfCiJy0yD3EPE1buktLEo
2025-03-27 13:14:52,930:INFO:certbot._internal.auth_handler:Performing the following challenges:
2025-03-27 13:14:52,930:INFO:certbot._internal.auth_handler:http-01 challenge for backyardbowl.ca
2025-03-27 13:14:52,946:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot/_internal/auth_handler.py", line 88, in handle_authorizations
resps = self.auth.perform(achalls)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 1240, in perform
http_response = http_doer.perform()
^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/http_01.py", line 63, in perform
self._mod_config()
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/http_01.py", line 134, in _mod_config
config = [self._make_or_mod_server_block(achall) for achall in self.achalls]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/http_01.py", line 223, in _make_or_mod_server_block
http_vhosts, https_vhosts = self.configurator.choose_auth_vhosts(achall.domain)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 581, in choose_auth_vhosts
vhosts = [m['vhost'] for m in self._get_ranked_matches(target_name) if m and 'vhost' in m]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 460, in _get_ranked_matches
vhost_list = self.parser.get_vhosts()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/parser.py", line 165, in get_vhosts
parsed_server = _parse_server_raw(server)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/parser.py", line 821, in _parse_server_raw
end_index = [i for i, param in enumerate(params) if param.startswith('\n')][0]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range
2025-03-27 13:14:52,929:DEBUG:acme.client:Storing nonce: 3u7v6Bm_lirtOMayx3q47JBl50Z4NebfCiJy0yD3EPE1buktLEo
2025-03-27 13:14:52,930:INFO:certbot._internal.auth_handler:Performing the following challenges:
2025-03-27 13:14:52,930:INFO:certbot._internal.auth_handler:http-01 challenge for backyardbowl.ca
2025-03-27 13:14:52,930:INFO:certbot._internal.auth_handler:http-01 challenge for wp.metalsforhumanity.com
2025-03-27 13:14:52,930:INFO:certbot._internal.auth_handler:http-01 challenge for www.backyardbowl.ca
2025-03-27 13:14:52,946:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot/_internal/auth_handler.py", line 88, in handle_authorizations
resps = self.auth.perform(achalls)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 1240, in perform
http_response = http_doer.perform()
^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/http_01.py", line 63, in perform
self._mod_config()
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/http_01.py", line 134, in _mod_config
config = [self._make_or_mod_server_block(achall) for achall in self.achalls]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/http_01.py", line 223, in _make_or_mod_server_block
http_vhosts, https_vhosts = self.configurator.choose_auth_vhosts(achall.domain)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 581, in choose_auth_vhosts
vhosts = [m['vhost'] for m in self._get_ranked_matches(target_name) if m and 'vhost' in m]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/configurator.py", line 460, in _get_ranked_matches
vhost_list = self.parser.get_vhosts()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/parser.py", line 165, in get_vhosts
parsed_server = _parse_server_raw(server)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/snap/certbot/4482/lib/python3.12/site-packages/certbot_nginx/_internal/parser.py", line 821, in _parse_server_raw
end_index = [i for i, param in enumerate(params) if param.startswith('\n')][0]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

2

Can you please share the output of the command sudo nginx -T?

There has been a change to the code recently which added the line that's throwing the error, see Remove internal comments from server_names directive (#10147) · certbot/certbot@6fd6a54 · GitHub.

Maybe your previous nginx was compatible and now it's not.. Which may be a bug in Certbot or maybe an incompatible nginx directive to begin with which went unnoticed until now.

4

The output is:

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

It failed on another domain to renew as well. It has been working with these configs for 5+ years. I removed the *.backyardbowl.ca as a suggestion from ChatGPT to see if that helped, but it didn't.

5

That was a lowercase t, not the uppercase T. Please try again. And no, ChatGPT most likely will not provide any sensible suggestions.

2 Likes
6

Not comfortable with posting all my nginx configs here. But here is the one that was problematic:

# configuration file /etc/nginx/sites-enabled/remote-folder:
server {

  root /var/www/backyard;

  index index.php;

  server_name remote-folder.club remote-folder.club backyardbowl.ca www.backyardbowl.ca;

  location / {
    try_files $uri $uri/ /index.php?q=$uri&$args;
  }

  client_max_body_size 10M;

  error_page 404 /404.html;

  # redirect server error pages to the static page /50x.html
  #
  error_page 500 502 503 504 /50x.html;
  location = /50x.html {
    root /usr/share/nginx/www;
  }

  # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
  #
  location ~ \.php$ {
    try_files $uri =404;
    fastcgi_pass unix:/var/run/php/php-fpm.sock;
    fastcgi_index index.php;
    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    include fastcgi_params;
  }


   listen 443 ssl; # managed by Certbot
   ssl_certificate /etc/letsencrypt/live/backyardbowl.ca/fullchain.pem; # managed by Certbot
   ssl_certificate_key /etc/letsencrypt/live/backyardbowl.ca/privkey.pem; # managed by Certbot
   include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
   ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = www.backyardbowl.ca) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = backyardbowl.ca) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen 80;

    server_name remote-folder.club backyardbowl.ca www.backyardbowl.ca;
    return 404; # managed by Certbot
}
7

This should be harmless (I think) but you don't need the same domain name twice in your port 443 server block

Below is the more correct line from your port 80 server block

Perhaps the mismatch is confusing Certbot. Just a wild guess but easy enough to change the one in port 443 and should be changed anyway.

2 Likes
8

Then I won't be able to help you. Or at least, not willing. Because the log also mentions the hostname wp.metalsforhumanity.com, which is not present in the configuration file you've shown.

Now, you can either help us help you by providing the necessary information or you can try to fix this yourself.

I could not reproduce the error with the provided configuration file. And I wouldn't expect it to, because the line the error is thrown deals with comments and only gets triggered when there's a # in a directive, in this case the server_name directive.

10

Sorry, the error I posted was from an earlier run before I cleaned up the conf file (removed superflous domain (wp.metalsforhumanity.com), removed a *.backyardbowl.ca and removed a bunch of a boilerplate comments. The full nginx -T is over a 1000 lines which is probably too much to have you look at anyway. I will just stick with pinning Certbot at 3.2 until I rebuild the VPS which I will have to do pretty soon anyway (it is probably the ancient version of Ubuntu that is causing the error) and then I can get all these old conf files cleaned up.

I appreciate the help and understand that it basically impossible to debug something like this without being able to replicate it exactly. Thanks for looking into it in the first place.

1 Like
11

Yeah, that might be the problem. Thanks for the heads up.

1 Like
12

No, we are pretty experienced at looking past what we don't need :slight_smile:

It is handy if you upload that output but not necessary. Like:

sudo nginx -T  >config.txt

and upload theconfig.txt

1 Like
13

As mentioned earlier, the bug seems to be triggered by a # in a server_name directive. It's not difficult to look for that yourself.

You could also debug it yourself by adding e.g. a print(directive) line just above the end_index = [i for i, param in enumerate(params) if param.startswith('\n')][0] line to see what the offending server_name line was.

Knowing the offending server_name directive would help reproducing this bug.

2 Likes
14

The quickest way to check all the active server_name lines is:

sudo nginx -T | grep server_name | grep -v '\$server_name'
2 Likes