Auto-Renewal fails with TypeError in nginxparser.py

Help
1

My domain is: katmail.de
I ran this command: certbot renew --dry-run
It produced this output: Attempting to renew cert from /etc/letsencrypt/renewal/katmail.de.conf produced an unexpected error: sequence item 2: expected str instance, NoneType found. Skipping. -> see below for full output
My web server is (include version): nginx 1.13.1
The operating system my web server runs on is (include version): Arch Linux 64bit
I can login to a root shell on my machine: yes
I’m using a control panel to manage my site: no

Hi, I’ve set up Let’s Encrypt for my sites using the nginx-plugin for certbot, worked without issues.
However, if I try to renew the certificates using certbot renew --dry-run I run into the following error:

root@hostname ~ # certbot renew --dry-run                  
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/katmail.de.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for katmail.de
tls-sni-01 challenge for calendar.katmail.de
tls-sni-01 challenge for cloud.katmail.de
tls-sni-01 challenge for mail.katmail.de
tls-sni-01 challenge for www.katmail.de
tls-sni-01 challenge for xmpp.katmail.de
Cleaning up challenges
2017/06/14 20:08:38 [notice] 19326#19326: signal process started
Attempting to renew cert from /etc/letsencrypt/renewal/katmail.de.conf produced an unexpected error: sequence item 2:     expected str instance, NoneType found. Skipping.
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/katmail.de/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
1 renew failure(s), 0 parse failure(s)

Judging from the Let’s Encrypt logs, it seems like there is an issue with parsing the nginx configuration files, but the server is up and running, and encryption is working without issues.

Part of letsencrypt.log that contains the error:

root@hostname ~ # cat /var/log/letsencrypt/letsencrypt.log
2017-06-14 18:08:38,504:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 115, in _solve_challenges
    resp = self.auth.perform(self.achalls)
  File "/usr/lib/python3.6/site-packages/certbot_nginx/configurator.py", line 809, in perform
    sni_response = chall_doer.perform()
  File "/usr/lib/python3.6/site-packages/certbot_nginx/tls_sni_01.py", line 75, in perform
    self.configurator.save("SNI Challenge", True)
  File "/usr/lib/python3.6/site-packages/certbot_nginx/configurator.py", line 721, in save
    self.parser.filedump(ext='')
  File "/usr/lib/python3.6/site-packages/certbot_nginx/parser.py", line 239, in filedump
    out = nginxparser.dumps(tree)
  File "/usr/lib/python3.6/site-packages/certbot_nginx/nginxparser.py", line 133, in dumps
    return str(RawNginxDumper(blocks.spaced))
  File "/usr/lib/python3.6/site-packages/certbot_nginx/nginxparser.py", line 97, in __str__
    return ''.join(self)
  File "/usr/lib/python3.6/site-packages/certbot_nginx/nginxparser.py", line 86, in __iter__
    for line in self.__iter__([parameter]): # negate "for b0 in blocks"
  File "/usr/lib/python3.6/site-packages/certbot_nginx/nginxparser.py", line 93, in __iter__
    yield "".join(item) + semicolon
TypeError: sequence item 2: expected str instance, NoneType found

2017-06-14 18:08:38,504:DEBUG:certbot.error_handler:Calling registered functions
2017-06-14 18:08:38,504:INFO:certbot.auth_handler:Cleaning up challenges
2017-06-14 18:08:39,697:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/katmail.de.conf produced an unexpected error: sequence item 2: expected str instance, NoneType found. Skipping.
2017-06-14 18:08:39,698:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/certbot/renewal.py", line 419, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 641, in renew_cert
    _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 77, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3.6/site-packages/certbot/renewal.py", line 297, in renew_cert
    new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
  File "/usr/lib/python3.6/site-packages/certbot/client.py", line 313, in obtain_certificate
    self.config.allow_subset_of_names)
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 74, in get_authorizations
    resp = self._solve_challenges()
  File "/usr/lib/python3.6/site-packages/certbot/auth_handler.py", line 115, in _solve_challenges
    resp = self.auth.perform(self.achalls)
  File "/usr/lib/python3.6/site-packages/certbot_nginx/configurator.py", line 809, in perform
    sni_response = chall_doer.perform()
  File "/usr/lib/python3.6/site-packages/certbot_nginx/tls_sni_01.py", line 75, in perform
    self.configurator.save("SNI Challenge", True)
  File "/usr/lib/python3.6/site-packages/certbot_nginx/configurator.py", line 721, in save
    self.parser.filedump(ext='')
  File "/usr/lib/python3.6/site-packages/certbot_nginx/parser.py", line 239, in filedump
    out = nginxparser.dumps(tree)
  File "/usr/lib/python3.6/site-packages/certbot_nginx/nginxparser.py", line 133, in dumps
    return str(RawNginxDumper(blocks.spaced))
  File "/usr/lib/python3.6/site-packages/certbot_nginx/nginxparser.py", line 97, in __str__
    return ''.join(self)
  File "/usr/lib/python3.6/site-packages/certbot_nginx/nginxparser.py", line 86, in __iter__
    for line in self.__iter__([parameter]): # negate "for b0 in blocks"
  File "/usr/lib/python3.6/site-packages/certbot_nginx/nginxparser.py", line 93, in __iter__
    yield "".join(item) + semicolon
TypeError: sequence item 2: expected str instance, NoneType found

2017-06-14 18:08:39,698:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.15.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 743, in main
    return config.func(config, plugins)
  File "/usr/lib/python3.6/site-packages/certbot/main.py", line 693, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3.6/site-packages/certbot/renewal.py", line 436, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2

can you show contents of:
/etc/letsencrypt/renewal/katmail.de.conf

3

Sure.

root@hostname ~ # cat /etc/letsencrypt/renewal/katmail.de.conf
# renew_before_expiry = 30 days
version = 0.15.0
archive_dir = /etc/letsencrypt/archive/katmail.de
cert = /etc/letsencrypt/live/katmail.de/cert.pem
privkey = /etc/letsencrypt/live/katmail.de/privkey.pem
chain = /etc/letsencrypt/live/katmail.de/chain.pem
fullchain = /etc/letsencrypt/live/katmail.de/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = <accountid>
authenticator = nginx
installer = nginx

The archive path is correct, the /live/-paths are all working symlinks.

4

@erica, could this be a known bug that would be fixed by updating to a newer Certbot release? Or something a bit trickier?

5

@schoen , 0.15.0 is already the newest release, isn’t it? I’d also be happy to try building directly from git if that helps.

6

@lcts, could you post the contents of your Nginx configuration file for this site? Probably found at /etc/nginx/sites-available/katmail.de.conf.

7

This is sort of a minimal testcase for the error.
For testing purposes I’ve set up most subdomains to redirect to www.katmail.de. cloud.katmail.de has its own config, but it’s very long (nextcloud instance) so I didn’t post it. In terms of the SSL config it is identical.

The only thing I’ve changed since the initial installation of the certificates is to move the redirections to HTTPS to a single server{}-block in nginx.conf instead of individual redirections in each block.

nginx.conf (redirects all traffic on port 80 to HTTPS):

root@hostname /etc/nginx # cat nginx.conf

<snip>
http {
    include		mime.types;
    include		fastcgi_params;
    include		servers.d/*.conf;
    <snip>
    # Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.
    server {
        listen 80 default_server;
        return 301 https://$host$request_uri;
    }

}

servers.d/katmail-root.conf (for testing purposes redirects everything except xmpp.katmail.de and cloud.katmail.de to www.katmail.de):

root@hostname /etc/nginx # cat servers.d/katmail-root.conf 
# redirect subdomains that don't serve web content to www
server {
    server_name mail.katmail.de calendar.katmail.de;
    return 301 $scheme://www.katmail.de$request_uri;
}

server {
    server_name  www.katmail.de katmail.de;
    root   /srv/http/katmail/root;
    listen 443 ssl http2;

    # Setup location of .well-known/acme-challenge/ and SSL/TLS options
    include conf.d/acme-challenge.conf;
    include /etc/letsencrypt/options-ssl-nginx.conf;

    ssl_certificate /etc/letsencrypt/live/katmail.de/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/katmail.de/privkey.pem; # managed by Certbot
}

conf.d/acme-challenge.conf (sets up a single directory for the challenge so I can include it in all configs):

root@hostname /etc/nginx # cat conf.d/acme-challenge.conf 
location ^~ /.well-known/acme-challenge {
  allow all;
  alias /var/lib/letsencrypt/.well-known/acme-challenge/;
  default_type "text/plain";
  try_files $uri =404;
}
8

Small update, in case this sheds some light on the issue for you: I tried a dry-run update using certbot certonly instead of certbot renew (webroot method, with /var/lib/letsencrypt as the web root to simulate acme-challenge.conf). That completes without issues, with nginx running as configured above.

2017-06-17 edited to add: The same is true when using renew, but with authenticator = webroot instead of = nginx in the config file.

So at least I can automatically update my certs that way, but I’d still like to find out what’s going wrong with the nginx plugin.

9

@erica , I checked to see what happens in nginxparser.py when the error occurs. This is the content of item in line 93 for the last few iterations before the error:

['#', ' Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.']
['listen', ' ', '80', ' ', 'default_server']
['#', '        listen [::]:80 default_server;']
['return', ' ', '301', ' ', 'https://$host$request_uri']
['#', " redirect subdomains that don't serve web content to www"]
['server_name', ' ', 'mail.katmail.de', ' ', 'calendar.katmail.de']
['return', ' ', '301', ' ', '$scheme://www.katmail.de$request_uri']
['listen', ' ', '80']
['#', ' managed by Certbot']
['listen', ' ', '443 ssl']
['#', ' managed by Certbot']
['ssl_certificate', ' ', '/var/lib/letsencrypt/snakeoil/0029_cert.pem']
['#', ' managed by Certbot']
['ssl_certificate_key', ' ', None]

I’m assuming that the problem is the path associated with ssl_certificate_key is not parsed correctly, but as everything after ['return', ' ', '301', ' ', '$scheme://www.katmail.de$request_uri'] was transiently added by Certbot I don’t know how that is supposed to look like.

Hope this helps.

10

Thanks @lcts. This looks like the same problem as Auto renewal dry run failed - sequence item 2: expected string, NoneType found, which will be fixed in our next release, at the beginning of next month. It’s an issue with --dry-run, so all production functionality should work as expected.

1 Like
11

Excellent, thank you.

12

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.