Certbot Could not parse NGINX server block because char 0 at line 1 is the "s" in "server"

My domain is: adamlein.com

I ran this command:

sudo certbot --installer nginx --dns-cloudflare --dns-cloudflare-credentials /etc/cloudflare/credential.ini --dns-cloudflare-propagation-seconds 30

It produced this output: For every domain in NGINX, the following error appears about one server block:

Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's' (at char 0), (line:1, col:1)

My web server is (include version): nginx version: nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version):

Virtualization: kvm
Operating System: Ubuntu 22.04.5 LTS
Kernel: Linux 6.8.0-1019-oracle
Architecture: arm64
Hardware Vendor: QEMU
Hardware Model: KVM Virtual Machine

My hosting provider, if applicable, is: Oracle Cloud

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.21.0

I can't figure out why this error keeps appearing. It appears for every domain in NGNIX when runnning certbot, but generating the certificates appears to be successful anyway. It just doesn't edit the server block file to include those certificates and configuration changes properly. Although I think automatic certbot renewals seem to fail (it did this week). I can edit the adamlein.com server block file manually in SSH and everything is fine. The error happens if I turn off Cloudflare DNS proxy and don't use the dns-cloudflare plug-ins as well.

The server block that causes the error is below:

server {

    root /var/www/adamlein.com/html;
    index index.html index.htm index.php;

    server_name adamlein.com www.adamlein.com beta.adamlein.com;
# Content security policy
#add_header Content-Security-Policy: default-src *; script-src * https://www.googletagmanager.com; frame-src *;
#HSTS enabled
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
# XSS protection header
add_header X-XSS-Protection "1; mode=block";
# X-Frame deny header
add_header X-Frame-Options "DENY";
# X-Content-Type-Options header
add_header X-Content-Type-Options "nosniff";

# custom log location
access_log  /var/log/nginx/adamlein.com.access.log;
#    location / {
#        try_files \$uri \$uri/ /index.php;
#    }

    location ~ \.php$ {
       # include snippets/fastcgi-php.conf;
       # fastcgi_pass unix:/var/run/php/php8.2-fpm.sock;
       # try_files $uri =404;
                fastcgi_split_path_info ^(.+\.php)(/.+)$;
                include snippets/fastcgi-php.conf;

                # With php-fpm (or other unix sockets):
                fastcgi_pass unix:/run/php/php8.4-fpm.sock;
                # With php-cgi (or other tcp sockets):
               # fastcgi_index index.php;
          #     fastcgi_pass 127.0.0.1:9000;
                fastcgi_param  SCRIPT_FILENAME $document_root$fastcgi_script_name;
                include fastcgi_params;
    }
#sitemap redirect to dynamic PHP file
location = /sitemap.xml
        {
            rewrite .* /sitemap.php redirect;
        }
#letter case issues
location = /Design.php
{ rewrite .* /design.php redirect;
}
location = /Infostrt.php
{ rewrite .* /infostrt.php redirect;
}

#asp redirect to php
location ~ \.asp$ {
    if (!-f $request_filename) {
        rewrite ^(.*)\.asp$ $1.php permanent;
    }
}


# cache policy
location ~* \.(ico|pdf|flv|jpg|jpeg|png|gif|js|css|swf|mp4|svg|json)$ {
    expires 3600h;
    add_header Cache-Control "public, no-transform";
}
# redirect without extensions
location = /contact
{ rewrite .* /contact.php redirect; }

location = /resume
#{ rewrite .* /resume.php redirect; }
{ rewrite .* /Files/Adam_Lein_resume.pdf redirect; }


location = /photography
{ rewrite .* /photography.php redirect; }

#hacked pages 410
location = /zeb.php {
        return 410;
}
location = /Files/zfa.php {
        return 410;
}
location = /Files/zeb.php {
        return 410;
}
location = /mds.php {
        return 410;
}



#block hacker attempts
location / {
    deny 141.98.81.44;
    deny 147.78.47.87;
    deny 83.147.52.49;
    deny 109.176.202.52;
    allow all;
}

#SSL stuff
    location ~ /\.ht {
        deny all;
    }

    listen [::]:443 ssl ipv6only=on; # managed by Certbot
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/adamlein.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/adamlein.com-0001/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
    if ($host = www.adamlein.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = adamlein.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    listen 80;
    listen [::]:80;

    server_name adamlein.com www.adamlein.com beta.adamlein.com;
    return 404; # managed by Certbot


}


The error is referring to the first character on the first line which is the "s" in "server", but that needs to be there for NGINX so maybe there's a different error somewhere and certbot isn't telling me the correct line/character?

Maybe the configuration file before the adamlein.com configuration file gets parsed ends incorrectly? Maybe a missing ;?

3 Likes

That is probably it. Nginx treats includes as macros, expanding them inline as encountered. There are a few other implementation details in the nginx config file system that could be related.

Another possible situation is if you're using an extension to nginx, like openresty. Certbot uses a library called Augeas to parse the config file, and that library can only handle "vanilla" (standard) nginx config file directives. It often completely breaks on the "advanced" directives from third party modules/forks.

What is the output of nginx -t -c {FILEPATH} of the main configuration file?

Look into fail2ban. It's available on ubuntu, and automatically manages something like this in the kernal via the system firewall.

3 Likes

Ah! Good idea! If they're processed alphabetically, adamlein.com is the first one though. I just looked at /etc/nginx/nginx.conf as well as the other server block files and don't see any issues there. The conf.d includes directory is empty, so probably nothing there. Hmm.
I would think if a semicolon was missing, then "sudo nginx -t" would say something about that, no?

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

If I put a ";" on the first line of the adamlein.com server block file before the "server", nginx -t has errors and certbot has even more errors. Same if I put a "}" as the first character.

I saw this Github issue 9942 saying to add a space after the # for commented lines, so I did that, but it didn't fix the certbot errors.

Dry run renewal (sudo certbot renew --dry-run --dns-cloudflare-propagation-seconds 30) gives me this:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/adamlein.com-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for adamlein.com and 3 more domains
Waiting 30 seconds for DNS changes to propagate

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/aowyn.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)
Simulating renewal of an existing certificate for aowyn.com and www.aowyn.com
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/bookofadamz.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)
Simulating renewal of an existing certificate for bookofadamz.com
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/lein.cc.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)
Simulating renewal of an existing certificate for lein.cc
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/lein.us.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)
Simulating renewal of an existing certificate for lein.us
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/leinassociates.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)
Simulating renewal of an existing certificate for leinassociates.com
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected stringEnd, found 's'  (at char 0), (line:1, col:1)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all simulated renewals succeeded:
  /etc/letsencrypt/live/adamlein.com-0001/fullchain.pem (success)
  /etc/letsencrypt/live/aowyn.com/fullchain.pem (success)
  /etc/letsencrypt/live/bookofadamz.com/fullchain.pem (success)
  /etc/letsencrypt/live/lein.cc/fullchain.pem (success)
  /etc/letsencrypt/live/lein.us/fullchain.pem (success)
  /etc/letsencrypt/live/leinassociates.com/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

So it seems like renewals should work now, but that's still a lot of "Could not parse file" errors.

Thanks! Yes, I later installed and configured that as well, so I probably don't need to block specific IPs in NGINX anymore.

This is fairly old. IMHO 99% of updates to certbot over the past few years have been on nginx/apache integration. you should update certbot (switching to snapd is the easiest; otherwise you may want pip). hopefully whatever issue you have will be fixed by that. if it doesn't , I have some ideas for troubleshooting and creating a reproducible test case.

This honestly isn't worth your time, or anyone else's, debugging against 1.21 though.

5 Likes

Thanks! I guess I didn't realize the apt repository had such an outdated certbot! Will try to upgrade before it's time for another renewal.

Certbot stopped supporting various linux distributions themselves several years ago, in favor of a single Snapd distribution. It's less work to support one channel, and the distributions were always months behind. After that happened, the distributions dropped their effort to support certbot to minimal levels.

4 Likes

I've updated to Certbot 4.0.0 and it has the same problem, but shows a slightly different error:

Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected string_end, found 'server' (at char 0), (line:1, col:1)

It thinks the whole first word "server" is the problem instead of just the first letter now. That error is repeated for every domain in NGINX when running:

sudo certbot renew --dry-run --dns-cloudflare-propagation-seconds 30

I don't know if I have to run the below again after updating certbot to the 4.0.0 snap version, but I tried it:

sudo certbot --installer nginx --dns-cloudflare --dns-cloudflare-credentials /etc/cloudflare/credential.ini --dns-cloudflare-propagation-seconds 30

And the results were that it didn't even recognize the adamlein.com domain as being configured in NGINX. It only listed all of the other domains and subdomains with no option to create a certificate for that domain.

Again, NGINX doesn't seem to have any problems with the /etc/nginx/sites-enabled/adamlein.com config file as "nginx -t" shows no errors and everything works properly with that site as long as I have a valid certificate there. It's just that certbot is showing errors during the renew.

If I remove the symlink to that specific server block config file:

sudo rm /etc/nginx/sites-enabled/adamlein.com

Then the certbot commands don't show the error anymore, so that would seem to indicate that the problem is in that file /etc/nginx/sites-enabled/adamlein.com and not some other config file, right? If it was somewhere else, the error should be referencing the next server block config file in that folder I would think.
Re-adding the symlink:

sudo ln -s /etc/nginx/sites-available/adamlein.com /etc/nginx/sites-enabled/

...brings the certbot error back:

sudo certbot --installer nginx --dns-cloudflare --dns-cloudflare-credentials /etc/cloudflare/credential.ini --dns-cloudflare-propagation-seconds 30
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected string_end, found 'server' (at char 0), (line:1, col:1)

Ok, so upgrading Certbot changed this error:

To this error:

That slight difference: stringEnd to string_end made all the difference.

A quick search for the new error code brought up this posting from a few years ago:

In that posting, the parsing error was attributed to commented out directives.

I suggest trying the following:

1- standardize all the comments - line the # up with the block indents and ensure there is one space after. e.g. # comment not #comment or # comment

2- if that doesn't work, try deleting the commented-out directives that span multiple lines first; then try the single lines

that should pinpoint what is causing the parsing error and allow you to file a bug report.

4 Likes

@adamz In addition to @jvanasco recent suggestions you should carefully review the output of:

sudo nginx -T >config.txt

An upper case T is essential. It will output the entire active nginx config including files that are included. The server block in question has at least one include so perhaps that is involved.

I'd also look carefully for stray null characters and similar. I am not sure if the -T output would retain those so you may need to check the original sources.

4 Likes

I FOUND IT!! THANK YOU!

Changing this:

 location = /resume
# { rewrite .* /resume.php redirect; }
{ rewrite .* /Files/Adam_Lein_resume.pdf redirect; }

to this:

 location = /resume
{ rewrite .* /Files/Adam_Lein_resume.pdf redirect; }

...Fixed everything and removed the error from certbot commands.
So really the thing it had a problem with was on line 68 not line 1. That line is supposed to be commented out, but I guess the bug is that certbot is reading it anyway and the second bug is that it's reporting the problem as line 1 when it was really line 68.

I had seen that before I posted this originally, but I didn't find any commented out lines with quotation marks in them like that person had.

3 Likes

Once you've seen enough of these issues, you start to realize which ones are probably related.

The parser Certbot uses is pretty fragile. To you, the quotation marks stood out; to me, a commented line in general stood out. That software throws a tantrum everytime it doesn't see whatever it think is perfectly normal (a tiny subset of what is actually normal and common in nginx configs).

5 Likes

Nginx themselves have config converter to json: GitHub - nginxinc/crossplane: Quick and reliable way to convert NGINX configurations into JSON and back.
not sure it can build part of split config file though

Cannot parse partial configs · Issue #99 · nginxinc/crossplane · GitHub have someone's hack to do about it.

With some more investigating, all I have to do to reproduce that error is to put a comment like this:

location = /resume
# {
{ rewrite .* /Files/Adam_Lein_resume.pdf redirect; }

in between two lines of a rewrite in an enabled NGINX config file (/etc/nginx/sites-enabled/). The # symbol and a { character is all that's needed.
Certbot will then have parsing errors on the first character of the config file like so:

Could not parse file: /etc/nginx/sites-enabled/adamlein.com due to Expected string_end, found 'server' (at char 0), (line:1, col:1)

1 Like

Before opening a bug report on the Certbot Github page, please update your Certbot to the latest version, currently 4.0.0.

Certbot 1.21.0 is quite old and perhaps this bug was fixed already.

As previously requested, they updated Certbot to 4.0 for renewal yesterday and recreated the issue.

Updating to 4.0 triggered the newer error message, which we used to identify the problem.

1 Like

Whoops, I scrolled the thread for some line like this, but scrolled by it too fast :rofl: My bad.

1 Like