[BUG] certbot --nginx generates PR_END_OF_FILE_ERROR

a Ubuntu 16.04.6 LTS VPS running nginx is presently bricked in terms of serving pages through port 443. This happened unexpectedly, I assume when a renewal kicked in automatically.

Following are twice replicated steps.

I removed all site definitions in sites-enabled and reduced the server to its simplest expression: one application in http mode only. The output of nginx -T is at bottom. the unencrypted pages serve as expected.

I then ran sudo certbot --nginx and selected 1 for the only 3rd level domain available to nginx

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: hf.iwant2go2.com
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1

and installed new certificate, the choosing option 2 to redirect port 80 calls to port 443.

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2

The dialog indicates a successful creation and installation. The differences with the new nginx -T output are as follows. the default site was appended with:

listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/hf.iwant2go2.com/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/hf.iwant2go2.com/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
server {
    if ($host = hf.iwant2go2.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        server_name hf.iwant2go2.com;
    listen 80;
    return 404; # managed by Certbot

}

and naturally, the configuration file was created:

# configuration file /etc/letsencrypt/options-ssl-nginx.conf:
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file.

ssl_session_cache shared:le_nginx_SSL:1m;
ssl_session_timeout 1440m;

ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
ssl_prefer_server_ciphers on;

ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS";

The suggestion, after successful processing, to [check with SSLlabs] (https://www.ssllabs.com/ssltest/analyze.html?d=hf.iwant2go2.com) confirms the failure, wheras the browser responds with:

An error occurred during a connection to hf.iwant2go2.com. PR_CONNECT_RESET_ERROR

This is therefore a direct bug in the certbot cert creation process, either in its own operation or interacting with some other object.

Initial nginx -T output

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	tcp_nodelay on;
	keepalive_timeout 130s;
        client_max_body_size 10M;
	types_hash_max_size 2048;
	# server_tokens off;
        ## Block spammers and other unwanted visitors  ##
        include blockips.conf;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;
        ssl_session_cache shared:SSL:10m;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;

# Security headers
        add_header X-XSS-Protection "1; mode=block";
        add_header X-Frame-Options "SAMEORIGIN";

        server {
            listen 80 default_server;
            listen 443 default_server;
            return 444;
        }
}

#mail {
#	# See sample authentication script at:
#	# http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#	# auth_http localhost/auth.php;
#	# pop3_capabilities "TOP" "USER";
#	# imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#	server {
#		listen     localhost:110;
#		protocol   pop3;
#		proxy      on;
#	}
#
#	server {
#		listen     localhost:143;
#		protocol   imap;
#		proxy      on;
#	}
#}

# configuration file /etc/nginx/modules-enabled/10-mod-http-ndk.conf:
load_module modules/ndk_http_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-auth-pam.conf:
load_module modules/ngx_http_auth_pam_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-cache-purge.conf:
load_module modules/ngx_http_cache_purge_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-dav-ext.conf:
load_module modules/ngx_http_dav_ext_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-echo.conf:
load_module modules/ngx_http_echo_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-fancyindex.conf:
load_module modules/ngx_http_fancyindex_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-geoip.conf:
load_module modules/ngx_http_geoip_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-headers-more-filter.conf:
load_module modules/ngx_http_headers_more_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:
load_module modules/ngx_http_image_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-lua.conf:
load_module modules/ngx_http_lua_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-passenger.conf:
load_module /usr/lib/nginx/modules/ngx_http_passenger_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-perl.conf:
load_module modules/ngx_http_perl_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-subs-filter.conf:
load_module modules/ngx_http_subs_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-uploadprogress.conf:
load_module modules/ngx_http_uploadprogress_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-upstream-fair.conf:
load_module modules/ngx_http_upstream_fair_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:
load_module modules/ngx_http_xslt_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-mail.conf:
load_module modules/ngx_mail_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-nchan.conf:
load_module modules/ngx_nchan_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-stream.conf:
load_module modules/ngx_stream_module.so;

# configuration file /etc/nginx/blockips.conf:
deny 185.213.20.172;
deny 104.129.18.6;
deny 185.224.91.73;
deny 176.126.83.15;
deny 69.49.102.227;
deny 60.191.38.77;
deny 209.17.96.66;
deny 80.82.77.139;
deny 184.105.247.194;
deny 184.105.247.195;
deny 104.131.133.207;
deny 80.82.77.33;
deny 151.239.67.109;
deny 77.246.234.70;
deny 213.85.3.11;
deny 45.227.253.36;
deny 185.82.216.97;
deny 185.222.211.54;
deny 82.118.242.240;
deny 97.177.247.162;
deny 118.138.101.236;
deny 213.183.146.74;
deny 5.188.211.15;
deny 85.93.20.58;
deny 203.172.211.4;
deny 77.161.231.173;
deny 35.175.132.238;
deny 45.227.255.149;
deny 162.62.26.7;

# configuration file /etc/nginx/mime.types:

types {
    text/html                             html htm shtml;
    text/css                              css;
    text/xml                              xml;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
    application/atom+xml                  atom;
    application/rss+xml                   rss;

    text/mathml                           mml;
    text/plain                            txt;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/x-component                      htc;

    image/png                             png;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/x-icon                          ico;
    image/x-jng                           jng;
    image/x-ms-bmp                        bmp;
    image/svg+xml                         svg svgz;
    image/webp                            webp;

    application/font-woff                 woff;
    application/java-archive              jar war ear;
    application/json                      json;
    application/mac-binhex40              hqx;
    application/msword                    doc;
    application/pdf                       pdf;
    application/postscript                ps eps ai;
    application/rtf                       rtf;
    application/vnd.apple.mpegurl         m3u8;
    application/vnd.ms-excel              xls;
    application/vnd.ms-fontobject         eot;
    application/vnd.ms-powerpoint         ppt;
    application/vnd.wap.wmlc              wmlc;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/x-7z-compressed           7z;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-perl                    pl pm;
    application/x-pilot                   prc pdb;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            der pem crt;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xspf+xml                  xspf;
    application/zip                       zip;

    application/octet-stream              bin exe dll;
    application/octet-stream              deb;
    application/octet-stream              dmg;
    application/octet-stream              iso img;
    application/octet-stream              msi msp msm;

    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
    application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

    audio/midi                            mid midi kar;
    audio/mpeg                            mp3;
    audio/ogg                             ogg;
    audio/x-m4a                           m4a;
    audio/x-realaudio                     ra;

    video/3gpp                            3gpp 3gp;
    video/mp2t                            ts;
    video/mp4                             mp4;
    video/mpeg                            mpeg mpg;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-m4v                           m4v;
    video/x-mng                           mng;
    video/x-ms-asf                        asx asf;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;
}

# configuration file /etc/nginx/conf.d/mod-http-passenger.conf:
### Begin automatically installed Phusion Passenger config snippet ###
passenger_ruby /home/deploy/.rbenv/shims/ruby; # If you use rbenv
passenger_root /usr/lib/ruby/vendor_ruby/phusion_passenger/locations.ini;
# passenger_ruby /usr/bin/passenger_free_ruby;
### End automatically installed Phusion Passenger config snippet ###

# configuration file /etc/nginx/sites-enabled/default:
server {
        # Make site accessible from http://localhost/
        server_name hf.iwant2go2.com;
        rails_env         development;
        root              /home/deploy/hf/current/public;

        # redirect server error pages to the static page /50x.html
        error_page 404 /404.html;
          location  /404.html {
        }
        error_page   500 502 503 504  /50x.html;
        location / {
           passenger_enabled on;
        }

        location = /50x.html {
            root   html;
        }
        location ~ /(wp-content|solr|jsonws|mifs|wp-includes|login.asp)/ {
            return 404;
        }
        location ~ \.php$ {
            return 404;
        }
        location ~ \.aspx$ {
            return 404;
        }
        location ~ \.env$ {
            return 404;
        }
        location ~ \.production$ {
            return 404;
        }
        location ~ \.git$ {
            return 404;
        }

}
1 Like

Thanks for writing this up in such a detailed way.

I suspect that the listen 443 default_server is the troublemaker here.

Essentially that statement is saying "listen with plain HTTP on port 443". (As opposed to listen 443 ssl; which would require you to configure a snakeoil certificate as well).

When Certbot then tries to enhance your actual virtualhost with HTTPS on port 443, it is clashing with the above directive.

Does this problem still occur if you comment that line out?

3 Likes

Is this normal?:

        server {
            listen 80 default_server;
            listen 443 default_server;
            return 444;
        }

No other listen statements found.

Not even in the single included file:

# configuration file /etc/nginx/sites-enabled/default:
server {
        server_name hf.iwant2go2.com;
        rails_env         development;
        root              /home/deploy/hf/current/public;
        error_page 404 /404.html;
          location  /404.html {
        }
        error_page   500 502 503 504  /50x.html;
        location / {
           passenger_enabled on;
        }
        location = /50x.html {
            root   html;
        }
        location ~ /(wp-content|solr|jsonws|mifs|wp-includes|login.asp)/ {
            return 404;
        }
        location ~ \.php$ {
            return 404;
        }
        location ~ \.aspx$ {
            return 404;
        }
        location ~ \.env$ {
            return 404;
        }
        location ~ \.production$ {
            return 404;
        }
        location ~ \.git$ {
            return 404;
        }
}
1 Like

My understanding is that nginx will take the default value of listen if it's not explicitly included: https://nginx.org/en/docs/http/ngx_http_core_module.html#listen

2 Likes

So it will, by default, listen on both?

1 Like

I believe that this will just listen on 801:

server {
        server_name example.com;
}

1. When running as root, anyway. Otherwise, it will listen on port 8000. That's why both are listed in the docs for listen.

1 Like

@_az, so, is the "fix"?:

        server {
            listen 80 default_server;
            listen 443 ssl default_server;
            return 444;
        }

Or does he have to add a listen statement in the default file?
OR
Does he have to edit both?

You can't listen 443 ssl without also specifying ssl_certificate and ssl_certificate_key.

To confirm the nature of the issue (since it is only my guess), I would suggest just getting rid of the listen 443 line entirely.

If the intention of this "default" server is to not accidentally expose which virtual hosts it is hosting, then do what you said, and also configure a self-signed snakeoil certificate.

1 Like

Then they should edit both:

        server {
            listen 80 default_server;
            #listen 443 default_server; # take this out
            return 444;
        }

# configuration file /etc/nginx/sites-enabled/default:
server {
        listen 443 ssl; # add this in
        server_name hf.iwant2go2.com;
...

And include the cert files generated.

1 Like

Oh yes, sure.

i was rather thinking about how to avoid the problem to begin with (starting from the "initial" config).

But it already did that?

So the whole problem is you can't listen on 443 with HTTP and then listen on 443 with SSL.

Yes, nicely put :slight_smile: .

1 Like

Then we're back to just one single edit:

        server {
            listen 80 default_server;
            #listen 443 default_server; # take this out
            return 444;
        }

Yes, I confirm that this is the fix.

This was a useful dialog to follow.

Still the certbot procedure may need to pick this up. nginx -t is also misleading somewhat.

1 Like

Yeah.
[prior to making any modifications]
If "nginx -T" contains an active line with "listen" & "443" and doesn't contain "ssl" then raise a flag!
"This configuration is not supported. You need to ..."