Nginx + http3 + letsencrypt

Is this possible to achieve?

I'm using this Nginx package built with Quic module. I configure it as described there except for the Stapling part because I'm not able to generate the ocsp file. Maybe is it for this reason it doesn't work? Is Stapling mandatory using HTTP/3 protocol?

ssl_stapling on;
ssl_stapling_file /path/to/ocsp; # generate by `openssl ocsp -no_nonce -issuer /path/to/intermediate -cert /path/to/cert -url "$(openssl x509 -in /path/to/cert -noout -ocsp_uri)" -respout /path/to/ocsp`

Using HTTP/2 no problem at all.
Using HTTP/3 Chrome returns NET::ERR_CERT_INVALID. Same on Firefox.


My domain is: https://dev.tecnicman.com/

I ran this command to acquire and install the certificate:

certbot -i nginx --dns-cloudflare --dns-cloudflare-credentials cloudflare.ini -d dev.tecnicman.com

My web server is (include version):

nginx version: nginx/1.23.0
built with OpenSSL 1.1.1 (compatible; BoringSSL) (running with BoringSSL)
TLS SNI support enabled
configure arguments: --prefix=/etc/nginx --sbin-path=/usr/sbin/nginx --add-module=/github/home/nginx/debian/modules/ngx_brotli --add-module=/github/home/nginx/debian/modules/headers-more-nginx-module --add-module=/github/home/nginx/debian/modules/ngx_http_geoip2_module --modules-path=/usr/lib/nginx/modules --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx.pid --lock-path=/var/run/nginx.lock --http-client-body-temp-path=/var/cache/nginx/client_temp --http-proxy-temp-path=/var/cache/nginx/proxy_temp --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp --user=nginx --group=nginx --with-file-aio --with-threads --with-http_ssl_module --with-http_v2_module --with-http_v3_module --without-select_module --without-poll_module --without-http_access_module --without-http_autoindex_module --without-http_browser_module --without-http_charset_module --without-http_empty_gif_module --without-http_limit_conn_module --without-http_memcached_module --without-http_mirror_module --without-http_referer_module --without-http_split_clients_module --without-http_scgi_module --without-http_ssi_module --without-http_upstream_hash_module --without-http_upstream_ip_hash_module --without-http_upstream_keepalive_module --without-http_upstream_least_conn_module --without-http_upstream_random_module --without-http_upstream_zone_module --without-http_userid_module --without-http_uwsgi_module --with-zlib=/github/home/nginx/debian/modules/zlib --with-cc-opt='-I../modules/boringssl/include -g -O2 -ffile-prefix-map=/github/home/nginx=. -fstack-protector-strong -Wformat -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -fPIC' --with-ld-opt='-ljemalloc -L../modules/boringssl/build/ssl -L../modules/boringssl/build/crypto -Wl,-z,relro -Wl,-z,now -Wl,--as-needed -pie'

The operating system my web server runs on is (include version):

Distributor ID:	Ubuntu
Description:	Ubuntu 22.04 LTS
Release:	22.04
Codename:	jammy

My hosting provider, if applicable, is: Not applicable

I can login to a root shell on my machine (yes or no, or I donā€™t know): Yes

Iā€™m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if youā€™re using Certbot):

certbot 1.29.0

CONFIGURATION


nginx.conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    use epoll;
    worker_connections  1024;
    multi_accept on;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;


    ##
    # Logging Settings
    ##

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;



    ##
    # Basic Settings
    ##

    aio threads;
        aio_write on;
        brotli on;
        brotli_types application/atom+xml application/javascript application/json application/rss+xml 
                            application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype 
                            application/x-font-ttf application/x-javascr>
    etag off;
    gzip on;
    gzip_comp_level 6;
    gzip_types application/atom+xml application/javascript application/json application/rss+xml 
                      application/vnd.ms-fontobject application/x-font-opentype application/x-font-truetype 
                      application/x-font-ttf application/x-javascrip>
    more_clear_headers server;
    quic_gso on;
    quic_retry on;
    sendfile on;
    tcp_nopush on;
    
    include /etc/nginx/security-headers.conf;

    include /etc/nginx/conf.d/*.conf;
}

security-headers.conf

# Hide from Header Server Name (nginx) (on to enable default)
server_tokens off;

# Security Headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";

# HSTS (HTTP Strict Transport Security)
add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";

ssl-params.conf

ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_buffer_size 1400;
ssl_session_tickets off;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305;
ssl_prefer_server_ciphers off;
ssl_ecdh_curve secp384r1;
ssl_dhparam /etc/nginx/dhparam.pem;

#ssl_stapling on;
#ssl_stapling_verify on;
#ssl_stapling_file /etc/nginx/isrg-root-ocsp-x1.der;
#ssl_trusted_certificate /etc/letsencrypt/live/dev.tecnicman.com/chain.pem;

resolver 1.1.1.1 1.0.0.1 valid=300s;
resolver_timeout 5s;

ssl_early_data on;

conf.d/dev.tecnicman.com.conf

server {
    if ($host = dev.tecnicman.com) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    listen   80;
    listen   [::]:80;
    server_name dev.tecnicman.com;
}

server {
    listen          443;
    listen          [::]:443;
    listen          443 http3;
    listen          [::]:443 http3;

    server_name     dev.tecnicman.com;

    add_header Alt-Svc 'h3=":443"; ma=3600';


    include ssl-params.conf;

    location / {
        proxy_pass  http://192.168.2.110;
        include proxy_params.conf;

        proxy_buffers              128 8k;
        proxy_buffer_size          512k;
        proxy_busy_buffers_size    512k;

        add_header Alt-Svc 'h3=":443"; ma=3600';
    }

#   include snippets/self-signed.conf;

    ssl_certificate /etc/letsencrypt/live/dev.tecnicman.com/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/dev.tecnicman.com/privkey.pem; # managed by Certbot
}

Hi @LetsKam and Welcome to the Community!
Unfortunately we will need more information from you to help you resolve your issue. I know there are experts here that might have crystal balls, but I don't.
Your cert error is not uncommon, stapling is not necissary but nice when you are ready and everything is configured correctly.... Please give us as much information as possible...

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=mydomain.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I donā€™t know):

Iā€™m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if youā€™re using Certbot):

https://certbot.eff.org/docs/

https://certbot.eff.org/docs/using.html#certbot-command-line-options

6 Likes

I would be interested in more answers to the questionnaire too. But, did you try asking that package maintainer about it? That is a very custom nginx build and they will be better informed about it than we would be.

I did a quick review of some docs and don't see anything unique about ocsp with http/3. And, that error message is not indicating an ocsp problem either. It talks about the ssl certs not being valid.

Have you checked your server blocks for http3 to ensure they send out the certs the same as your http2 server blocks?

If you click on the error icon in chrome or FF what does the detailed explanation about the "invalid cert" say? Is it a name mismatch or expired cert or ?

What URL are you using for http2? What about for http3?

Please answer the other questions on the form the best you can. The more info we have the better, and quicker, we can give advice.

7 Likes

I updated the question

1 Like

Something about that doesn't look right.
[it might be right (I haven't used http3 yet... just added that to my todo list) - it just doesn't seem right to me]

4 Likes

That's the recommended configuration found here. What's wrong?

I can't say for sure.
My gut tells me that it should be using something like "reuseport" and what about http2?

5 Likes

Do you mean something like this?

    listen          443;
    listen          [::]:443;
    listen          443 reuseport http3;
    listen          [::]:443 reuseport http3;

HTTP/2 seems to work great.

However there's one strange thing: to enable OSCP stapling with Letsencrypt I usually do:

ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;

and it works. But here I receive an error if I do so:

test-nginx  āŒ root  /etc/nginx  nginx -t
nginx: [emerg] Read OCSP response file "/etc/nginx/" failed: -1 (SSL: error:02000015:system library:OPENSSL_internal:Is a directory error:11000002:BIO routines:OPENSSL_internal:system library)
nginx: configuration file /etc/nginx/nginx.conf test failed

Therefore I disabled it now.

1 Like

I'm not getting any answer on port 443/UDP. To recall, HTTP/3 uses the QUIC protocol which uses UDP. While your TCP/TLS port + HTTP/2 seems fine, I cannot complete any QUIC handshake on 443/UDP. I used Quiche's client for my tests.

This may be due to firewalls, but I suspect that a missing reuseport directive may be the issue. Any documentation I've seen mentions that the UDP port (which seems to be enabled via the http3 command in the listen directive nowadays, it used to be called udp) requires that directive. I think that's because connection and socket management is a lot different on QUIC. However that seems to be missing in your nginx config.

8 Likes

I recall that OCSP stapling support in nginx with BoringSSL was broken. No idea if this is still the case, but if you're getting errors, probably yes.

6 Likes

443/UDP is closed. It's opened just the 443/TCP. I think you are right but why does this check say HTTP/3 is supported?

1 Like

I have no idea. Maybe the tool's broken? Sadly QUIC support is still early in many libs, and so tooling support isn't that great at the moment.

Does your nginx error log have anything to say about issues with the socket?

The listen directive I would expect to work is this one:

listen 443 ssl http2;
listen 443 http3 reuseport;  # UDP listener for QUIC+HTTP/3
listen [::]:443 ssl http2;
listen [::]:443 http3 reuseport;
5 Likes

Shame of me. Yeah, now that 443/UDP is opened it works :smiley: (that's definitely helped :D). However, I cannot still enable OCSP stapling as I usually do by putting:

ssl_stapling on;
ssl_stapling_verify on;

However, if I do:

ssl_stapling on;
ssl_stapling_verify on;
ssl_stapling_file /etc/nginx/isrg-root-ocsp-x1.der;

I receive no errors from nginx -t but ssl-labs.com returns no OCSP.

3 Likes

Yeah, I just re-read and it appears this is simply not supported by BoringSSL

This isn't doing what you think it does. This tells nginx to serve a stapled OCSP response you have aquired yourself. BoringSSL + nginx, at least with a patch, can send stapled OCSP, but you need to figure out how to aquire the responses first.

There's definetly progress! But quiche is still not happy:

cargo run --bin quiche-client -- https://dev.tecnicman.com/
     Running `target\debug\quiche-client.exe 'https://dev.tecnicman.com/'`
[2022-07-13T16:39:13.669191500Z ERROR quiche_apps::client] [::]:50815: recv failed: TlsFail
[2022-07-13T16:39:13.788542600Z ERROR quiche_apps::client] connection timed out after 3.1981937s
error: process didn't exit successfully: `target\debug\quiche-client.exe 'https://dev.tecnicman.com/'` (exit code: 0xffffffff)

Scratch that, it works on Linux, but not on Windows. Probably an issue with quiche.

7 Likes

However, must be something other wrong.

From Chrome on iOS I receivegg: ā€˜ ERR_INVALID_RESPONSEā€™

From Firefox on iOS: ā€˜cannot+parse+response NSURLErrorDomainā€™

Safari on iOS: ā€˜impossible to parse the responseā€™

Maybe should I try using OpenLiteSpeed? Does it work better on HTTP/3?

OCSP Stapling patch is already applied to the package.

That explains why ssl_stapling_file does not error, but this type of OCSP stapling will not require the OCSP response automatically - it will only serve OCSP responses on-disk, while aquiring them is left as an exercise to the admin.

I have no idea, I don't run any software from them. I also don't know the cause of your other issues, quiche did talk HTTP/3 just fine in the end. If you continue to have these problems, I suggest you might try seeking help elsewhere, as we're now pretty far off from Let's Encrypt services and other communities (such as those around OpenLiteSpeed) might be able to help you better.

7 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.