SSLLabs saying "This server's certificate chain is incomplete."

Jukelyn · August 7, 2024, 5:59pm

My web server is (include version): nginx 1.27.0

The operating system my web server runs on is (include version):
Ubuntu 22.04.4 LTS x86_64 (5.15.0-116-generic)
Docker version 27.1.1, build 6312585
Docker Compose version v2.29.1

My hosting provider, if applicable, is: Myself

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): N/A

On most browsers I have https working with my site but on some other browers (Brave on mobile for example) I get a certificate not trusted message. I ran a test on SSLLabs and it says that I have an incomplete chain. Any tips and help appreciated.

MikeMcQ · August 7, 2024, 6:07pm

What ACME client program did you use to get your certificate?

Because your nginx server is only sending the "leaf" cert, not the leaf and intermediates. Your ACME Client is responsible for placing the correct file(s). On Certbot for example, you would use the fullchain.pem file (not just cert.pem), for the nginx ssl_certificate

You should probably remove your local IP from the public DNS system too. Use your hosts file to direct your local requests directly to your local server if you wish.

Jukelyn · August 7, 2024, 6:18pm

As for the ACME, I used Porkbun (my registrar). They have a feature to let you download a bundle with the following files: domain.cert.pem intermediate.cert.pem private.key.pem public.key.pem

I took domain.cert.pem and appended intermediate.cert.pem to it and renamed it to fullchain.pem. I then placed them into my nginx config file like this:

...
http {
    ...
    server {
        listen 443 ssl;
        server_name jukelyn.com;

        ssl_certificate /etc/nginx/ssl/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/private.key.pem;
        ssl_trusted_certificate /etc/nginx/ssl/intermediate.cert.pem;
        ...
        }
    ...

Also how come I should remove my local IP? I have it setup like that because when I go to my domain from my home network I can't access it, only via IP but I want to be able to access it via the domain as well. I'm not really stringent about it since it wouldn't really matter to me personally but I do wonder.

I guess I should also add that I tried using SSLForWeb and have that SSL cert on my testing site but it has the same issue as my main site so I'm confused. (I am not sure if my wildcard cert for jukelyn.com overwrites the one for test.jukelyn.com though)

MikeMcQ · August 7, 2024, 6:30pm

That sounds correct but that is not happening. Did you first just use the cert.pem? And maybe did not reload nginx after appending the intermediate to it?

If reloading nginx does not help please post contents of that fullchain.pem. Just be sure it is as you describe with the cert and intermediate. Those are publicly known. Your private key should never be shown.

MikeMcQ · August 7, 2024, 6:32pm

I don't know what you mean but the file named by nginx in ssl_certificate is the one it sends out during an HTTPS connection

Jukelyn · August 7, 2024, 6:32pm

I see, nevermind that then.

Jukelyn · August 7, 2024, 6:34pm

That's odd. Here is the fullchain.pem though:

The intermediate.pem is

MikeMcQ · August 7, 2024, 6:41pm

Huh. The intermediate.pem is for R3

But, the fullchain.pem has your leaf, followed by the R10 intermediate that signed your leaf, followed by a (now obsolete) R3

Yet, connections to your nginx see only your leaf.

SSL Labs is very good at showing all the pieces. My own test server also only sees the leaf.

I can't make sense of these pieces. Your nginx isn't using that fullchain.pem. You might try restarting nginx or even rebooting your server. Then try SSL Labs and refresh its cache

rg305 · August 7, 2024, 6:43pm

What shows?:

`nginx -T`

Jukelyn · August 7, 2024, 6:53pm

I see, unfortunately I have restarted nginx and rebooted the entire server a few time but to no avail.

Jukelyn · August 7, 2024, 6:55pm

It actually that it fails..

[Aug 7 02:52 PM] 26 juke:[main]$ sudo nginx -T
nginx: [emerg] getpwnam("nginx") failed in /etc/nginx/nginx.conf:1
nginx: configuration file /etc/nginx/nginx.conf test failed

this is my nginx config file:

[Aug 7 02:52 PM] 27 juke:[main]$ sudo cat /etc/nginx/nginx.conf 
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    server {
        listen 80;
        server_name jukelyn.com;

        location / {
            root /usr/share/nginx/html;
            index index.html;
    }
    }

    server {
        listen 443 ssl;
        server_name jukelyn.com;

        ssl_certificate /etc/nginx/ssl/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/private.key.pem;
        ssl_trusted_certificate /etc/nginx/ssl/intermediate.cert.pem;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:!aNULL:!MD5';
        ssl_prefer_server_ciphers on;

        ssl_stapling on;
        ssl_stapling_verify on;

        location / {
            root /usr/share/nginx/html;
            index index.html;
        }
    }
}

rg305 · August 7, 2024, 7:16pm

Is there a user defined for "nginx" ?

rg305 · August 7, 2024, 7:20pm

Some systems use this user for nginx:
user www-data;

You could review all your users with:
getent passwd | sort
[don't show that list here]

Jukelyn · August 7, 2024, 7:43pm

Ah! No there isn't!

Jukelyn · August 7, 2024, 8:12pm

@rg305

Correction: there isn't an nginx user on the host machine but the docker container that the webserver is running from has the nginx user on it...

docker exec -it jukelyn_nginx /bin/sh
# getent passwd
www-data:x:33:33:www-data:/var/www:/usr/sbin/nologin
nginx:x:101:101:nginx user:/nonexistent:/bin/false

(I removed most of the other entries except these since these are the relevant ones)

Jukelyn · August 7, 2024, 8:18pm

Within the docker container this is what is says:

[Aug 7 04:17 PM] 11 juke:[main]$ docker exec -it jukelyn_nginx /bin/sh
# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;

    # Define your server blocks here
    server {
        listen 80;
        server_name jukelyn.com;

        location / {
            root /usr/share/nginx/html;
            index index.html;
    }
    }

    server {
        listen 443 ssl;
        server_name jukelyn.com;

        ssl_certificate /etc/nginx/ssl/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/private.key.pem;
        ssl_trusted_certificate /etc/nginx/ssl/intermediate.cert.pem;

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:!aNULL:!MD5';
        ssl_prefer_server_ciphers on;

        ssl_stapling on;
        ssl_stapling_verify on;

        location / {
            root /usr/share/nginx/html;
            index index.html;
        }
    }
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

Let me know if this helps at all...

Osiris · August 7, 2024, 8:37pm

Are these paths/volumes mapped to outside of Docker? Or at least to the Docker where Certbot is running?

Jukelyn · August 7, 2024, 8:48pm

Osiris:

Jukelyn:
        ssl_certificate /etc/nginx/ssl/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/private.key.pem;
Are these paths/volumes mapped to outside of Docker? Or at least to the Docker where Certbot is running?

Yes, here is the docker-compose file for it:

...
  nginx:
    container_name: jukelyn_nginx
    image: nginx
    labels:
      - traefik.enable=true
      - traefik.http.routers.nginx.rule=Host(`jukelyn.com`) || Host(`www.jukelyn.com`)
      - traefik.http.routers.nginx.entrypoints=web,websecure
      - traefik.http.routers.nginx.tls=true
      - traefik.http.services.nginx.loadbalancer.server.port=80
    volumes:
      - ~/sites/mysite/public:/usr/share/nginx/html
      - /etc/nginx/nginx.conf:/etc/nginx/nginx.conf
      - /etc/nginx/ssl:/etc/nginx/ssl
    restart: always
    networks:
      - default
    environment:
      - TZ=America/New_York
...
networks:
  default:
    driver: bridge

btw I have tls enabled from traefik but that is actually just pointing to the same files.

  traefik:
    container_name: traefik
    image: traefik
    ports:
      - "80:80"
      - "443:443"
      - "8081:8080"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/traefik/traefik.yaml:/etc/traefik/traefik.yaml
      - /etc/traefik/certs/acme.json:/acme.json
      - /etc/nginx/ssl/fullchain.pem:/certs/fullchain.pem
      - /etc/nginx/ssl/intermediate.cert.pem:/certs/intermediate.cert.pem
      - /etc/nginx/ssl/private.key.pem:/certs/private.key.pem
    restart: unless-stopped
    environment:
      - TZ=America/New_York
    networks:
      - default
      - kuma_net
    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.traefik-ipallowlist.ipallowlist.sourcerange=127.0.0.1/32, 192.168.1.0/24"
      - "traefik.http.routers.traefik.rule=Host(`traefik.jukelyn.com`)"
      - "traefik.http.routers.traefik.entrypoints=websecure"
      - "traefik.http.routers.traefik.service=api@internal"
      - "traefik.http.routers.traefik.tls.certresolver=production"
      - "traefik.http.services.api.loadbalancer.server.port=8080"

(the acme.json is empty as well but it's there as placeholder)

Jukelyn · August 7, 2024, 8:50pm

Osiris:

Jukelyn:
        ssl_certificate /etc/nginx/ssl/fullchain.pem;
        ssl_certificate_key /etc/nginx/ssl/private.key.pem;
Are these paths/volumes mapped to outside of Docker? Or at least to the Docker where Certbot is running?

I don't have Certbot running as of yet. I wanted to figure out my issues before that so that when I do set it up, I will be able to know that it's working correctly and then forget about it.

Jukelyn · August 12, 2024, 4:54am

It's been a few days and I still haven't figured out this issue. If anybody is able to provide any additional insight, it would be much appreciated!