Edit: removed my previous answer.
I suspect that this might have something to do with your website being a PWA.
Perhaps when your Chrome installation first downloaded your site/manifest, it also recorded the characteristics of the SSL certificate.
Then when you re-visit the PWA, Chrome re-uses that information for its initial offline load.
And when you force a refresh, you actually perform a real SSL handshake and get updated SSL information.
(Alternatively, it might just be that the nature of a PWA just reliably triggers the generic behavior for cached browser responses - if you do not need to revalidate an HTTP resource, avoid opening any SSL connection and show the cached information).
One way to prove this may be to start Chrome with --profile-directory=blah (where your PWA wasn’t already downloaded) and see what happens.