Some clients unable to get local issuer certificate after latest renewal

Help
1

My domain is: git.ecodoo.eu
I ran this command: echo | openssl s_client -connect git.ecodoo.eu:443 -servername git.ecodoo.eu

It produced this output:

Only from a few broken clients (same OS, same openssl version):

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = git.ecodoo.eu
verify error:num=10:certificate has expired
notAfter=Oct 8 09:36:01 2023 GMT
verify return:1
depth=0 CN = git.ecodoo.eu
notAfter=Oct 8 09:36:01 2023 GMT
verify return:1
-- Certificate chain --
0 s:CN = git.ecodoo.eu
i:C = US, O = Let's Encrypt, CN = R3
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jul 10 09:36:02 2023 GMT; NotAfter: Oct 8 09:36:01 2023 GMT
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
i:O = Digital Signature Trust Co., CN = DST Root CA X3
a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT
Server certificate
-----BEGIN CERTIFICATE-----
MIIE6DCCA9CgAwIBAgISBGncszgTRA2ZKTlHneOWKd5FMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMzA3MTAwOTM2MDJaFw0yMzEwMDgwOTM2MDFaMBgxFjAUBgNVBAMT
DWdpdC5lY29kb28uZXUwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3
cy8nBL5SPIY0Cb00bnwmHpi2ymSZmkxgPJNXMUsdf/oGOJGmUCRvYmvvES7NTGOe
eF/beNWI61dhbuta4jhLSdV7A5/BAxZl5wwFlthP1JnlaPsEybNEGv4Cf/bbVS8D
+7Jus/Z+fwiKTAqADtGy3vg8VF6ztSt/51Tm3CgfEUAWRUxi6JMGRcx+7wdJj+zn
KiR6lOSLQcS6xVo5ffFnmAkmZW/b0cBR3RCLgjYYujMEY6rziiLYjN84h+MFrGOf
I5ZA+UCWjm3xA1XH20oz5kly2Mc9XaFG3lx7MAENFCyqa/4ABk2h7SZi5e1BKZAC
ecQSCDkdxmYsQ3qWKxVHAgMBAAGjggIQMIICDDAOBgNVHQ8BAf8EBAMCBaAwHQYD
VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwGA1UdEwEB/wQCMAAwHQYDVR0O
BBYEFBzw6+bNLzw/B1xXAJ+HrCunqURSMB8GA1UdIwQYMBaAFBQusxe3WFbLrlAJ
QOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAhBggrBgEFBQcwAYYVaHR0cDovL3Iz
Lm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZodHRwOi8vcjMuaS5sZW5jci5vcmcv
MBgGA1UdEQQRMA+CDWdpdC5lY29kb28uZXUwEwYDVR0gBAwwCjAIBgZngQwBAgEw
ggEFBgorBgEEAdZ5AgQCBIH2BIHzAPEAdwC3Pvsk35xNunXyOcW6WPRsXfxCz3qf
NcSeHQmBJe20mQAAAYk/X2APAAAEAwBIMEYCIQDdeQDOexOIUhVvMCjpQ/wzIRpT
waakexvul9ug94GcoAIhAIG7tbERZ1sO5ICPzP6jLrk5U0GjJpnSNKwTf9G0xnFu
AHYArfe++nz/EMiLnT2cHj4YarRnKV3PsQwkyoWGNOvcgooAAAGJP19iOgAABAMA
RzBFAiEAi2h9GIfErZj9ixRu8/LZUiRMbvkWNzZ/uPCYO6CHne4CIEhQeNG0we3P
Z2WcEMzXfL/0idiX5jLnDvdnQXfp4uV/MA0GCSqGSIb3DQEBCwUAA4IBAQAI1UkO
5x6EQyf48LIB4cN3iINqGZd+6w19Z0LSmJxi7C5xEbxrcOgB0s44tBoDiER2S/YE
PmCwkqKlaQ2Wnv3fwzKYv4Dcn/grX/IO+O6jKlFeoiLSZkueQ3sJrswzAB34BXNu
n+FRi/JD52iY/A3w2+FWCDkwiu7r0zj7tAfYd/bgnMZHZClHf445lY2WN/zIA9g0
LBIjcH2RqYqX4lFZ/bWePQgvr5sRforfZKKOlQ/WrIp8E/2d3kGqZcP6vQZSufYh
Ng+60Dal29M2flzChIFaz825cXnSKRR++vopVuRd61SdWcUCMTLZSVkoSVXq0YxZ
/E1zeXLL3+dXrq6+
-----END CERTIFICATE-----
subject=CN = git.ecodoo.eu
issuer=C = US, O = Let's Encrypt, CN = R3
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
SSL handshake has read 4516 bytes and written 395 bytes
Verification error: certificate has expired
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 10 (certificate has expired)

Other working clients return:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R11
verify return:1
depth=0 CN = git.ecodoo.eu
verify return:1
--- Certificate chain ---
0 s:CN = git.ecodoo.eu
i:C = US, O = Let's Encrypt, CN = R11
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Aug 13 00:57:53 2024 GMT; NotAfter: Nov 11 00:57:52 2024 GMT
1 s:C = US, O = Let's Encrypt, CN = R11
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
Server certificate
-----BEGIN CERTIFICATE-----
MIIFEDCCA/igAwIBAgISA7uXXPF4m3xKK4Cr3SVoUsVlMA0GCSqGSIb3DQEBCwUA
MDMxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQwwCgYDVQQD
EwNSMTEwHhcNMjQwODEzMDA1NzUzWhcNMjQxMTExMDA1NzUyWjAYMRYwFAYDVQQD
Ew1naXQuZWNvZG9vLmV1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA
4zMT2BWVDMglws73+29Aim1JldSKrKkhdorP+kzlZoxdblF8iPv5d39mjkznETSS
LQLrwDyDxceA4T7/TVT/+0UEUA4f7ebBpuTrhPxk6ieeEQJlDfaP28rlc6hJmPoa
4O8LN4Bb/gm3YCrjFMcYbpfGdEfwKGDdn5caOr12KhwCHWKEA91jtYinxdP1dPLd
qgJKEAuE7edDpjgGcZZz6YOf302ybLjKBOlD6jmYVKEsW7HH3bYeaQyTR6iUv1NQ
xGsXZ2T8sgRwKPT+CT7RnrG+86rvh6//cOsncjiKL59YGDtkc+ZpRCakCKtYxkeL
5T2G2ls33WD1NAlWMoW06wIDAQABo4ICNzCCAjMwDgYDVR0PAQH/BAQDAgWgMB0G
A1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0GA1Ud
DgQWBBSClnDfavzjnsrPIkfazv87DDAKIzAfBgNVHSMEGDAWgBTFz0ak6vTDwHps
lcQtsF6SLybjuTBXBggrBgEFBQcBAQRLMEkwIgYIKwYBBQUHMAGGFmh0dHA6Ly9y
MTEuby5sZW5jci5vcmcwIwYIKwYBBQUHMAKGF2h0dHA6Ly9yMTEuaS5sZW5jci5v
cmcvMD0GA1UdEQQ2MDSCDWdpdC5lY29kb28uZGWCDWdpdC5lY29kb28uZXWCFGdp
dGxhYi5lY29zZXJ2aWNlLmRlMBMGA1UdIAQMMAowCAYGZ4EMAQIBMIIBBQYKKwYB
BAHWeQIEAgSB9gSB8wDxAHYAdv+IPwq2+5VRwmHM9Ye6NLSkzbsp3GhCCp/mZ0xa
OnQAAAGRSXNg8wAABAMARzBFAiEAuAm08ynygJJAum2as1bRui7kvRyM/1wwLqFT
p/q6FV4CIA0nhV6XZNaoBjFmtpouCiBunIMZDbpeUWBAz6PWGfdfAHcA3+FW66oF
r7WcD4ZxjajAMk6uVtlup/WlagHRwTu+UlwAAAGRSXNhvgAABAMASDBGAiEAvebR
3Z8vCHum16toXjKbypCZgmnyVOUC//C2BZvt7fgCIQCls5/z0ghFQe6l9OaSTlds
UTJwzgLyKPwMghBmRxUCajANBgkqhkiG9w0BAQsFAAOCAQEAelhA/Wo1qnbL18ON
sNPUnG5B+Gyk4WNaHDb7HRrdFe8rQrKA1t+1AyTYUAwXblO+cjFe+XHCQKHLV1b2
qmsTJGyDzl/xOBlpDQlbdJexYbYwtt1oWoUg4Ij8rOoUlg3D2ovrKQvI4IEL7Mm1
jSy581T23BGkIlu6sMBg66FQk8enxB29YJAJSnxxpXmhA73sX1H8bVA1uNrUPxyV
X5bpa4bGmstaQfMO7z4vDzhMqKVBKIlYPBgEVo7S4j0LyP3WgKqW6GSuObU1y/dl
2bH4hG3AveFTlEHK7KXVw/NOLHhLZoY8XLo0bkQJg6tbJ9LA7FmBEp5mPO0N3chT
u8dIpQ==
-----END CERTIFICATE-----
subject=CN = git.ecodoo.eu
issuer=C = US, O = Let's Encrypt, CN = R11
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
SSL handshake has read 3155 bytes and written 395 bytes
Verification: OK
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
DONE

  • Webserver: nginx version: nginx/1.18.0 (Ubuntu), built with OpenSSL 3.0.2 15 Mar 2022, TLS SNI support enabled
  • Operating System: Ubuntu 22.04.4 LTS
  • Root shell access: yes
  • I'm using a control panel to manage my site: no
  • The version of my client is: certbot 1.21.0

My first question is if this is a client or a server issue?

I noticed, that the clients that get an expired certificate, get it from "i:C = US, O = Let's Encrypt, CN = R3" which seems expired according to Chains of Trust - Let's Encrypt while the working clients get it from "C = US, O = Let's Encrypt, CN = R11" .. however, I have no idea how to make the "broken" clients use the correct chain.

The server should provide the fullchain (nginx config):

ssl_client_certificate /etc/letsencrypt/live/git.ecodoo.eu/fullchain.pem;
ssl_certificate /etc/letsencrypt/live/git.ecodoo.eu/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/git.ecodoo.eu/privkey.pem;
ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384';
ssl_protocols TLSv1.2 TLSv1.3;

However, I noticed that the fullchain.pem only contains 2 certificates instead of 3 which - i guess - has to do with Shortening the Let's Encrypt Chain of Trust - Let's Encrypt ?

What am I missing? Any form of help or hints would be appreciated :wink:

1 Like
2

Welcome to the community @Armitxes

I think the problem is your DNS. You have both an A and an AAAA record defined for IPv4 and IPv6. But, these point to different servers or at least give different responses.

Your IPv6 looks correct but IPv4 is not. See results at SSL Labs
https://www.ssllabs.com/ssltest/analyze.html?d=git.ecodoo.eu&hideResults=on&latest

6 Likes
3

Thanks for the welcome and feedback.

That's a bit weird. We have another domain with the same configuration
https://www.ssllabs.com/ssltest/analyze.html?d=gitlab.ecoservice.de&hideResults=on&latest
https://www.ssllabs.com/ssltest/analyze.html?d=git.ecodoo.eu&hideResults=on&latest
both return the same IPv6 and IPv4 address yet one is successful while the other one fails

Both use the exact same nginx configuration and Let's Encrypt certificate.

server {
server_name gitlab.ecoservice.de;
include /etc/nginx/gitlab_locations.conf;
}
server {
server_name git.ecodoo.eu;
include /etc/nginx/gitlab_locations.conf;
}

Trying to get access to the DNS Entires of gitlab.ecoservice.de to compare, but it's only a simple A record entry pointing from a domain to an IP, there shouldn't be much going wrong there.

1 Like
4

I do not see any listen statement in that nginx server block you show. Can you post the contents of that include file?

3 Likes
5

Sure, here it is: gitlab_locations.conf ($39) · Snippets · GitLab

6

A bit of a puzzler. The DNS A and AAAA records are the same so that is not the source of the problem.

But, the IPv4 HTTP requests to port 80 for git.ecodoo.eu uses an expired cert. And, one that expired nearly a year ago.

So, something is either intercepting IPv4 requests before it reaches your nginx. Or, something in your nginx is grabbing those requests. Can you upload the config.txt file from this command?

sudo nginx -T >config.txt

An uppercase T is esential. The config.txt file will be long.

5 Likes
7

:man_facepalming: Thank you so much, thanks to your hint to the config (which took me a bit to find at /var/opt/gitlab/nginx/conf/nginx.conf).

Here is the config: config.txt ($40) · Snippets · GitLab and I'm really not a fan of how it's generated.

While include /etc/nginx/gitlab.conf; is my config, what immediately spotted by eye is include /var/opt/gitlab/nginx/conf/gitlab-http.conf; : gitlab-http.conf ($41) · Snippets · GitLab

Inside that file: Different certificates :man_facepalming: ... I commented these lines out in the gitlab.rb file years ago but it doesn't seem to have any effect when reconfiguring. Only thing that still confuses me, is how gitlab.ecoservice.de worked different from git.ecodoo.eu and how it was working in the past at all...

Commenting in and fixing the path within gitlab.rb followed by the gitlab-ctl reconfigure command solved the issue: SSL Server Test: git.ecodoo.eu (Powered by Qualys SSL Labs)

nginx['ssl_certificate'] = "/etc/letsencrypt/live/git.ecodoo.eu/fullchain.pem"
nginx['ssl_certificate_key'] = "/etc/letsencrypt/live/git.ecodoo.eu/privkey.pem"

Will try to get a proper nginx config generated by gitlab, or even better, try to get GitLab working with the standalone nginx service completly disabling gitlab nginx.

Thank you again so much @MikeMcQ for your patience and help. This issue is solved.

5 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.