Generated cert rejected for "bad signature"

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: causeuse.org

I ran this command:

% openssl s_client -connect oak.causeuse.org:443 -servername www.causeuse.org

It produced this output:
[....]
FFFFFFFFFFFFFFFF:error:02000068:rsa routines:RSA_verify_PKCS1_PSS_mgf1:bad signature:/u/sources/netbsd-10/src/crypto/external/bsd/openssl/dist/crypto/rsa/rsa_pss.c:132:
FFFFFFFFFFFFFFFF:error:1C880004:Provider routines:rsa_verify:RSA lib:/u/sources/netbsd-10/src/crypto/external/bsd/openssl/dist/providers/implementations/signature/rsa_sig.c:815:
FFFFFFFFFFFFFFFF:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:/u/sources/netbsd-10/src/crypto/external/bsd/openssl/dist/ssl/statem/statem_lib.c:538:
[...]

My web server is (include version): Hiawatha 11.6

The operating system my web server runs on is (include version): NetBSD 10

My hosting provider, if applicable, is: n/a (VPS)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): uacme 1.7.5

Hi,

in an installation that worked fine for the first three months, I now get an SEC_ERROR_BAD_SIGNATURE from Firefox, and the same result from the above openssl(1) incantation.

An uacme(8) run renews the cert properly, it's just the result isn't accepted.

Funny enough, an ancient Safari 13 is just fine with the server cert, so it's not complete breakage.

The moduli of the server's private key and public cert match.

letsdebug.net has no complaints.

I have exhausted things to try for now - does anybody see where I took a wrong turn?

Cheerio,
Hauke

1 Like
3

A different volunteer should be able to give more details about this specific error.

I just wanted to clarify the '3 months'. Do you mean it worked back in April but began to fail starting Jun30? Because your cert in Apr used the R3 intermediate but starting Jun30 uses R10/R11.

Let's Encrypt changed the issuing Intermediates from R3 to randomly chosen R10 or R11 (and possibly others). I see you return the cert using the R10 intermediate last used. So, not sure why you'd get this TLS connect failure. I'd guess it's related but, as I said, a different volunteer will need to be involved.

I was also puzzled why you used that format of openssl command. Even without that signature error it would fail with cert validation since no cert was issued that includes your oak subdomain. But, even this openssl which should validate fails with same error

echo | openssl s_client -connect causeuse.org:443 | head

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R10
verify return:1
depth=0 CN = causeuse.org
verify return:1
800BC38E0F770000:error:02000068:rsa routines:RSA_verify_PKCS1_PSS_mgf1:bad signature:../crypto/rsa/rsa_pss.c:132:
800BC38E0F770000:error:1C880004:Provider routines:rsa_verify:RSA lib:../providers/implementations/signature/rsa_sig.c:815:
800BC38E0F770000:error:0A00007B:SSL routines:tls_process_cert_verify:bad signature:../ssl/statem/statem_lib.c:537:
CONNECTED(00000003)
5 Likes
4

The reason old browsers work is that TLSv1.2 works fine. However, when connecting via TLSv1.3, your server sends an invalid signature in the TLS CertificateVerify message.

TLSv1.2 uses a different algorithm (rsa_pkcs1_sha256) compared to TLSv1.3 (rsa_pss_rsae_sha256), and apparently one works while the other doesn't.

I can reproduce the invalid signatures. They have the correct length, but not the correct value. This is likely a bug in your server software, operating system, or hardware.

PS: Since that cert is fairly fresh, have you tried restarting your system? It is possible that your server software somehow mixed up an older private key with a newer certificate.

6 Likes
5

Hi @hfath,

What version of openssl?
Please share the output from this command openssl version

4 Likes
6

@Nummer378 Do these changelog notes shed any light on the issue?
From: About - Hiawatha webserver

8.0: January 2012, ... OpenSSL replaced with mbed TLS.
...
11.0: July 2021, mbed TLS library updated to v3.0.0.

@hfath Were there any changes to this server or system about the time it started failing?

4 Likes
7

Also @hfath how did you get or install Hiawatha?
Did you build it from the source code?
https://hiawatha.leisink.net/download

Edit

Also assuming you built it from source, which build options did you enable / disable?

This one kind of jumps out to me
" -DENABLE_TLS=ON off Enable TLS (mbed TLS) support in Hiawatha."

4 Likes
8

Also interesting: TLSv1.3 support in Hiawatha is quite recent, just added in the latest release:

image
image526×225 5.93 KB

May be worthwhile to report this upstream, possibly a bug on their end.

5 Likes
9 
% openssl version
OpenSSL 3.0.12 24 Oct 2023 (Library: OpenSSL 3.0.12 24 Oct 2023)
%

as shipped with netbsd-10.

1 Like
10

I installed from the pkgsrc package (maintained by yours truly).

TLS support is enabled.

1 Like
11

Hmmm... the certificate started to fail after the first renewal, which I didn't really notice at the time, being swamped with real-world family problems.

I updated the package to 11.6 in August, but may well not have looked at the ssl functionality at the time, since the server is still under construction.

So you may be onto something here - thanks, hadn' thought of that.

3 Likes
12

Forcing TLS 1.2 with -tls1_2 makes OpenSSL connect succesfully. With -tls1_3 it's indeed broken.

4 Likes
13

Here are the Security Advisories for Mbed TLS.

This one is recent 30 August 2024 CVE-2024-45159
Limited authentication bypass in TLS 1.3 optional client authentication

Edit

The Work-around is "The issue can be avoided by forcing use of TLS 1.2 with connections that use optional client authentication. That is, all servers that call mbedtls_ssl_conf_authmode(..., MBEDTLS_SSL_VERIFY_OPTIONAL); should also call mbedtls_ssl_conf_max_tls_version(..., MBEDTLS_SSL_VERSION_TLS1_2); ."

And that seem to be in alignment with Osiris' observation.

2 Likes
14

Greping through the sources, Hiawatha doesn't do that.

I tried disabling TLS 1.3 support in the enclosed mbedtls sources, but that ended up disabling all TLS support... I guess I need to reach out to Hiawatha's upstream about an mbedtls update.

2 Likes
15

Lowering the max. TLS version to 1.2 as per mbedtls upstream's ticket works around the immediate issue.

2 Likes
16

Hi @hfath, yeah I was suppling more supplemental information; if it assisted great.
But at last the information didn't prove to be of any real assistance, sorry! :frowning:

3 Likes
17

Well, it helped, in that the other (dynamic) way of limiting the TLS version worked.

So you got me thinking - thanks!

4 Likes