Thanks, this does indeed seem to be the case. This did involve a jump from Debian 10 to Debian 12 and a corresponding jump in OpenSSL, but it seems like it should be fixable.
I did notice in the packet captures, a successful handshake results in the following cipher being chosen:
TLS_RSA_WITH_AES_256_CBC_SHA (0x0035)
This is the third cipher in the Client Hello list of ciphers, interestingly.
In the failing handshake, the Server Hello contains TLS_DHE_RSA_WITH_AES_256_CBC_SHA (0x0039), which is the first cipher in the Client Hello, but the server hello is immediately followed by the Fatal, Internal Error packet.
Even more interestingly, according to nmap, in the working scenario, it's client cipher preference and in the nonworking, it's server cipher preference, though the test above seems to suggest the opposite.
I'm trying to see if there's a way to force TLS_RSA_WITH_AES_256_CBC_SHA, since I know that works, but I've been having difficulty in picking that out. It doesn't show up in the output of openssl s_client -cipher on either machine:
AES128-GCM-SHA256 DHE-PSK-AES256-CBC-SHA384 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-PSK-AES256-CBC-SHA384 PSK-AES128-CBC-SHA256 RSA-PSK-AES256-CBC-SHA384
AES128-SHA DHE-PSK-AES256-GCM-SHA384 ECDHE-ECDSA-AES128-SHA ECDHE-PSK-CHACHA20-POLY1305 PSK-AES128-GCM-SHA256 RSA-PSK-AES256-GCM-SHA384
AES128-SHA256 DHE-PSK-CHACHA20-POLY1305 ECDHE-ECDSA-AES128-SHA256 ECDHE-RSA-AES128-GCM-SHA256 PSK-AES256-CBC-SHA RSA-PSK-CHACHA20-POLY1305
AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES256-GCM-SHA384 ECDHE-RSA-AES128-SHA PSK-AES256-CBC-SHA384 SRP-AES-128-CBC-SHA
AES256-SHA DHE-RSA-AES128-SHA ECDHE-ECDSA-AES256-SHA ECDHE-RSA-AES128-SHA256 PSK-AES256-GCM-SHA384 SRP-AES-256-CBC-SHA
AES256-SHA256 DHE-RSA-AES128-SHA256 ECDHE-ECDSA-AES256-SHA384 ECDHE-RSA-AES256-GCM-SHA384 PSK-CHACHA20-POLY1305 SRP-RSA-AES-128-CBC-SHA
DHE-PSK-AES128-CBC-SHA DHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-CHACHA20-POLY1305 ECDHE-RSA-AES256-SHA RSA-PSK-AES128-CBC-SHA SRP-RSA-AES-256-CBC-SHA
DHE-PSK-AES128-CBC-SHA256 DHE-RSA-AES256-SHA ECDHE-PSK-AES128-CBC-SHA ECDHE-RSA-AES256-SHA384 RSA-PSK-AES128-CBC-SHA256 TLS_AES_128_GCM_SHA256
DHE-PSK-AES128-GCM-SHA256 DHE-RSA-AES256-SHA256 ECDHE-PSK-AES128-CBC-SHA256 ECDHE-RSA-CHACHA20-POLY1305 RSA-PSK-AES128-GCM-SHA256 TLS_AES_256_GCM_SHA384
DHE-PSK-AES256-CBC-SHA DHE-RSA-CHACHA20-POLY1305 ECDHE-PSK-AES256-CBC-SHA PSK-AES128-CBC-SHA RSA-PSK-AES256-CBC-SHA TLS_CHACHA20_POLY1305_SHA256
The applications themselves show what ciphers are available, too, but even there I don't see it listed exactly:
TLS_AES_256_GCM_SHA384, TLS_CHACHA20_POLY1305_SHA256, TLS_AES_128_GCM_SHA256, ECDHE-ECDSA-AES256-GCM-SHA384, ECDHE-RSA-AES256-GCM-SHA384, DHE-DSS-AES256-GCM-SHA384, DHE-RSA-AES256-GCM-SHA384, ECDHE-ECDSA-CHACHA20-POLY1305, ECDHE-RSA-CHACHA20-POLY1305, DHE-RSA-CHACHA20-POLY1305, ECDHE-ECDSA-AES256-CCM8, ECDHE-ECDSA-AES256-CCM, DHE-RSA-AES256-CCM8, DHE-RSA-AES256-CCM, ECDHE-ECDSA-ARIA256-GCM-SHA384, ECDHE-ARIA256-GCM-SHA384, DHE-DSS-ARIA256-GCM-SHA384, DHE-RSA-ARIA256-GCM-SHA384, ADH-AES256-GCM-SHA384, ECDHE-ECDSA-AES128-GCM-SHA256, ECDHE-RSA-AES128-GCM-SHA256, DHE-DSS-AES128-GCM-SHA256, DHE-RSA-AES128-GCM-SHA256, ECDHE-ECDSA-AES128-CCM8, ECDHE-ECDSA-AES128-CCM, DHE-RSA-AES128-CCM8, DHE-RSA-AES128-CCM, ECDHE-ECDSA-ARIA128-GCM-SHA256, ECDHE-ARIA128-GCM-SHA256, DHE-DSS-ARIA128-GCM-SHA256, DHE-RSA-ARIA128-GCM-SHA256, ADH-AES128-GCM-SHA256, ECDHE-ECDSA-AES256-SHA384, ECDHE-RSA-AES256-SHA384, DHE-RSA-AES256-SHA256, DHE-DSS-AES256-SHA256, ECDHE-ECDSA-CAMELLIA256-SHA384, ECDHE-RSA-CAMELLIA256-SHA384, DHE-RSA-CAMELLIA256-SHA256, DHE-DSS-CAMELLIA256-SHA256, ADH-AES256-SHA256, ADH-CAMELLIA256-SHA256, ECDHE-ECDSA-AES128-SHA256, ECDHE-RSA-AES128-SHA256, DHE-RSA-AES128-SHA256, DHE-DSS-AES128-SHA256, ECDHE-ECDSA-CAMELLIA128-SHA256, ECDHE-RSA-CAMELLIA128-SHA256, DHE-RSA-CAMELLIA128-SHA256, DHE-DSS-CAMELLIA128-SHA256, ADH-AES128-SHA256, ADH-CAMELLIA128-SHA256, ECDHE-ECDSA-AES256-SHA, ECDHE-RSA-AES256-SHA, DHE-RSA-AES256-SHA, DHE-DSS-AES256-SHA, DHE-RSA-CAMELLIA256-SHA, DHE-DSS-CAMELLIA256-SHA, AECDH-AES256-SHA, ADH-AES256-SHA, ADH-CAMELLIA256-SHA, ECDHE-ECDSA-AES128-SHA, ECDHE-RSA-AES128-SHA, DHE-RSA-AES128-SHA, DHE-DSS-AES128-SHA, DHE-RSA-SEED-SHA, DHE-DSS-SEED-SHA, DHE-RSA-CAMELLIA128-SHA, DHE-DSS-CAMELLIA128-SHA, AECDH-AES128-SHA, ADH-AES128-SHA, ADH-SEED-SHA, ADH-CAMELLIA128-SHA, RSA-PSK-AES256-GCM-SHA384, DHE-PSK-AES256-GCM-SHA384, RSA-PSK-CHACHA20-POLY1305, DHE-PSK-CHACHA20-POLY1305, ECDHE-PSK-CHACHA20-POLY1305, DHE-PSK-AES256-CCM8, DHE-PSK-AES256-CCM, RSA-PSK-ARIA256-GCM-SHA384, DHE-PSK-ARIA256-GCM-SHA384, AES256-GCM-SHA384, AES256-CCM8, AES256-CCM, ARIA256-GCM-SHA384, PSK-AES256-GCM-SHA384, PSK-CHACHA20-POLY1305, PSK-AES256-CCM8, PSK-AES256-CCM, PSK-ARIA256-GCM-SHA384, RSA-PSK-AES128-GCM-SHA256, DHE-PSK-AES128-GCM-SHA256, DHE-PSK-AES128-CCM8, DHE-PSK-AES128-CCM, RSA-PSK-ARIA128-GCM-SHA256, DHE-PSK-ARIA128-GCM-SHA256, AES128-GCM-SHA256, AES128-CCM8, AES128-CCM, ARIA128-GCM-SHA256, PSK-AES128-GCM-SHA256, PSK-AES128-CCM8, PSK-AES128-CCM, PSK-ARIA128-GCM-SHA256, AES256-SHA256, CAMELLIA256-SHA256, AES128-SHA256, CAMELLIA128-SHA256, ECDHE-PSK-AES256-CBC-SHA384, ECDHE-PSK-AES256-CBC-SHA, SRP-DSS-AES-256-CBC-SHA, SRP-RSA-AES-256-CBC-SHA, SRP-AES-256-CBC-SHA, RSA-PSK-AES256-CBC-SHA384, DHE-PSK-AES256-CBC-SHA384, RSA-PSK-AES256-CBC-SHA, DHE-PSK-AES256-CBC-SHA, ECDHE-PSK-CAMELLIA256-SHA384, RSA-PSK-CAMELLIA256-SHA384, DHE-PSK-CAMELLIA256-SHA384, AES256-SHA, CAMELLIA256-SHA, PSK-AES256-CBC-SHA384, PSK-AES256-CBC-SHA, PSK-CAMELLIA256-SHA384, ECDHE-PSK-AES128-CBC-SHA256, ECDHE-PSK-AES128-CBC-SHA, SRP-DSS-AES-128-CBC-SHA, SRP-RSA-AES-128-CBC-SHA, SRP-AES-128-CBC-SHA, RSA-PSK-AES128-CBC-SHA256, DHE-PSK-AES128-CBC-SHA256, RSA-PSK-AES128-CBC-SHA, DHE-PSK-AES128-CBC-SHA, ECDHE-PSK-CAMELLIA128-SHA256, RSA-PSK-CAMELLIA128-SHA256, DHE-PSK-CAMELLIA128-SHA256, AES128-SHA, SEED-SHA, CAMELLIA128-SHA, PSK-AES128-CBC-SHA256, PSK-AES128-CBC-SHA, PSK-CAMELLIA128-SHA256, ECDHE-ECDSA-NULL-SHA, ECDHE-RSA-NULL-SHA, AECDH-NULL-SHA, NULL-SHA256, ECDHE-PSK-NULL-SHA384, ECDHE-PSK-NULL-SHA256, ECDHE-PSK-NULL-SHA, RSA-PSK-NULL-SHA384, RSA-PSK-NULL-SHA256, DHE-PSK-NULL-SHA384, DHE-PSK-NULL-SHA256, RSA-PSK-NULL-SHA, DHE-PSK-NULL-SHA, NULL-SHA, NULL-MD5, PSK-NULL-SHA384, PSK-NULL-SHA256, PSK-NULL-SHA, DEFAULT, @SECLEVEL=1, @SECLEVEL=2, @SECLEVEL=3, @SECLEVEL=4, @SECLEVEL=5'
This one above is from the working machine, so somehow the available ciphers must make it work. And this actually does show up on the nonworking machine, but that isn't the cipher the server chooses. I'm thinking that the server is choosing the wrong cipher here, and for some reason, one that's not compatible with the client, but it seems strange I can't force this particular cipher just to make it work - does it go by another name I can use? Been having trouble finding what else it could be in these lists.