Support mod_gnutls with Apache

Good catch!

New line:
GnuTLSPriorities PFS:-VERS-ALL:+VERS-TLS1.3:+VERS-TLS1.2:+VERS-DTLS1.2:-GROUP-ALL:+GROUP-X25519:+GROUP-X448:+GROUP-SECP521R1:+GROUP-SECP384R1:+GROUP-SECP256R1:-SHA1:-MD5:-AES-256-CBC:-AES-256-CCM:-AES-128-CBC:-AES-128-CCM:-CIPHER-ALL:+CHACHA20-POLY1305:+AES-256-GCM:+AES-128-GCM:+SHA384:+SHA256:+AES-256-GCM:+ECDHE-ECDSA:+ECDHE-RSA:%SERVER_PRECEDENCE

1 Like

Sorry. This was my confusion. What does --apache do combined with --webroot?

2 Likes

In my case: nothing, it returns an error:

Too many flags setting configurators/installers/authenticators 'apache' -> 'webroot'

:rofl:

But the combination -i apache --webroot does seem to work :slight_smile:

Although the whole issue here was that the apache installer doesn't handle GnuTLS, so this wouldn't be an option for you.

What @griffin tried to say, I think, is that you could have used --apache in combination with the certonly subcommand, which would ignore the non-functional (due to GnuTLS) installer part of the apache plugin, but would enable the authenticator part. This would of course mean you wouldn't be using the webroot authenticator.

2 Likes

I can't seem to get much right here :smiley: Sorry!

What is the difference between certbot certonly -i apache --webroot and certbot certonly --webroot? I used the latter which seems to work fine for me. I do get an error Could not find ssl_module; not disabling session tickets. with the `-i apache' option.

# certbot certonly -i apache --webroot --dry-run -w /var/www/domains/tnonline.net/htdocs/ -d tnonline.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Could not find ssl_module; not disabling session tickets.
Simulating renewal of an existing certificate for tnonline.net
The dry run was successful.

NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.
# certbot certonly --webroot --dry-run -w /var/www/domains/tnonline.net/htdocs/ -d tnonline.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating renewal of an existing certificate for tnonline.net
The dry run was successful.

NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

I do have a warning in my apache log about OSCP stapling:

[Sun Jul 04 17:48:30.645502 2021] [gnutls:warn] [pid 9196:tid 140017591871360] Could not create OCSP stapling configuration for certificate 1 in chain (CN=R3,O=Let's Encrypt,C=US): No OCSP URI in the certificate nor a GnuTLSOCSPResponseFile setting, cannot configure OCSP stapling.
2 Likes

When using the apache installer plugin, certbot gives the user a menu where it can choose hostnames from the Apache configuration. But the certonly subcommand will tell certbot not to actually install the certificate after issuance. The apache installer plugin is this case only used to retrieve the hostnames from the Apache configuration. With certbot certonly --webroot, you'd have to enter the hostnames yourself manually.

The above is only the case when not using the -d option by the way.

With the -d option, combining certonly with -i apache is quite useless I would say.

That's correct: the R3 intermediate certificate does not contain an OCSP URI. Your end leaf certificate however does, so OCSP should work for that cert.

4 Likes

Not sure I understand what a leaf certificate is. Is it not the fullchain.pem that is loaded with

GnuTLSCertificateFile   /etc/letsencrypt/live/tnonline.net/fullchain.pem

Trying oscptool with fullchain.pem fails:

 # ocsptool  --ask --load-cert cert.pem --load-issuer fullchain.pem --load-signer fullchain.pem
Connecting to OCSP server: r3.o.lencr.org...
Resolving 'r3.o.lencr.org:80'...
Connecting to '2a02:26f0:18::5c7b:9b9a:80'...
OCSP Response Information:
        Response Status: unauthorized

-----BEGIN OCSP RESPONSE-----
MAMKAQY=
-----END OCSP RESPONSE-----

Verifying OCSP Response: ASN1 parser: Value was not found.

Using the shorter chain.pem it works:

# ocsptool  --ask --load-cert cert.pem --load-issuer chain.pem --load-signer chain.pem
OCSP Response Information:
        Response Status: Successful
        Response Type: Basic OCSP Response
        Version: 1
        Responder ID: CN=R3,O=Let's Encrypt,C=US
        Produced At: Sat Jul 03 02:10:00 UTC 2021
        Responses:
                Certificate ID:
                        Hash Algorithm: SHA1
                        Issuer Name Hash: 48dac9a0fb2bd32d4ff0de68d2f567b735f9b3c4
                        Issuer Key Hash: 142eb317b75856cbae500940e61faf9d8b14c2c6
                        Serial Number: 04a22feac6122ced6ea93ea2f95e61e745b8
                Certificate Status: good
                This Update: Sat Jul 03 02:00:00 UTC 2021
                Next Update: Sat Jul 10 02:00:00 UTC 2021
        Extensions:
        Signature Algorithm: RSA-SHA256

But the problem is that I cannot use chain.pem instead of fullchain.pem with GnuTLSCertificateFile as browsers seems to not get a valid cert then.

The README file says:

`privkey.pem`  : the private key for your certificate.
`fullchain.pem`: the certificate file used in most server software.
`chain.pem`    : used for OCSP stapling in Nginx >=1.3.7.
`cert.pem`     : will break many server configurations, and should not be used without reading further documentation (see link below).
1 Like

The "leaf" comes from the tree analogy which is used to describe the certificate chain, with at the utmost top the root certificate. If the root is at the top, leafs are at the "bottom". I.e., when describing a full chain, the leaf certificate is the actual certificate you're issued by the CA for your website or other service.

fullchain.pem consists of a concatenation of cert.pem and chain.pem so you can use it directly in services such as Apache.

You shouldn't want to use chain.pem, there's no reason to. Verifying the OCSP response using ocsptool is very differently than configuring your webserver.

It's quite unfortunate that the README doesn't actually describe the files properly, probably for shortness. The correct descriptions would be:

privkey.pem : the private key for your certificate.
fullchain.pem: the certificate file used in most server software*, which is a concatenation of > cert.pem and chain.pem, see below.
chain.pem : the intermediate certificate used to sign cert.pem, used for OCSP stapling in Nginx >=1.3.7.
cert.pem : *the file containing just your certificate. This file should not be used directly, as using it *will break many server configurations, and should not be used without reading further documentation (see link below).

3 Likes

Thanks for your answers. I was just not liking that warning in my apache logs :slight_smile:

As fas as I can tell nginx and Apache support OSCP stapling (User Guide — Certbot 1.16.0.dev0 documentation), but perhaps not with GnuTLS at this time.

2 Likes

GnuTLS does support OCSP. That said, the manual at https://mod.gnutls.org/browser/mod_gnutls/doc/mod_gnutls_manual.md is down. Earlier today I could get it refreshing a few times, but now it seems pretty much totally dead. Quite embarrassing I would say.

What issue do you currently have with OCSP than? You can ignore the error/warning for R3, as that error doesn't say anything about the OCSP status for your actual certificate.

1 Like

In my ssl_error.log I am getting lots of these:

[Sun Jul 04 18:49:20.256973 2021] [gnutls:error] [pid 56107:tid 140017197176384] Invalid HTTP response status from r3.o.lencr.org: HTTP/1.0 400 Bad Request
[Sun Jul 04 18:49:20.257071 2021] [gnutls:error] [pid 56107:tid 140017197176384] (104)Connection reset by peer: [client 64.41.200.106:33788] Caching a fresh OCSP response failed
[Sun Jul 04 18:49:24.305678 2021] [gnutls:error] [pid 56076:tid 140016470718016] Invalid HTTP response status from r3.o.lencr.org: HTTP/1.0 400 Bad Request
[Sun Jul 04 18:49:24.305774 2021] [gnutls:error] [pid 56076:tid 140016470718016] (104)Connection reset by peer: [client 162.158.183.160:15528] Caching a fresh OCSP response failed

in my tnonline.net-error.log I am getting:

[Sun Jul 04 16:21:20.858437 2021] [gnutls:warn] [pid 9196:tid 140017591871360] Could not create OCSP stapling configuration for certificate 1 in chain (CN=R3,O=Let's Encrypt,C=US): No OCSP URI in the certificate nor a GnuTLSOCSPResponseFile setting, cannot configure OCSP stapling.
[Sun Jul 04 16:21:33.870787 2021] [gnutls:warn] [pid 9196:tid 140017591871360] Could not create OCSP stapling configuration for certificate 1 in chain (CN=R3,O=Let's Encrypt,C=US): No OCSP URI in the certificate nor a GnuTLSOCSPResponseFile setting, cannot configure OCSP stapling.
[Sun Jul 04 17:48:30.645502 2021] [gnutls:warn] [pid 9196:tid 140017591871360] Could not create OCSP stapling configuration for certificate 1 in chain (CN=R3,O=Let's Encrypt,C=US): No OCSP URI in the certificate nor a GnuTLSOCSPResponseFile setting, cannot configure OCSP stapling.
[Sun Jul 04 18:24:48.885198 2021] [gnutls:warn] [pid 9196:tid 140017591871360] Could not create OCSP stapling configuration for certificate 0 in chain (CN=R3,O=Let's Encrypt,C=US): No OCSP URI in the certificate nor a GnuTLSOCSPResponseFile setting, cannot configure OCSP stapling.
[Sun Jul 04 18:28:56.723892 2021] [gnutls:warn] [pid 9196:tid 140017591871360] Could not create OCSP stapling configuration for certificate 1 in chain (CN=R3,O=Let's Encrypt,C=US): No OCSP URI in the certificate nor a GnuTLSOCSPResponseFile setting, cannot configure OCSP stapling.
1 Like

That's the OCSP URI used indeed, strange to see a HTTP 400 error. Might be a configuration issue (still can't see the GnuTLS manual! :frowning: ) or a GnuTLS implementation error.

It seems those are all for the R3 intermediate cert, which is to be expected. Not sure how you could silence GnuTLS about that tho.

1 Like

I could bring it to the mod_gnutls mailing lists and see if anyone there is using mod_gnutls with Let's Encrypt. It would seem likely, no?

1 Like

You could try that indeed.

The OCSP stapeling on my locally fake Pebble CA isn't working either, but I'm getting ZERO warnings or errors in my Apache logs, which is quite frustrating.. However, when I try to get the OCSP response with openssl on the command line, it works.

So it's not an issue with the ACME server, but GnuTLS somehow.

Found it: GnuTLS uses the SHA-256 hash algorithm for signing its OCSP requests, while OpenSSL uses the older SHA-1 algorithm. SHA-256 results in an error, while SHA-1 does not. So it seems Boulder/Pebble does not accept SHA-256 hashes in the OCSP request.

2 Likes

Thanks for confirming this. Did you get similar errors?

2 Likes

I'm getting:

[Sun Jul 04 19:24:09.171905 2021] [gnutls:error] [pid 4091] Invalid HTTP response status from 127.0.0.1:4002: (null)
[Sun Jul 04 19:24:09.171998 2021] [gnutls:error] [pid 4091] (104)Connection reset by peer: [client ::1:60330] Caching a fresh OCSP response failed
[Sun Jul 04 19:24:09.172014 2021] [gnutls:error] [pid 4091] OCSP request for example.com failed, next try after Sun, 04 Jul 2021 17:29:09 GMT.
[Sun Jul 04 19:24:10.486234 2021] [gnutls:info] [pid 4091] (70014)End of file found: [client ::1:60330] GnuTLS: Error reading data. (-54) 'Error in the pull function.'

With LogLevel at "info".

It seems Boulder, the Let's Encrypt CA software, indeed also doesn't support SHA-256:

openssl ocsp -sha1 -issuer chain.pem -cert cert.pem -text -url http://r3.o.lencr.org

Works. However:

openssl ocsp -sha256 -issuer chain.pem -cert cert.pem -text -url http://r3.o.lencr.org

does not..

Maybe @lestaff could enlighten us why SHA-256 isn't supported for OCSP requests? (Not asking for SHA-256 signatures here :wink:) As it seems, this is an incompatibility with GnuTLS' OCSP.

1 Like

Does this indicate an error on let's encrypt servers or in the gnutls client? I. E. Should sha256 not be used, or does the server not support it?

1 Like

In my personal opinion the Let's Encrypt server is lacking SHA-256 support. That said, I think GnuTLS is a liiiiiitle bit too strict by using SHA-256 for OCSP requests.. And I can't find an option to select SHA-1 anywhere.

You can see in the mod_gnutls source code that the use of SHA-256 is hardcoded:

1 Like

https://duckduckgo.com/?q=sha1+deprecation&t=ofa&ia=answer

Interestingly it seems SHA1 should be considered deprecated. This is probably why it's hardcoded in GnuTLS.

Maybe we should bring this to some Let's Encrypt bug tracker instead?

That said I believe the error message in the Apache log could have been clearer.

1 Like

It already exists:

Perhaps one of us (probably you, as I don't actually use GnuTLS on a production system) should add our findings from this thread there. I.e.: GnuTLS is another example of software using SHA-256 as the algorithm to generate IssuerNameHash / IssuerKeyHash.

2 Likes

Alas, SHA-1 is still specified in some parts of RFC 6960. This thread from MozDev may be relevant: https://groups.google.com/g/mozilla.dev.security.policy/c/ImCmDRMj-JU

2 Likes