Mod_md fails - status 20014

[nsayer]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

Geppetto Electronics

I ran this command:

curl https://acme-v02.api.letsencrypt.org/

It produced this output:

{
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
"zCBgPRZUOSg": "Adding random entries to the directory"

My web server is (include version):

Apache 2.4.41

The operating system my web server runs on is (include version):

FreeBSD 11.4

My hosting provider, if applicable, is:

rootbsd

I can login to a root shell on my machine (yes or no, or I don't know):

Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

mod_md, came with Apache

md_status shows

    "last": {
      "status": 20014,
      "status-description": "Internal error (specific information not available)",
      "detail": "Unsuccessful in contacting ACME server at <https://acme-v02.api.letsencrypt.org/directory>. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.",
      "activity": "Contacting ACME server for www.geppettoelectronics.com at https://acme-v02.api.letsencrypt.org/directory"
    },
    "log": {
      "entries": [
        {
          "when": "Wed, 23 Sep 2020 04:22:03 GMT",
          "type": "renewal-error",
          "detail": "Unsuccessful in contacting ACME server at <https://acme-v02.api.letsencrypt.org/directory>. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this."
        },

but, again, curl works just fine. I've tried adding /etc/hosts entries for acme-v02 (temporarily), but none of them helped.

[_az]

Does anything useful show up if you enable trace logging for mod_md?

LogLevel md:trace1

(or even upto trace5).

[nsayer]

This seems apropos:

[Tue Sep 22 21:58:47.926132 2020] [md:debug] [pid 33778:tid 34390135040] md_curl.c(259): (20014)Internal error (specific information not available): request failed(59): Couldn't use specified SSL cipher

[rg305]

What version of OpenSSL do you have?
[or whatever you use]

[nsayer]

OpenSSL 1.1.1g (pad pad pad)

[rg305]

Well that was a dead end.

This has me thinking:

@nsayer

Couldn't use specified SSL cipher

[rg305]

Is there a newer version/update of mod_md or md_curl.c ?

[nsayer]

Both of those were built as part of the Apache port. I can't imagine there's anything newer.

More tracing:

[Tue Sep 22 22:35:12.721550 2020] [md:debug] [pid 8361:tid 34510921984] md_acme.c(769): get directory from https://acme-staging-v02.api.letsencrypt.org/directory

[Tue Sep 22 22:35:12.721589 2020] [md:trace3] [pid 8361:tid 34510921984] md_curl.c(322): req[0]: GET https://acme-staging-v02.api.letsencrypt.org/directory

[Tue Sep 22 22:35:12.722595 2020] [md:trace4] [pid 8361:tid 34510921984] md_curl.c(199): req[0]: info Trying 172.65.46.172:443...\n

[Tue Sep 22 22:35:12.724264 2020] [md:trace4] [pid 8361:tid 34510921984] md_curl.c(199): req[0]: info Connected to acme-staging-v02.api.letsencrypt.org (172.65.46.172) port 443 (#0)\n

[Tue Sep 22 22:35:12.724397 2020] [md:trace4] [pid 8361:tid 34510921984] md_curl.c(199): req[0]: info ALPN, offering h2\n

[Tue Sep 22 22:35:12.724434 2020] [md:trace4] [pid 8361:tid 34510921984] md_curl.c(199): req[0]: info ALPN, offering http/1.1\n

[Tue Sep 22 22:35:12.724476 2020] [md:trace4] [pid 8361:tid 34510921984] md_curl.c(199): req[0]: info failed setting cipher list: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH\n

[Tue Sep 22 22:35:12.724522 2020] [md:trace4] [pid 8361:tid 34510921984] md_curl.c(199): req[0]: info Closing connection 0\n

[Tue Sep 22 22:35:12.724582 2020] [md:debug] [pid 8361:tid 34510921984] md_curl.c(385): (20014)Internal error (specific information not available): request failed(59): Couldn't use specified SSL cipher

[Tue Sep 22 22:35:12.724616 2020] [md:trace3] [pid 8361:tid 34510921984] md_curl.c(358): (20014)Internal error (specific information not available): req[0] fire callbacks

[Tue Sep 22 22:35:12.724703 2020] [md:warn] [pid 8361:tid 34510921984] (20014)Internal error (specific information not available): md[www.geppettoelectronics.com] while[Contacting ACME server for www.geppettoelectronics.com at https://acme-staging-v02.api.letsencrypt.org/directory] detail[Unsuccessful in contacting ACME server at https://acme-staging-v02.api.letsencrypt.org/directory. If this problem persists, please check your network connectivity from your Apache server to the ACME server. Also, older servers might have trouble verifying the certificates of the ACME server. You can check if you are able to contact it manually via the curl command. Sometimes, the ACME server might be down for maintenance, so failing to contact it is not an immediate problem. Apache will continue retrying this.]

[_az]

@rg305

It appears to me to be there is an unset variable "STRENGTH" somewhere in the code.

https://www.openssl.org/docs/man1.1.1/man1/ciphers.html

Additionally the cipher string @STRENGTH can be used at any point to sort the current cipher list in order of encryption algorithm key length.

This error is propagating up from SSL_CTX_set_cipher_list - OpenSSL Documentation.

SSL_CTX_set_cipher_list() and SSL_set_cipher_list() return 1 if any cipher could be selected and 0 on complete failure.

Sounds to me like something has gone wrong with the way the package was built. I assume the output of openssl ciphers is sane, since a normal curl works just fine.

[rg305]

@_az

I assume the output of openssl ciphers is sane, since a normal curl works just fine.

I would think so too.
I see this works on my system:
@nsayer, does this work on yours:
openssl ciphers 'ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH'

[nsayer]

Recompiling the curl port made this go away.

I suspect it's possible that curl begets libcurl and it may not have been built against openssl 1.1.1g, but rather the openssl that came with base FreeBSD.