Let´s Encrypt denies ALPN challenge, domains can´t be authorized

thomaskl · February 9, 2022, 10:41am

Hi!

We have an issue to generate certificates in two cases. In both cases Let´s Encrypt denies ALPN challenge and the domains can´t be authorized. Could you assist please and give feedback why those domains can´t be authorized via ALPN?

printhouseservice.com

jentzsch-gmbh.de

Osiris · February 9, 2022, 10:48am

To me, it's not very clear what kind of threads should be opened here in the #issuance-tech-questions category, but I think it's more about discussing issuance tech? While your post seems more about "seeking help on how to fix generating a certificate". I believe this thread gets more attention in the #help section, so I'm moving it there. In the #help section, you would also be provided with a (mandatory) questionnaire. We really need more info than just "It doesn't work, help" (99 % here are just volunteers and not associated with Let's Encrypt, so we don't have access to the LE systems/logs et cetera), so please answer the questions to the best of your knowledge:

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

rg305 · February 9, 2022, 1:35pm

Hi @thomaskl and welcome to the LE community forum

In addition to answering those questions, please be sure to provide log messages of the failures.

thomaskl · February 9, 2022, 2:09pm

Hi! Thank´s for the fast replies!

Please find more informations below:

Domains with mentioned issue:
printhouseservice.com

jentzsch-gmbh.de

I ran this command:

cm4all-certdb acme --account-db --alpn new-order database-handle example.com www.example.com

It produced this output:

No compatible challenge found

My web server is (include version):
CM4all beng-proxy 17.0.82

The operating system my web server runs on is (include version):
debian 11 (bullseye)

My hosting provider, if applicable, is:
I am... German Telekom

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is:
17.0.82

Full protocol:

root@c4aweb11:~# cm4all-certdb acme --debug --account-db --alpn new-order database-handle  example.com www.example.com
*   Trying 2606:4700:60:0:f53d:5624:85c7:3a2c:443...
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=acme-v02.api.letsencrypt.org
*  start date: Jan 28 15:17:20 2022 GMT
*  expire date: Apr 28 15:17:19 2022 GMT
*  subjectAltName: host "acme-v02.api.letsencrypt.org" matched cert's "acme-v02.api.letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x56230b989b90)
> GET /directory HTTP/2
Host: acme-v02.api.letsencrypt.org
user-agent: CM4all beng-proxy 17.0.82
accept: */*

* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< server: nginx
< date: Wed, 09 Feb 2022 13:16:40 GMT
< content-type: application/json
< content-length: 658
< cache-control: public, max-age=0, no-cache
< x-frame-options: DENY
< strict-transport-security: max-age=604800
<
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
* Found bundle for host acme-v02.api.letsencrypt.org: 0x56230b99be70 [can multiplex]
* Re-using existing connection! (#0) with host acme-v02.api.letsencrypt.org
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* Using Stream ID: 3 (easy handle 0x56230b989b90)
> HEAD /acme/new-nonce HTTP/2
Host: acme-v02.api.letsencrypt.org
user-agent: CM4all beng-proxy 17.0.82
accept: */*

< HTTP/2 200
< server: nginx
< date: Wed, 09 Feb 2022 13:16:40 GMT
< cache-control: public, max-age=0, no-cache
< link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
< replay-nonce: xxxxxxxx
< x-frame-options: DENY
< strict-transport-security: max-age=604800
<
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
* Found bundle for host acme-v02.api.letsencrypt.org: 0x56230b99be70 [can multiplex]
* Re-using existing connection! (#0) with host acme-v02.api.letsencrypt.org
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* Using Stream ID: 5 (easy handle 0x56230b9a4680)
> POST /acme/new-order HTTP/2
Host: acme-v02.api.letsencrypt.org
user-agent: CM4all beng-proxy 17.0.82
accept: */*
content-type: application/jose+json
content-length: 1144

* We are completely uploaded and fine
< HTTP/2 201
< server: nginx
< date: Wed, 09 Feb 2022 13:16:41 GMT
< content-type: application/json
< content-length: 488
< boulder-requester: 89641109
< cache-control: public, max-age=0, no-cache
< link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
< location: https://acme-v02.api.letsencrypt.org/acme/order/89641109/60890678660
< replay-nonce: xxxxxxxxxx
< x-frame-options: DENY
< strict-transport-security: max-age=604800
<
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
* Found bundle for host acme-v02.api.letsencrypt.org: 0x56230b99be70 [can multiplex]
* Re-using existing connection! (#0) with host acme-v02.api.letsencrypt.org
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* Using Stream ID: 7 (easy handle 0x56230b9a41f0)
> POST /acme/authz-v3/xxxxxxxxx HTTP/2
Host: acme-v02.api.letsencrypt.org
user-agent: CM4all beng-proxy 17.0.82
accept: */*
content-type: application/jose+json
content-length: 1005

* We are completely uploaded and fine
< HTTP/2 200
< server: nginx
< date: Wed, 09 Feb 2022 13:16:41 GMT
< content-type: application/json
< content-length: 839
< boulder-requester: 89641109
< cache-control: public, max-age=0, no-cache
< link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
< replay-nonce: xxxxxxxx
< x-frame-options: DENY
< strict-transport-security: max-age=604800
<
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
No compatible challenge found
root@c4aweb11:~#

MikeMcQ · February 9, 2022, 3:59pm

Can you show more of that log output? There is no error displayed in the part you showed.

Also, can you explain why your printhouseservice.com domain returns a Let's Encrypt Staging cert but jentzsch-gmbh.de returns one from Deutsche Telekom?

Phil · February 9, 2022, 5:23pm

Can you provide details of the challenge certificate generation?

Do you know which OID your TLS-ALPN capable acme client is attempting to use?

Does your challenge certificate only contain a dnsName?

Is your challenge certificate self signed?

jvanasco · February 9, 2022, 6:54pm

Looks to be the right one since the first commit

github.com

CM4all/beng-proxy/blob/master/src/certdb/AcmeAlpn.cxx#L53


      
          
          	const auto sha256 = SHA256(AsBytes(host));
          	const auto hex = HexFormat(std::span{sha256}.first<20>());
          
          	std::string result{"acme-tls-alpn-01:"};
          	result.append(std::string_view{hex.data(), hex.size()});
          	return result;
          }
          
          Alpn01ChallengeRecord::Alpn01ChallengeRecord(CertDatabase &_db,
          					     std::string_view _host)
          	:db(_db), host(_host),
          	 handle(std::string{"acme-tls-alpn-01:"} + host),
          	 cert(MakeSelfIssuedDummyCert(MakeCommonName(host)))
          {
          	std::string alt_name = std::string{"DNS:"} + host;
          
          	AddExt(*cert, NID_subject_alt_name, alt_name.c_str());
          }
          
          Alpn01ChallengeRecord::~Alpn01ChallengeRecord() noexcept

@thomaskl two questions:

Have you ever successfully gotten a certificate using TLS-ALPN-01 from this client? If not, you may get more support from that client's authors.
The output is for "example.com", can you show a log from running this on your actual domains?

thomaskl · February 14, 2022, 8:29am

Yes, we did 102.391 successful. Only these two do not get offered ALPN. It's not a client problem. We're the authors of the client and the web server. It's let's encrypt that doesn't offer ALPN in these two cases only. I'm not sure if this isn't some LE bug.

I added the entire log, which is from the actual domains. Please ignore example.com in there.

Osiris · February 14, 2022, 8:51am

That is yet to be determined if you'd ask me.

It seems the client output you've posted above are just the headers of the requests. I'd very much like to see the actual return data too, as for the latest request for your "POST /acme/authz-v3/xxxxxxxxx HTTP/2" action Boulder returned "HTTP/2 200" with 839 bytes of JSON output, but no output is shown and your client decides that no compatible challenge was found.

I think it would be important to show what Boulder actually answered in that JSON and why your client decided it didn't find a compatible challenge.

jvanasco · February 14, 2022, 3:08pm

This is a very unlikely, but the only explanation that comes to mind right now – can you check your system to see the last few times these domains tried to renew?

As part of the TLS-ALPN-01 bug fix a few weeks ago, LetsEncrypt not only revoked all the certificates, but also destroyed all the authorizations. There was a potential race condition issue, that should not have been possible from how the situation was handled, in which an ACME Authorization was created after the TLS-ALPN-01 challenge was disabled but before it was re-enabled. Again, I don't think this would be possible, but that would have created an ACME-Authorization object that has no TLS-ALPN-01 challenges.

A quick way to check that would be to manually debug the AcmeOrder/AcmeAuthorization objects to see what challenges are available for that domain. If you don't see a TLS-ALPN-01, I would invalidate those objects (by the api, or answering a challenge that is doomed to fail) and then try again. If you did somehow experience a race condition, TLS-ALPN-01 should be an option in the next order.

I've encountered more edge-cases when developing a client than I expected. I don't have an answer for your problem, but I can offer some pointers that have helped me figure out weird situations:

Add a debug level/facility to your system where you can log the entire response, and requests, from the ACME servers. Drop down the error to one domain if possible, and compare the ERROR request to a SUCCESS request. Sometimes the client makes a error very early in the acme-order, and it was doomed to fail.
Make sure you read the divergences document and implementation details document on the Boulder source code. Some decisions in Boulder can be from a different interpretation of the ACME spec's requirements.

thomaskl · February 22, 2022, 8:45am

In the meanwhile we tried generating certs again and now it works, for whatever reason. Thx for your fast replies and support!

system · March 24, 2022, 8:45am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
A test authorization for your domain to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued. Error creating new order :: Policy forbids issuing for name Help	10	2478	February 28, 2021
Struggling with certificate Help	3	526	July 18, 2022
Take the certifikate Help	4	417	August 27, 2023
I am not able to generate cert Help	4	544	November 22, 2023
Issue with issuing lets encrypt certificate Help	3	412	September 7, 2023

Let´s Encrypt denies ALPN challenge, domains can´t be authorized

Related topics