Pebble acme-tls/1 failed to connect on challenge, Jetty shows connections

Maurice-Betzel · December 22, 2022, 4:30pm

My domain is: localhost

I ran this command: renew

It produced this output: Failed to connect to [::1]:8444 for the tls-alpn-01 challenge

My web server is (include version): Jetty 9.4.43 / Pebble 2.4.0

The operating system my web server runs on is (include version): Windows 10 64 bit

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my acme4j client is: 2.15

Renew is initiated and the certificate and key-pair for the challenge are stored in the Java key-store, after triggering the acme-tls/1 challenge i see 3 connections to my Jetty server and the right SNI gets matched from the keystore. But then I get the error from acme4j

2022-12-22T17:17:04,436 | DEBUG | qtp592782151-170 | SniX509ExtendedKeyManager | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | Chose explicit alias=null keyType=EC on sun.security.ssl.SSLEngineImpl@544a1748
2022-12-22T17:17:04,437 | DEBUG | qtp592782151-170 | SslContextFactory | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | Selecting alias: keyType=RSA, sni=localhost, sniRequired=false, certs=[X509@672013bf(acme_challenge_key,h=[localhost, acme.invalid],a=,w=), X509@7bbfee12(digital.gaston-schul.com (r3),h=[digital.gaston-schul.com],a=,w=)]
2022-12-22T17:17:04,437 | DEBUG | qtp592782151-170 | SslContextFactory | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | Selected alias=acme_challenge_key
2022-12-22T17:17:04,437 | DEBUG | qtp592782151-170 | SniX509ExtendedKeyManager | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | Matched SNI localhost with alias acme_challenge_key, certificate X509@672013bf(acme_challenge_key,h=[localhost, acme.invalid],a=,w=) from aliases [acme_challenge_key, digital.gaston-schul.com (r3)]
2022-12-22T17:17:04,437 | DEBUG | qtp592782151-170 | SniX509ExtendedKeyManager | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | Chose explicit alias=acme_challenge_key keyType=RSA on sun.security.ssl.SSLEngineImpl@544a1748
2022-12-22T17:17:04,439 | DEBUG | qtp592782151-170 | ACMEServerConnectionFactory | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | proto=acme-tls/1 tls=TLSv1.2 cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 acceptable=true
2022-12-22T17:17:04,445 | DEBUG | qtp592782151-170 | ACMEServerConnectionFactory | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | Creating ACME server connection
2022-12-22T17:17:04,445 | DEBUG | qtp592782151-170 | ACMEServerConnection | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | Closing endpoint
2022-12-22T17:17:04,459 | INFO | AcmeQuartzScheduler_Worker-1 | JobRunShell | 190 - org.quartz-scheduler.quartz - 2.3.2 | Job acme.action.job.group.ACMERenewJob threw a JobExecutionException:
org.quartz.JobExecutionException: Challenge has failed due to Failed to connect to [::1]:8444 for the tls-alpn-01 challenge
at eu.abeel.platform.security.acme.api.job.ACMERenewJob.executeJob(ACMERenewJob.java:128) ~[?:?]
at eu.abeel.platform.security.acme.api.spi.ACMEJob.execute(ACMEJob.java:42) ~[?:?]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [!/:?]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [!/:?]

Pebble:

Pebble 2022/12/22 17:17:01 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"localhost"}, Challenge:(*core.Challenge)(0xc000077720), Account:(*core.Account)(0xc00006e0a0)}
Pebble 2022/12/22 17:17:01 Starting 3 validations.
Pebble 2022/12/22 17:17:01 Sleeping for 3s seconds before validating
Pebble 2022/12/22 17:17:01 Sleeping for 0s seconds before validating
Pebble 2022/12/22 17:17:01 Sleeping for 1s seconds before validating
Pebble 2022/12/22 17:17:01 authz PC7yhZA0dMK8fH4r1TajsD6hR0OVK0Hiwb1SoEYOA8M set INVALID by completed challenge fr5hb6uVAutsvQxxt-OeqGO7NGrj2sbHZXMegLXcBWg
Pebble 2022/12/22 17:17:01 order mFhea8wwms4h2WYRnW90W6eXHhQCuX4l4-WPtgWNzwg set INVALID by invalid authz PC7yhZA0dMK8fH4r1TajsD6hR0OVK0Hiwb1SoEYOA8M
Pebble 2022/12/22 17:17:04 POST /chalZ/ -> calling handler()
Pebble 2022/12/22 17:17:04 POST /chalZ/ -> calling handler()

Any ideas?

_az · December 22, 2022, 10:08pm

Mmm. I'm not sure whether Windows automatically dual stacks listening sockets, like it does on Linux.

Is Jetty definitely listening on [::1]:8444 or [::]:8444 for HTTPS?

Not just 127.0.0.1:8444 or 0.0.0.0:8444?

Maurice-Betzel · December 23, 2022, 9:25am

Jetty is set on localhost, as is Pebble, to listen to all networking interfaces being 0.0.0.0.
I see the 3 connection attempts from Pebble in the Jetty logs and the SNI getting selected, see above Jetty log where i printed the last log entry being the third Pebble connection.
Testing if Jetty consumes your proposed host settings.

Maurice-Betzel · December 23, 2022, 10:25am

I removed the Jetty host configuration leaving only the port, that should make Jetty bind to IPV4 and 6 addresses and according to netstat it does:

[java.exe]
TCP 0.0.0.0:8444 LT-VLO-IT-11:0 LISTENING

[java.exe]
TCP [::]:8444 LT-VLO-IT-11:0 LISTENING

To no avail:

So this [::1] is the stacking part?

_az · December 23, 2022, 10:31am

Okay. I don't think the dual stacking is the problem after all. That looks like it should work.

What makes this harder to troubleshoot is that Pebble swallows the error when it tries to connect and do the TLS handshake. It returns the generic "failed to connect to ..." error without any detail, which could be a networking issue or a TLS protocol issue or certificate validation issue. We don't know . If you compiled Pebble from source, you could put something at that line:

log.Printf("va: %s/%s TLS error: %v", config.ServerName, hostPort,  err)

and see whether it yields any extra information.

I'm really not sure otherwise.

Maurice-Betzel · December 23, 2022, 10:34am

Compiling, my go debugger is not working in itellij, thanks for the line of code, that helped a lot:

2022/12/23 11:44:44 va: localhost/[::1]:8444 TLS error: read tcp [::1]:57719->[::1]:8444: wsarecv: An established connection was aborted by the software in your host machine.
2022/12/23 11:44:45 va: localhost/[::1]:8444 TLS error: EOF

Had a feeling that closing the connection within Jetty after the SSL handshake was bad advice from the Jetty forum. Usually Jetty throws an exception on the ALPN tls-acme part and I created a connection factory for it to avoid unwanted log entries. so I have to find out how to handle this:

"When the ACME client connects, Jetty will select the "acme-tls/1"
ConnectionFactory, complete the TLS handshake, find that the
"acme-tls/1" has no correspondent ConnectionFactory and close the
connection (which is expected)."

Maurice-Betzel · December 23, 2022, 11:12am

Not closing results in a TLS error:

Pebble 2022/12/23 12:06:20 Starting 3 validations.
Pebble 2022/12/23 12:06:20 Sleeping for 4s seconds before validating
Pebble 2022/12/23 12:06:20 Sleeping for 2s seconds before validating
Pebble 2022/12/23 12:06:20 Sleeping for 1s seconds before validating
Pebble 2022/12/23 12:06:23 POST /chalZ/ -> calling handler()
Pebble 2022/12/23 12:06:26 POST /chalZ/ -> calling handler()
Pebble 2022/12/23 12:06:29 POST /chalZ/ -> calling handler()
Pebble 2022/12/23 12:06:32 POST /chalZ/ -> calling handler()
Pebble 2022/12/23 12:06:35 POST /chalZ/ -> calling handler()
2022/12/23 12:06:37 va: localhost/[::1]:8444 TLS error: context deadline exceeded
Pebble 2022/12/23 12:06:37 authz 2nebQXWBe78zAF-jNR-5KX2_MmgAHQsG61mQ349t-Aw set INVALID by completed challenge lLPGz1XZDP2p4MwuABd54eIckozPiI2ijshrD31VUDk
Pebble 2022/12/23 12:06:37 order D-Tsjkv9sRMH-RJukf0bGe52x1UxT-vCJCBKspZ1WPY set INVALID by invalid authz 2nebQXWBe78zAF-jNR-5KX2_MmgAHQsG61mQ349t-Aw
Pebble 2022/12/23 12:06:38 POST /chalZ/ -> calling handler()
2022/12/23 12:06:39 va: localhost/[::1]:8444 TLS error: context deadline exceeded

Maurice-Betzel · December 23, 2022, 11:17am

Using the default Jetty setup without my connection factory resulted in a similar error:

2022-12-23T12:15:22,522 | INFO | qtp33227220-133 | NegotiatingServerConnection | 100 - org.eclipse.jetty.util - 9.4.43.v20210629 | ALPNServerConnection@628753fb::DecryptedEndPoint@66b8db73{l=/0:0:0:0:0:0:0:1:8444,r=/0:0:0:0:0:0:0:1:58332,OPEN,fill=-,flush=-,to=20/30000} application selected protocol 'acme-tls/1', but no correspondent org.eclipse.jetty.server.ConnectionFactory has been configured

org.quartz.JobExecutionException: Challenge has failed due to Failed to connect to [::1]:8444 for the tls-alpn-01 challenge

Pebble 2022/12/23 12:15:20 Starting 3 validations.
Pebble 2022/12/23 12:15:20 Sleeping for 2s seconds before validating
Pebble 2022/12/23 12:15:20 Sleeping for 1s seconds before validating
Pebble 2022/12/23 12:15:20 Sleeping for 0s seconds before validating
2022/12/23 12:15:21 va: localhost/[::1]:8444 TLS error: EOF
Pebble 2022/12/23 12:15:21 authz BF0MU3ZeM1TrlW5NR4fNsTeWKRS4nNlXDL98XYPnQvY set INVALID by completed challenge aJq13ks9J1-4Uqc-wCxwu7Tn18WmZO56p3jtE4Fm7F0
Pebble 2022/12/23 12:15:21 order p7UkzZ1MA9AJV7Cgbh0QQSkWiGULL4A915LkGhbnHbw set INVALID by invalid authz BF0MU3ZeM1TrlW5NR4fNsTeWKRS4nNlXDL98XYPnQvY
2022/12/23 12:15:22 va: localhost/[::1]:8444 TLS error: EOF
Pebble 2022/12/23 12:15:23 POST /chalZ/ -> calling handler()

Maurice-Betzel · December 23, 2022, 11:41am

Should the SSL handshake not suffice on the ACME server side? Why keep the connection and die on close?

Bruce5051 · December 23, 2022, 7:14pm

No, one needs to be using TLS v1.2 or TLS v1.3; here are 3 posts on the subject.

_az · December 23, 2022, 10:08pm

Are you writing your own ACME client?

If Pebble is reporting that EOF when you close it, or "context deadline exceeded" (timeout) when left open, presumably the TLS handshake is incomplete.

If you connect to your Jetty's TLS server with OpenSSL, do you see that both client and server send a "Finished" message? If not, the handshake is incomplete.

openssl s_client -connect localhost:8444 -servername localhost -msg -tls1_2 -alpn acme-tls/1

Maurice-Betzel · December 24, 2022, 11:09am

Sorry about the inaccurate question, i am using TLS 1.2 and thought the client hello response from the server would suffice. The ACME server has the challenge certificate then.

Maurice-Betzel · December 24, 2022, 11:31am

I had a discussion about this with Simone Bordet on the Jetty server mailing list and he provided me with a Jetty connection factory that does the job, details: Support RFC 8737 · Issue #9085 · eclipse/jetty.project · GitHub

system · January 23, 2023, 11:31am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.