org.shredzone.acme4j.exception.AcmeLazyLoadingException

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: auth-test.phs.org

I ran this command:
Error while renewing certificate for above domain using acme client. Error is thrown when letsecnrypt verifies http challenge. client will fetch a new order from letsecnrypt and then get authorizations from the new order. Acme client throws below error while fetching identifier from the authorization.
Can someone share the logs for this domain from Letsecnrypt server, are there any problems with domain valdiation while validating the DNS records? OR if there are logs suggesting any firewall or ip blocklist?

It produced this output:
Error at below line

logs from 05/10 for the same domain

org.shredzone.acme4j.exception.AcmeLazyLoadingException: Authorization https://acme-v02.api.letsencrypt.org/acme/authz-v3/348472255457
at org.shredzone.acme4j.AcmeJsonResource.getJSON(AcmeJsonResource.java:66)
at org.shredzone.acme4j.Authorization.getIdentifier(Authorization.java:55)

The version of my client is : acme client version is 2.11

Also noticed there are AcmeRateLimitedException & AcmeProtocolException exception from previous renewal attempts below while fetching new order from letsecnrypt LetsEncryptProxy.getNewOrder()

logs from 05/09 for the same domain

org.shredzone.acme4j.exception.AcmeRateLimitedException: Service busy; retry later.
at org.shredzone.acme4j.connector.DefaultConnection.throwAcmeException(DefaultConnection.java:545)
at org.shredzone.acme4j.connector.DefaultConnection.performRequest(DefaultConnection.java:479)
at org.shredzone.acme4j.connector.DefaultConnection.sendSignedRequest(DefaultConnection.java:407)
at org.shredzone.acme4j.connector.DefaultConnection.sendSignedRequest(DefaultConnection.java:168)
at org.shredzone.acme4j.AccountBuilder.createLogin(AccountBuilder.java:212)
at org.shredzone.acme4j.AccountBuilder.create(AccountBuilder.java:173)
at com.saasure.core.services.org.LetsEncryptProxy.findOrRegisterAccount(LetsEncryptProxy.java:102)
at com.saasure.core.services.org.LetsEncryptProxy.getAccount(LetsEncryptProxy.java:164)
at com.saasure.core.services.org.LetsEncryptProxy.getNewOrder(LetsEncryptProxy.java:169)

logs from 05/07 for the same domain

org.shredzone.acme4j.exception.AcmeProtocolException: JSON response is empty
at org.shredzone.acme4j.connector.DefaultConnection.readJsonResponse(DefaultConnection.java:188)
at org.shredzone.acme4j.connector.DefaultConnection.throwAcmeException(DefaultConnection.java:526)
at org.shredzone.acme4j.connector.DefaultConnection.resetNonce(DefaultConnection.java:126)
at org.shredzone.acme4j.connector.DefaultConnection.performRequest(DefaultConnection.java:445)
at org.shredzone.acme4j.connector.DefaultConnection.sendSignedRequest(DefaultConnection.java:407)
at org.shredzone.acme4j.connector.DefaultConnection.sendSignedRequest(DefaultConnection.java:168)
at org.shredzone.acme4j.AccountBuilder.createLogin(AccountBuilder.java:212)
at org.shredzone.acme4j.AccountBuilder.create(AccountBuilder.java:173)
at com.saasure.core.services.org.LetsEncryptProxy.findOrRegisterAccount(LetsEncryptProxy.java:102)
at com.saasure.core.services.org.LetsEncryptProxy.getAccount(LetsEncryptProxy.java:164)
at com.saasure.core.services.org.LetsEncryptProxy.getNewOrder(LetsEncryptProxy.java:169)

2

Can you try updating acme4j? Version 2.11 looks to be a few years old, and it wouldn't shock me if you're hitting a bug, especially if you're getting errors like AcmeProtocolException: JSON response is empty which makes me think it's not sending or understanding something in the protocol correctly.

Also, see if you can do further testing against the Staging Environment. It has much more forgiving rate limits and may help you diagnose whatever the underlying issue is better.

4 Likes
3

Here is what I presently see

The request gets HTTP/1.1 302 Found and redirect HTTP to HTTPS
on a Server: Apache

$ curl -i http://auth-test.phs.org/.well-known/acme-challenge/sometestfile
HTTP/1.1 302 Found
Date: Mon, 13 May 2024 14:31:07 GMT
Server: Apache
Location: https://auth-test.phs.org//.well-known/acme-challenge/sometestfile
Content-Length: 250
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://auth-test.phs.org//.well-known/acme-challenge/sometestfile">here</a>.</p>
</body></html>

And the redirect from HTTP to HTTPS get HTTP/1.1 200 OK
on a Server: nginx

$ curl -i https://auth-test.phs.org//.well-known/acme-challenge/sometestfile
HTTP/1.1 200 OK
Date: Mon, 13 May 2024 14:31:12 GMT
Server: nginx
Content-Type: application/octet-stream
Content-Length: 0
x-okta-request-id: ZkIkMOF1zBklGgQNHTe9_wAAAOc
x-xss-protection: 0
p3p: CP="HONK"
content-security-policy: frame-ancestors 'self'
x-rate-limit-limit: 100
x-rate-limit-remaining: 98
x-rate-limit-reset: 1715610706
cache-control: no-cache, no-store
pragma: no-cache
expires: 0
referrer-policy: strict-origin-when-cross-origin
accept-ch: Sec-CH-UA-Platform-Version
x-content-type-options: nosniff
Strict-Transport-Security: max-age=315360000; includeSubDomains
X-Robots-Tag: noindex,nofollow
set-cookie: sid="";Version=1;Path=/;Max-Age=0
set-cookie: autolaunch_triggered=""; Expires=Thu, 01 Jan 1970 00:00:10 GMT; Path=/
set-cookie: JSESSIONID=E4AA5D9F73F39A24BE8DA57A5A310422; Path=/; Secure; HttpOnly

Edit:
Attempt using a different more random < TOKEN >
namely ZQ9QvtC57WWI67m8DpJbQ4uJzk5lSxStWgxDs1zl

$ curl -i http://auth-test.phs.org/.well-known/acme-challenge/ZQ9QvtC57WWI67m8DpJbQ4uJzk5lSxStWgxDs1zl
HTTP/1.1 302 Found
Date: Mon, 13 May 2024 14:38:58 GMT
Server: Apache
Location: https://auth-test.phs.org//.well-known/acme-challenge/ZQ9QvtC57WWI67m8DpJbQ4uJzk5lSxStWgxDs1zl
Content-Length: 278
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://auth-test.phs.org//.well-known/acme-challenge/ZQ9QvtC57WWI67m8DpJbQ4uJzk5lSxStWgxDs1zl">here</a>.</p>
</body></html>

And still the redirected response still get HTTP/1.1 200 OK

I believe that is an issue; one that would be difficult for Let's Debug to find.

$ curl -i https://auth-test.phs.org//.well-known/acme-challenge/ZQ9QvtC57WWI67m8DpJbQ4uJzk5lSxStWgxDs1zl
HTTP/1.1 200 OK
Date: Mon, 13 May 2024 14:39:19 GMT
Server: nginx
Content-Type: application/octet-stream
Content-Length: 0
x-okta-request-id: ZkImF9VrJ3EVWob4I3xftgAABc4
x-xss-protection: 0
p3p: CP="HONK"
content-security-policy: frame-ancestors 'self'
x-rate-limit-limit: 100
x-rate-limit-remaining: 99
x-rate-limit-reset: 1715611219
cache-control: no-cache, no-store
pragma: no-cache
expires: 0
referrer-policy: strict-origin-when-cross-origin
accept-ch: Sec-CH-UA-Platform-Version
x-content-type-options: nosniff
Strict-Transport-Security: max-age=315360000; includeSubDomains
X-Robots-Tag: noindex,nofollow
set-cookie: sid="";Version=1;Path=/;Max-Age=0
set-cookie: autolaunch_triggered=""; Expires=Thu, 01 Jan 1970 00:00:10 GMT; Path=/
set-cookie: JSESSIONID=25B127048639B1BC0D26C55CA177D406; Path=/; Secure; HttpOnly
1 Like
4

Why are there two slashes after .org ?

3 Likes
5

I found it here; I too thought that was odd. Didn't notice until after I posted. :man_shrugging:

2 Likes
6

same version of acme client is working for other domains.

We saw another domain failing with AcmeLazyLoadingException exception. After the exception is thrown, all the subsequent retry attempts will return auth status as pending.

org.shredzone.acme4j.exception.AcmeLazyLoadingException: Authorization https://acme-v02.api.letsencrypt.org/acme/authz-v3/350126430617
	at org.shredzone.acme4j.AcmeJsonResource.getJSON(AcmeJsonResource.java:66)
	at org.shredzone.acme4j.Authorization.getIdentifier(Authorization.java:55)
	at com.saasure.core.services.org.AcmeServerServiceImpl.verifyHttpChallenge(AcmeServerServiceImpl.java:353)
	at

lets encrypt returns a 200 response on triggering the http challenge
for eg. below is the log from letsencrypt GET http challenge for the domain "mastersrx-auth.prod.mckesson.com"

"GET /.well-known/acme-challenge/FlYEVAnjDouyx5eS74eMSJtZa2Dq0itSQIIHFjXIO1E HTTP/1.1" 200 87 "http://mastersrx-auth.prod.mckesson.com/.well-known/acme-challenge/FlYEVAnjDouyx5eS74eMSJtZa2Dq0itSQIIHFjXIO1E" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "34.209.204.123" "mastersrx-auth.prod.mckesson.com" ok10-majorecs03b.aue2-ok10.internal 0.022 "0.021" . rid=ZkGvFF1p5G2AnuN9YCpVOQAACnc "-" "127.0.0.1:22020" "-" "orgId=00o5vz6g1l49ySSXF4h7 APP=0.003 MOCA=0.010 rlInfo=CAT_C dbri=NO" "" ""

Is it possible to get logs from lets encrypt server for the below 2 authorizations ->

Authorization https://acme-v02.api.letsencrypt.org/acme/authz-v3/348472255457

domain name = auth-test.phs.org

https://acme-v02.api.letsencrypt.org/acme/authz-v3/350126430617

domain=mastersrx-auth.prod.mckesson.com

7

It might not fix the problem, but I see an issue in the redirection:

"url": "http://mastersrx-auth.prod.mckesson.com/.well-known/acme-challenge/FlYEVAnjDouyx5eS74eMSJtZa2Dq0itSQIIHFjXIO1E",
"url": "https://mastersrx-auth.prod.mckesson.com//.well-known/acme-challenge/FlYEVAnjDouyx5eS74eMSJtZa2Dq0itSQIIHFjXIO1E"

Where it adds an extra slash after .com

2 Likes
8

tried with double slash in the url and it still works and returns the token

9

could there be any pending authorizations./orders for these domains? Since the api failed with an exception after creating a new order, and the subsequent attempts do not use the older order but instead create a new order in each retry attempt. Would that leave the old order(from the first attempt that failed with exception) in pending state?? Can it be confirmed from lets encrypt server logs

10

I get an empty return.
Did you try it from the Internet?

2 Likes
11

url is working from the internet too, returning the token.

This domain is not propagated to all locations.

Can someone help verify acme server logs whether domain validation was successful and if any pending authorizations?
this is customer owned domain and certificate will be expiring soon.

12

@adongare,

Summary of below is ultimately a response of HTTP/1.1 200 OK is being return when there is no file;
that is an issue. Also HTTP is on Apache and is redirected to HTTPS on Nginx; does your ACME client handle putting the < TOKEN > where Nginx will server it?

Here is what I see presently https://letsdebug.net/auth-test.phs.org/1955124?debug=y

Using nmap to show Ports 80 &443 are Open and that
DNS has 2 IPv4 Addresses 13.248.244.122 and 76.223.106.8

$ nmap -Pn -p80,443 auth-test.phs.org
Starting Nmap 7.80 ( https://nmap.org ) at 2024-05-14 21:39 UTC
Nmap scan report for auth-test.phs.org (13.248.244.122)
Host is up (0.0091s latency).
Other addresses for auth-test.phs.org (not scanned): 76.223.106.8
rDNS record for 13.248.244.122: a556120ce37110a35.awsglobalaccelerator.com

PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.09 seconds

Trying the HTTP-01 challenge with a random token of iqu7zRtwmVzJrKKJ8KV4yKT5gHRid5Jc4Yew3f7L
Expecting a 301 HTTP Response code, but got HTTP/1.1 302 Found; but probably not a big deal as the redirect seems to happen.

$ curl -i http://auth-test.phs.org/.well-known/acme-challenge/iqu7zRtwmVzJrKKJ8KV4yKT5gHRid5Jc4Yew3f7L
HTTP/1.1 302 Found
Date: Tue, 14 May 2024 21:36:20 GMT
Server: Apache
Location: https://auth-test.phs.org//.well-known/acme-challenge/iqu7zRtwmVzJrKKJ8KV4yKT5gHRid5Jc4Yew3f7L
Content-Length: 278
Content-Type: text/html; charset=iso-8859-1

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>302 Found</title>
</head><body>
<h1>Found</h1>
<p>The document has moved <a href="https://auth-test.phs.org//.well-known/acme-challenge/iqu7zRtwmVzJrKKJ8KV4yKT5gHRid5Jc4Yew3f7L">here</a>.</p>
</body></html>

HTTP being redirect to HTTPS with double slash after the .org in the URL
Expecting a 404 HTTP Response code, but got HTTP/1.1 200 OK

$ curl -i https://auth-test.phs.org//.well-known/acme-challenge/iqu7zRtwmVzJrKKJ8KV4yKT5gHRid5Jc4Yew3f7L
HTTP/1.1 200 OK
Date: Tue, 14 May 2024 21:42:20 GMT
Server: nginx
Content-Type: application/octet-stream
Content-Length: 0
x-okta-request-id: ZkPavNqFUvYeZoyBl8vK2QAADQw
x-xss-protection: 0
p3p: CP="HONK"
content-security-policy: frame-ancestors 'self'
x-rate-limit-limit: 100
x-rate-limit-remaining: 99
x-rate-limit-reset: 1715723000
cache-control: no-cache, no-store
pragma: no-cache
expires: 0
referrer-policy: strict-origin-when-cross-origin
accept-ch: Sec-CH-UA-Platform-Version
x-content-type-options: nosniff
Strict-Transport-Security: max-age=315360000; includeSubDomains
X-Robots-Tag: noindex,nofollow
set-cookie: sid="";Version=1;Path=/;Max-Age=0
set-cookie: autolaunch_triggered=""; Expires=Thu, 01 Jan 1970 00:00:10 GMT; Path=/
set-cookie: JSESSIONID=8EE32FC958F8859CC14343C880E7B65B; Path=/; Secure; HttpOnly

HTTP being redirect to HTTPS with double slash REMOVED after the .org in the URL
Same as previous expecting a 404 HTTP Response code, but got HTTP/1.1 200 OK

$ curl -i https://auth-test.phs.org/.well-known/acme-challenge/iqu7zRtwmVzJrKKJ8KV4yKT5gHRid5Jc4Yew3f7L
HTTP/1.1 200 OK
Date: Tue, 14 May 2024 21:43:43 GMT
Server: nginx
Content-Type: application/octet-stream
Content-Length: 0
x-okta-request-id: ZkPbD-um-bbjHqdLw687cwAAB_A
x-xss-protection: 0
p3p: CP="HONK"
content-security-policy: frame-ancestors 'self'
x-rate-limit-limit: 100
x-rate-limit-remaining: 99
x-rate-limit-reset: 1715723083
cache-control: no-cache, no-store
pragma: no-cache
expires: 0
referrer-policy: strict-origin-when-cross-origin
accept-ch: Sec-CH-UA-Platform-Version
x-content-type-options: nosniff
Strict-Transport-Security: max-age=315360000; includeSubDomains
X-Robots-Tag: noindex,nofollow
set-cookie: sid="";Version=1;Path=/;Max-Age=0
set-cookie: autolaunch_triggered=""; Expires=Thu, 01 Jan 1970 00:00:10 GMT; Path=/
set-cookie: JSESSIONID=D07EE3B0F1CE2D8B566DDB97D371A5AA; Path=/; Secure; HttpOnly

Edit
Also this is the double slash
image
and
image

Edit:

Let's Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt
Seems like you may have some geo blocking happening Website Uptime and Availability of auth-test.phs.org at 14 May 2024 03:23:30 PM : Site24x7 Tools

image
image1133×793 12.3 KB

Please read these:

1 Like
13

Actually, this is a bug in acme4j. What's happening here is that acme4j needs a nonce and performs a HEAD request against the newNonce endpoint. This request fails on server side, and now acme4j is trying to read the Problem JSON from the body. However, the body is empty because it's a HEAD request, which results in the AcmeProtocolException. I will fix that with the next release.

The problem of the OP could now be that since acme4j is unable to get a valid nonce, it is also unable to retrieve the current auth status, so the authorization seems to be stuck.

1 Like
14

Is it a "known bug" that it's not handling 429s/503s on newNonce correctly? Is it fixed in a newer version than the 2.11 that they're using?

4 Likes
15

No... It's the first time I heard of this problem. It will be fixed in acme4j v3.2.2.

The fix will just give a more speaking exception than AcmeProtocolException though. The problem is the rate limit that was hit. Without a nonce it's impossible to go on.

1 Like
16

Sure it can't go on, but I could think that it would retry (and I think that they work to ensure that their 429 & 503 responses have a Retry-After header). I haven't used the library myself so I don't know what kind of retry logic its users would expect, or if the user integrating the logic would expect to handle retry logic themselves.

4 Likes
17

The library will throw an AcmeRetryAfterException with the v3.2.2 fix. However it is up to the user to wait until the retry-after time is reached. The reason is that simple clients can just let their thread sleep for the given time, while more sophisticated clients can set a timer and come back later.

2 Likes
18

But even without reworking how the exception is handled, in the case here the user should be able to just try again anyway, right?

@adongare, are you consistently seeing these errors? By any chance, is it when running as a part of a scheduled task at exactly the top of an hour? (0:00 UTC is busiest for Let's Encrypt, but other hour:00 are busy too.)

4 Likes
19

Yes... As soon as the rate limit situation is resolved, the user can just try again.

Unless of course the user is causing the rate limit by polling the status too frequently. The user should either wait for the time given in the AcmeRetryAfterException, or should wait for a reasonable time before retrying. The documentation gives an example for that.

3 Likes
20

Thank you for the info; it's great to have library authors staying on top of issues! Here's the link to the info specifically about the "Service busy" message which may be happening slightly more often than it used to, just in case you didn't see it,

6 Likes