403 Forbidden when trying to generate new LetsEncrypt cert

Help
1

Hey guys, getting stuck on this renewal for some reason. Not sure what's unique about it. We do lots of new certs and renewals without difficulty.

My domain is: geraldtoncityspeedway.com.au

The challenge url appears to be responding as expected: http://geraldtoncityspeedway.com.au/.well-known/acme-challenge/dNEpdSg7NSzf3zM3UFztKti0Y2X1ac05GXGL_X3YzLo

I ran this command: Our system automatically generates certs using a custom AWS lambda. We renew many certs without issue, but are getting stuck on this one for some reason.

output: [{"msg":"Updating cert for geraldtoncityspeedway.com.au, received err Error: Forbidden: {\n "type": "urn:ietf:params:acme:error:orderNotReady",\n "detail": "Order's status (\"invalid\") is not acceptable for finalization",\n "status": 403\n}, Error: Forbidden: {\n "type": "urn:ietf:params:acme:error:orderNotReady",\n "detail": "Order's status (\"invalid\") is not acceptable for finalization",\n "status": 403\n}\n at agent.post.type.send.catch.err (/var/task/src/acme/v2/sendSignedRequestV2.js:17:15)\n at \n at process._tickDomainCallback (internal/process/next_tick.js:228:7)","err":true}]

web server: Apache (version unknown)

operating system: CentOS (version unknown)

hosting provider: AWS

login as root: No

using control panel: No

client: custom implementation

Any guidance is much appreciated.

2

Hey, it's somebody else using a custom AWS Lambda! I'm curious how big this club is. Are you using an existing ACME library to handle making the requests, or is it just completely custom?

That error message means that the order is now in an invalid state (see the diagram for it in the RFC section 7.1.6):

Order objects are created in the "pending" state. Once all of the
authorizations listed in the order object are in the "valid" state,
the order transitions to the "ready" state. The order moves to the
"processing" state after the client submits a request to the order's
"finalize" URL and the CA begins the issuance process for the
certificate. Once the certificate is issued, the order enters the
"valid" state. If an error occurs at any of these stages, the order
moves to the "invalid" state. The order also moves to the "invalid"
state if it expires or one of its authorizations enters a final state
other than "valid" ("expired", "revoked", or "deactivated").

I'm guessing that your authorization challenges failed, and your code isn't logging or diagnosing that properly. My best guess as to why it failed is that your challenge URL you posted only works on IPv4-only networks. That is, the IPv4 address works but the IPv6 address is returning a file-not-found.

$ curl -6 http://geraldtoncityspeedway.com.au/.well-known/acme-challenge/dNEpdSg7NSzf3zM3UFztKti0Y2X1ac05GXGL_X3YzLo
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p>The requested URL was not found on this server.</p>
<p>Additionally, a 404 Not Found
error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>
$ curl -4 http://geraldtoncityspeedway.com.au/.well-known/acme-challenge/dNEpdSg7NSzf3zM3UFztKti0Y2X1ac05GXGL_X3YzLo
dNEpdSg7NSzf3zM3UFztKti0Y2X1ac05GXGL_X3YzLo._nH4Ef7Db7YQ597Gk6blCA3vIRiVC89ME1WZzQmfteo

So you likely have your AAAA DNS records pointed at the wrong server. (And you should probably add some more handling around the challenge validation, since if it fails and the order becomes invalid then finalizing it obviously won't work.)

4 Likes
3

Peter, first of all, THANK YOU for your detailed response. Very, very helpful.

We had run into this a long time ago, and I couldn't recall what was going on. Sure enough, this client had their AAAA records pointing to another unrelated server, which apparently was causing the validation to fail.

Hey, it's somebody else using a custom AWS Lambda! I'm curious how big this club is. Are you using an existing ACME library to handle making the requests, or is it just completely custom?

Good question! I am actually not 100% sure on this as we had a third party put the lambda together; I think it might be a completely custom integration (written in Node). Our system looks up domains in our own db and, when they are ready to be renewed, automatically calls the Lambda. Well, two Lambdas actually as our first lambda gets the "verification string," (which we store), and if that works as expected, our second lambda then attempts to generate the cert. It tries a several times automatically, and if it keeps failing it stops trying.

I'm guessing that your authorization challenges failed, and your code isn't logging or diagnosing that properly. My best guess as to why it failed is that your challenge URL you posted only works on IPv4-only networks. That is, the IPv4 address works but the IPv6 address is returning a file-not-found.

I don't know that there is anything I can do in a case like this where there is a faulty AAAA record as it falls outside of our system. Does something come to mind on your end how I could programmatically respond better in the case of an unclear response like 403 forbidden (which I imagine could mean many things)?

That said, if I could ask a few questions to get a better understanding of what's going on, perhaps I can make some adjustments:

the IPv4 address works but the IPv6 address is returning a file-not-found.

This is confusing to me. The vast majority of our clients do NOT have IPv6 DNS entries, and they are processed without difficulty. I guess the issue here is that the ones which DO have IPv6 entries are going to be tested during the LetsEncrypt verification process, and if they exist, well then they need to be correct. Is that right?

I'm not a networking guy, so I am confused... how is it that LetsEncrypt is issuing requests that use both the IPv4 and the IPv6 DNS entries? I guess their code is designed to look for those DNS entries and if it finds an AAAA record, it will issue both the v4 and v6 requests?

That is baffling to me as I thought all they were doing was hitting the verification url, and if it serves up the correct string, it is valid.

I saw in your curl statement that you used a -6 param. I tried that on my end (with a Macbook), and it did not come back with the 404 that you received, mine came back with the expected verification string response. Any idea why that happened?

I guess, taking a step back, my question is this: what causes a client (any client really, not just LetsEncrypt) to make use of the IPv4 address (the A record) vs. the IPv6 address (the AAAA record) to complete a request? Is it just based on how the browser (or other client) is written? Sorry, again I am not very well versed in networking.

Any insight you can provide is again much appreciated Peter! I'm a knucklehead with this stuff and Googling/researching these questions is not providing me with much clarity.

2 Likes
4

Well, you could have your system check that the validation file is served correctly (on both IPv4 and IPv6) before attempting to validate the challenge with Let's Encrypt. That's probably more custom coding somewhere, though, to go with the rest of your custom work. For digging into a failure, there are also tools like Let's Debug and DNSViz (and the regulars here use a few other tools that are out there that I don't remember at the moment) that can help you find out that your DNS isn't working or your server isn't accessible from the outside world or whatnot.

Also, your custom integration should be returning you the actual error from the validation attempt (which would be more helpful for diagnosing), whereas it looks like it is just trudging through anyway and showing you an error from a later step of the process once there's no chance of it working anymore. So it's not giving you nearly as much to go on as you should be getting.

Sure, they just hit the verification URL and make sure it works. But there are really two "Internets", if you want to think of it that way, the old busted broken IPv4 and the new modern hotness IPv6. And in the meantime where you don't know which platforms your users will have access to, it's best practice to be "dual stack" and ensure that everything is set up on both. Systems that have access to both IPv4 and IPv6 will usually try the IPv6 first (or only), though web browsers sometimes will try both at once and go with whichever responds first. (That's why just looking in a web browser at a server or at a certificate doesn't really tell you everything, since web browsers work hard to muddle through misconfigurations in order to keep users happy.) Let's Encrypt has access to IPv6, so if your server does as well (as can be determined by the existence of an AAAA DNS record) then that's what it'll try, and if some server that's not expecting the challenge is what's answers then they can't issue the certificate (because the server listening for that name can't prove that it's the one requesting it).

I don't have a Mac handy so I don't know maybe if their curl works differently, but my first thought is that you may want to ensure your computer can connect to the IPv6 Internet. Maybe see what http://test-ipv6.com returns for you? If your computer doesn't have access to it, then you're not going to be seeing things the way many other people on the Internet (like Let's Encrypt) do.

The simple version is that if it has access to the IPv6 Internet (meaning that the ISP, router, and network are set up for it, and the program is new enough to understand it) then it'll use the AAAA IPv6 address, and if it isn't then it'll use the A IPv4 address. But like I said, some browsers just try both at once to try to work around various network misconfigurations (like a system that thinks it's on IPv6 but their ISP doesn't actually have IPv6 connectivity). This trying both at once is sometimes called "Happy Eyeballs" if that helps your web searching.

3 Likes
5

And if you want specifics on how Let's Encrypt decides when to fall back to IPv4, they actually have quite a bit described in their documentation:

But of course other software may use other heuristics to determine which protocol to use, and it's usually not documented nearly as well as Let's Encrypt does here.

2 Likes
6

Thanks Peter! Great stuff all around, and again, much appreciated.

Well, you could have your system check that the validation file is served correctly (on both IPv4 and IPv6) before attempting to validate the challenge with Let's Encrypt.

This sounds pretty darn logical to me. That said, I know in the language I typically program in, while I can readily issue http requests programmatically, there doesn't seem to be any way to force that http request to use IPv6 instead of IPv4. Perhaps that is just a shortcoming of the language I am using though :thinking:

In general though, I certainly like your idea of strengthening up the error checking a bit.

But there are really two "Internets", if you want to think of it that way, the old busted broken IPv4 and the new modern hotness IPv6

What is the difference between these "two internets" aside from on just having a zillion more available IP addresses than the other? Whenever I look up IPv4 vs IPv6, all I hear people talking about are the huge amount of IP addresses offered by the latter, but nothing else of significance. Am I missing an entire different portion of the discussion, or is that all there is to it? Like, is it faster or something else?

The rest of what you said about LetsEncrypt "having access" to IPv6 and therefore using the AAAA record if available all made perfect sense to me, so thank you.

Maybe see what http://test-ipv6.com returns for you?

I did check this and according to the report: "It looks like you have only IPv4 Internet service at this time." Guess my ISP (here at the office) doesn't support it :man_shrugging:

But like I said, some browsers just try both at once to try to work around various network misconfigurations (like a system that thinks it's on IPv6 but their ISP doesn't actually have IPv6 connectivity). This trying both at once is sometimes called "Happy Eyeballs " if that helps your web searching.

Interesting stuff, thanks for all the insight Peter!

OH, one last question: we have resolved this everything it has come up by removing the erroneous "quad A" DNS entry... but what if we instead wanted to explicitly support IPv6, and rather update the IPv6 DNS entry to point to our servers? I guess this would potentially open a can of worms in terms of ensuring the entry point to our system was fully setup with a static IPv6 address. Is there more to it than that, or do you think that is all we would need to consider?

2 Likes
7

The short story: Then it should work just as the IPv4 works. [ensure that it does in test/staging first]

The needlessly and ridiculously long story:

Summary

Think of IPv6 as a cell phone while IPv4 as a land line.
They both place calls and receive calls.
[Although phones can call each other directly, that is not exactly the case with IPv4 <> IPv6]
So maybe it's a bit more like FM and XM Radio or similar Digital and Analog devices.
In the end they can do very similar things; But they do them is sometimes very different ways.

So..., The caller doesn't know, nor care much about, the calling mechanisms involved nor the technology being used; They just care about the call quality and the overall user experience.
Similarly, like with your programming language, most internet things don't really care about how things happen; They just want to get from A to B and have that "experience" be at their expected level.
They don't try to reinvent a working wheel, they just use whatever transportation system is available to them.

Soo...... HTTP (or HTTPS) is still HTTP (or HTTPS); Whether it is via IPv4 or IPv6 or BOTH. The systems that expect them, will continue to expect them as they always have. The multiple "paths" scenario can present an implementation decision that increases the connection possibilities from just 0,1 (pass/fail) to 00,01,10,11; Which, of course, there is no set standard decision in place (that I know of) to dictate order of operations/preference/etc. for handling what to do when it deviates from 11.

Again, all these things mostly happen way below the things most of us interact with directly.
Like who ever talks directly to an Internet router, yet every single IP thing that reaches us travels through them.

If your programming language "dictionary" doesn't even have "words" for it... how can you program anything with it?
Well... I'm glad you asked.
You could delegate one system to check IPv4 and one system to check IPv6.
[you probably guessed by now, they each only have one externally facing IP stack - so they can only do things one way]
From where you sit (program), you merely call an app on two different systems (appserver4 & appserver6) and deal with their results generically.

If you want to read more about this story subscribe to my weekly newsletter... where the saga continews (JK)

2 Likes
8

There's probably a way to do it, though that doesn't mean it'll be easy for some of them. You mentioned your code was written in Node, and Node does have a "family" parameter for its http request method to specify whether to use IPv4 or v6.

It's probably not really much faster in practice, though it is for some people. My main point was really just that there are two Internets, so you need to be able to handle both of them. The larger address space of IPv6 though allows you to do away with a lot of the hacks added to IPv4, so you can do things like each system can actually know its "real" public IP address and you don't need systems in-between to try to translate between "internal" and "real" IPs. And then things like routing tables between networks can consolidate routes a lot more since a company can just have one big contiguous space of IPv6 rather than whatever haphazard blocks of IPv4 they've managed to buy. It just makes things a lot easier to not need to worry about conserving those addresses. But most of that doesn't matter if you're not being a network engineer, you just (for your own use case) need to understand that IPv4 hosts can only talk to IPv4, and IPv6 hosts can only talk to IPv6. And most hosts are (or at least should be) on both if their network teams know what they're doing (and have had the funding and such to make it a priority).

Then it's going to be impossible to test how Let's Encrypt (or any customers with IPv6 access) will see your web sites directly from your office. It may be that your ISP supports it and your company router doesn't, or just hasn't been configured to use it yet. In the meantime, you probably want to try using tools that do have IPv6 access, and bugging your IT department (or whomever would be making the decisions and setting up your office's network) that a lack of IPv6 is making it harder to do your job.

Well, your servers would need an IPv6 address that it was listening to and the AAAA record pointed to it, just like they currently have an IPv4 address with an A record pointed to it. Whether that's all you need to consider depends on whether your servers' Internet provider and network makes it easy to get you that IPv6 address and access.

2 Likes
9

Thanks for this great response. The analogies were very helpful, and what you said about configuring different servers to do the IPv4 check and the IPv6 check made a lot of sense.

Lol'd a the newsletter bit as well. Sign me up!

2 Likes
10

Thanks a lot for the extra insight Peter! I appreciate you helping my wrap my melon around this, and I plan to re-read it several times and also refer back to it down the road as well :pray:

2 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.