No, it has not. If these errors have been resolved, I'll be happy to manually try another request now.
1 Like
Yes, the DNSSEC errors have been resolved according to dnsviz and @CC980
And, you can see from the Let's Debug link I provided a cert request should succeed. Or, at least not be prevented by DNS failures
EDIT: I probably should have shown the Summary result from Let's Debug instead of the Detailed display. See that here
4 Likes
We've already retried several different certs through section.io, and all are still failing. We still get a few that randomly succeed because they're on a different source, but that's been the case throughout this entire issue.
There's never been an issue with validating ownership via HTTP or DNS. The issue is only on checking the CAA which requires DNS.
Confirmed, the retry results in the same timeout error. Here's the full output from our logs:
Field: ExceptionType from null to Pardot\Acme\Client\DnsException
Field: Error from null to {"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: query timed out looking up CAA for realpage.com","instance":"","code":400}
Can you show the command and the error messages? I am just looking at your first post
What is section.io ?
4 Likes
I agree that they aren't going to do it, and they aren't going to provide direct support.
I can't see it as I'm just one cog in the machine. We were just on an internal call and the person with access was retrying requests (slowly and on different domains) and they were still failing post correcting DNSSEC.
The company that opened this discussion is Salesforce. They sell a product that utilizes LE certs. Section.io is another vendor that also utilizes LE certs to sell a product. It's the same issue with both vendors. I can't get into Salesforce and poke their buttons, but one of our product managers can get into this vendor to poke theirs.
Can you provide some of the domain names that randomly succeed?
And, what do you mean they work because they are on a "different source"
3 Likes
There was only one that succeeded in our tests this AM, and it was in the middle. I don't know the exact full URL, but these are all subdomains under this domain.
By "different source", I mean LE uses multiple sources when querying DNS. It's usually been failing, but sometimes it will succeed because they're using a source address that can reach us. The failure is related to LE's source IPs. As far as we can tell this just started this month.
For reference, we have indeed issued a cert for this domain in the past. Per our logs, the last cert was successfully issued Jul 3, 2023 11:06:52 PM EDT.
On Sep 1, 2023 10:07:51 PM EDT the system started its automated renewal process, and by Sep 1, 2023 10:19:00 PM EDT we hit the same error we're still seeing now.
I manually cleared the error and forced a refresh today at Sep 14, 2023 12:47:44 PM EDT, and the error returned moments later.
1 Like
Hmm. I see these 4 domains got certs most recently. And, there were 22 Let's Encrypt certs for subdomains of that root domain in the past week. The dnsviz report for ellipse subdomain is unusual so surprised that cert request worked. Not sure this is helpful.
I see several certs for subdomains of that root issued nearly every day going back at least several weeks. The history includes certs from several other providers mostly Entrust and Amazon.
@CC980 When you say some LE IP's can't reach you do you know if it is because you actively block them or does it really just never reach your DNS server? I assume you are the domain or DNS operator so could you check the DNS firewall logs to see?
4 Likes
Can someone try obtaining a cert with a single FQDN?
[I can't see anything from crt.sh, so I'm flying blind]
1 Like
We block things through several methods, and I have to be vague with that answer. These are not hitting the DNS server.
We've been trying to associate specific rejections with blocked traffic, but we haven't been able to identify any yet. I believe that's where the issue lies, but without anything to identify the source traffic it's a very large effort. This is still ongoing so I should have some more information later today.
1 Like
We had another pow-wow and turned on all the debugging. This was with section.io and not Salesforce (because we have the button to test).
The requests are coming in and going out, but still failing.
Most of the source IPs from LE were from Amazon in the US.
Comparing the source IPs that we traced during this process with our logs, we have not been rejecting traffic from them. These requests are coming in and going out, but LE says there's an error.
@MikePardotDude Can you press the button from your end and capture the error message for this team?
{"issued":false,"message":"The certificate has not been renewed","error":"Challenge failed"}
9/14/2023 3:06:26 PM 0F64 PACKET 00000210FD8504B0 UDP Rcv 33bb Q [1000 NOERROR] 257 (4)link(8)PaymEnts(8)rEALpAGe(3)cOM(0)
9/14/2023 3:06:26 PM 0F64 PACKET 00000210FD8504B0 UDP Snd 33bb R Q [1084 A NOERROR] 257 (4)link(8)PaymEnts(8)rEALpAGe(3)cOM(0)
9/14/2023 2:53:08 PM 0F64 PACKET 00000210FCE3CD00 UDP Rcv 5497 Q [1000 NOERROR] 257 (6)Qa-mNP(5)mEdIA(8)REaLpAGE(3)COM(0)
9/14/2023 2:53:08 PM 0F64 PACKET 00000210FCE3CD00 UDP Snd 5497 R Q [1084 A NOERROR] 257 (6)Qa-mNP(5)mEdIA(8)REaLpAGE(3)COM(0)
1 Like
{"type":"urn:ietf:params:acme:error:dns","detail":"DNS problem: query timed out looking up CAA for realpage.com","instance":"","code":400}
Is the error we get when we try to clear the SSL error status in our application.
1 Like
Yep, we just tried to refresh twice and got the same error again (posted above by Aritra) both times.
But you can control when the renewals are being attempted.
2 Likes
No, we don't use LetsEncrypt. Various vendors we've purchased services from use LetsEncrypt. When things happen is up to them.
Regardless, we've determined this is something on LetsEncrypts end. We are sending responses to their requests, but LE is still failing.
OK, I see now.
With 200M+ certs in their rotation... I'm confident LE is not to blame for whatever is going on here.
3 Likes