403: Order's status ("invalid") is not acceptable for finalization

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
we are a hosting provider but one example of a failing domain is:
visitpearlofafrica.com

I ran this command:
we are using various Perl modules (LE, ACME, etc) to send customer domain SSL requests via the api: https://acme-v02.api.letsencrypt.org/acme/

It produced this output:
[Sun Jul 9 13:47:37 2023] [error] Param: domain => visitpearlofafrica.com
[Sun Jul 9 13:47:37 2023] [error] Param: hostname => visitpearlofafrica.com
[Sun Jul 9 13:47:37 2023] [error] [can_get_cert] info hash for visitpearlofafrica.com: {
'domain' => 'visitpearlofafrica.com',
'full_domain' => 'visitpearlofafrica.com',
'root_domain' => 'visitpearlofafrica.com',
[Sun Jul 9 13:47:37 2023] [debug] Need a cert for: /wildcard.visitpearlofafrica.com.crtc
[Sun Jul 9 13:47:37 2023] [error] Call get_acme_order with: , visitpearlofafrica.com, visitpearlofafrica.com, /,31306, 0
Sun Jul 9 13:47:37 2023: [get_acme_order] Domains: *.visitpearlofafrica.com, visitpearlofafrica.com
'value' => 'visitpearlofafrica.com'
'value' => 'visitpearlofafrica.com'
Sun Jul 9 13:47:38 2023: [redis_set_pending_authz] Adding pending status for visitpearlofafrica.com::https://acme-v02.api.letsencrypt.org/acme/authz-v3/244110561827
Sun Jul 9 13:47:38 2023: [redis_set_pending_authz] Adding pending status for visitpearlofafrica.com::https://acme-v02.api.letsencrypt.org/acme/authz-v3/244110561837
Sun Jul 9 13:47:38 2023: [_authz_handler] Started with zone: visitpearlofafrica.com, with challenges: bless( {
Sun Jul 9 13:47:38 2023: [_authz_handler] Challenge for: visitpearlofafrica.com => bless( {
Sun Jul 9 13:47:38 2023: [_authz_handler] [visitpearlofafrica.com] Need a DNS record at _acme-challenge.visitpearlofafrica.com with content: 0xB_l4IefvzNv44NvWwKQl7BZg5K5cSBy4Jf4U-t2_Y
Sun Jul 9 13:47:38 2023: [handle_dns] Started with base_domain==visitpearlofafrica.com, hostname_from_challenge==visitpearlofafrica.com, content==0xB_l4IefvzNv44NvWwKQl7BZg5K5cSBy4Jf4U-t2_Y
Sun Jul 9 13:47:38 2023: [add_cname_record] Args: visitpearlofafrica.com, _acme-challenge.visitpearlofafrica.com, visitpearlofafrica.com.letsencrypt.vdeck.eigdyn.com
Sun Jul 9 13:47:38 2023: [add_cname_record] CNAME record for _acme-challenge.visitpearlofafrica.com already exists (domain: visitpearlofafrica.com, id: 60604353)
Sun Jul 9 13:47:38 2023: [add_or_replace_txt_record] Inserting record for visitpearlofafrica.com.letsencrypt.vdeck.eigdyn.com (87732662): 0xB_l4IefvzNv44NvWwKQl7BZg5K5cSBy4Jf4U-t2_Y
Sun Jul 9 13:47:38 2023: [redis_add_txt] Add TXT to redis: visitpearlofafrica.com.letsencrypt.vdeck.eigdyn.com, 0xB_l4IefvzNv44NvWwKQl7BZg5K5cSBy4Jf4U-t2_Y
Sun Jul 9 13:47:38 2023: [_authz_handler] Started with zone: visitpearlofafrica.com, with challenges: bless( {
Sun Jul 9 13:47:38 2023: [_authz_handler] Challenge for: visitpearlofafrica.com => bless( {
Sun Jul 9 13:47:38 2023: [_authz_handler] [visitpearlofafrica.com] Need a DNS record at _acme-challenge.visitpearlofafrica.com with content: XhGkFtSuXahFB7UOv5cIgmfU0ZtO7UaOScVRaPuZnNY
Sun Jul 9 13:47:38 2023: [handle_dns] Started with base_domain==visitpearlofafrica.com, hostname_from_challenge==visitpearlofafrica.com, content==XhGkFtSuXahFB7UOv5cIgmfU0ZtO7UaOScVRaPuZnNY
Sun Jul 9 13:47:38 2023: [add_cname_record] Args: visitpearlofafrica.com, _acme-challenge.visitpearlofafrica.com, visitpearlofafrica.com.letsencrypt.vdeck.eigdyn.com
Sun Jul 9 13:47:38 2023: [add_cname_record] CNAME record for _acme-challenge.visitpearlofafrica.com already exists (domain: visitpearlofafrica.com, id: 60604353)
Sun Jul 9 13:47:38 2023: [add_or_replace_txt_record] Inserting record for visitpearlofafrica.com.letsencrypt.vdeck.eigdyn.com (87732662): XhGkFtSuXahFB7UOv5cIgmfU0ZtO7UaOScVRaPuZnNY
Sun Jul 9 13:47:38 2023: [redis_add_txt] Add TXT to redis: visitpearlofafrica.com.letsencrypt.vdeck.eigdyn.com, XhGkFtSuXahFB7UOv5cIgmfU0ZtO7UaOScVRaPuZnNY
Sun Jul 9 13:47:39 2023: [handle_authorizations] Accepting challenges for *.visitpearlofafrica.com
Sun Jul 9 13:47:39 2023: [handle_authorizations] Dump of challenge for *.visitpearlofafrica.com: bless( {
Sun Jul 9 13:47:39 2023: [handle_authorizations] Dump of challenge for *.visitpearlofafrica.com: bless( {
Sun Jul 9 13:47:40 2023: [handle_authorizations] Done accepting challenges for *.visitpearlofafrica.com
Sun Jul 9 13:47:40 2023: [redis_clear_pending_authz] Deleting pending status for visitpearlofafrica.com::https://acme-v02.api.letsencrypt.org/acme/authz-v3/244110561827 (0/1)
Sun Jul 9 13:47:40 2023: [redis_clear_pending_authz] Deleting pending status for visitpearlofafrica.com::https://acme-v02.api.letsencrypt.org/acme/authz-v3/244110561837 (0/1)
Sun Jul 9 13:47:40 2023: [_make_key_for_domains] Looking for existing key for *.visitpearlofafrica.com at: /wildcard.visitpearlofafrica.com.key
Sun Jul 9 13:47:40 2023: [_make_key_for_domains] Use existing key for *.visitpearlofafrica.com
Sun Jul 9 13:47:40 2023: [handle_authorizations] Creating CSR for *.visitpearlofafrica.com
Sun Jul 9 13:47:40 2023: [_make_key_for_domains] Generating CSR for *.visitpearlofafrica.com
Sun Jul 9 13:47:40 2023: [_make_key_for_domains] CSR req for .visitpearlofafrica.com: openssl req -new -sha384 -key /wildcard.visitpearlofafrica.com.key -subj '/CN=.visitpearlofafrica.com' -reqexts SAN -extensions SAN -config /wildcard.visitpearlofafrica.com.cnf
Sun Jul 9 13:47:40 2023: [handle_authorizations] Calling finalize_order for .visitpearlofafrica.com
[Sun Jul 9 13:47:43 2023] [error] [call_acme] [visitpearlofafrica.com/
.visitpearlofafrica.com] Exception in handle_authorizations: Net::ACME2::x::ACME: “https://acme-v02.api.letsencrypt.org/acme/finalize/36261705/193867515727” indicated an ACME error: 403 Forbidden (403 urn:ietf:params:acme:error:orderNotReady (Order's status ("invalid") is not acceptable for finalization)).SCALAR(0x1e5ad364c7b0) ==> Net::ACME2::x::Generic::new('Net::ACME2::x::ACME', '“https://acme-v02.api.letsencrypt.org/acme/finalize/36261705/193867515727” indicated an ACME error: 403 Forbidden (403 urn:ietf:params:acme:error:orderNotReady (Order's status ("invalid") is not acceptable for finalization)).

it seems to be dying here, when the csr/order is sent to acme:

    # Paths
    my $cert_path = $cert_dir . "/" . $output_name . ".crtc";
    my $key_path = $cert_dir . "/" . $output_name . ".key";
    my $cert_path_final = $final_dir . "/" . $output_name . ".crtc";
    my $key_path_final = $final_dir . "/" . $output_name . ".key";

    # Load existing key or make a new one
    my $key = _make_key_for_domains( $domains[0], $key_path );

    # If we couldn't make a key, fail it
    unless ( $key ) {
            _log( 1, "[handle_authorizations] Unable to create key for $domains[0]" );
            return;
    } # end unless

    # Debug
    _log( 128, "[handle_authorizations] Creating CSR for $domains[0]" );

    # Generate CSR
    my $csr = _make_csr_for_domains( $key, $key_path, @domains );
    # my $csr = _make_csr_for_domains( $key, @domains );

    # Debug
    _log( 128, "[handle_authorizations] Calling finalize_order for $domains[0]" );

    # Sleep for a sec to let things settle
    sleep 3;

    # Send CSR to LE
    $acme->finalize_order( $order, $csr ); <---- doesn't make it past here, as we don't see the "begin order wait loop" in the line below this

    # Debug
    _log( 128, "[handle_authorizations] Begin order wait loop for $domains[0]" );

I can see that the TXT/CNAME records are generated properly, but for some reason the status is "invalid".

[cmauldin@C02XK26TJG5L-M ~]
[Sun Jul 09 11:42:25]$ dig TXT _acme-challenge.visitpearlofafrica.com +short
visitpearlofafrica.com.letsencrypt.vdeck.eigdyn.com.

[cmauldin@C02XK26TJG5L-M ~]
[Sun Jul 09 11:43:03]$ dig CNAME _acme-challenge.visitpearlofafrica.com +short
visitpearlofafrica.com.letsencrypt.vdeck.eigdyn.com.

The cnr/key files for the csr generation do exist (we keep them in the 'tmp' dir and then the final crt is placed in the same path but without 'tmp'):

ll / |grep visitpearl

-rw-rw-rw- 1 nobody nogroup 11080 Jul 8 16:12 wildcard.visitpearlofafrica.com.cnf
-rw-rw-rw- 1 nobody nogroup 288 Jul 8 16:12 wildcard.visitpearlofafrica.com.key

For some background, this was all architected by someone no longer with our company and I'm pretty new to LE. If any additional information, such as more info from the perl subroutines, is required please let me know

My web server is (include version):
nginx 1:1.17.8-1

The operating system my web server runs on is (include version): Ubuntu 18.04.5

My hosting provider, if applicable, is: we're the provider :slight_smile:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): the customers do, we're troubleshooting via SSH

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot isn't installed, not sure where I would find this information

Certbot is just an example, you're using a different ACME client, so this question requests the version of that piece of software.

That said, it's unfortunate the software used doesn't relay the actual error from the ACME server:

urn:ietf:params:acme:error:dns", "detail": "DNS problem: SERVFAIL looking up TXT for _acme-challenge.visitpearlofafrica.com - the domain's nameservers may be malfunctioning

You can see this error in both the authorizations (links with authz-v3 in them).

This means something isn't going correctly when trying to resolve the TXT record. You can use services like DNSViz to see what's going on with your DNS: _acme-challenge.visitpearlofafrica.com | DNSViz

Another good site to check DNS resolving as close as how LE resolves the DNS records is unboundtest.com, but the unbound library logs are harder to read: https://unboundtest.com/m/TXT/_acme-challenge.visitpearlofafrica.com/WKCG2TPB

Both sites clearly indicate a DNS issue, which should be your current main problem to resolve.

3 Likes

Hey Osiris,

Thanks for the quick reply.

I see what you mean about the TXT record not being seen but instead the CNAME is returned.

Although I'm still a bit confused. This error isn't occurring on every domain and here is an example of the same DNS setup but the cert being issued yesterday via the same scripts:

[cmauldin@C02XK26TJG5L-M ~]
[Sun Jul 09 02:03:23]$ sslcheck aranval.com
Issuer: C=US, O=Let's Encrypt, CN=R3
Not After : Oct 7 09:40:35 2023 GMT
CA Issuers - URI:http://r3.i.lencr.org/
DNS:*.aranval.com, DNS:aranval.com
[cmauldin@C02XK26TJG5L-M ~]
[Sun Jul 09 02:03:33]$ dig TXT _acme-challenge.aranval.com

; <<>> DiG 9.10.6 <<>> TXT _acme-challenge.aranval.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56674
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4000
;; QUESTION SECTION:
;_acme-challenge.aranval.com. IN TXT

;; ANSWER SECTION:
_acme-challenge.aranval.com. 3600 IN CNAME aranval.com.letsencrypt.vdeck.eigdyn.com.

;; Query time: 346 msec
;; SERVER: 10.250.2.34#53(10.250.2.34)
;; WHEN: Sun Jul 09 14:03:50 MST 2023
;; MSG SIZE rcvd: 107

Log file:

[Sat Jul 8 18:10:46 2023] [error] Param: domain => aranval.com
[Sat Jul 8 18:10:46 2023] [error] Param: hostname => webmail.aranval.com
[Sat Jul 8 18:10:46 2023] [error] [can_get_cert] info hash for aranval.com: {
'domain' => 'aranval.com',
'full_domain' => 'aranval.com',
'root_domain' => 'aranval.com',
[Sat Jul 8 18:10:46 2023] [debug] Need a cert for: /wildcard.aranval.com.crtc
[Sat Jul 8 18:10:46 2023] [error] Call get_acme_order with: redacted, aranval.com, webmail.aranval.com, , 0
Sat Jul 8 18:10:46 2023: [get_acme_order] Domains: *.aranval.com, aranval.com
'hostname' => 'aranval.com'
'value' => 'aranval.com'
'hostname' => 'aranval.com'
'value' => 'aranval.com'
Sat Jul 8 18:10:46 2023: [redis_set_pending_authz] Adding pending status for aranval.com::https://acme-v02.api.letsencrypt.org/acme/authz-v3/237047161257
Sat Jul 8 18:10:46 2023: [redis_set_pending_authz] Adding pending status for aranval.com::https://acme-v02.api.letsencrypt.org/acme/authz-v3/237047161267
Sat Jul 8 18:10:46 2023: [redis_clear_pending_authz] Deleting pending status for aranval.com::https://acme-v02.api.letsencrypt.org/acme/authz-v3/237047161257 (0/1)
Sat Jul 8 18:10:46 2023: [handle_authorizations] This account is already authorized on aranval.com
Sat Jul 8 18:10:46 2023: [redis_clear_pending_authz] Deleting pending status for aranval.com::https://acme-v02.api.letsencrypt.org/acme/authz-v3/237047161267 (0/1)
Sat Jul 8 18:10:46 2023: [handle_authorizations] This account is already authorized on aranval.com
Sat Jul 8 18:10:47 2023: [handle_authorizations] Accepting challenges for *.aranval.com
Sat Jul 8 18:10:47 2023: [handle_authorizations] Done accepting challenges for *.aranval.com
Sat Jul 8 18:10:47 2023: [_make_key_for_domains] Looking for existing key for *.aranval.com at: /wildcard.aranval.com.key
Sat Jul 8 18:10:47 2023: [_make_key_for_domains] Use existing key for *.aranval.com
Sat Jul 8 18:10:47 2023: [handle_authorizations] Creating CSR for *.aranval.com
Sat Jul 8 18:10:47 2023: [_make_key_for_domains] Generating CSR for *.aranval.com
Sat Jul 8 18:10:47 2023: [_make_key_for_domains] CSR req for .aranval.com: openssl req -new -sha384 -key /wildcard.aranval.com.key -subj '/CN=.aranval.com' -reqexts SAN -extensions SAN -config /wildcard.aranval.com.cnf
Sat Jul 8 18:10:47 2023: [handle_authorizations] Calling finalize_order for *.aranval.com
Sat Jul 8 18:10:51 2023: [handle_authorizations] Begin order wait loop for *.aranval.com
Sat Jul 8 18:10:51 2023: [handle_authorizations] Begin order wait loop for *.aranval.com
Sat Jul 8 18:10:51 2023: [_get_cert_from_le] Get cert for *.aranval.com from https://acme-v02.api.letsencrypt.org/acme/cert/03102be68764154b931dc4ad383157f46d8c and write to /wildcard.aranval.com.crtc

1 Like

This might have to do with the success, as the authz was already validated in June. See https://acme-v02.api.letsencrypt.org/acme/authz-v3/237047161267. When an authz is reused, no validation has to be done at the time of re-use.

That said, it looks like the TXT RR are removed after any attempt, so that might also be a cause of the current DNSViz results. Although any SERVFAIL is NOT suppose to happen. If a TXT RR cannot be found, the DNS server should respond with "NXDOMAIN" or simply with "NOERR" but with 0 answers. Not SERVFAIL.

My money is still on a problem with the DNS server of the letsencrypt.vdeck.eigdyn.com. zone.

3 Likes

Ah ok. Roger, I'll keep digging into that then. Thanks again for the quick replies.

3 Likes

It might be helpful with debugging if you could somehow pause the ACME client just after the TXT RR was added to the DNS zone.

3 Likes

Hey Osiris, thanks a ton for helping out. We resolved the issue with the DNS server and certs are being generated successfully. Much appreciated!

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.