Proxmox failed to issue prod cert but successful with staging

[sgp]

Hi there!

First of all, I want to thank the LE team. I was using wildcard LE certs for my free duckdns domains for a couple of years now. No major issues so far (I was using linuxserver's swag container for issuing certs).

Recently, I've finally registered my own domain with desec.io as a DNS hoster. I'm trying to issue a non-wildcard cert for my home proxmox 8.2.2 node (subdomain is pointing to LAN address) via dns-01 challenge. I was following this instruction: https://www.youtube.com/watch?v=2_PhwHOxytM with changes for desec. At first I wasn't able to issue a cert, nor with prod, nor with staging. A couple of hours later I was able to issue a cert from staging (no settings had been changed). I assumed that some kind of DNS propagation had occurred because it was the same day I registered the domain. So, I decided to wait another 24 hours. A day passed but still no success.

My question is: should I wait more or is there another issue involved? I'm adding proxmox logs for both successful staging task run and failed prod task run.

Fail

Loading ACME account details
Placing ACME order
Order URL: https://acme-v02.api.letsencrypt.org/acme/order/2044487217/321301252657

Getting authorization details from 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/427486193567'
The validation for pve-topton.internal.ahyc.link is pending!
[Sat Nov  9 02:54:11 MSK 2024] Using desec.io api
[Sat Nov  9 02:54:12 MSK 2024] Adding record
[Sat Nov  9 02:54:13 MSK 2024] Added, OK
Add TXT record: _acme-challenge.pve-topton.internal.ahyc.link
Sleeping 30 seconds to wait for TXT record propagation
Triggering validation
Sleeping for 5 seconds
[Sat Nov  9 02:54:49 MSK 2024] Using desec.io api
[Sat Nov  9 02:54:50 MSK 2024] Deleting record
[Sat Nov  9 02:54:50 MSK 2024] Deleted, OK
Remove TXT record: _acme-challenge.pve-topton.internal.ahyc.link
TASK ERROR: validating challenge 'https://acme-v02.api.letsencrypt.org/acme/authz-v3/427486193567' failed - status: invalid

Great success!

Loading ACME account details
Placing ACME order
Order URL: https://acme-staging-v02.api.letsencrypt.org/acme/order/170561893/20318917083

Getting authorization details from 'https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/14803711193'
The validation for pve-topton.internal.ahyc.link is pending!
[Fri Nov  8 05:02:54 MSK 2024] Using desec.io api
[Fri Nov  8 05:02:55 MSK 2024] Adding record
[Fri Nov  8 05:02:56 MSK 2024] Added, OK
Add TXT record: _acme-challenge.pve-topton.internal.ahyc.link
Sleeping 30 seconds to wait for TXT record propagation
Triggering validation
Sleeping for 5 seconds
Status is still 'pending', trying again in 10 seconds
Status is 'valid', domain 'pve-topton.internal.ahyc.link' OK!
[Fri Nov  8 05:03:43 MSK 2024] Using desec.io api
[Fri Nov  8 05:03:43 MSK 2024] Deleting record
[Fri Nov  8 05:03:44 MSK 2024] Deleted, OK
Remove TXT record: _acme-challenge.pve-topton.internal.ahyc.link

All domains validated!

Creating CSR
Checking order status
Order is ready, finalizing order
valid!

Downloading certificate
Setting pveproxy certificate and key
Restarting pveproxy
TASK OK

[MikeMcQ]

@sgp

I assumed that some kind of DNS propagation had occurred

Yes, probably this. Try a longer --dnssleep value with the acme.sh command. Try a couple minutes to start.

All the authoritative DNS servers must sync world-side to ensure Let's Encrypt validation. LE checks from multiple points around the world. You probably got lucky once and desec servers sync'd fast but may generally need longer than 30s.

Note this is not ttl propagation but sync between the auth servers themselves.

[Bruce5051]

Hi @sgp,

Not that long ago I had to increase the delay with deSEC's Name Servers ns1.desec.io and ns2.desec.org using the option --dns-desec-propagation-seconds=300; I had arbitrary selected 300 seconds (5 minutes).
See CAA record prevents issuance - #53 by Bruce5051

[sgp]

Thanks. Changing the proxmox challenge plugin's verification delay to 120 resolved the issue.

Seems kinda strange to me though, as I saw the txt record during every failed validation in desec ui (at least I saw it every time I'd checked).

[sgp]

Thanks.

[MikeMcQ]

@sgp

I saw the txt record during every failed validation in desec

Sure, but does it give any indication of when the sync amongst its servers is complete?

deSEC uses AnyCast which is a broadly distributed system of servers. Their UI might show you made the change request but it takes longer for all the servers to see it. Some DNS systems show the status of this and others do not. That's probably a question for them but this kind of thing is common for DNS queries. It is a main reason ACME Clients even have options like --dnssleep

Perhaps it was just a rare event like this. This also discusses the sync (replication) issues generally

Ns1.desec.io replication issues

[sgp]

Makes sense.
I didn't give it a thought before, but it looks like in the case of all that DNS stuff, it is consistency from the CAP theorem has been sacrificed.