Failed to issue the certificate even http01 challenge succeeded

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: davidshen84.duckdns.org

I ran this command:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  acme:
    email: davidshen84@---
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-staging-issuer-account-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: traefik-gateway
                namespace: kube-system
                kind: Gateway
---

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: letsencrypt-staging
  namespace: cert-manager
spec:
  issuerRef:
    name: letsencrypt-staging
    kind: Issuer
  secretName: letsencrypt-staging-tls
  dnsNames:
    - davidshen84.duckdns.org
  emailAddresses:
    - davidshen84@---

It produced this output:

After a while, after the solver pod as been deleted, I got this:

Failed to wait for order resource "letsencrypt-staging-1-2542508985" to become ready: order is in "invalid" state:

My web server is (include version): traefik gateway

The operating system my web server runs on is (include version): linux, k3s

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don't know): no

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): n/a


From the log of the challenge solver, I found these messages:

I0802 06:55:57.992856       1 solver.go:104] "comparing token" logger="cert-manager.acmesolver" host="davidshen84.duckdns.org" path="/.well-known/acme-challenge/toke-redacted" base_path="/.well-known/acme-challenge" token="toke-redacted" headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.18.2 (linux/amd64) cert-manager/686afa62160249022a0205d3a9bf5f059476c684"],"X-Forwarded-For":["2001:cafe:42::1"],"X-Forwarded-Host":["davi shen84.duckdns.org"[],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["traefik-5dd8bf4ff-lzgsd"],"X-Real-Ip":["2001:cafe:42::1"]} expected_token="toke-redacted"
I0802 06:55:57.992872       1 solver.go:112] "got successful challenge request, writing key" logger="cert-manager.acmesolver" host="davidshen84.duckdns.org" path="/.well-known/acme-challenge/toke-redacted" base_path="/.well-known/acme-challenge" token="toke-redacted" headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.18.2 (linux/amd64) cert-manager/686afa62160249022a0205d3a9bf5f059476c684"],"X-Forwarded-For":["2001:cafe:42:1"[],"X-Forwarded-Host":["davidshen84.duckdns.org"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["traefik-5dd8bf4ff-lzgsd"],"X-Real-Ip":["2001:cafe:42::1"]}
I0802 06:55:59.995587       1 solver.go:89] "validating request" logger="cert-manager.acmesolver" host="davidshen84.duckdns.org" path="/.well-known/acme-challenge/toke-redacted" base_path="/.well-known/acme-challenge" token="toke-redacted" headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.18.2 (linux/amd64) cert-manager/686afa62160249022a0205d3a9bf5f059476c684"],"X-Forwarded-For":["2001:cafe:42::1"],"X-Forwarded-Host":["daidshen84.duckdns.org"[],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["traefik-5dd8bf4ff-lzgsd"],"X-Real-Ip":["2001:cafe:42::1"]}
I0802 06:55:59.995633       1 solver.go:97] "comparing host" logger="cert-manager.acmesolver" host="davidshen84.duckdns.org" path="/.well-known/acme-challenge/toke-redacted" base_path="/.well-known/acme-challenge" token="toke-redacted" headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.18.2 (linux/amd64) cert-manager/686afa62160249022a0205d3a9bf5f059476c684"],"X-Forwarded-For":["2001:cafe:42::1"],"X-Forwarded-Host":["davidsen84.duckdns.org"[],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["traefik-5dd8bf4ff-lzgsd"],"X-Real-Ip":["2001:cafe:42::1"]} expected_host="davidshen84.duckdns.org"
I0802 06:55:59.995662       1 solver.go:104] "comparing token" logger="cert-manager.acmesolver" host="davidshen84.duckdns.org" path="/.well-known/acme-challenge/toke-redacted" base_path="/.well-known/acme-challenge" token="toke-redacted" headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.18.2 (linux/amd64) cert-manager/686afa62160249022a0205d3a9bf5f059476c684"],"X-Forwarded-For":["2001:cafe:42::1"],"X-Forwarded-Host":["davishen84.duckdns.org"[],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["traefik-5dd8bf4ff-lzgsd"],"X-Real-Ip":["2001:cafe:42::1"]} expected_token="toke-redacted"
I0802 06:55:59.995680       1 solver.go:112] "got successful challenge request, writing key" logger="cert-manager.acmesolver" host="davidshen84.duckdns.org" path="/.well-known/acme-challenge/toke-redacted" base_path="/.well-known/acme-challenge" token="toke-redacted" headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.18.2 (linux/amd64) cert-manager/686afa62160249022a0205d3a9bf5f059476c684"],"X-Forwarded-For":["2001:cafe:42:1"[],"X-Forwarded-Host":["davidshen84.duckdns.org"],"X-Forwarded-Port":["80"],"X-Forwarded-Proto":["http"],"X-Forwarded-Server":["traefik-5dd8bf4ff-lzgsd"],"X-Real-Ip":["2001:cafe:42::1"]}

Apparently, the letsencrypt issuer server can access my service. But evaultally, it failed to issue the certificate.

The only thing I am not certain is the ""X-Real-Ip":["2001:cafe:42::1"]" in the log. That is the IP of the service in my k8s cluster, not the public IP. I hope it doesn't matter.

Add more log from the "cert-manager" pod:

E0802 06:56:30.991319       1 sync.go:371] "error waiting for authorization" err="acme: authorization error for davidshen84.duckdns.org: 400 urn:ietf:params:acme:error:connection: 202.128.117.26: Fetching http://davidshen84.duckdns.org/.well-known/acme-challenge/FQlJNt1vA_3GX8hV2PK7cmzLF76cG0Tuo5yTLEKzkh4: Timeout during connect (likely firewall problem)" logger="cert-manager.controller.acceptChallenge" resource_name="letsencrypt-staging-1-2542508985-3648348455" resource_namespace="cert-manager" resource_kind="Challenge" resource_version="v1" dnsName="davidshen84.duckdns.org" type="HTTP-01"
I0802 06:56:31.201025       1 conditions.go:201] "Found status change for Certificate condition; setting lastTransitionTime" logger="cert-manager" certificate="cert-manager/letsencrypt-staging" condition="Issuing" oldStatus="True" status="False" lastTransitionTime="2025-08-02 06:56:31.201012414 +0000 UTC m=+6943.505474906"
I0802 06:56:31.212706       1 trigger_controller.go:204] "Backing off from issuance due to previously failed issuance(s). Issuance will next be attempted at 2025-08-03 14:56:31.00000104 +0000 UTC m=+122143.304463541" logger="cert-manager.controller" key="cert-manager/letsencrypt-staging"
I0802 06:56:31.228530       1 trigger_controller.go:204] "Backing off from issuance due to previously failed issuance(s). Issuance will next be attempted at 2025-08-03 14:56:31.000001094 +0000 UTC m=+122143.304463589" logger="cert-manager.controller" key="cert-manager/letsencrypt-staging"
I0802 06:56:31.231977       1 controller.go:152] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" error="Operation cannot be fulfilled on certificates.cert-manager.io \"letsencrypt-staging\": the object has been modified; please apply your changes to the latest version and try again"
E0802 06:56:31.370724       1 sync.go:78] "failed to update status" logger="cert-manager.controller" resource_name="letsencrypt-staging-1-2542508985" resource_namespace="cert-manager" resource_kind="Order" resource_version="v1"
I0802 06:56:31.370756       1 controller.go:152] "re-queuing item due to optimistic locking on resource" logger="cert-manager.controller" error="Operation cannot be fulfilled on orders.acme.cert-manager.io \"letsencrypt-staging-1-2542508985\": the object has been modified; please apply your changes to the latest version and try again"

The error:

err="acme: authorization error for davidshen84.duckdns.org: 400 urn:ietf:params:acme:error:connection: 202.128.117.26

That is my ISP's IP. I don't have a public IPv4 address. According to letsencrypt document, it prefers ipv6. I was betting it would never try my ipv4 address.

@davidshen84, welcome to the community! :slightly_smiling_face:

I do not know how the name servers of duckdns is working, but it looks quite bad. For an NS type query, it gave an A type answer (and there you can see the offending IPv4 address):

tumbleweed:~ # dig NS davidshen84.duckdns.org @35.182.183.211
;; communications error to 35.182.183.211#53: timed out

; <<>> DiG 9.20.10 <<>> NS davidshen84.duckdns.org @35.182.183.211
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26624
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 9, ADDITIONAL: 10
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;davidshen84.duckdns.org.       IN      NS

;; ANSWER SECTION:
davidshen84.duckdns.org. 60     IN      A       202.128.117.26

;; AUTHORITY SECTION:
duckdns.org.            600     IN      NS      ns8.duckdns.org.
duckdns.org.            600     IN      NS      ns9.duckdns.org.
duckdns.org.            600     IN      NS      ns1.duckdns.org.
duckdns.org.            600     IN      NS      ns2.duckdns.org.
duckdns.org.            600     IN      NS      ns3.duckdns.org.
duckdns.org.            600     IN      NS      ns4.duckdns.org.
duckdns.org.            600     IN      NS      ns5.duckdns.org.
duckdns.org.            600     IN      NS      ns6.duckdns.org.
duckdns.org.            600     IN      NS      ns7.duckdns.org.

;; ADDITIONAL SECTION:
ns8.duckdns.org.        600     IN      A       15.223.106.16
ns9.duckdns.org.        600     IN      A       15.222.19.97
ns1.duckdns.org.        600     IN      A       99.79.143.35
ns2.duckdns.org.        600     IN      A       35.182.183.211
ns3.duckdns.org.        600     IN      A       35.183.157.249
ns4.duckdns.org.        600     IN      A       3.97.51.116
ns5.duckdns.org.        600     IN      A       99.79.16.64
ns6.duckdns.org.        600     IN      A       3.97.58.28
ns7.duckdns.org.        600     IN      A       15.223.21.81

;; Query time: 103 msec
;; SERVER: 35.182.183.211#53(35.182.183.211) (UDP)
;; WHEN: Sat Aug 02 08:04:05 UTC 2025
;; MSG SIZE  rcvd: 374

tumbleweed:~ #

I wanted to check if an AAAA record exists for davidshen84.duckdns.org, but DuckDNS looks to be down for like 80 % at the moment...

I guess you're lucky Let's Encrypt tried your IPv4 address at all.. It probably had trouble resolving the AAAA RR you need.

That said, I'm getting a timeout on port 80 from 2400:a844:5bd5::cafe:2, so even IPv6 wouldn't work.

curl -6v http://davidshen84.duckdns.org/hi

Works...but yeah, very high timeout rate! I'm not sure if it's due to my ISP or my network setup.

What do you mean "offending ipv6 address"? Are you referring to the "202.*" address? It is not my real public ipv4 address. I don't have a public ipv4 address. My server is behind my ISP's CGNAT.

I was betting letsencrypt issuer server will prefer ipv6 record and ignore that invalid ipv4 address.

Does letsencrypt require both ipv4 and ipv6 addresses to be valid?

It does prefer IPv6. However, if IPv6 somehow fails (timeout at the DNS level getting an AAAA RR or when trying to connect to the IPv6 address), it falls back to IPv4.

Nope.

I wrote IPv4 not IPv6. And yes, that is the same IP address supplied by duckdns as visible in the log.

I read somewhere that I have to set an ipv4 address, I'd prefer not to set a useless ipv4 address there.

I dont know if it is duckdns nor working properly, or my ISP's network. I can access davidshen84.duckdns.org from my home computer. But if I use my mobile phone network or a vpn, I cannot access that domain. I got connection reset error in the browser.

Traceroute Online - Tracert Tracks Full Path of IP Packet or Domain gives me this:

Start: 2025-08-03T04:43:03+0500
HOST: DNSChecker.org                                                                                      Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                                                                                                 100.0     3    0.0   0.0   0.0   0.0   0.0
  2.|-- fd00:0:7::2b4                                                                                        0.0%     3    0.5   0.9   0.5   1.3   0.4
  3.|-- 2604:a880:ffff::1:a                                                                                  0.0%     3   21.0   7.8   1.1  21.0  11.4
  4.|-- 2a03:b0c0:fffd::a4                                                                                   0.0%     3    0.4   0.6   0.4   0.8   0.2
  5.|-- 2a03:b0c0:fffe::15c                                                                                  0.0%     3    0.9   1.3   0.9   1.6   0.3
  6.|-- 2a03:b0c0:fffe::113                                                                                  0.0%     3    0.7   0.9   0.7   1.0   0.2
  7.|-- ???                                                                                                 100.0     3    0.0   0.0   0.0   0.0   0.0
  8.|-- po3.chi-eqxch1-cr7.globalsecurelayer.com (2401:3cc0:10:50::1:44)                                     0.0%     3   20.2  20.3  20.2  20.5   0.2
  9.|-- po20.lax-csla2-bb5.globalsecurelayer.com (2401:3cc0:10:50::1:4)                                      0.0%     3   65.9  66.1  65.9  66.3   0.2
 10.|-- e1.syd-eqxsy5-bb1.globalsecurelayer.com (2401:3cc0:10:50::1:3)                                       0.0%     3  198.6 198.8 198.6 199.0   0.2
 11.|-- po3.syd-eqxsy5-cr6.globalsecurelayer.com (2401:3cc0:10:50::1:180)                                    0.0%     3  198.5 198.7 198.5 198.9   0.2
 12.|-- 2401:3cc0::4:45d                                                                                     0.0%     3  198.6 198.8 198.6 198.8   0.1
 13.|-- ndcs1-pe2-po3004.nsw.leaptel.network (2402:2c80:0:22f::66)                                           0.0%     3  199.0 199.0 199.0 199.0   0.0
 14.|-- 2400-a844-0-1--56.nsw.leaptel.network (2400:a844:0:1::56)                                            0.0%     3  199.0 199.1 199.0 199.3   0.1
 15.|-- 2400-a844-8000-5ba2-359b-e9eb-43b-5b58.nsw.leaptel.network (2400:a844:8000:5ba2:                     0.0%     3  200.8 201.0 200.8 201.3   0.3
 16.|-- 2400-a844-5bd5-0-c492-3bdb-58bf-f9a5.nsw.leaptel.network (2400:a844:5bd5:0:c492:                     0.0%     3  209.8 209.3 208.5 209.8   0.6
 17.|-- ???                                                                                                 100.0     3    0.0   0.0   0.0   0.0   0.0

The 58bf:f9a5 node is my server's temporary GUA. So, maybe the problem is on in my networking setup?

Oh my sweet Google WiFi...it blocked the traffic.

I though I only need to ipv4 port nat management, and ipv6 traffic will go straght to the targeted node.

Now, I got my letsencrypt certificate.

Sorry for wasting your time!

You should not set an incorrect IPv4 address in the public DNS. Anyone trying to use it would fail not just Let's Encrypt (who only uses it on IPv6 timeout as a fallback). Not sure where you read that but you should contact them.