Sudden issues with Caddy to get HTTPS for my local IP (DNS and ACME related i guess)

Below config used to work flawlessly 2 months ago.

Okay so I downloaded the Caddy module for Duckdns for Linux AMD 64 from website.

I use Duckdns for giving https to my local ip 192.168.1.197 with domain: adguardcad.duckdns.org
And my API key for DuckDNS is [redacted]

Now I use caddy for doing it, where my CaddyFile is

adguardcad.duckdns.org:443 {
  

  # Use the ACME DNS-01 challenge to get a cert for the configured domain.
  tls {
    dns duckdns [redacted]
  }

  # This setting may have compatibility issues with some browsers
  # (e.g., attachment downloading on Firefox). Try disabling this
  # if you encounter issues.
  encode gzip
  
  reverse_proxy adguardhome:80
  reverse_proxy /notifications/hub/ adguardhome:3012

  # Proxy everything else to Rocket
}

Now my code for Caddy Docker container is

docker run -d \
  --name adguardcaddy \
  --restart unless-stopped \
  --ip 192.168.1.197 \
  --dns 1.1.1.1 \
  --network="net" \
  -p 80:80 \
  -p 443:443 \
  -v /drive1t/dockerconf/caddy/caddy:/usr/bin/caddy \
  -v /drive1t/dockerconf/caddy/Caddyfile:/etc/caddy/Caddyfile:ro \
  -v /drive1t/dockerconf/caddy/caddy-config:/config \
  -v /drive1t/dockerconf/caddy/caddy-data:/data \
  -e DOMAIN=adguardcad.duckdns.org \
  -e EMAIL=home@xyz.com \
  -e TOKEN=[redacted] \
  -e LOG_FILE=/data/access.log \
  -e DUCKDNS_DOMAIN=https://adguardcad.duckdns.org \
  -e DUCKDNS_TOKEN=[redacted] \
  caddy:latest

But I get error in container logs:

INF ts=1714099311.3780174 logger=tls.obtain msg=obtaining certificate identifier=adguardcad.duckdns.org
INF ts=1714099312.7734187 logger=tls.issuance.acme msg=waiting on internal rate limiter identifiers=["adguardcad.duckdns.org"] ca=https://acme-v02.api.letsencrypt.org/directory account=
INF ts=1714099312.7734914 logger=tls.issuance.acme msg=done waiting on internal rate limiter identifiers=["adguardcad.duckdns.org"] ca=https://acme-v02.api.letsencrypt.org/directory account=
INF ts=1714099313.3487773 logger=tls.issuance.acme.acme_client msg=trying to solve challenge identifier=adguardcad.duckdns.org challenge_type=dns-01 ca=https://acme-v02.api.letsencrypt.org/directory
ERR ts=1714099360.3287015 logger=tls.obtain msg=could not get certificate from issuer identifier=adguardcad.duckdns.org issuer=acme-v02.api.letsencrypt.org-directory error=[adguardcad.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of "_acme-challenge.adguardcad.duckdns.org": dial tcp 3.97.51.116:53: i/o timeout (order=https://acme-v02.api.letsencrypt.org/acme/order/1693662117/264145700467) (ca=https://acme-v02.api.letsencrypt.org/directory)
WRN ts=1714099360.3292382 logger=tls.issuance.zerossl msg=missing email address for ZeroSSL; it is strongly recommended to set one for next time
INF ts=1714099361.5142632 logger=tls.issuance.zerossl msg=generated EAB credentials key_id=NYlXS6Fv5Pl9UuKzcH5JXA
INF ts=1714099363.5542395 logger=tls.issuance.zerossl msg=waiting on internal rate limiter identifiers=["adguardcad.duckdns.org"] ca=https://acme.zerossl.com/v2/DV90 account=
INF ts=1714099363.5543056 logger=tls.issuance.zerossl msg=done waiting on internal rate limiter identifiers=["adguardcad.duckdns.org"] ca=https://acme.zerossl.com/v2/DV90 account=
INF ts=1714099364.2849708 logger=tls.issuance.zerossl.acme_client msg=trying to solve challenge identifier=adguardcad.duckdns.org challenge_type=dns-01 ca=https://acme.zerossl.com/v2/DV90
ERR ts=1714099392.3516834 logger=tls.obtain msg=could not get certificate from issuer identifier=adguardcad.duckdns.org issuer=acme.zerossl.com-v2-DV90 error=[adguardcad.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of "_acme-challenge.adguardcad.duckdns.org": dial tcp 35.183.157.249:53: i/o timeout (order=https://acme.zerossl.com/v2/DV90/order/-mGHWGkMGrmqEIIl5eKjwQ) (ca=https://acme.zerossl.com/v2/DV90)
ERR ts=1714099392.3518286 logger=tls.obtain msg=will retry error=[adguardcad.duckdns.org] Obtain: [adguardcad.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of "_acme-challenge.adguardcad.duckdns.org": dial tcp 35.183.157.249:53: i/o timeout (order=https://acme.zerossl.com/v2/DV90/order/-mGHWGkMGrmqEIIl5eKjwQ) (ca=https://acme.zerossl.com/v2/DV90) attempt=1 retrying_in=60 elapsed=80.97429205 max_duration=2592000

I have also tried ping 1.1.1.1 from inside the container and it works:

root@omv:~# docker exec -it adguardcaddy /bin/sh
/srv # ping 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=51 time=14.364 ms
64 bytes from 1.1.1.1: seq=1 ttl=51 time=14.050 ms
64 bytes from 1.1.1.1: seq=2 ttl=51 time=14.132 ms
64 bytes from 1.1.1.1: seq=4 ttl=51 time=14.134 ms
64 bytes from 1.1.1.1: seq=5 ttl=51 time=14.052 ms
64 bytes from 1.1.1.1: seq=6 ttl=51 time=13.622 ms
64 bytes from 1.1.1.1: seq=7 ttl=51 time=13.499 ms
64 bytes from 1.1.1.1: seq=9 ttl=51 time=13.405 ms

@fedonr, this is a public forum, so please never post API keys, access tokens, or private key material.

6 Likes

Thank you so much, I would keep that in mind, I did edit out the email forgot to remove the token. Sorry about it

3 Likes

Have you checked the outbound DNS rules?

5 Likes

Yes I did try 8.8.8.8 and 1.1.1.1 both, Idk what changed still a noob. 2 months ago same config worked.

DNS happens over UDP [mostly] and also TCP.
These are outbound TCP DNS requests that seem to fail.

5 Likes

I have the Vaultwarden in the same, where via admin access I was able to check the Vaultwarden container is able to connect to internet, so if my outbound connections are blocked, won't that also cause Vaultwarden to not connect with Github (accessible via Admin console)

I have also tried Ping 1.1.1.1 from inside the container and it works. I have also updated the same in the main post after edit

@fedonr I strongly recommend you obtain a new API key as Google has proven to be faster than your redaction.
RIP

5 Likes

The failures are for TCP port 53 requests - Ping will not simulate those types of requests.
Also, 1.1.1.1 was able to resolve the destination name(s).
So, the DNS server used is NOT part of the problem.
Taking a closer look at the error message:

We can see that a nameserver IP has been resolved [in this case "3.97.51.116"].
But the DNS request to it on TCP port 53 returned with a "timeout".

So, again, the DNS server used ["1.1.1.1"] is NOT part of that problem.

5 Likes

Ahh! I see, any idea how to solve this? As I'm stuck here since last 2 days.

I ensured my router also uses 1.1.1.1, but still this is persistent.

As this config was working for more than a year now, and last I checked 2 months ago it was all good and now suddenly this.

I'm still a little noob, so I'll try my best to understand but ELI5 would be better :sweat_smile:

2 Likes

I'd do some DNS testing from within that container.

4 Likes

@fedonr I strongly suggest getting new (or regenerate) and different DuckDNS API Tokens!

Never post Private Keys, Credentials, nor secret API keys.
Editing them out is nice, but they are still loose in the wild for bad actors to get and use.

Now Public Keys are just that; anyone can and should be able to see them, so posting them is OK.

4 Likes

Thank you so much, I did that.

3 Likes

"I did that?" Hope so.
A lot of times "unaware or inexperienced admins" drop tons of stuff on the forum.

We need information to solve issues, but credentials and keys are another story. I don't know how many times I have reported these kinds of breeches... But they happen.

And on the other hand there are those that won't post the domain for one reason or another. It is a hand off. Good on the Leaders here and (moderators?) that catch this stuff and help fix it.

5 Likes

Still facing the issues, I tried to do some trial and error. Where I did factory reset on my router and it worked for the first time, so I kept on it for 3 days trying to delete certs and restart container every 5 hours after deleting caddy data and config files. It's working 1 out of 4 times in a day.

Logs

INF ts=xxx logger=tls.issuance.zerossl.acme_client msg=trying to solve challenge identifier=gharnas.duckdns.org challenge_type=dns-01 ca=https://acme.zerossl.com/v2/DV90
ERR ts=xxx logger=tls.issuance.zerossl.acme_client msg=cleaning up solver identifier=*.gharnas.duckdns.org challenge_type=dns-01 error=no memory of presenting a DNS record for “_acme-challenge.gharnas.duckdns.org” (usually OK if presenting also failed)
ERR ts=xxx logger=tls.issuance.zerossl.acme_client msg=cleaning up solver identifier=gharnas.duckdns.org challenge_type=dns-01 error=no memory of presenting a DNS record for “_acme-challenge.gharnas.duckdns.org” (usually OK if presenting also failed)
ERR ts=xxx logger=tls.obtain msg=could not get certificate from issuer identifier=.gharnas.duckdns.org issuer=acme.zerossl.com-v2-DV90 error=[.gharnas.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain “_acme-challenge.gharnas.duckdns.org”: unexpected response code ‘SERVFAIL’ for gharnas.duckdns.org. (order=https://acme.zerossl.com/v2/DV90/order/KqujzNEER2dkJKK5RpFzvg) (ca=https://acme.zerossl.com/v2/DV90)
ERR ts=xxx logger=tls.obtain msg=will retry error=[.gharnas.duckdns.org] Obtain: [.gharnas.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain “_acme-challenge.gharnas.duckdns.org”: unexpected response code ‘SERVFAIL’ for gharnas.duckdns.org. (order=https://acme.zerossl.com/v2/DV90/order/KqujzNEER2dkJKK5RpFzvg) (ca=https://acme.zerossl.com/v2/DV90) attempt=1 retrying_in=60 elapsed=10.854302968 max_duration=2592000
ERR ts=xxx logger=tls.obtain msg=could not get certificate from issuer identifier=gharnas.duckdns.org issuer=acme.zerossl.com-v2-DV90 error=[gharnas.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain “_acme-challenge.gharnas.duckdns.org”: unexpected response code ‘SERVFAIL’ for _acme-challenge.gharnas.duckdns.org. (order=https://acme.zerossl.com/v2/DV90/order/ODv6ymYqrO5Z9lkO9-MRCg) (ca=https://acme.zerossl.com/v2/DV90)
ERR ts=xxx logger=tls.obtain msg=will retry error=[gharnas.duckdns.org] Obtain: [gharnas.duckdns.org] solving challenges: presenting for challenge: could not determine zone for domain “_acme-challenge.gharnas.duckdns.org”: unexpected response code ‘SERVFAIL’ for _acme-challenge.gharnas.duckdns.org. (order=https://acme.zerossl.com/v2/DV90/order/ODv6ymYqrO5Z9lkO9-MRCg) (ca=https://acme.zerossl.com/v2/DV90) attempt=1 retrying_in=60 elapsed=10.85496946 max_duration=2592000

Thank you so much, I would ensure I never repeat those mistakes.

2 Likes

Can this be the case?

Jio block tcp and udp requests for dns

Jio, a major Indian ISP, has been observed to employ various methods to block certain websites and services. According to the search results, Jio is using a technique called SNI (Server Name Indication) inspection to block websites for its users. This method involves inspecting the SNI field in the TLS protocol to determine the intended destination of a connection and block it if necessary.

Additionally, Jio has also been reported to block UDP ports required for Cloudflare Warp, a service that provides a secure and fast way to access the internet. This blocking of UDP ports is likely done to prevent users from accessing certain websites or services that Jio may deem inappropriate.

It’s also worth noting that Jio has been known to change its public IP addresses frequently to maintain efficiency and manage its network resources. This can cause issues for users who rely on static IP addresses or have specific configurations that rely on a specific IP address.

In summary, Jio has been observed to block TCP and UDP requests for DNS, as well as block specific ports and services, in order to manage its network and block certain websites or services.

Okay, so I did try once again and this time I kept on swapping resolvers.

It seems that Clouldflare is failing each time (out of 3), Google works everytime, and ControlD fails 2 out of 3 times.

I keep goggle Dns as of now and would test it for 3 more days and would update here.

Hey, thanks you so much for all your help.

It is resolved now, as I had to change the resolver to google 8.8.8.8

As cloudflare and controld didn't work.

So, for the solution was resetting router and change resolver to Google.

Initially all google, cloudflare and controld but after resetting the router Google started working.

Still I do have a question, why suddenly cloudflare stopped working it worked for more than a year for me. And why do some resolvers work while others don't?

1 Like