I have a strange redirection problem during a renewal

atanas-vladimirov · January 22, 2024, 6:25pm

Hi,

I have problems renewing my domain and I can't figure it out why. I believe that it could be related to the HTTP -> HTTPS redirection because it seems that it works for IPv6 but not for IPv4.

When I run the below command (OpenBSD have their own implementation - acme-client) I can see the following in the nginx logs:

2600:3000:2710:300::83 - - [22/Jan/2024:19:58:33 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 200 87 "http://cloud.bsdbg.net/.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:58:33 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:58:34 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 200 87 "http://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:58:35 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:58:36 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 200 87 "http://juri.bsdbg.net/.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:58:43 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:58:45 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::83 - - [22/Jan/2024:19:58:45 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::83 - - [22/Jan/2024:19:58:46 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 200 87 "http://cloud.bsdbg.net/.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::83 - - [22/Jan/2024:19:58:46 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::83 - - [22/Jan/2024:19:58:47 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 200 87 "http://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::83 - - [22/Jan/2024:19:58:47 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::83 - - [22/Jan/2024:19:58:48 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 200 87 "http://juri.bsdbg.net/.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:58:52 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:58:52 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:58:53 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:58:54 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:58:55 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:58:56 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:58:57 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:58:57 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:58:58 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 200 87 "http://cloud.bsdbg.net/.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:58:58 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:58:59 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 200 87 "http://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::83 - - [22/Jan/2024:19:58:59 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::83 - - [22/Jan/2024:19:59:00 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 200 87 "http://juri.bsdbg.net/.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:59:05 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:59:06 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:59:07 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:59:08 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:59:09 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:59:10 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:59:10 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:59:10 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 200 87 "http://cloud.bsdbg.net/.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::88 - - [22/Jan/2024:19:59:11 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 200 87 "http://juri.bsdbg.net/.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:59:17 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:59:17 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:59:18 +0200] "GET /.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0 HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:59:19 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:59:19 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:59:29 +0200] "GET /.well-known/acme-challenge/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
18.216.244.154 - - [22/Jan/2024:19:59:30 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
35.91.114.106 - - [22/Jan/2024:19:59:30 +0200] "GET /.well-known/acme-challenge/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM HTTP/1.1" 301 162 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

As you can see the IPv6 works, but IPv4 won't. I only get the HTTP/1.1" 301 and nothing more.

Any help will be much appreciated!

My domain is: hodor.bsdbg.net

I ran this command: acme-client -vv hodor.bsdbg.net

It produced this output - here part of the output

acme-client: dochngreq: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10759819874
acme-client: transfer buffer: [{ "identifier": { "type": "dns", "value": "cloud.bsdbg.net" }, "status": "pending", "expires": "2024-01-29T17:58:28Z", "challenges": [ { "type": "http-01", "status": "pending", "url": "https:/
/acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819874/ltyblw", "token": "wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" }, { "type": "dns-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.or
g/acme/chall-v3/10759819874/PMZV7g", "token": "wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" }, { "type": "tls-alpn-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819874/tTs
Djg", "token": "wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" } ] }] (820 bytes)
acme-client: challenge, token: wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY, uri: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819874/ltyblw, status: 0
acme-client: /var/www/acme/wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY: created
acme-client: dochngreq: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10759819884
acme-client: transfer buffer: [{ "identifier": { "type": "dns", "value": "hodor.bsdbg.net" }, "status": "invalid", "expires": "2024-01-29T17:58:28Z", "challenges": [ { "type": "http-01", "status": "invalid", "error": { "typ
e": "urn:ietf:params:acme:error:connection", "detail": "During secondary validation: 78.130.168.61: Fetching https://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0: Timeout during con
nect (likely firewall problem)", "status": 400 }, "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819884/MxcCAw", "token": "jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0", "validationRecord": [ { "url"
: "http://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0", "hostname": "hodor.bsdbg.net", "port": "80", "addressesResolved": [ "78.130.168.61", "2001:67c:21bc:68::1", "2001:67c:21bc:4
2::1" ], "addressUsed": "2001:67c:21bc:68::1" }, { "url": "https://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0", "hostname": "hodor.bsdbg.net", "port": "443", "addressesResolved": 
[ "78.130.168.61", "2001:67c:21bc:68::1", "2001:67c:21bc:42::1" ], "addressUsed": "2001:67c:21bc:68::1" } ], "validated": "2024-01-22T17:58:33Z" } ] }] (1564 bytes)
acme-client: challenge, token: jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0, uri: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819884/MxcCAw, status: -1
acme-client: dochngreq: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10759819894
acme-client: transfer buffer: [{ "identifier": { "type": "dns", "value": "juri.bsdbg.net" }, "status": "pending", "expires": "2024-01-29T17:58:28Z", "challenges": [ { "type": "http-01", "status": "pending", "url": "https://
acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819894/o_riIg", "token": "bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" }, { "type": "dns-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org
/acme/chall-v3/10759819894/DJFRQQ", "token": "bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" }, { "type": "tls-alpn-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819894/-C4D
qA", "token": "bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" } ] }] (819 bytes)
acme-client: challenge, token: bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM, uri: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819894/o_riIg, status: 0
acme-client: /var/www/acme/bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM: created
acme-client: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819874/ltyblw: challenge
acme-client: transfer buffer: [{ "type": "http-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819874/ltyblw", "token": "wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" }] (194
 bytes)
acme-client: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819894/o_riIg: challenge
acme-client: transfer buffer: [{ "type": "http-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819894/o_riIg", "token": "bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" }] (194
 bytes)
acme-client: transfer buffer: [{ "status": "invalid", "expires": "2024-01-29T17:58:28Z", "identifiers": [ { "type": "dns", "value": "cloud.bsdbg.net" }, { "type": "dns", "value": "hodor.bsdbg.net" }, { "type": "dns", "value
": "juri.bsdbg.net" } ], "authorizations": [ "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10759819874", "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10759819884", "https://acme-staging-v02.api.
letsencrypt.org/acme/authz-v3/10759819894" ], "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/133398274/13864003414" }] (643 bytes)
acme-client: order.status -1
acme-client: dochngreq: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10759819874
acme-client: transfer buffer: [{ "identifier": { "type": "dns", "value": "cloud.bsdbg.net" }, "status": "pending", "expires": "2024-01-29T17:58:28Z", "challenges": [ { "type": "http-01", "status": "pending", "url": "https:/
/acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819874/ltyblw", "token": "wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" }, { "type": "dns-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.or
g/acme/chall-v3/10759819874/PMZV7g", "token": "wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" }, { "type": "tls-alpn-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819874/tTs
Djg", "token": "wqLKC0CfDQyambovY83T5dIHf1fMbG8F7KNoYaJd4cY" } ] }] (820 bytes)
acme-client: dochngreq: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10759819884
acme-client: transfer buffer: [{ "identifier": { "type": "dns", "value": "hodor.bsdbg.net" }, "status": "invalid", "expires": "2024-01-29T17:58:28Z", "challenges": [ { "type": "http-01", "status": "invalid", "error": { "typ
e": "urn:ietf:params:acme:error:connection", "detail": "During secondary validation: 78.130.168.61: Fetching https://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0: Timeout during con
nect (likely firewall problem)", "status": 400 }, "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819884/MxcCAw", "token": "jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0", "validationRecord": [ { "url"
: "http://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0", "hostname": "hodor.bsdbg.net", "port": "80", "addressesResolved": [ "78.130.168.61", "2001:67c:21bc:68::1", "2001:67c:21bc:4
2::1" ], "addressUsed": "2001:67c:21bc:68::1" }, { "url": "https://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0", "hostname": "hodor.bsdbg.net", "port": "443", "addressesResolved": 
[ "78.130.168.61", "2001:67c:21bc:68::1", "2001:67c:21bc:42::1" ], "addressUsed": "2001:67c:21bc:68::1" } ], "validated": "2024-01-22T17:58:33Z" } ] }] (1564 bytes)
acme-client: During secondary validation: 78.130.168.61: Fetching https://hodor.bsdbg.net/.well-known/acme-challenge/jrhFZ6mdiMK6E4F3_mhSXCBx1GgkXAjbYpUL1sA-Cg0: Timeout during connect (likely firewall problem)
acme-client: dochngreq: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/10759819894
acme-client: transfer buffer: [{ "identifier": { "type": "dns", "value": "juri.bsdbg.net" }, "status": "pending", "expires": "2024-01-29T17:58:28Z", "challenges": [ { "type": "http-01", "status": "pending", "url": "https://
acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819894/o_riIg", "token": "bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" }, { "type": "dns-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org
/acme/chall-v3/10759819894/DJFRQQ", "token": "bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" }, { "type": "tls-alpn-01", "status": "pending", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/10759819894/-C4D
qA", "token": "bqujGRZYZ8B6nOMknd-p7H2zQn4gAtlNQkq7ELuVtaM" } ] }] (819 bytes)
acme-client: bad exit: netproc(39956): 1

My web server is (include version): nginx/1.24.0

I have the following in the config:

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;
        return 301 https://$host$request_uri;
}


    server {
        listen 443 ssl;
        listen [::]:443 ssl;
        server_name hodor.bsdbg.net;
    
        root         /htdocs;

        ssl_certificate      /etc/ssl/bsdbg.fullchain.pem;
        ssl_certificate_key  /etc/ssl/private/bsdbg.key;

        ssl_session_timeout  5m;
        ssl_session_cache    shared:SSL:1m;

        ssl_protocols TLSv1.2;
        ssl_ciphers  HIGH:!aNULL:!MD5:!RC4;
        ssl_prefer_server_ciphers   on;

        location /.well-known/acme-challenge {
                alias /acme/;
                try_files $uri =404;
        }
.............

The operating system my web server runs on is (include version): OpenBSD 7.4

I can login to a root shell on my machine (yes or no, or I don't know): yes

rg305 · January 22, 2024, 6:50pm

Hi @atanas-vladimirov, and welcome to the LE community forum

Please show us the complete running nginx config, with the output of:

`nginx -T`

atanas-vladimirov · January 22, 2024, 7:24pm

Thanks for your reply and nice to meet you!

Here is the requested config.
Let me know if you need anything else.

Best wishes,
Atanas

rg305 · January 22, 2024, 7:48pm

What is this client?

Can it do --webroot?

This code seems unbalanced:

the location does NOT end with "/"
the alias ends with a "/"

atanas-vladimirov · January 22, 2024, 8:25pm

This is the client - acme-client(1) - OpenBSD manual pages

Can it do --webroot ?

I'm not sure, probably not.

This code seems unbalanced:

But this has been working this way for years. I think that it started to fail last year, on my previous renewal.
I have workarounded it back then by placing the following:

server {
        listen 80 default_server;
        listen [::]:80 default_server;
        server_name _;

        location /.well-known/acme-challenge {
                alias /acme/;
                try_files $uri =404;
        }

#       return 301 https://$host$request_uri;
}

i.e. removing the HTTP -> HTTPS redirection.

But this time I decided to debug it further and understand why it happens.
I found https://letsdebug.net/ and it is returning a green OK.
Then I run multiple curl commands from 5-6 places against my server /acme/ URL, and it works like a charm.

If you wish, you can try it on your end:

curl -4 -L -vv http://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test

And just a reminder that the IPv6 is working with the same nginx rules, but IPv4 is not.

And now I'm out of ideas.

Thanks for helping me!
Atanas

MikeMcQ · January 22, 2024, 8:57pm

You have two IPv6 addresses for that domain name. Is that intentional?

I see that Let's Debug sees them both. Oddly, I get timeouts to both of your IPv6 addresses but IPv4 works fine for me.

Should your nginx server block be redirecting HTTP requests right now? Your last server block had the redirect commented out but I get redirected on IPv4.

curl -I4 -L http://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test
HTTP/1.1 301 Moved Permanently
Server: nginx
Location: https://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test

HTTP/2 200
server: nginx
content-length: 5

# IPv6 times out 
curl -6 -L -m8 http://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test
curl: (28) Failed to connect to hodor.bsdbg.net port 80 after 6002 ms: 
Connection timed out

I tried using the IPv6 addresses directly and both timeout the same

hodor.bsdbg.net.	0	IN	AAAA	2001:67c:21bc:68::1
hodor.bsdbg.net.	0	IN	AAAA	2001:67c:21bc:42::1

atanas-vladimirov · January 22, 2024, 9:12pm

Hi MikeMcQ,

You have two IPv6 addresses for that domain name. Is that intentional?

Heh, yes, it was temporary ... But I know about that and it should be fine.

I see that Let's Debug sees them both. Oddly, I get timeouts to both of your IPv6 addresses but IPv4 works fine for me.

Can you tell me your IPv6 address so I can see if it is not block by the firewall?

Should your nginx server block be redirecting HTTP requests right now? Your last server block had the redirect commented out but I get redirected on IPv4.

Yes, the redirection is in place and should work. Please ignore my last server block, it was just to illustrate how I was workaround it last time.

I tried using the IPv6 addresses directly and both timeout the same

They both should work:

$ curl -6 -L -m8 http://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test
test

and with more details

$ curl -6 -L -vv http:/hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test
*   Trying 2001:67c:21bc:42::1:80...
* Connected to hodor.bsdbg.net (2001:67c:21bc:42::1) port 80 (#0)
> GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1
> Host: hodor.bsdbg.net                 
> User-Agent: curl/7.74.0
> Accept: */*                                 
>                   
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Mon, 22 Jan 2024 21:10:28 GMT                                                              
< Content-Type: text/html                                                                                                                                                                             
< Content-Length: 162                
< Connection: keep-alive
< Location: https://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test
< 
* Ignoring the response-body
* Connection #0 to host hodor.bsdbg.net left intact
* Issue another request to this URL: 'https://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test'
*   Trying 2001:67c:21bc:42::1:443...
....

* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Mon, 22 Jan 2024 21:10:28 GMT
< content-type: application/octet-stream
< content-length: 5
< last-modified: Mon, 22 Jan 2024 18:05:45 GMT
< etag: "65aeae79-5"
< accept-ranges: bytes
< 
test
* Connection #1 to host hodor.bsdbg.net left intact

atanas-vladimirov · January 22, 2024, 9:20pm

And just another confirmation that my IPv6 works - https://www.mythic-beasts.com/ipv6/health-check?domain=hodor.bsdbg.net

But my problem is not with IPv6 - if you check the nginx logs in the first post, you will see that the Let's encrypt IPv6 addresses are reaching mines (the first GET is with code 301, then 200).

The issues is with IPv4 - my nginx is returning 301 but the Let's encrypt IPs are not following the redirection ....

MikeMcQ · January 22, 2024, 9:29pm

No, if you have AAAA records for IP's that work Let's Encrypt Servers will not use IPv4 at all. Let's Debug yes. LE no.

If the IPv6 connection cleanly times out then LE will fallback to IPv4. But, just for the initial HTTP request. If the inbound IPv4 request is then redirected, say to HTTPS, then LE will only try that redirect with IPv6. It is likely that will timeout just like the first try and the challenge will fail.

I think this is what is happening here. But, the signals are a little confusing.

MikeMcQ · January 22, 2024, 9:33pm

Both AAAA records are still there. See https://unboundtest.com

Not helpful. My sandbox uses ephemeral IP addresses. If you are blocking IP addresses how do you know you are not blocking Let's Encrypt? See my previous comment about IPv6 / IPv4 fallback.

It is best to not redirect HTTP challenge requests. Even ignoring the complication with IPv6/4 fallback. It is quicker and more reliable since less traffic.

PS: The -v isn't too helpful for this case. You don't have to post all that

atanas-vladimirov · January 22, 2024, 10:02pm

Thanks, MikeMcQ!

What you're saying makes sense.
Can you please try to ping one of my IPv6 addresses and then try a traceroute6? It will help me find the problem easily.

I just checked the firewall and there are no any IPv6 rules that should block the HTTP(S).
Maybe it is something with my IPv6 provider ....

Both AAAA records are still there. See https://unboundtest.com

Sorry, I didn't express myself very clear - what I wanted to say was that the first time when I added the second (additional) IPv6 address was just temporary (more than a year ago

Best wishes,
Atanas

MikeMcQ · January 22, 2024, 10:12pm

I can but not sure how helpful that will be. I am not Let's Encrypt so what affects me may not be affecting their servers. I am testing from an AWS EC2 instance and Let's Encrypt also uses AWS for some of their farms. But, that is a very large pool

I think you should first try each of your IPv6 addresses independently. Perhaps only one of them is an issue and this will narrow the scope. That is, remove one of them, wait for your DNS to sync and try it. Then, do the other one.

Use https://unboundtest.com to confirm the DNS changes appear in your authoritive servers. Let's Encrypt walks the auth tree and so is not subject to TTL delays.

Bruce5051 · January 22, 2024, 10:12pm

From a FreeBSD 13.2-RELEASE-p8 GENERIC amd64 system

>ping6 hodor.bsdbg.net
PING6(56=40+8+8 bytes) 2001:418:1::62 --> 2001:67c:21bc:68::1
16 bytes from 2001:67c:21bc:68::1, icmp_seq=0 hlim=55 time=155.747 ms
16 bytes from 2001:67c:21bc:68::1, icmp_seq=1 hlim=55 time=155.808 ms
16 bytes from 2001:67c:21bc:68::1, icmp_seq=2 hlim=55 time=155.724 ms
16 bytes from 2001:67c:21bc:68::1, icmp_seq=3 hlim=55 time=155.628 ms
^C
--- hodor.bsdbg.net ping6 statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 155.628/155.727/155.808/0.065 ms

>traceroute6 2001:67c:21bc:68::1
traceroute6 to 2001:67c:21bc:68::1 (2001:67c:21bc:68::1) from 2001:418:1::62, 64 hops max, 28 byte packets
 1  r1.sea.rg.net  0.223 ms  0.295 ms  0.135 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  *
 * *
 8  *
    e0-35.core2.bts1.he.net  143.288 ms  142.942 ms
 9  marla6.ludost.net  154.711 ms
    e0-36.core2.beg1.he.net  150.322 ms
    marla6.ludost.net  154.628 ms
10  hodor.bsdbg.net  156.017 ms *  156.281 ms

Bruce5051 · January 22, 2024, 10:16pm

And as @MikeMcQ suggested I see this https://unboundtest.com/m/AAAA/hodor.bsdbg.net/B22KV3WU

Query results for AAAA hodor.bsdbg.net

Response:
;; opcode: QUERY, status: NOERROR, id: 38234
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 512

;; QUESTION SECTION:
;hodor.bsdbg.net.	IN	 AAAA

;; ANSWER SECTION:
hodor.bsdbg.net.	0	IN	AAAA	2001:67c:21bc:68::1
hodor.bsdbg.net.	0	IN	AAAA	2001:67c:21bc:42::1

----- Unbound logs -----
Jan 22 22:14:34 unbound1.19[455054:0] debug: creating udp6 socket ::1 1053
Jan 22 22:14:34 unbound1.19[455054:0] debug: creating tcp6 socket ::1 1053
Jan 22 22:14:34 unbound1.19[455054:0] debug: creating udp4 socket 127.0.0.1 1053
Jan 22 22:14:34 unbound1.19[455054:0] debug: creating tcp4 socket 127.0.0.1 1053
Jan 22 22:14:34 unbound1.19[455054:0] debug: chdir to .
Jan 22 22:14:34 unbound1.19[455054:0] debug: switching log to stderr
Jan 22 22:14:34 unbound1.19[455054:0] debug: module config: "validator iterator"
Jan 22 22:14:34 unbound1.19[455054:0] notice: init module 0: validator
Jan 22 22:14:34 unbound1.19[455054:0] debug: reading autotrust anchor file root.key
Jan 22 22:14:34 unbound1.19[455054:0] info: trust point . : 1

.
.
.

atanas-vladimirov · January 22, 2024, 10:18pm

Thank you, Bruce5051,

And what happens if you try to connect with the following curl command?

`curl -6 -L -m8 http://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test`

Bruce5051 · January 22, 2024, 10:19pm

I get this

>curl -6 -L -m8 http://hodor.bsdbg.net/.well-known/acme-challenge/letsdebug-test
test

Bruce5051 · January 22, 2024, 10:23pm

Also here hodor.bsdbg.net | DNSViz there is a warning

And given that SOA and NS Records themselves are referencing hodor.bsdbg.net seems like it might be significant

NS	bsdbg.net	5m	hodor.bsdbg.net.
					ns.bsdbg.net.
SOA	bsdbg.net	5m	hodor.bsdbg.net. vlado.bsdbg.net. 2023012401 3600 7200 1209600 600

from here DNS Spy report for bsdbg.net,

atanas-vladimirov · January 22, 2024, 10:24pm

Thanks!

It seems that it works. So I guess that @MikeMcQ assumption is correct - there is a problem between me and AWS.

But I still wonder how I can see the LE IPv6 addresses hitting my nginx logs, such as 2600:3000:2710:300::83 or 2600:3000:2710:300::88

rg305 · January 22, 2024, 10:29pm

Maybe you are not seeing enough of them.
LE requires verification from multiple points.

Bruce5051 · January 22, 2024, 10:31pm

And expanding on that see Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt

Topic		Replies	Views
Renew failure with certbot or letsencrypt-auto Help	25	6961	July 14, 2017
LE fails to verify domain when site has been set up only to listen to port 443 and all requests for http are redirected to https Help	34	15296	May 2, 2016
Renewal 404 fail Help	29	1074	March 26, 2019
Letsencrypt renewal failure fetching timeout Ubuntu 18.04LTS Help	23	2053	January 19, 2019
Renewal is failing Server	31	2027	December 27, 2018

I have a strange redirection problem during a renewal

nginx -T

Related topics

`nginx -T`