Certbot fails to create a certificate for the main domain, when subdomain has a certificate

[antanasramana]

Hello,
I was trying to use certbot to create certificates for both api.auto-nuoma.lt and auto-nuoma.lt. Both of them are on the same server which has an IPv4 of 194.31.55.108 and IPv6 of 2a02:4780:c:8c89::1. I am using nginx as a reverse proxy to route traffic to my internal node.js and .net applications.
Unfortunately, the certbot fails to authorize auto-nuoma.lt and throws and error of

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: auto-nuoma.lt
  Type:   unauthorized
  Detail: 2a02:4780:c:8c89::1: Invalid response from http://auto-nuoma.lt/.well-known/acme-challenge/GvcSONag5C8A3mRMZttY0aDYE_SdiON5H16dJJs5yMs: 404

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

I managed to hack my way around it, I removed nginx config file for api.auto-nuoma.lt, and then ran

sudo certbot --nginx

Then selected auto-nuoma.lt domain. When it was alone - everything went well and I managed to get a certificate for it. After that I added back the api.auto-nuoma.lt nginx config file and I was happy with my fix.

But that is not the end, I tried to renew the certificates and got the same problem

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: auto-nuoma.lt
  Type:   unauthorized
  Detail: 2a02:4780:c:8c89::1: Invalid response from http://auto-nuoma.lt/.well-known/acme-challenge/GvcSONag5C8A3mRMZttY0aDYE_SdiON5H16dJJs5yMs: 404

Any ideas why this is happening and how to fix it? I would like to get a benefit of automatic certificate renewal process.

The operating system my web server runs on is (include version): Ubuntu 22.04.3 LTS

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.8.0

[Bruce5051]

Hello @antanasramana, welcome to the Let's Encrypt community.

Using the online tool Let's Debug yields these results https://letsdebug.net/auto-nuoma.lt/1743562

[rg305]

From the error msg shown above and the fact that LE prefers IPv6 over IPv4...
I would guess that your nginx vhost [for this specific domain] isn't using/listening on IPv6.
Other parts of nginx [serving other vhosts] may be doing IPv6 correctly.

[Bruce5051]

IPv4 response (note there is a redirection from HTTP to HTTPS; not a problem)

>curl -4 -Ii http://auto-nuoma.lt/.well-known/acme-challenge/sometestfile
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 19 Dec 2023 21:19:08 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://auto-nuoma.lt/.well-known/acme-challenge/sometestfile

>curl -4 -Ii https://auto-nuoma.lt/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 19 Dec 2023 21:19:20 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 2009
Connection: keep-alive
x-powered-by: Next.js
etag: "12e7brw381x1jt"
vary: Accept-Encoding

IPv6 Response (note this response is different than that from
curl -4 -Ii http://auto-nuoma.lt/.well-known/acme-challenge/sometestfile )

>curl -6 -Ii http://auto-nuoma.lt/.well-known/acme-challenge/sometestfile
HTTP/1.1 404 Not Found
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 19 Dec 2023 21:19:28 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive

[antanasramana]

Thanks guys, for such a quick response. I managed to track it down with your suggestions. Turns out nginx for auto-nuoma.lt domain (which was failing) was not listening on IPv6, only on IPv4. Just added one liner into the config to listen for IPv6 as well and certbot went through without any issues. Thanks guys for help.
TLDR: Issue is solved, nginx was not listening on IPv6.