Time to renew and I am getting the Challenge failed for domain

ebonsi · March 28, 2019, 2:11am

My domain is:
bonsi.org
I ran this command:
./certbot-auto certonly --webroot --webroot-path /Users/user2/Sites/ --email webmaster@bonsi.org -d bonsi.org -d www.bonsi.org

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for bonsi.org
http-01 challenge for www.bonsi.org
Using the webroot path /Users/User2/Sites for all unmatched domains.
Waiting for verification…
Challenge failed for domain bonsi.org
Challenge failed for domain www.bonsi.org
http-01 challenge for bonsi.org
http-01 challenge for www.bonsi.org
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: bonsi.org
Type: connection
Detail: Fetching
https://bonsi.org/.well-known/acme-challenge/KKjjd8yl5IZqnkw1yOIspKzmFdre8QDexWZEjyxN0pI:
Error getting validation data

Domain: www.bonsi.org
Type: connection
Detail: Fetching
https://www.bonsi.org/.well-known/acme-challenge/PmrQPrFQmaANMOUoxeCltJgr8dd_gHeV-xCMjyKNT8k:
Error getting validation data

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you’re using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.
My web server is (include version):
Apache version 2.4.37
The operating system my web server runs on is (include version):

MacOS X (High Sierra) Client Server, with Bind and Apache

My hosting provider, if applicable, is:
ISP AT&T
I can login to a root shell on my machine (yes or no, or I don’t know):
YES
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
No
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
root# certbot --version || /Users/user/letsencrypt/certbot-auto --version
certbot 0.32.0

Chief complaint: Cannot renew certs, Time to renew and I am getting the Challenge failed for domain

Actions:
ping6 bonsi.org
root# ping6 bonsi.org
PING6(56=40+8+8 bytes) 2600:1700:b310:c2e0::3 --> 2600:1700:b310:c2e0::3
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=0 hlim=64 time=0.083 ms
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=1 hlim=64 time=0.090 ms
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=2 hlim=64 time=0.140 ms
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=3 hlim=64 time=0.084 ms
root# dig bonsi.org

; <<>> DiG 9.10.6 <<>> bonsi.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47502
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;bonsi.org. IN A

;; ANSWER SECTION:
bonsi.org. 21600 IN A 162.201.66.177

;; Query time: 165 msec
;; SERVER: 192.168.1.254#53(192.168.1.254)
;; WHEN: Wed Mar 27 18:44:45 PDT 2019
;; MSG SIZE rcvd: 54

root# dig aaaa bonsi.org

; <<>> DiG 9.10.6 <<>> aaaa bonsi.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62670
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;bonsi.org. IN AAAA

;; ANSWER SECTION:
bonsi.org. 3600 IN AAAA 2600:1700:b310:c2e0::2
bonsi.org. 3600 IN AAAA 2600:1700:b310:c2e0::4
bonsi.org. 3600 IN AAAA 2600:1700:b310:c2e0::3

;; Query time: 161 msec
;; SERVER: 192.168.1.254#53(192.168.1.254)
;; WHEN: Wed Mar 27 18:45:20 PDT 2019
;; MSG SIZE rcvd: 122

Trying to connect through Tor Ok!

Connecting through Tor to the acme challenge returned positive.

Recently AT&T changed my ipv6 address and I had to configure all the network with the new prefix.

The IntoDNS reporte seems fine for ipv4
https://intodns.com/bonsi.org

I checked with:
http://ipv6-test.com/validate.php
and it returned “web server is unreachable : Permission denied”
I am not blocking my ipv6 through firewalls
I can successfully ping
root# ping6 2600:1700:b310:c2e0::3
PING6(56=40+8+8 bytes) 2600:1700:b310:c2e0::3 --> 2600:1700:b310:c2e0::3
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=0 hlim=64 time=0.088 ms
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=1 hlim=64 time=0.172 ms
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=2 hlim=64 time=0.086 ms
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=3 hlim=64 time=0.175 ms
16 bytes from 2600:1700:b310:c2e0::3, icmp_seq=4 hlim=64 time=0.083 ms

but not able to connect directly to the server using the ipv6 https://[2600:1700:b310:c2e0::3]

Apache Server is working fine, I can reach all the domains from the regular internal browser.

It could be (Maybe) the AT&T router. Lately they updated the router and not sure how far they block things over. I only use the Server for Teaching purposes, none of the ips are assigned by ATT. The costs of assigned ip does not justify the means.

Any thoughts?
Thanks!

JuergenAuer · March 28, 2019, 8:27am

Hi @ebonsi

I can confirm the problem ( https://check-your-website.server-daten.de/?q=bonsi.org ):

You have 3 ipv6

Host	T	IP-Address	is auth.	∑ Queries	∑ Timeout
bonsi.org	A	162.201.66.177	yes	1	0
	AAAA	2600:1700:b310:c2e0::2	yes
	AAAA	2600:1700:b310:c2e0::3	yes
	AAAA	2600:1700:b310:c2e0::4	yes
www.bonsi.org	A	162.201.66.177	yes	1	0
	AAAA	2600:1700:b310:c2e0::2	yes
	AAAA	2600:1700:b310:c2e0::3	yes
	AAAA	2600:1700:b310:c2e0::4	yes

but only ipv4 works:

Domainname	Http-Status	redirect	Sec.	G
• http://bonsi.org/
162.201.66.177	302	https://bonsi.org/	0.404	A

• http://www.bonsi.org/
162.201.66.177	302	https://www.bonsi.org/	0.493	A

• http://bonsi.org/
2600:1700:b310:c2e0::2	-14		10.036	T
Timeout - The operation has timed out

• http://bonsi.org/
2600:1700:b310:c2e0::3	-14		10.027	T
Timeout - The operation has timed out

• http://bonsi.org/
2600:1700:b310:c2e0::4	-14		10.027	T
Timeout - The operation has timed out

• http://www.bonsi.org/
2600:1700:b310:c2e0::2	-14		10.027	T
Timeout - The operation has timed out

• http://www.bonsi.org/
2600:1700:b310:c2e0::3	-14		10.027	T
Timeout - The operation has timed out

• http://www.bonsi.org/
2600:1700:b310:c2e0::4	-14		10.026	T
Timeout - The operation has timed out

• https://bonsi.org/
162.201.66.177	200		2.390	I

• https://bonsi.org/
2600:1700:b310:c2e0::2	-14		10.030	T
Timeout - The operation has timed out

Skipped the list.

Is it possible to remove the ipv6, create a new certificate, then fix the ipv6 problem?

Do you have [::] or somewhere explicit (now wrong) ipv6 addresses (in vHosts)?

ebonsi · March 28, 2019, 5:24pm

Thanks, @JuergenAuer,

I guess I will have to do just that! AT&T recently changed my ipv6 addresses and network configuration as they do not have enough ipv6s to play with it. Worse, they did not even give me a courtesy warning. I had to find that for myself checking the router. I am having to edit every piece of data related to it on 3 servers. I already started the process and might have left something. On top of that, I am testing the Mojave upgrade in one of the servers and there were some issues with the network card and BIND. The network card won’t stick and Bind need to be practically re-installed. Then we have to scramble here to find out why things are not working as before. Every time I see one upgrade now, I am running to get my aspirin bottle.

ebonsi · March 28, 2019, 7:53pm

TXT - Entries

Domainname	TXT Entry	Status	∑ Queries
bonsi.org	v=spf1 include:_spf.google.com ~all	ok	1
www.bonsi.org		ok	1
_acme-challenge.bonsi.org		Name Error - The domain name does not exist	1
_acme-challenge.www.bonsi.org		Name Error - The domain name does not exist	1
_acme-challenge.bonsi.org.bonsi.org		Name Error - The domain name does not exist	1
_acme-challenge.www.bonsi.org.www.bonsi.org		Name Error - The domain name does not exist	1

I can see and It seems to me that acme is requiring to validate a domain by a reversing ip and that is a mistake and is not going to happen! Not because of any network mistake but because nx domains are just that. As I said before, asking AT&T for anything else than what they have will imply in an extra cost, (you all know how monopolized greedy corporations works, don’t you?). This, not a commercial server. It is a test server only and I use for free educational purposes on open source software.

The Server exists and it is accessible through the www over Tor.

I have control of the server and shell access.

The domain resolve on the www, (Otherwise I wouldn’t be able to access their challenge through Tor)
I think that is the main things they should be looking for, not the if the ip resolve to the domain name. I turned off the ipv6 and still getting an error.

Tried to issue the Certbot commands and the same things happens.

clientuser$ sudo certbot --apache
Password:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
No names were found in your configuration files. Please enter in your domain
name(s) (comma and/or space separated) (Enter ‘c’ to cancel): bonsi.org, www.bonsi.org
Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for bonsi.org
http-01 challenge for www.bonsi.org
Cleaning up challenges
Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.

Right after that, I tested the domain using Tor and it logged fine. I am assuming Certbot is failing somewhere to recognize the host. Maybe Certbot’s mistake is trying to connect using an ip address and not the host.

Not sure what to go from here.

JuergenAuer · March 28, 2019, 8:12pm

The TXT entries are irrelevant if you use http-01 validation.

These entries are only relevant if someone uses dns-01 - validation.

But my tool checks that to see if entries are defined.

The first block with the ip addresses and the url-check ( https://check-your-website.server-daten.de/?q=bonsi.org#url-checks ) is relevant if you use http-01 validation.

Do you have an ipv6 listen directive?

Listen [::]:80
Listen [::]:443

JuergenAuer · March 28, 2019, 8:14pm

Every browser and every tool connects only the ip address.

The domain is sent in the header field.

Domainname -> dns query to find an ip address -> connecting the ip address with the domain name -> vHost-selection with the certificate -> Ssl connection -> https connection (http over ssl).

ebonsi · March 28, 2019, 8:20pm

I do have these directives on the Apache httpd.conf and on httpd-ssl.conf for ipv4. I’ll enter the ipv6 and try again.

ebonsi · March 28, 2019, 8:41pm

Apache is not accepting these directives for ipv6. I am reading the new semantics and seems that the directive should be only

Listen 80

https://httpd.apache.org/docs/2.4/en/bind.html
Looks like there is no need for colon anymore.
Testing…
Simple directives working fine

root# scutil -r bonsi.org

Reachable

schoen · March 28, 2019, 8:53pm

Unfortunately, I’m still seeing an inability to connect via IPv6 to any of the three IPv6 addresses listed in your DNS.

ebonsi · April 7, 2019, 4:37pm

Thanks for testing @schoen ! I am able to ping all the ipv6 addresses from the server terminal.
The ipv6 addresses from the account have been changed recently by AT&T. I did my best to reconfigure but it seems to have some glitches happening somewhere in the network blocking the address.

Anyway, I do not want people to feel that they can be stranded by the Certbot software and not being able to renew their domains.

The solution is to get a certificate using Zerossl where you can validate your Domain through DNS Challenge inserting the challenge code as txt in the DNS Table at the Register (the one that host your domain) of your domain.

JuergenAuer · April 7, 2019, 4:53pm

It’s the same picture: Only timeouts ( https://check-your-website.server-daten.de/?q=bonsi.org ).

There are Certbot installations which support dns-01 - validation.

Check the list of clients:

Acme.sh supports a lot of dns-plugins. So it’s possible to automate that.

JuergenAuer · April 7, 2019, 5:01pm

That’s interesting:

Ping works. Same with tracert:

tracert 2600:1700:b310:c2e0::2

Routenverfolgung zu 2600:1700:b310:c2e0::2 über maximal 30 Hops

  1    <1 ms    <1 ms    <1 ms  vl1991.sw-sbs1-2.11.as6724.net [2a01:238:3000::3]
  2    <1 ms    <1 ms    <1 ms  xe-4-0-1.0.core-b2.as6724.net [2a01:238:b:2b2::1]
  3    <1 ms    <1 ms    <1 ms  ae2.0.core-b30.as6724.net [2a01:238:0:b230::2]
  4    <1 ms    <1 ms    <1 ms  bei-b2-link.telia.net [2001:2000:3080:6b::1]
  5     *        *        *     Zeitüberschreitung der Anforderung.
  6     *        *        *     Zeitüberschreitung der Anforderung.
  7     *        *        *     Zeitüberschreitung der Anforderung.
  8   114 ms   114 ms   113 ms  ash-b1-v6.telia.net [2001:2000:3018:99::1]
  9   104 ms   107 ms   107 ms  2001:1890:1fff:21c:192:205:33:25
 10   170 ms   170 ms   171 ms  wshdc82crs.ipv6.att.net [2001:1890:ff:ffff:12:122:135:102]
 11   173 ms   175 ms   175 ms  2001:1890:ff:ffff:12:122:135:250
 12   176 ms   174 ms   175 ms  2001:1890:ff:ffff:12:122:28:205
 13   177 ms   175 ms   175 ms  sffca21crs.ipv6.att.net [2001:1890:ff:ffff:12:122:1:174]
 14   170 ms   166 ms   167 ms  scaca403cts.ipv6.att.net [2001:1890:ff:ffff:12:122:137:185]
 15   170 ms   170 ms   170 ms  2001:1890:ff:e167:12:83:105:126
 16     *        *        *     Zeitüberschreitung der Anforderung.
 17     *        *        *     Zeitüberschreitung der Anforderung.
 18   190 ms   189 ms   190 ms  2600:1700:b310:c2e0::2

So it looks that your Apache configuration doesn’t work.

ebonsi · April 7, 2019, 6:33pm

Your help is mostly appreciated @JuergenAuer I prefer to think that this is all on myself instead of throwing the responsibility of this issue to Certbot or AT&T (Even thou they changed my ipv6 without even tell me). The MacOSX does have it’s particularity on configuring the routing stack local and the www.
In this discussion, I am trying to make things as open source as possible – (even exposing my network, I can see in the logs that [some people] uses the network information for other purposes in attempt to see if they can break something) – so, on my side the hope is to help others to understand these type of issues. But that is what open source is all about… to test the security and resilience of the system. I am not sure in other systems but Apache on OSX is no longer accepts the listen directives, [::]:80 or listen [::]:443 I am still reading about that.

> One issue about ipv6 (For further discussion) is that even if one configures ok, the Certificate does not allow people to connect to ipv6 directly. If one tries to connect using the ipv6 address their browser will say you are connecting to an insecure site even thou you got the certificate working on the domain ok.

By the way I just fixed on issue, happening locally using the commands;

ifconfig en0

and

ifconfig en0 | grep inet6 | awk -F " " '{print $2}' | sed 's/%en0//'

The routing now between local (inet6 fe80:: …) and the www addresses is flowing fine through the Network Stack.

JuergenAuer · April 7, 2019, 6:58pm

PS: There is a running recheck. You can check one ipv6 address direct to check if your server answers.

system · May 7, 2019, 6:58pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.