Strange: authorization fails with timeout but it connected

Hi,
I run a number of servers and already have 3 certificates running, 1 on Fedora and 2 on Ubuntu.
These have all been generated from Fedora box which is acting as a central repository. The authorization has been achieved by placing redirects on the target domains to a url on the central server.
I am in the process of moving some domains from the second Ubuntu server to a third but have tripped over a weird problem. The domains not yet using SSL have been moved. All of these target domains have been authorized but the fqdn of the host server has not!
When I watch the traffic on the target host with tcpdump I see the following exchange with the authentication server 66.133.109.36.:
< SYN
SYN/ACK >
< ACK
< FIN/ACK
FIN/ACK >
ACK >

There is no data exchange and the FIN/ACK follows the ACK by 0.55 ms.
Any ideas?
John

Please fill out the fields below so we can help you better.

My domain is:
jaytee-solutions.co.uk

I ran this command:
certbot certonly --webroot -w /var/www/html -d vesta.jaytee-solutions.co.uk,,,…
dom? are other domains

It produced this output:

My web server is (include version):
apache 2.4

The operating system my web server runs on is (include version):
certbot 0.14.1 running on Fedora 25
webhost ubuntu 16.04

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

Hi @jaytee,

It gives me a time out too.

$ curl -IkL http://vesta.jaytee-solutions.co.uk/.well-known/acme-challenge/mJgjK4gvJhAr-JJMHLUEwXbQHESgS-Ps0II1izDCbOo
HTTP/1.1 301 Moved Permanently
Date: Sun, 10 Sep 2017 15:01:00 GMT
Server: Apache/2.4.18 (Ubuntu)
Location: http://noc.jaytee-solutions.co.uk/.well-known/acme-challenge/mJgjK4gvJhAr-JJMHLUEwXbQHESgS-Ps0II1izDCbOo
Content-Type: text/html; charset=iso-8859-1

curl: (7) Failed to connect to noc.jaytee-solutions.co.uk port 80: Connection timed out

So, requests to your doman are being redirected to http://noc.jaytee-solutions.co.uk/* and noc.jaytee-solutions.co.uk has a CNAME pointing to oracle.jaytee-solutions.co.uk that resolves to 82.152.50.4 but or you have a firewall or nothing is listening on port 80 on that ip.

Cheers,
sahsanu

Hi @sahsanu
Yes that is exactly the mechanism I am using. However oracle is firewalled on port 80 with only the validation server being allowed so no surprise you timed out at that point.
However, you’ve got further than the validation bot. It doesn’t send the GET or anything for that matter so there is no reply for it to see the 301.

The situation I currently have is 2 guest domains on vesta and 2 on slarti. I can see the validations for all 4 guest domains in oracle’s logs as well as on their host server. Here is the redacted extract from vesta. slarti adn oracle would just be repeating the same.

icarus:80 66.133.109.36 - - [09/Sep/2017:15:55:52 +0000] “GET /.well-known/acme-challenge/TLCqb…S1Q HTTP/1.1” 301 700 “-” "Mozilla/5.0 (compatible; Let’s Encrypt validation server; +https://www.letsencrypt.org)"
realise:80 66.133.109.36 - - [09/Sep/2017:15:55:51 +0000] “GET /.well-known/acme-challenge/e5cT…9AI HTTP/1.1” 301 701 “-” “Mozilla/5.0 (compatible; Let’s Encrypt validation server; +https://www.letsencrypt.org)”

I can see your two requests in vesta’s logs and one in slarti where it appears you tried the top domain. From that I am 100% sure my servers are behaving so are we looking at a validations process issue?

There are 3 situations that might have an impact.

1 Initially I had a firewall issue on vesta. That was fixed yesterday.

2 I have an IPv6 address for vesta but had not confirmed apache’s operation on v6. I have dropped the v6 address from DNS and am waiting for it to time out.

3 jaytee-solutions. is being server from slarti but is on the list for vesta’s certificate.

Any clues in that?

John

If you are blocking me then you are blocking Let's Encrypt too. Let's Encrypt source ips are not public available, even if you get one ip from Let's Encrypt and you are allowing it, since one or two weeks ago, Let's Encrypt validates your domains from at least 3 different sources at the same time so I'm afraid you should not use a firewall to block access to oracle domain on port 80 if you want to validate your domains.

You can always use DNS-01 challenge so no need of web servers, just the ability to update a dns record (your dns servers should provide an API or a method to create/delete records in an automatic way or could be a pain in the neck) ;).

Cheers,
sahsanu

I am not convinced this was the problem as the publicly available server vesta never received a valid call and so did not issue a redirect for this domain. The conversation could be likened to a telephone call where the caller hangs up as soon as the other party answers without either of them saying a word.

You may gather I have managed to obtain a certificate but I suspect something has timed out. This time I modified oracle’s firewall with one rule for each of the known IP addresses and one rule for everyone else. The request appears in both vesta and oracle’s apache log and trips the firewall rule for a known IP address. I have turned it off for now while I implement countermeasures.

I looked at that but potential issues with propagation delays worried me.

What would be nice is if certbot has a hook that announced the validation source(s) so access could be dynamically controlled. Call me paranoid but I don’t want all and sundry calling /.well-known/… You should see my server’s logs since my first post on this topic. Apple,Google,etc all having a go at that url plus a few variants!

Thanks for your help.

The security goals of you, and of Let's Encrypt and the relying parties (i.e. everyone else), are in conflict here. Allowing all and sundry to access /.well-known/acme-challenge/ is unfortunately exactly what they require.

Since someone in a position to hijack BGP between your server and Let's Encrypt could fraudulently pass validation and obtain a certificate, using variable and undocumented IP addresses (somewhat) increases trust. While it's not hard to obtain the list of critical IP addresses, they don't want to make it any easier.

Additionally, if they were required to provide the IP addresses ahead of time, it would make it technically more difficult or impossible to enable future validation using, for example, Tor or a commercial VPN provider.

This isn't an issue - Let's Encrypt always asks the authoritative nameserver for your domain, so the only timing factor is how quickly your DNS provider updates this information.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.