Hi .....
New to this method of renewing certs and I am trying to get me NetSecEng head around this. When I look at something, I am mainly looking at the data communication paths going through my security appliances. Is there some form of drawing that describes the communication paths of the recertifcation method?
netseceng
Are you looking for IP addresses? I am not sure I understand your question, if there is one.. AND we don't have any information relating to your configuration:
Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=mydomain.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
My domain is:
I ran this command:
It produced this output:
My web server is (include version):
The operating system my web server runs on is (include version):
My hosting provider, if applicable, is:
I can login to a root shell on my machine (yes or no, or I don’t know):
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
https://certbot.eff.org/docs/using.html#certbot-command-line-options
Your success with curl shows that this is not an IP communication problem. I suggest you open a new topic and use the template to fully describe your errors.
Short(TXT) version:
A connects with CA
If HTTP(S) authentication path: Vs connect to W at F
If DNS authentication path: CA connects to D
If valid, then CA gives A the new(renewed) cert.
Longer version:
Acme client[A] connects to Certificate Authority[CA] (via HTTPS)
If HTTP(S) authentication path chosen:
Validation Servers connect to Web Service at FQDN (IP) and request validation token file.
If DNS authentication path:
CA connects to (global) DNS and retrieves validation TXT record
If valid, then CA gives ACME client the new(renewed) cert.
So the basic flow if using http validation is that the ACME client (Certify The Web in this case) talks to the Let's Encrypt API using outgoing https. Then Let's Encrypt make an http (TCP port 80) request to your server which Certify The Web (running on your server) responds to, using a temporary http server that sits in the http.sys pipeline before IIS. If you are using other webservers like apache then these don't support that configuration and you need to use filesystem based validation (write files that are served by your actual webserver software) or use DNS validation (which talks to your DNS API provider and sets a special TXT record).
By default Certify The Web also has a reporting API for config validation and failure notifications etc, but it'll work without that.
If you're in the habit of blocking outgoing https you may want to allow exceptions for the Certify background service as none of these APIs have fixed IPs.
Ok ..... Drawing is a start however I need IPs and/or FQDNs
And it seems I forgot some information:
This is an internal Microsoft Exchange server not an web server.
We do not allow OWA so there is no external (Internet) access via HTTP or HTTPS unless we write a specific FW rule inbound.
We installed this method for renewing the cert back in January.
It renewed fine in Feb/Mar/Apr/May/Jun
It started failing in July
What we see in the log is:
2022-08-09 15:24:59.740 -04:00 [INF] ---- Beginning Request [mail.colonialmotorcars.com] ----
2022-08-09 15:24:59.753 -04:00 [INF] Certify/5.6.8.0 (Windows; Microsoft Windows NT 10.0.14393.0)
2022-08-09 15:24:59.988 -04:00 [INF] Beginning Certificate Request Process: mail.colonialmotorcars.com using ACME Provider:Certes
2022-08-09 15:24:59.989 -04:00 [INF] Requested identifiers to include on certificate: mail.colonialmotorcars.com
2022-08-09 15:24:59.989 -04:00 [INF] Beginning certificate order for requested domains
2022-08-09 15:25:00.407 -04:00 [INF] BeginCertificateOrder: creating/retrieving order. Retries remaining:2
2022-08-09 15:25:01.099 -04:00 [INF] Created ACME Order: https://acme-v02.api.letsencrypt.org/acme/order/403336350/114673871856
2022-08-09 15:25:01.260 -04:00 [INF] Fetching Authorizations.
2022-08-09 15:25:01.692 -04:00 [INF] Got http-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/140204922836/FsGq7g
2022-08-09 15:25:01.809 -04:00 [INF] Got dns-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/140204922836/EMFdWQ
2022-08-09 15:25:04.115 -04:00 [INF] Http Challenge Server process available.
2022-08-09 15:25:04.116 -04:00 [INF] Attempting Domain Validation: mail.colonialmotorcars.com
2022-08-09 15:25:04.116 -04:00 [INF] Registering and Validating mail.colonialmotorcars.com
2022-08-09 15:25:04.116 -04:00 [INF] Preparing automated challenge responses (mail.colonialmotorcars.com)
2022-08-09 15:25:04.127 -04:00 [INF] Preparing challenge response for the issuing Certificate Authority to check at: http://mail.colonialmotorcars.com/.well-known/acme-challenge/tSc3kle3cHNhDbnl8jvUSqNltx0vbT2gxS-5PN5TCJQ with content tSc3kle3cHNhDbnl8jvUSqNltx0vbT2gxS-5PN5TCJQ.5vC3jaQsn4pIq3-h5gdT-CmX_c-LrbuzalrxlvRDKyw
2022-08-09 15:25:04.127 -04:00 [INF] If the challenge response file is not accessible at this exact URL the validation will fail and a certificate will not be issued.
2022-08-09 15:25:04.131 -04:00 [INF] Using website path C:\inetpub\wwwroot
2022-08-09 15:25:04.133 -04:00 [INF] Requesting Validation: mail.colonialmotorcars.com
2022-08-09 15:25:04.156 -04:00 [INF] Attempting Challenge Response Validation for Domain: mail.colonialmotorcars.com
2022-08-09 15:25:04.156 -04:00 [INF] Registering and Validating mail.colonialmotorcars.com
2022-08-09 15:25:04.156 -04:00 [INF] Checking automated challenge response for Domain: mail.colonialmotorcars.com
2022-08-09 15:25:14.928 -04:00 [INF] Domain validation failed: mail.colonialmotorcars.com
65.211.26.38: Fetching http://mail.colonialmotorcars.com/.well-known/acme-challenge/tSc3kle3cHNhDbnl8jvUSqNltx0vbT2gxS-5PN5TCJQ: Timeout during connect (likely firewall problem) BadRequest urn:ietf:params:acme:error:connection
So from what I surmising, the ????? server does validate the correct IP address from DNS however it is failing on the HTTP check??????
I need the FQDN(s) of the ????? server to check my FW logs and to rewrite rule if necessary.
OR IS THERE AN EASIER WAY WITHOUT THE HTTP CHECK?
LetsEncrypt does not publish their validation IPs as they are subject to change at any point. Their engineers have shared some hints about what they are in several posts, but that's about it.
You can consider using DNS-01 validation.
Does your environment use PaloAlto firewalls? If so, you might be running into this, but the timing of when it started failing doesn't quite match. Worth looking into though if you do.
Specifically, for the last several months, the IPs that perform multi-perspective validation change every few hours, and our primary datacenter IPs have changed a couple of times too.
Our blog post about MPV has some reasons why: Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt
Yes, as mentioned there is also the option of DNS validation instead of http validation. This involves (automatically) creating a specific TXT record in your domain DNS DNS Validation (dns-01) | Certify The Web Docs
Note that you don't have to run a web server on your servers to be able to use http challenges, Certify The Web has it's own built in http listener during validation. You would just need to open TCP port 80 so the traffic can get through. This is the easiest option, DNS is the alternative but it can be trickier depending on how your DNS service is hosted.
You noted that this started failing in July so it looks like you updated your firewall in July and your quickest and easiest solution would be to fix that.
No we use a different vendor. I can't be certain it is FW until I can find the communication in the FW logs. And I cant do that without knowing all the data paths and IP addressing or FQDNs. Since they dont give the IPs then is there someone here that knows the FQDNs? I can use those to convert to IPs and find the latest attempts.
I dont think the DNS option will work as we do not mange our external DNS ..... So is there someone who knows the FQDNs for the servers that will be contacting my mail server?
You should be able to send a sample challenge request from any IP outside your local network. Even use a mobile phone with wifi disabled.
A challenge request always has this format:
http://(domain)/.well-known/acme-challenge/ChallengeTokenValue
I just tried a couple from my own test server (in AWS) and it times out. The Let's Debug site times out too.
You should make a rule in your FW to allow a URL with /.well-known/acme-challenge from any IP and it should work.
This information does not exist for the public, or possibly at all.
Your options are to:
Thank you MikeMcQ for the attempt however I am trying to validate the requests came from the actual servers in my FW logs. This tells me there isn't a problem between the Lets Encrypt servers on the Internet and my server. Unfortunately, your solution will not use the correct source IP address of the lets Encrypt servers.
Correct. You cannot know those IP's and they are prone to change from hour to hour.
So, testing from any IP should be adequate to test a rule that allows the challenge URL.
Once that unit-test is complete you can try a --dry-run or actual cert request
No ..... not a test attempt ..... I want to verify the actual attempt is getting to my FW. The only way I can do that is if I have the IP addresses or the FQDNs. If no IP addresses ranges, at the time of failure, I can use the nslookup and the FQDNs to get an IP address and check my FW logs.
Some of our validation endpoints do not have FQDNs (forward or reverse).
Best solution:
Get the FQDNs and then validate the communication attempts are hitting FW and failing there. If yes, write a FW rule that tracks those FQDNs through DNS as source and my mail server as destination on HTTP port and allow. Sounds like the best plan.
Is there anyone out there who can provide the FQDNs?
Negative.
No.