Connecting to old dynamic IP (DNS issue?)

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: cr.tuz.ovh

I ran this command: n/a

It produced this output:
"when": "Tue, 12 Sep 2023 11:00:04 GMT",
"type": "renewal-error",
"status": "urn:ietf:params:acme:error:connection",
"detail": "62.10.1.74: Fetching http://cr.tuz.ovh/.well-known/acme-challenge/SlGqPZ6PSflGMKlrVoyGL655iQrk4yHCIZ9tETu2Kgc: Error getting validation data"
},
{
"when": "Tue, 12 Sep 2023 11:00:04 GMT",
"type": "progress",
"detail": "Starting challenges for domains: 62.10.1.74: Fetching http://cr.tuz.ovh/.well-known/acme-challenge/SlGqPZ6PSflGMKlrVoyGL655iQrk4yHCIZ9tETu2Kgc: Error getting validation data, problem: urn:ietf:params:acme:error:connection"

My web server is (include version): FortiGate

The operating system my web server runs on is (include version): 7.0.12

My hosting provider, if applicable, is: ovh.net

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): n/a

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): n/a

Hello,
I've been using the FortiGate's ACME feature since many months with Letsencrypt without issues. But a few days ago I received the email warning that the cert was not auto-renewed and expiring in 19 days (I left the default auto-renew at 30 days) so I checked the FortiGate logs and I noticed that Letsencrypt is trying to connect to my old dynamic IP. Current dynamic IP changed a few days ago and is resolved correctly.
The FortiGate tries automatically every hour or 2 and it's been days now, still the same old IP.

tuzhao@ciuchino:~ $ dig A +short cr.tuz.ovh @dns101.ovh.net
62.10.4.240

This is my current dynamic IP

Not sure what I can do to fix this, any help appreciated :slight_smile:

2

Could you potentially have an entry in /etc/hosts overriding DNS for that hostname on that machine? Your dig command would bypass that but what about a normal nslookup with no nameserver specified, or just ping the hostname from the machine in question?

It consistently resolves to 62.10.4.240 for me from everywhere I checked, so it seems like you have something abnormal happening locally, either a hard-coded host file entry or abnormal DNS caching far beyond your 1-minute TTL.

I'm not familiar with FortiGate but can it be restarted to flush its DNS cache? A quick search indicated it has commands to view its DNS cache so you might want to try that first.

3 Likes
3

A side note Port 80 is Open, but Port 443 is filtered

$ nmap -Pn -p80,443 cr.tuz.ovh
Starting Nmap 7.80 ( https://nmap.org ) at 2023-09-12 15:55 UTC
Nmap scan report for cr.tuz.ovh (62.10.4.240)
Host is up (0.18s latency).

PORT    STATE    SERVICE
80/tcp  open     http
443/tcp filtered https

Nmap done: 1 IP address (1 host up) scanned in 3.76 seconds
2 Likes
4

The staging environment currently seems to be using the correct IP address 62.10.4.240. The production environment should to so too.

Can you perhaps try again? Maybe some propogation issue previously.

2 Likes
5

There is more going wrong than just the IP in the error. Repeated tests using staging (https://letsdebug.net) shows the same "got" value for each fresh challenge test. This is wrong. An actual cert request will have a unique value returned. And, in a Let's Debug test no value should be returned (should get 404 error) as the tested server is not aware of the challenge token.

Something seems "stuck" - probably Fortigate.

acme: error code 403 "urn:ietf:params:acme:error:unauthorized": 
The key authorization file from the server did not match this challenge. 
Expected "AmoBXsO1Vx6M9pmw9xUQQH0bYxrHzxgWecuhXOdkJGQ.49wknCPN_3HICrKF6BR-V-a-E_ipoaGro7D1Fju_2ec" 
(got "SlGqPZ6PSflGMKlrVoyGL655iQrk4yHCIZ9tETu2Kgc.1O3tf1CnKIwR3pgiQPjftg07gQzX4qQsw-Fzg1L5A4c")
3 Likes
6

I get:

# dig A +short cr.tuz.ovh @dns101.ovh.net
62.10.1.250

# dig A +short cr.tuz.ovh @ns101.ovh.net
62.10.1.250
3 Likes
7

Not only is the IP different the Let's Debug test now fails with a 403 instead of returning faulty response data. Actually, any URL gets a 403 with the same HTML error text including one formatted like an ACME HTTP Challenge. And, even a "home" page request.

This at least seems like some kind of progress

curl -i http://cr.tuz.ovh/.well-known/acme-challenge/E21YBg3n8Na9F4cdDIIhatyw4xhjyom5hxiutyXq31g

HTTP/1.1 403 Forbidden
Date: Wed, 13 Sep 2023 03:04:18 GMT

<!DOCTYPE html><html><head>
<title>ACME Access Only</title></head>
<body>ACME Access Only</body></html>
3 Likes
8

Hello,
thanks for all the replies.
I did reboot the FortiGate last night just to see if that would help, that's probably why the dynamic IP address changed. So that's the correct current dynamic IP: 62.10.1.250 and it's properly assigned to the WAN interface.
Looking at the currently installed cert it seems that it's still the old one but looking at the logs Letsencrypt was able to verify the domain (I can't upload TXT files apparently so I'll copy paste):

CRfw-61E # diag sys acme status-full cr.tuz.ovh
{
  "name": "cr.tuz.ovh",
  "finished": true,
  "notified": false,
  "next-run": "Thu, 14 Sep 2023 00:35:20 GMT",
  "last-run": "Wed, 13 Sep 2023 01:35:10 GMT",
  "valid-from": "Thu, 14 Sep 2023 00:35:20 GMT",
  "errors": 0,
  "last": {
    "status": 0,
    "detail": "The certificate for the managed domain has been renewed successfully and can be used from Thu, 14 Sep 2023 00:35:20 GMT on.",
    "valid-from": "Thu, 14 Sep 2023 00:35:20 GMT"
  },
  "log": {
    "entries": [
      {
        "when": "Wed, 13 Sep 2023 01:35:23 GMT",
        "type": "finished"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:23 GMT",
        "type": "progress",
        "detail": "The certificate for the managed domain has been renewed successfully and can be used from Thu, 14 Sep 2023 00:35:20 GMT on."
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:22 GMT",
        "type": "progress",
        "detail": "Retrieving certificate chain for cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:21 GMT",
        "type": "progress",
        "detail": "Waiting for finalized order to become valid"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:19 GMT",
        "type": "progress",
        "detail": "Submitting CSR to CA for cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:19 GMT",
        "type": "progress",
        "detail": "Creating CSR for cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:18 GMT",
        "type": "progress",
        "detail": "Finalizing order for cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:17 GMT",
        "type": "progress",
        "detail": "Waiting for order to become ready"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:17 GMT",
        "type": "progress",
        "detail": "Monitoring challenge status for cr.tuz.ovh: domain authorization for cr.tuz.ovh is valid"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:16 GMT",
        "type": "progress",
        "detail": "Monitoring challenge status for cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:15 GMT",
        "type": "progress",
        "detail": "Setting up challenge 'http-01' for domain cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:14 GMT",
        "type": "progress",
        "detail": "Starting challenges for domains"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:12 GMT",
        "type": "progress",
        "detail": "Creating new order"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:11 GMT",
        "type": "progress",
        "detail": "Selecting account to use for cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:11 GMT",
        "type": "progress",
        "detail": "Driving ACME protocol for renewal of cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:11 GMT",
        "type": "progress",
        "detail": "Resetting staging for cr.tuz.ovh"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:10 GMT",
        "type": "progress",
        "detail": "Contacting ACME server for cr.tuz.ovh at https://acme-v02.api.letsencrypt.org/directory"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:10 GMT",
        "type": "progress",
        "detail": "Assessing current status"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:10 GMT",
        "type": "progress",
        "detail": "Resetting staging area"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:10 GMT",
        "type": "progress",
        "detail": "Checking staging area"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:10 GMT",
        "type": "progress",
        "detail": "Starting challenges for domains"
      },
      {
        "when": "Wed, 13 Sep 2023 01:35:10 GMT",
        "type": "starting"
      },
      {
        "when": "Wed, 13 Sep 2023 00:33:19 GMT",
        "type": "message-errored"
      },
      {
        "when": "Wed, 13 Sep 2023 00:33:19 GMT",
        "type": "renewal-error",
        "status": "urn:ietf:params:acme:error:connection",
        "detail": "62.10.1.74: Fetching http://cr.tuz.ovh/.well-known/acme-challenge/SlGqPZ6PSflGMKlrVoyGL655iQrk4yHCIZ9tETu2Kgc: Error getting validation data"
      },

It looks like the cert will be available starting from tomorrow so I'll report back.

The last error was at 00:33, I need to check when the FortiGate actually rebooted or when the dynamic IP changed to understand if one or the other solved the issue.

10

All certs are always available immediately [upon issuance].
I think it may be misrepresenting the time zone/date information.
What do you see in: System | Certificates

2 Likes
11

It still shows the previous one, that's why I'd like to check what happens tonight:

Issuer:
Common Name (CN)	R3
Organization (O)	Let's Encrypt
Country/Region (C)	US

Validity Period:
Valid From			2023/06/30 11:15:20
Valid To			2023/09/28 11:15:19

The device is in GMT+2 zone, the reboot happened at 3 AM, so if the logs are UTC it might mean that at 2:33 (logs say 0:33) there was the last failed attempt, then the reboot happened, and at 3:33 (logs say 1:33) the other attempt was successful. But I'm just speculating :slight_smile:

1 Like
13

In review of such logs, it seems that is the expected behavior employed by FortiOS.
I had never noticed that before...
It seems to only affect renewals.
New issuances can be used right away.

3 Likes
14

Yeah, the cert was renewed and valid now but looks like Fortigate won't start using it until tomorrow

image
image1947×495 80.4 KB

2 Likes
15

I found a somewhat similar discussion on reddit:
(2) Let's encrypt certificate renewal : fortinet (reddit.com)

It seems there may be some parameter that maintains the renewal implementation schedule.
And even though the cert may be expired, it will wait until the next scheduled window to switch certs.
You might want to ask Fortinet about that.

3 Likes
16

Bingo:

Issuer:
Common Name (CN)		R3
Organization (O)		Let's Encrypt
Country/Region (C)		US

Validity Period:
Valid From				2023/09/13 02:35:20
Valid To				2023/12/12 01:35:19

Still not sure whether the reboot fixed the issue, or Letsencrypt took 7-8 days to refresh its DNS cache for my domain :slight_smile:

17

That is not possible.

I think it was just time that fixed the problem.
[waiting for the firewall to apply the new cert]

3 Likes
18

The error message was referencing the old dynamic IP for a week or so, the FortiGate only waits 1 day to install the new certificate (which it did after the cert was generated). Not sure where that old IP came from.
I'm assuming Letsencrypt was resolving my domain and getting the old dynamic IP.
It would be interesting to see logs from Letsencrypt side just out of curiosity.

19

That is not possible.
LE doesn't use any DNS caching, nor any non-authoritative DNS systems.
If LE went to the "wrong IP", then the entire authoritative DNS system had the "wrong IP".

4 Likes
20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.