Unable to renew using Sophos UTM 9 (dehydrated)

Uniquenospacesshort · July 8, 2022, 8:11pm

My domain is: Blueboxit.ca

I ran this command: Sophos UTM, Webserver Protection, Certificate Management, Renew

It produced this output:

handling CSR REF_CaCsrMYNAMElet for domain set [autodiscover.blueboxit.ca,utm.blueboxit.ca,mail.blueboxit.ca,secure.blueboxit.ca,sip.blueboxit.ca,spam.blueboxit.ca,email.blueboxit.ca,vpn.blueboxit.ca]
running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain autodiscover.blueboxit.ca --domain utm.blueboxit.ca --domain mail.blueboxit.ca --domain secure.blueboxit.ca --domain sip.blueboxit.ca --domain spam.blueboxit.ca --domain email.blueboxit.ca --domain vpn.blueboxit.ca
command completed with exit code 256
 COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: ["type"] "http-01"
COMMAND_FAILED: ["status"] "invalid"
COMMAND_FAILED: ["error","type"] "urn:ietf:params:acme:error:unauthorized"
COMMAND_FAILED: ["error","detail"] "155.138.131.34: Invalid response from https://mail.blueboxit.ca:443/.well-known/acme-challenge/ZB1WqE0VgRkNjvBC9KAFoUX1xKpOWBZjSoi3L5aZAHs: 403"
COMMAND_FAILED: ["error","status"] 403
COMMAND_FAILED: ["error"] {"type":"urn:ietf:params:acme:error:unauthorized","detail":"155.138.131.34: Invalid response from https://mail.blueboxit.ca:443/.well-known/acme-challenge/ZB1WqE0VgRkNjvBC9KAFoUX1xKpOWBZjSoi3L5aZAHs: 403","status":403}
COMMAND_FAILED: ["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/128386722946/9qYOgQ"
COMMAND_FAILED: ["token"] "ZB1WqE0VgRkNjvBC9KAFoUX1xKpOWBZjSoi3L5aZAHs"
COMMAND_FAILED: ["validationRecord",0,"url"] "http://mail.blueboxit.ca/.well-known/acme-challenge/ZB1WqE0VgRkNjvBC9KAFoUX1xKpOWBZjSoi3L5aZAHs"
COMMAND_FAILED: ["validationRecord",0,"hostname"] "mail.blueboxit.ca"
COMMAND_FAILED: ["validationRecord",0,"port"] "80"
COMMAND_FAILED: ["validationRecord",0,"addressesResolved",0] "155.138.131.34"
COMMAND_FAILED: ["validationRecord",0,"addressesResolved"] ["155.138.131.34"]
COMMAND_FAILED: ["validationRecord",0,"addressUsed"] "155.138.131.34"
COMMAND_FAILED: ["validationRecord",0] {"url":"http://mail.blueboxit.ca/.well-known/acme-challenge/ZB1WqE0VgRkNjvBC9KAFoUX1xKpOWBZjSoi3L5aZAHs","hostname":"mail.blueboxit.ca","port":"80","addressesResolved":["155.138.131.34"],"addressUsed":"155.138.13
COMMAND_FAILED: ["validationRecord",1,"url"] "https://mail.blueboxit.ca:443/.well-known/acme-challenge/ZB1WqE0VgRkNjvBC9KAFoUX1xKpOWBZjSoi3L5aZAHs"
COMMAND_FAILED: ["validationRecord",1,"hostname"] "mail.blueboxit.ca"
COMMAND_FAILED: ["validationRecord",1,"port"] "443"
COMMAND_FAILED: ["validationRecord",1,"addressesResolved",0] "155.138.131.34"
COMMAND_FAILED: ["validationRecord",1,"addressesResolved"] ["155.138.131.34"]
COMMAND_FAILED: ["validationRecord",1,"addressUsed"] "155.138.131.34"
COMMAND_FAILED: ["validationRecord",1] {"url":"https://mail.blueboxit.ca:443/.well-known/acme-challenge/ZB1WqE0VgRkNjvBC9KAFoUX1xKpOWBZjSoi3L5aZAHs","hostname":"mail.blueboxit.ca","port":"443","addressesResolved":["155.138.131.34"],"addressUsed":"155.
COMMAND_FAILED: ["validationRecord"] [{"url":"http://mail.blueboxit.ca/.well-known/acme-challenge/ZB1WqE0VgRkNjvBC9KAFoUX1xKpOWBZjSoi3L5aZAHs","hostname":"mail.blueboxit.ca","port":"80","addressesResolved":["155.138.131.34"],"addressUsed":"155.138.131
COMMAND_FAILED: ["validated"] "2022-07-08T20:00:16Z")
sending notification WARN-603
[WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service
execution completed (CSRs renewed: 0, failed: 1)

My web server is: Sophos UTM, Virtual Appliance. v9

The operating system my web server runs on is: 9.711-5

I can login to a root shell on my machine: Yes

I'm using a control panel to manage my site: no

The version of my client is: unknown, built in

Uniquenospacesshort · July 8, 2022, 8:24pm

rg305 · July 8, 2022, 8:51pm

Hi @Uniquenospacesshort, and welcome to the LE Community forum

I can't say I specialize in Dehydrated or Sophos...
But I am pretty good at trobleshooting.

It seems that the FQDN "mail" isn't handled as others are, see:

curl -Ii http://mail.blueboxit.ca/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 301 Moved Permanently
Date: Fri, 08 Jul 2022 20:42:21 GMT
Server: Apache
Location: https://mail.blueboxit.ca:443/.well-known/acme-challenge/Test_File-1234
Content-Type: text/html; charset=iso-8859-1

curl -Ii https://mail.blueboxit.ca:443/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 403 Forbidden
Date: Fri, 08 Jul 2022 20:42:31 GMT
Server: Apache
Content-Type: text/html; charset=iso-8859-1

While:

curl -Ii http://email.blueboxit.ca/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 301 Moved Permanently
Date: Fri, 08 Jul 2022 20:44:36 GMT
Server: Apache
Location: https://email.blueboxit.ca:443/.well-known/acme-challenge/Test_File-1234
Content-Type: text/html; charset=iso-8859-1

curl -Ii https://email.blueboxit.ca:443/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 503 Service Unavailable
Date: Fri, 08 Jul 2022 20:44:48 GMT
Server: Apache
Connection: close
Content-Type: text/html; charset=iso-8859-1

[still prpblematic - but clearly some other problem]
Also:

curl -Ii http://autodiscover.blueboxit.ca/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 301 Moved Permanently
Date: Fri, 08 Jul 2022 20:49:07 GMT
Server: Apache
Location: https://autodiscover.blueboxit.ca:443/.well-known/acme-challenge/Test_File-1234
Content-Type: text/html; charset=iso-8859-1

root@ul18ipv46:/var/tmp/trash# curl -Ii https://autodiscover.blueboxit.ca:443/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 404 Not Found
Date: Fri, 08 Jul 2022 20:49:20 GMT
Server: Apache

What has "changed" since your last renewal?
Since it is running Apache, let's try looking at:
apachectl -t -D DUMP_VHOSTS

Uniquenospacesshort · July 9, 2022, 9:08pm

Thanks for this insight, there is an option to allow for redirecting pages to other locations on UTM, like a reverse proxy, I had forced the URL request to "/.well-known/acme-challenge/" over to the respective windows servers behind the UTM.

I took those rules out, and hit refresh again - same errors (I think).

I cant say what I've changed since the initial certificate was issued, in mind review, nothing...

I Renew certificate: handling CSR REF_CaCsrBlueboxlet for domain set [autodiscover.blueboxit.ca,utm.blueboxit.ca,mail.blueboxit.ca,secure.blueboxit.ca,sip.blueboxit.ca,spam.blueboxit.ca,email.blueboxit.ca,vpn.blueboxit.ca]
I Renew certificate: running command: /var/storage/chroot-reverseproxy/usr/dehydrated/bin/dehydrated -x -f /var/storage/chroot-reverseproxy/usr/dehydrated/conf/config -c --accept-terms --domain autodiscover.blueboxit.ca --domain utm.blueboxit.ca --domain mail.blueboxit.ca --domain secure.blueboxit.ca --domain sip.blueboxit.ca --domain spam.blueboxit.ca --domain email.blueboxit.ca --domain vpn.blueboxit.ca
command completed with exit code 256
COMMAND_FAILED: ERROR: Challenge is invalid! (returned: invalid) (result: ["type"]	"http-01"
COMMAND_FAILED: ["status"]	"invalid"
COMMAND_FAILED: ["error","type"]	"urn:ietf:params:acme:error:unauthorized"
COMMAND_FAILED: ["error","detail"]	"155.138.131.34: Invalid response from https://sip.blueboxit.ca:443/.well-known/acme-challenge/qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs: 404"
COMMAND_FAILED: ["error","status"]	403
COMMAND_FAILED: ["error"]	{"type":"urn:ietf:params:acme:error:unauthorized","detail":"155.138.131.34: Invalid response from https://sip.blueboxit.ca:443/.well-known/acme-challenge/qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs: 404","status":403}
COMMAND_FAILED: ["url"]	"https://acme-v02.api.letsencrypt.org/acme/chall-v3/128386722966/KVn2Kw"
COMMAND_FAILED: ["token"]	"qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs"
COMMAND_FAILED: ["validationRecord",0,"url"]	"http://sip.blueboxit.ca/.well-known/acme-challenge/qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs"
COMMAND_FAILED: ["validationRecord",0,"hostname"]	"sip.blueboxit.ca"
COMMAND_FAILED: ["validationRecord",0,"port"]	"80"
COMMAND_FAILED: ["validationRecord",0,"addressesResolved",0]	"155.138.131.34"
COMMAND_FAILED: ["validationRecord",0,"addressesResolved"]	["155.138.131.34"]
COMMAND_FAILED: ["validationRecord",0,"addressUsed"]	"155.138.131.34"
COMMAND_FAILED: ["validationRecord",0]	{"url":"http://sip.blueboxit.ca/.well-known/acme-challenge/qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs","hostname":"sip.blueboxit.ca","port":"80","addressesResolved":["155.138.131.34"],"addressUsed":"155.138.131.34"}
COMMAND_FAILED: ["validationRecord",1,"url"]	"https://sip.blueboxit.ca:443/.well-known/acme-challenge/qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs"
COMMAND_FAILED: ["validationRecord",1,"hostname"]	"sip.blueboxit.ca"
COMMAND_FAILED: ["validationRecord",1,"port"]	"443"
COMMAND_FAILED: ["validationRecord",1,"addressesResolved",0]	"155.138.131.34"
COMMAND_FAILED: ["validationRecord",1,"addressesResolved"]	["155.138.131.34"]
COMMAND_FAILED: ["validationRecord",1,"addressUsed"]	"155.138.131.34"
COMMAND_FAILED: ["validationRecord",1]	{"url":"https://sip.blueboxit.ca:443/.well-known/acme-challenge/qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs","hostname":"sip.blueboxit.ca","port":"443","addressesResolved":["155.138.131.34"],"addressUsed":"155.138.131.34"}
COMMAND_FAILED: ["validationRecord"]	[{"url":"http://sip.blueboxit.ca/.well-known/acme-challenge/qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs","hostname":"sip.blueboxit.ca","port":"80","addressesResolved":["155.138.131.34"],"addressUsed":"155.138.131.34"},{"url":"https://sip.blueboxit.ca:443/.well-known/acme-challenge/qfdaFK5RrZLHH29cG76HmUWxRCsg7oNySJrNWLUPIIs","hostname":"sip.blueboxit.ca","port":"443","addressesResolved":["155.138.131.34"],"addressUsed":"155.138.131.34"}]
COMMAND_FAILED: ["validated"]	"2022-07-09T21:01:27Z")
sending notification WARN-603
[WARN-603] Let's Encrypt certificate renewal failed accessing Let's Encrypt service
execution completed (CSRs renewed: 0, failed: 1)

Uniquenospacesshort · July 9, 2022, 9:11pm

FYI - - https://email.blueboxit.ca ......... this is another server, its a server on my hosting platform in the cloud. I turned this server off, as right now its not being used. So it's down, down. email.blueboxit.ca is another server with another certificate.

But - the autodiscover, utm and sip etc are all in another cluster. So hoping that we can disregard any email.blueboxit.ca entries for now, as its not in the same cluster.

rg305 · July 9, 2022, 9:33pm

Again, I'm not a Sophos expect, but...
Since they all resolve to the same IP, I must assume that the firewall is terminating HTTP and HTTPS - and then doing some sort of reverse proxy to the multiple internal systems.

If that is the case, and the certs are issued to the firewall, then we must focus our search there.

It seems to be running dehydrated - I'll update the topic title.

Uniquenospacesshort · July 9, 2022, 9:44pm

I managed to get it resolved, I started a new cert request off, this time without email.blueboxit.ca - it handled it without issue, certificate issued (if it will renew, I do not know). But I'm all good now.... one thing to note (which may or may not help:

The interface, I noticed the interface on the troublesome cert-renewal was "INTERNAL"... one would assume this is incorrect.

I added "EXTERNAL" Interface to the new cert, and it was sucessful.

Thanks again for your support - not sure if this helps, but case resolved.

system · August 8, 2022, 9:44pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.