Acme failure on NetGate pfSense

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: santafe.weeksrobinson.com

I ran this command: issue

It produced this output:

> 
> santafe.weeksrobinson.com
> Renewing certificate 
> account: Weeks Santa Fe 
> server: letsencrypt-production-2 
> 
> /usr/local/pkg/acme/acme.sh  --issue  --domain 'santafe.weeksrobinson.com' --dns 'dns_cf'  --home '/tmp/acme/santafe.weeksrobinson.com/' --accountconf '/tmp/acme/santafe.weeksrobinson.com/accountconf.conf' --force --reloadCmd '/tmp/acme/santafe.weeksrobinson.com/reloadcmd.sh' --log-level 3 --log '/tmp/acme/santafe.weeksrobinson.com/acme_issuecert.log'
> Array
> (
>     [path] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
>     [PATH] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
>     [CF_Key] => [ ]
>     [CF_Email] => admin@weeksrobinson.com
>     [CF_Token] => 
>     [CF_Account_ID] => 
>     [CF_Zone_ID] => 
> )
> [Thu Oct 27 08:57:53 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:58:04 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:58:14 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:58:24 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:58:35 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:58:45 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:58:55 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:59:05 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:59:16 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:59:26 MDT 2022] Sleep 10 and retry.
> [Thu Oct 27 08:59:36 MDT 2022] Can not init api, for https://acme-v02.api.letsencrypt.org/directory

My web server is (include version):

The operating system my web server runs on is (include version):
pfSense+ 22.05
Acme 0.7.3

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
n/a

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Acme 0.7.3

I have tried restarting the machine w/ no change, and everything else is working correctly. This is not the weird encryption extension failure thing as I can update and re-install packages as needed.

I am wondering if the IP for this device has been blocked do to excessive failed retries. Current IP is: 98.60.135.79

Many thanks.

2

Hello @jvedman, welcome to the Let's Encrypt community. :slightly_smiling_face:

Looks like a connectivity issue.

Seems to indicate you cannot connect out from your network.

Here Check report was removed: Check host - online website monitoring shows unable to connect to HTTP
and here Check report was removed: Check host - online website monitoring shows unable to connect to HTTPS

Do you have a firewall blocking Ports 80 and 443?

And https://letsdebug.net/ HTTP-01
Let's Debug

image
image1077×729 13.1 KB

2 Likes
3

Bruce,
Thank you for the response.
I know I have outbound connectivity because a) I have clients behind that firewall that aren't screaming and b) I am accessing it remotely.
Ports 80 and 443 are blocked inbound from outside with the exception of a handful of trusted IPs that we access the device from. Since we are masking the u/i login to the device I cannot open those ports to the world. I can, however, turn on ping on that address to prove outbound connectivity.

1 Like
4

@jvedman, and you are pingable from around the world Ping server, ping website: Check host - online website monitoring

Let’s Encrypt offers Domain Validation (DV) certificates.

Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

Thus you need to own and have control over the Domain Name (or have a subdomain under an existing domain name, for example pointed to your server by your employer or school) you wish to obtain a certificate for, from an ICANN Accredited Registrar.

For Let’s Encrypt to issue a Domain Validation (DV) certificate Domain Validation must happen
and it is a CA/Browser Forum Baseline Requirement.

Best Practice - Keep Port 80 Open

What IP addresses does Let’s Encrypt use to validate my web server?
Let’s Encrypt does not publish a list of IP addresses we use to validate,
and these IP addresses may change at any time.

Let's Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt

Since these are Domain Validation (DV) certificates the Domain Name System (DNS) is used extensively in the validation process as well a allowing us to assist here on Let's Encrypt community.
DNS Queries need to give consistent results from any location on the Internet, all your authoritative DNS Servers for the Domain need to also give consistent results as well.

Testing and debugging are best done using the Staging Environment as the Rate Limits are much higher. Rate Limits are per week (rolling).

And to assist with debugging there is a great place to start is Let's Debug.

1 Like
5

Please try:

curl -LIv https://acme-v02.api.letsencrypt.org/directory

To check connectivity to the ACME API.

You should allow access to the path /.well-known/acme-challenge/ regarding the remote IP address. Let's Encrypt validates from multiple points across the globe and there is not a list of IP addresses used. If that's not possible, you could use the dns-01 challenge.

2 Likes
6

I think they are using DNS challenge. Right?

3 Likes
7

And https://letsdebug.net/ DNS-01 passes here: Let's Debug

2 Likes
8

@jvedman

The LE ACME endpoint supports both IPv4 and IPv6. Does your log show what IP it was trying to connect to? You could also try these to check both

curl -4v https://acme-v02.api.letsencrypt.org/directory
curl -6v https://acme-v02.api.letsencrypt.org/directory

Do either succeed?

3 Likes
9

Whoops, true true.

4 Likes
10

Oh, I missed it also.

3 Likes
11

Yes, using the Cloudflare DNS challenge with all of the requisite information. I should also note that this system has been in place about 2 years and has been working fine until the last several weeks.

12

So, this is weird. Curl is installed. Issuing which curl gives me /usr/local/bin/curl, but running any of your suggested curl requests just drops back to the command line with no output in either the diagnostics -> shell command or from the console. I know this isn't right as I can run the command from another pfsense device and get a full response.

1 Like
13

After forcing a reinstall of curl, I get this:

[**22.05-RELEASE**][**admin****@****Weeks-SF-Sense.localdomain**]/root**:** curl -4v https://acme-v02.api.letsencrypt.org/directory

* Trying 172.65.32.248:443...

* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)

* ALPN: offers h2

* ALPN: offers http/1.1

* CAfile: /usr/local/share/certs/ca-root-nss.crt

* CApath: none

* TLSv1.3 (OUT), TLS handshake, Client hello (1):

* TLSv1.3 (IN), TLS handshake, Server hello (2):

* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):

* TLSv1.3 (IN), TLS handshake, Certificate (11):

* TLSv1.3 (IN), TLS handshake, CERT verify (15):

* TLSv1.3 (IN), TLS handshake, Finished (20):

* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):

* TLSv1.3 (OUT), TLS handshake, Finished (20):

* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384

* ALPN: server accepted h2

* Server certificate:

* subject: CN=acme-v02.api.letsencrypt.org

* start date: Sep 8 19:36:39 2022 GMT

* expire date: Dec 7 19:36:38 2022 GMT

* subjectAltName: host "acme-v02.api.letsencrypt.org" matched cert's "acme-v02.api.letsencrypt.org"

* issuer: C=US; O=Let's Encrypt; CN=R3

* SSL certificate verify ok.

* Using HTTP2, server supports multiplexing

* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0

* h2h3 [:method: GET]

* h2h3 [:path: /directory]

* h2h3 [:scheme: https]

* h2h3 [:authority: acme-v02.api.letsencrypt.org]

* h2h3 [user-agent: curl/7.83.1]

* h2h3 [accept: */*]

* Using Stream ID: 1 (easy handle 0x41a54800)

> GET /directory HTTP/2

> Host: acme-v02.api.letsencrypt.org

> user-agent: curl/7.83.1

> accept: */*

>

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):

* old SSL session ID is stale, removing

* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!

< HTTP/2 200

< server: nginx

< date: Thu, 27 Oct 2022 18:16:46 GMT

< content-type: application/json

< content-length: 659

< cache-control: public, max-age=0, no-cache

< x-frame-options: DENY

< strict-transport-security: max-age=604800

<

{

"LUcCJ2ttgME": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",

"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",

"meta": {

"caaIdentities": [

"letsencrypt.org"

],

"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",

"website": "https://letsencrypt.org"

},

"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",

"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",

"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",

"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"

* Connection #0 to host acme-v02.api.letsencrypt.org left intact

Going to attempt another issue / renew just to see if this was the problem.

14

Bingo! This was it. Well, not it, but pointed to it. Apparently curl got borked on the system and forcing a re-install fixed curl which fixed the problem.

Thanks to everyone for helping me get to the core issue.

4 Likes
15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.