ACME failure on Expressway-E 15

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: ncagr.gov

I ran this command: Not sure as it is a button on the Expressway-E virtual appliance

It produced this output: sign operation failed : The server could not connect to validation target. Connection reset by peer: agrexedge.ncagr.gov

My web server is (include version): X15.2.4

The operating system my web server runs on is (include version): TANDBERG Video Communication Server

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): Yes, kind of.

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): I am using a GUI interface on the Expressway-E appliance

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Not sure.

This is your firewall blocking TCP port 80 (incoming HTTP). If you are not part of your IT dept I would suggest you talk to them as there are security issues to consider here.

2 Likes

I asked them about that and they say port 80 is open. I pulled a PCAP from the Expressway E and I can see two tries.

Jason A. Reed
Systems Administrator I
2 West Edenton St.
Raleigh, NC 27601
(919) 707-3094

E-mail correspondence to and from this address may be subject to the North Carolina Public Records Law and may be disclosed to third parties.

(attachments)

The Cisco docs have more info on the process: Cisco Expressway Certificate Creation and Use Deployment Guide (X15.0) - Using ACME on Expressway-E [Cisco Expressway Series] - Cisco

3 Likes

Do you mean, that you see two attempts from outside your network to access the /.well-known/acme-challenge path on your server? You should be seeing five. Is your firewall maybe blocking some but not all incoming connections?

Though one test I tried couldn't connect from anywhere.

4 Likes

The "RST, ACK" in the PCAP is a "Reset" which we see reported by HTTP requests to your domain. You are actively rejecting requests.

IssueFromLetsEncrypt
Error
A test authorization for agrexedge.ncagr.gov to the Let's Encrypt staging service has revealed issues that may prevent any certificate for this domain being issued.
207.4.160.37: Fetching http://agrexedge.ncagr.gov/.well-known/acme-challenge/r2D1Ms-Spe3s0EPwnQ-O531OVSUYMsmZpPQPw9_B4dk: Connection reset by peer

From: Let's Debug

5 Likes

I have no idea why we are rejecting these requests. Anyone have any ideas on what I can change on the Expressway-E?

I'm still working towards getting this certificate signing working. I'm getting a different error after our firewall team made some adjustments. Now I'm getting the following in Expressway Edge.

ACME sign operation failed : The server could not connect to validation target. Timeout during connect: agrexedge.ncagr.gov

Here's a snapshot of the PCAP filtered to port 80

It still looks to only be accessible from a few places.

I only see three attempts in that screenshot, though maybe others are cut off. It'd be possible to be more helpful if you could get whatever logs on that system might show the actual error message that the ACME server is replying with.

3 Likes

Here's the error.

The Certificate Authority reported these problems: Domain: agrexedge.ncagr.gov Type: connection Detail: During secondary validation: 207.4.160.37: Fetching http://agrexedge.ncagr.gov/.well-known/acme-challenge/RK9LRdG3EJkySkM5JmT-WJIiWH4jpec9mEoWXlduruk: Timeout during connect (likely firewall problem) Domain: collab-edge.ncagr.gov Type: connection Detail: During secondary validation: 207.4.160.37: Fetching http://collab-edge.ncagr.gov/.well-known/acme-challenge/kOoN-zyw7d4A56ghDUSotcs_b_uNXLqNb_yxfLJ6Rko: Timeout during connect (likely firewall problem) Hint: The Certificate Authority failed to verify the challenge files created by the --manual-auth-hook. Ensure that this hook is functioning correctly. Refer to "certbot --help manual" and the Certbot User Guide.

Thank you!

This means, as I suspected, that some locations (the "primary" validation) could connect and confirm your control over the name, but that at least some other locations (the "secondary" validations) could not.

And from those locations, the symptom is a timeout: Meaning that the validation server attempts to connect to your server, but just doesn't get a response in time. Usually, this means that a firewall has blocked it.

You (and/or your network/firewall administration team) may find this FAQ useful, about how Let's Encrypt needs to check from multiple places in order to validate that you actually control a domain name:

5 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.