Cannot connect to acme-v2.api.letsencrypt.org from web server

Help
1

Yesterday my organization renewed our certificates using wacs.exe, through which we were able to connect to acme-v2.api.letsencrypt.org without issue. Today, that is not the case. We have been through every similar post I could find but they were either closed without resolution or did not resolve our issue. Please shed any light you can on this, we are currently at a loss as to what is going on.

My domain is: (www.)itmmarketing.com (we have multiple subdomains, all with valid certificates)

I ran this command: ping, tracert, openssl s_client -connect, wacs.exe, opened in browser

It produced this output:

1 * * * Request timed out.
etc.

  • openssl s_client -connect:

12160:error:0200274C:system library:connect:reason(1868):crypto\bio\b_sock2.c:110:
12160:error:2008A067:BIO routines:BIO_connect:connect error:crypto\bio\b_sock2.c:111:
connect:errno=0

  • wacs.exe: MicrosoftTeams-image (4)
  • opened in browser: ERR_CONNECTION_TIMED_OUT

No such errors occur when using acme-staging-v02.api.letsencrypt.org.

My web server is (include version): IIS Version 10.0.14393.0

The operating system my web server runs on is (include version):Microsoft Windows Server 2016 Version 1607

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Windows ACMEv2 client version 2.1.18.1119

2

That's interesting, it seems like your packets are getting dropped somewhere along the way. Are you able to successfully ping other hosts from the same machine? Did anything change about your network topology, firewall, or routing recently? Do you have a firewall that restricts egress?

3

Yes we can ping other hosts from the same machine, including acme-staging-v02.letsencrypt.org. Nothing has changed since yesterday when we were able to successfully connect. We do not restrict egress, all outbound traffic is allowed.

5

I'm investigating this now.

Edit: All staging/prod loadbalancers are serving the following chain, the example below is from prod as shown by the CN.

Certificate chain
 0 s:CN = acme-v02.api.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
6

Is it possible that we are being blocked for some reason, similar to this post? I can't imagine why, but we are at a total loss as to what else the problem could be at this point.

7

PM me IP addresses and I will see if there's anything in the Cloudflare block list.

Edit: The IP address was not found in the block list.

8

@bb_dev welcome to the LE community forum :slight_smile:

Have you ruled out local firewall and antivirus?

9

I believe so. We were able to connect without issue on Thursday and nothing has changed in either. If there is any additional information I can provide to help troubleshoot this, please let me know. We still have no clue what the problem is at this time.

11

What does that mean exactly?

13
17

This is the output I get when I run wacs.exe with --verbose:

[EROR] Initial connection failed, retrying with TLS 1.2 forced
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at PKISharp.WACS.Services.ProxyService.LoggingHttpClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at PKISharp.WACS.Clients.Acme.AcmeClient.CheckNetwork()
[DBUG] Send GET request to https://acme-v02.api.letsencrypt.org/directory
[EROR] Unable to connect to ACME server
System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 10 seconds elapsing.
---> System.TimeoutException: A task was canceled.
---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
at PKISharp.WACS.Services.ProxyService.LoggingHttpClientHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at PKISharp.WACS.Clients.Acme.AcmeClient.CheckNetwork()

19

Yes, the server can successfully tracert other addresses including acme-staging-v02.api.letsencrypt.org

21

We have finally fixed this issue. Why the problem occurred is still unclear, but resetting TCP/IP via the command netsh int ip reset and restarting the server has corrected whatever was stopping us from connecting.