Renewal requests rejected

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: passwords.sharonblain.com

I ran this command: certbot renew

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/passwords.sharonblain.com.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator webroot, Installer None
Attempting to renew cert (passwords.sharonblain.com) from /etc/letsencrypt/renewal/passwords.sharonblain.com.conf produced an unexpected error: Requesting acme-v02.api.letsencrypt.org/directory: Network is unreachable. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/passwords.sharonblain.com/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/passwords.sharonblain.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

My web server is (include version): Apache

The operating system my web server runs on is (include version): Debian v9

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 0.31.0

I am trying to renew a certificate. I can resolve the domain name and I try to connect to it but I get a TCP reset back killing the connection. I have reduced the MTU on my NIC to 1200 as I heard that can help but it didn’t. As you can see in the FW logs below, my connect reaches the server but it rejected with a TCP reset in bold.

1: 20:01:26.321638 802.1Q vlan#97 P0 192.168.xx.xx.55302 > 192.168.xx.xx.53: udp 39
2: 20:01:26.322737 802.1Q vlan#97 P0 192.168.xx.xx.53 > 192.168.xx.xx.55302: udp 103
~~
10: 20:01:30.542193 802.1Q vlan#97 P0 192.168.xx.xx.37192 > 172.65.32.248.443: S 2877232600:2877232600(0) win 23200 <mss 1160,sackOK,timestamp 47460041 0,nop,wscale 7>
11: 20:01:30.542238 802.1Q vlan#97 P0 172.65.32.248.443 > 192.168.xx.xx.37192: R 0:0(0) ack 2877232601 win 23200

Any advice would be appreciated.

Thanks

2

Does your machine have outbound connectivity?

I don't get the reset, I timeout, as if iptables drops me. Is 180.92.193.62 the correct ip address?

Also:

% nmap passwords.sharonblain.com -Pn
Starting Nmap 7.80 ( https://nmap.org ) at 2020-03-19 10:22 CET
Nmap scan report for passwords.sharonblain.com (180.92.193.62)
Host is up.
All 1000 scanned ports on passwords.sharonblain.com (180.92.193.62) are filtered

(ignore "host is up", it's meaningless when using -Pn)

3

Thanks for the response. As this is not a publicly accessible server it is not normally accessible from the internet but I do make it temporarily available to renew the cert.

That being said, I had thought the TCP reset in my original post was coming from the letencrypt server but it was coming from our firewall. I have resolved that now and can see healthy communications between the our server and letsencrypt as shown below but the renewal is still failing.

4: 00:51:23.875870 180.92.193.62.54972 > 172.65.32.248.443: S 76436964:76436964(0) win 23200 <mss 1160,sackOK,timestamp 1709954653 0,nop,wscale 7>
5: 00:51:23.876312 172.65.32.248.443 > 180.92.193.62.54972: S 1272044911:1272044911(0) ack 76436965 win 65535 <mss 1460,nop,nop,sackOK,nop,wscale 10>
6: 00:51:23.876755 180.92.193.62.54972 > 172.65.32.248.443: . ack 1272044912 win 182
7: 00:51:23.887206 180.92.193.62.54972 > 172.65.32.248.443: P 76436965:76437321(356) ack 1272044912 win 182
8: 00:51:23.887557 172.65.32.248.443 > 180.92.193.62.54972: . ack 76437321 win 66
9: 00:51:24.258592 172.65.32.248.443 > 180.92.193.62.54972: . 1272044912:1272046072(1160) ack 76437321 win 66
10: 00:51:24.258607 172.65.32.248.443 > 180.92.193.62.54972: P 1272046072:1272046372(300) ack 76437321 win 66
11: 00:51:24.258623 172.65.32.248.443 > 180.92.193.62.54972: . 1272046372:1272047532(1160) ack 76437321 win 66
12: 00:51:24.258638 172.65.32.248.443 > 180.92.193.62.54972: P 1272047532:1272047832(300) ack 76437321 win 66
13: 00:51:24.258638 172.65.32.248.443 > 180.92.193.62.54972: P 1272047832:1272048299(467) ack 76437321 win 66
14: 00:51:24.259340 180.92.193.62.54972 > 172.65.32.248.443: . ack 1272046372 win 205
15: 00:51:24.259340 180.92.193.62.54972 > 172.65.32.248.443: . ack 1272047832 win 227
16: 00:51:24.259355 180.92.193.62.54972 > 172.65.32.248.443: . ack 1272048299 win 245

I am getting the error “Failed Authorisation Procedure, the server could not connect to the client to verify the domain. Timeout during connection”.

When I did the capture on the firewall shown above, I could see outbound connections from my server to letsencrypt working fine. When renewing in the past I would then see a new connection made back to my server to verify the domain but I never saw this on the firewall when it failed most recently.

While I have now disabled internet connectivity for my server until I try to renew again, I can confirm I have been able to reach this server from the internet when I open it up on my firewall. This is shown below:

1: 01:04:03.603835 203.206.xx.xx.50686 > 180.92.193.62.443: S 2746238189:2746238189(0) win 64240 <mss 1380,sackOK,timestamp 1174803913 0,nop,wscale 7>
2: 01:04:03.604430 203.206.xx.xx.443 > 203.206.234.44.50686: S 800322711:800322711(0) ack 2746238190 win 22960 <mss 1160,sackOK,timestamp 2827480731 1174803913,nop,wscale 7>
3: 01:04:03.616560 203.206.xx.xx.50686 > 180.92.193.62.443: . ack 800322712 win 502 <nop,nop,timestamp 1174803926 2827480731>
4: 01:04:03.619337 203.206.xx.xx.50686 > 180.92.193.62.443: P 2746238190:2746238707(517) ack 800322712 win 502 <nop,nop,timestamp 1174803929

So I am not sure why this is still failing. It seems like the connection back to my server to verify the domain is not occuring.

Thanks

4

if you don't want your machine to be publicly reachable during validation by ANY ip (see "multi-perspective validation") you should switch to dns-01 validation.

the connection is happening, multiple times, from multiple IPs.

5

Just to confirm the expected flow, this is how I understand it:

  • My server initiates a connection to https://acme-v02.api.letsencrypt.org/directory which can be seen above in packets between by server and 172.65.32.248.
  • A second connection(s), the one you are referring to in your last message, is initiated from LetsEncrypt back to my server address. This is the validation step.

That is how I understand it but I am just not seeing any new connections hit my firewall to 180.92.193.62:443 after the connection from 180.92.193.62 to 172.65.32.248:443 is initiated. I assume I am right in thinking the second connection will target TCP443 on my server, I can’t imagine what other port it would use.

I only had DNS set up 6-7 hours ago, that should be plenty of time but maybe it hasn’t propagated yet though the failure didn’t mention anything about DNS not resolving, it was about a connection timeout. I assume that connection it is referring to is the second, validation connection, but I just don’t see it hitting my firewall.

Thanks

6

Then initiates several others to different endpoints, to do what ACME does.

When the client tells LE that a challenge is ready,

But it doesn't happen like this. It's initiated from arbitrary IP addresses, always more than one. And it connects on port 80. (for http-01)

7

I think I see the issue. it looks as though the connection back for validation is using HTTP not HTTPS. I was looking for inbound connections on TCP443

8

Thanks, one or many addresses, it doesn’t really matter, the firewall would let them all through while I do this work. The issue is I expected the connections to be on TCP443, so I just need to allow TCP80 and it should be fine, that is once my rate-limit hour has expired.

Thanks for your assistance.

9

you know about --pre-hook and --post-hook already, do you?

10

Nope, no idea what they are.

11

certbot options that specify commands to be run before and after obtaining certificates: https://certbot.eff.org/docs/using.html

  --pre-hook PRE_HOOK   Command to be run in a shell before obtaining any
                        certificates. Intended primarily for renewal, where it
                        can be used to temporarily shut down a webserver that
                        might conflict with the standalone plugin. This will
                        only be called if a certificate is actually to be
                        obtained/renewed. When renewing several certificates
                        that have identical pre-hooks, only the first will be
                        executed. (default: None)
  --post-hook POST_HOOK
                        Command to be run in a shell after attempting to
                        obtain/renew certificates. Can be used to deploy
                        renewed certificates, or to restart any servers that
                        were stopped by --pre-hook. This is only run if an
                        attempt was made to obtain/renew a certificate. If
                        multiple renewed certificates have identical post-
                        hooks, only one will be run. (default: None)