LE API 'Connection reset by peer'

MikeMcQ · October 29, 2021, 7:04pm

@techpad Sounds good on gathering more info. Not sure if this is helpful but more poking I see these items:

crt.sh shows you got a cert 2 days ago.

Yet, the https response for that server returns a self-signed cert:

curl -I https://wkweiea9ixstat.mgddns.io
curl: (60) SSL certificate problem: self signed certificate

Also see sslshopper.com/sslchecker.html

Trying -k to bypass verification gives a failure that it should not:

curl -Ik https://wkweiea9ixstat.mgddns.io
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

This is starting to "feel" like a comms issue at your end but debugging that is getting beyond my area of expertise.

JamesLE · October 29, 2021, 7:14pm

We're not blocking this IP address.

rg305 · October 29, 2021, 7:27pm

Yeah, this is far from a standard config.
[OR the MiTM has been exposed! - LOL]

techpad · October 29, 2021, 8:58pm

I've actually left this as is to troubleshoot. This is setting up an endpoint to connect to a Monit stats page protected by NGNIX http password. The entire script bailed out at this error so it's an incomplete configuration at this point.

techpad · October 30, 2021, 12:38am

This is the command that was run from the script.

certbot-auto certonly --webroot \
  -w /var/www/wkweiea9ixstat.mgddns.io/htdocs \
  --cert-name "wkweiea9ixstat.mgddns.io" \
  -d "wkweiea9ixstat.mgddns.io" \
  --email hostservices@mgdwp.io --no-eff-email --agree-tos \
  --staple-ocsp --must-staple \
  --preferred-chain "ISRG Root X1" \
  2>&1

I just ran it and it worked without any issue.

techpad · October 30, 2021, 1:23am

@rg305 & @MikeMcQ

I went back to the let's encrypt logs and i pasted all the rotated logs in order and I think I'm seeing a trend. I would like to post the whole thing, but it's spitting out everything including the certs. Besides having to rerun the cert or even just rebuilding the server. Is posting this whole script a no no on this forum or is it OK.

I suspect the script that is building this might be hammering the API, or maybe the server is running the script quickly and it's hammering the API. In any event I'd love a second set of eyes on this.

MikeMcQ · October 30, 2021, 2:16am

@techpad Before posting huge logs, there looks to be a problem with the command you used. You show certbot-auto but that was deprecated in favor of just certbot. This was primarily due to it needing python2.

Yet, your initial log showed a snap install of certbot with python3. This is the "modern" install. And, you noted certbot version 1.20 which is recent. I do not understand the migration of that older install method well but Rudy and others will be better able to help sort this out. Let's see what they have to say.

Here is a post about that from Dec 2020:

And, the current install page that mentions it too:
https://certbot.eff.org/docs/install.html#certbot-auto

rg305 · October 30, 2021, 3:37am

I think @MikeMcQ is on to something.
Please check the your system doesn't have multiple versions of certbot(-auto).
which certbot
which certbot-auto

techpad · October 30, 2021, 3:47am

@rg305 @MikeMcQ

Thank you this was the clue I needed. I found the problem. The author of the script is doing this for some unknown reason which I'm about to ask.

root@mgdwp_denver-carpet-32:~# grep -rnw '/var/log' -e 'certbot-auto'
/var/log/auth.log.1:1726:Oct 19 04:26:30 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log.1:3567:Oct 20 04:26:27 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log.1:5275:Oct 21 04:26:47 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log.1:6868:Oct 22 04:26:25 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log.1:8472:Oct 23 04:26:25 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log.1:10389:Oct 24 04:26:32 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log:1879:Oct 25 04:26:30 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log:3587:Oct 26 04:27:20 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log:5317:Oct 27 04:26:26 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log:7184:Oct 28 04:26:22 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto
/var/log/auth.log:9039:Oct 29 04:26:58 localhost sudo:     root : TTY=unknown ; PWD=/root ; USER=root ; COMMAND=/usr/bin/snap alias certbot certbot-auto

rg305 · October 30, 2021, 3:51am

Looks like a retrofit "catch-all" attempt.
[which actually missed two: "letsencrypt" & "letsencrypt-auto"]

Still. Are there any of those other programs laying around?

techpad · October 30, 2021, 4:02am

yes, it's living in /snap/bin/certbot-auto

techpad · October 30, 2021, 4:03am

wait, it is, but its a symlink to /snap/bin/certbot which is a symlink to usr/bin/snap

rg305 · October 30, 2021, 4:04am

Ok. What about?:
which certbot
which certbot-auto

techpad · October 30, 2021, 4:04am

root@wpms-shost1:~# which certbot
/snap/bin/certbot
root@wpms-shost1:~# which certbot-auto
/snap/bin/certbot-auto
root@wpms-shost1:~#

techpad · October 30, 2021, 4:09am

Is it strange for me to wonder... why not find replace certbot-auto -> certbot. Is there some purpose for backwards compatibility w/ 16.04... 18.04 that I'm not clued into or does this just really look like a short shortcut?

rg305 · October 30, 2021, 4:12am

I'm not familiar with /snap/bin/certbot-auto
That must have been manually added by a previous admin of that system.
And, yes, it seems like a "short shortcut".

Anywho, that doesn't seem to be part of the problem...

techpad · October 30, 2021, 4:18am

So this leads me to my next theory... and kind of wanting to post the whole story.... Which I guess I could edit out the uninteresting parts for sanity.

The script seems to use the same block for initializing new domains as it would a renewal, so it's hitting the API over again, in my opinion somewhat unnecessarily. I'm going to search my 18.04 boxes for similar errors to see if it's really a 20.04 issue. Right now my working theory is the guys who wrote this are chalking it up to 20.04 is new yadadada we're not supporting it. Which is fine, but still why would the endpoint kill the session. Unless, 20.04 has the new kernal with a really efficient network stack. The majority of users are in commercial LAAS providers with lots of limits on IO, lots of CPU steal so these scripts are executing at a limited clip.

Now I have zero restraints, no IO limits, no CPU steal, its the next best thing to bare metal. My theory here is that it's running quick enough that these unnecessary loops are hitting the endpoint too fast and it's closing the session out, either because it isn't fast enough to respond or it's rate limiting my IP.

I'd have to prove this theory somehow...

rg305 · October 30, 2021, 4:20am

If you can alter the coding...
Throw in a small delay (just a few seconds) here and there.
See if that normalizes things.

techpad · October 30, 2021, 4:22am

Yah, that is something I can do. I'll try that and also spin up an unedited version and run the processes together and see what happens.

I'll report back when I have something new. Thanks' for the time and help. Much appreciated!

system · November 29, 2021, 4:23am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.