It can be tricky to diagnose this, but temporarily setting MTU to 1100 on the VM and seeing whether that makes a difference, can be helpful to confirm.
3 Likes
Hi _az
Thanks a lot for the suggestion. Here is the result - seems unchanged. I don't think MTU is the root cause.
[root@mail joe.dutka]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether 52:54:00:d4:07:1a brd ff:ff:ff:ff:ff:ff
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1100 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:5d:b8:bf brd ff:ff:ff:ff:ff:ff
inet 60.250.195.22/24 brd 60.250.195.255 scope global noprefixroute enp7s0
valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:13:19:ea brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
5: cni-podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 82:2a:e6:3a:a1:f7 brd ff:ff:ff:ff:ff:ff
inet 10.88.0.1/16 brd 10.88.255.255 scope global cni-podman0
valid_lft forever preferred_lft forever
inet6 fe80::802a:e6ff:fe3a:a1f7/64 scope link
valid_lft forever preferred_lft forever
7: vethf743bd74@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni-podman0 state UP group default
link/ether b2:55:30:89:88:f4 brd ff:ff:ff:ff:ff:ff link-netns netns-c2fcce68-046a-5763-9279-e2181061c8af
inet6 fe80::b055:30ff:fe89:88f4/64 scope link
valid_lft forever preferred_lft forever
[root@mail joe.dutka]# certbot renew -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/acer-isu.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for acer-isu.com and 2 more domains
Failed to renew certificate acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mail.acer-isu.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer None
Renewing an existing certificate for mail.acer-isu.com
Failed to renew certificate mail.acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/acer-isu.com/fullchain.pem (failure)
/etc/letsencrypt/live/mail.acer-isu.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
Is there a way to pass a longer timeout to the python in certbot? I am wondering if there is a timeout issue for some reason.
1 Like
This doesn't look like a timeout to me. You're getting a connection reset in what looks like 150ms, which is very fast.
Can you try:
head -c 35000 /dev/urandom | base64 | curl -d @- -i -X POST -m 10 -H 'Expect:' https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735505864
Do you get a connection reset for this as well? Or just an HTTP 415?
3 Likes
Hi _az,
Here is what I receive on the box.
[joe.dutka@mail ~]$ head -c 35000 /dev/urandom | base64 | curl -d @- -i -X POST -m 10 -H 'Expect:' https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735505864
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
Thank you.
1 Like
I have the same output when I use this command:
head -c 35000 /dev/urandom | base64 | curl -d @- -i -X POST -m 10 -H 'Expect:' https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735505864
curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104
Rocky Linux 8.6
What version of Python 3 do you have? Mine is 3.6.8.
I know Rocky is pretty close to the CentOS Stream I have, I wonder if we have some strange bug, but I really doubt it.
Now installed snapd.
certbot 1.29.0 2192 latest/stable certbot-eff✓ classic
Until that time it was. There was a mistake.
python36 3.6.8-38 и certbot 1.22
Try reduce that 35000 number until the error goes away. Let us know what the number is where it doesn't error.
2 Likes
reduced in the flesh to 500, the error is random.
Full
nothing has changed on the system. It worked great two months ago.
If it's truly random and the payload size has nothing to do with it, I think we'd probably need to ask Let's Encrypt or Cloudflare to look into it.
I am mindful though that you and @JoeAcerIsu may not experiencing the exact same issue, though they may look similar.
2 Likes
According to the archived threads of this forum. I made a conclusion after studying them briefly. That maybe my ip address got blocked on the Let's Encrypt side.
I can’t say, but perhaps a verification of such a probability is required.
I have a consistent error - unfortunately. I can pass with a value of 770 - that was the highest I could get it. 779 still returns a value but has "Invalid number of bytes" warning. 780 or above had a connection reset.
HTTP/2 415
server: nginx
date: Mon, 22 Aug 2022 10:36:06 GMT
content-type: application/problem+json
content-length: 168
cache-control: public, max-age=0, no-cache
link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: 0001Q_KtIKvL6M4aSuXd1hE0XEO6fkIV41uEEU7uREam5H8
{
"type": "urn:ietf:params:acme:error:malformed",
"detail": "Invalid Content-Type header on POST. Content-Type must be \"application/jose+json\"",
"status": 415
}
So you consistently get the HTTP 415 error at 770 (which is ~1KB), but higher gives you the connection reset? To me that still sounds like an MTU issue! You might try lowering MTU even further, maybe to 900, and see whether that makes any difference when you use Certbot.
@ebkjiud I think your problem might have a separate cause to @JoeAcerIsu's, it's probably better to keep the discussion in the thread you opened.
2 Likes
I get an issue with a count beyond 770 - 780 and up has an issue. I just tried to lower my MTU to 750 and the original issue persists.
Also I even tried to revoke one of my certificates - mail.acer-isu.com as it is actually a duplicate. That has the same problem.
Thanks again for your attention to this.
[root@mail ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp1s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
link/ether 52:54:00:d4:07:1a brd ff:ff:ff:ff:ff:ff
3: enp7s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 750 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:5d:b8:bf brd ff:ff:ff:ff:ff:ff
inet 60.250.195.22/24 brd 60.250.195.255 scope global noprefixroute enp7s0
valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:13:19:ea brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
5: cni-podman0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 12:c9:98:62:02:42 brd ff:ff:ff:ff:ff:ff
inet 10.88.0.1/16 brd 10.88.255.255 scope global cni-podman0
valid_lft forever preferred_lft forever
inet6 fe80::10c9:98ff:fe62:242/64 scope link
valid_lft forever preferred_lft forever
7: veth69bb8a50@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master cni-podman0 state UP group default
link/ether de:1a:b4:fb:2f:be brd ff:ff:ff:ff:ff:ff link-netns netns-480660fd-c783-85f5-077c-45d613107892
inet6 fe80::dc1a:b4ff:fefb:2fbe/64 scope link
valid_lft forever preferred_lft forever
[root@mail ~]# certbot renew -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/acer-isu.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate for acer-isu.com and 2 more domains
Failed to renew certificate acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mail.acer-isu.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer None
Renewing an existing certificate for mail.acer-isu.com
Failed to renew certificate mail.acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/acer-isu.com/fullchain.pem (failure)
/etc/letsencrypt/live/mail.acer-isu.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
1 Like
I have a floating problem and breaks in the work of cerbot in a random place, each time a different filling of the error. Anyone who can help please post.
Does the curl fail on the physical host with reduced MTU as well, not only the VM?
3 Likes
Hi _az
The host seems okay honestly. It's weird because there is no firewall on the host system - so it can't be limiting or throttling the ports to the VM. head -c 35000 .... worked. MTU is 1500 on the host as well.
I tried to adjust a network setting on the VM, but that didn't seem to help the certbot renew command.
Thanks.
2 Likes
If the curl works on the host system but not the guest VM, the problem sounds environmental but I don't have much of a clue about the cause.
You might have some luck by simultaneously recording packet captures on both the VM and the host, then seeing whether the TCP RST packet is coming from the internet or from the host system.
4 Likes
Hi _az,
Thanks so much for all the help. I have it working so we can close this bug.
Resolution: The KVM hypervisor set the VM to bridge using e1000e device type. I guess that was causing an issue, because I adjusted the NIC device type to virtio and the connection succeeded.
Found the following certs:
Certificate Name: acer-isu.com
Serial Number:
Key Type: RSA
Domains: acer-isu.com mail.acer-isu.com www.acer-isu.com
Expiry Date: 2022-11-22 02:27:29+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/acer-isu.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/acer-isu.com/privkey.pem
Certificate Name: mail.acer-isu.com
Serial Number:
Key Type: RSA
Domains: mail.acer-isu.com
Expiry Date: 2022-11-22 02:27:34+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/mail.acer-isu.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/mail.acer-isu.com/privkey.pem
Thanks so much for all the teams patience and help on this! Really impressed with the Community's response!
4 Likes