Connection Reset by Peer - Certbot Renew Fails

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: acer-isu.com

I ran this command: certbot renew

It produced this output: Failed to renew certificate acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

My web server is (include version): Apache

The operating system my web server runs on is (include version): Apache/2.4.37 (centos)

My hosting provider, if applicable, is: China Telecom HiNet

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

More details:
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.22.0

I am having an issue with the connection getting reset by peer on my webserver. It had worked grand for more than a year, but recently the team moved the server to a new location and changed the server IP.

When going through the debugging of this issue I found the output of is different than from my client:
curl acme-v02.api.letsencrypt.org
curl: (56) Recv failure: Connection reset by peer

This is the same as the error, but I don't know why this would be blocked. I have ports open and can ping.

Any thoughts would be very helpful.

Thank you.

1 Like
2

Welcome to the community @JoeAcerIsu

First, the above will always fail like that. That endpoint requires https like:

curl https://acme-v02.api.letsencrypt.org/directory

As to your failed cert request, have you gotten that resolved? Because I don't get a connection failure to your site and neither does Let's Debug test site.

If you still have problems please show the contents of this file:

/etc/letsencrypt/renewal/acer-isu.com.conf
3 Likes
3

We should review more of the log file:
/var/log/letsencrypt/letsencrypt.log

2 Likes
4

Thanks, both for the prompt response.

Output of /var/log/letsecrypt/letsencrypt.log:
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 833, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 590, in read
v = self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1441, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 127, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 345, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/usr/lib/python3.6/site-packages/certbot/_internal/client.py", line 440, in obtain_certificate
cert, chain = self.obtain_certificate_from_csr(csr, orderr)
File "/usr/lib/python3.6/site-packages/certbot/_internal/client.py", line 339, in obtain_certificate_from_csr
orderr, deadline, fetch_alternative_chains=self.config.preferred_chain is not None)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 978, in finalize_order
orderr, deadline, fetch_alternative_chains)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 793, in finalize_order
self._post(orderr.body.finalize, wrapped_csr)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 101, in _post
return self.net.post(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1269, in post
return self._post_once(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1282, in _post_once
response = self._send_request('POST', url, data=data, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1174, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

2022-08-19 15:10:37,501:DEBUG:certbot._internal.display.obj:Notifying user:

2022-08-19 15:10:37,501:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2022-08-19 15:10:37,501:ERROR:certbot._internal.renewal: /etc/letsencrypt/live/acer-isu.com/fullchain.pem (failure)
/etc/letsencrypt/live/mail.acer-isu.com/fullchain.pem (failure)
2022-08-19 15:10:37,502:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-08-19 15:10:37,502:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in
load_entry_point('certbot==1.22.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3.6/site-packages/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1632, in main
return config.func(config, plugins)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1518, in renew
renewal.handle_renewal_request(config)
File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 512, in handle_renewal_request
len(renew_failures), len(parse_failures)))
certbot.errors.Error: 2 renew failure(s), 0 parse failure(s)
2022-08-19 15:10:37,503:ERROR:certbot._internal.log:2 renew failure(s), 0 parse failure(s)

Output of /etc/letsencrypt/renewal/acer-isu.com.conf:

renew_before_expiry = 30 days

version = 1.22.0
archive_dir = /etc/letsencrypt/archive/acer-isu.com
cert = /etc/letsencrypt/live/acer-isu.com/cert.pem
privkey = /etc/letsencrypt/live/acer-isu.com/privkey.pem
chain = /etc/letsencrypt/live/acer-isu.com/chain.pem
fullchain = /etc/letsencrypt/live/acer-isu.com/fullchain.pem

Options used in the renewal process

[renewalparams]
account = 70d40c13a35806fad44ce45aa94bee6f
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory

Note that this had worked fine for a year+ just after changing the IP address of the system did I get this failure.

Thanks again.

1 Like
5

Can you show the log before this part? It seems more a problem with outbound comms than inbound to your server.

Also, can you show result of this (using https not http like you did earlier):

curl https://acme-v02.api.letsencrypt.org/directory

And, just the first few lines of output from this (if lots of lines):

curl -I https://letsencrypt.org
3 Likes
6

Hi MikeMcQ,

Thanks again. A little unsure how much of the log you need - here is a bit more, but I'm removing the payload keys (unsure if that would be a security risk or not).

CURL OUTPUT curl -I https://letsencrypt.org:
HTTP/2 200
age: 65833
cache-control: public, max-age=0, must-revalidate
content-security-policy: default-src 'none'; font-src 'self'; style-s

CURL OUTPUT curl https://acme-v02.api.letsencrypt.org/directory:
{
"HEcRSMQyXLE": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}

LOG OUTPUT
2022-08-19 15:10:34,736:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/137572949586 HTTP/1.1" 200 766
2022-08-19 15:10:34,737:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 19 Aug 2022 07:10:34 GMT
Content-Type: application/json
Content-Length: 766
Connection: keep-alive
Boulder-Requester: 117921551
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce:
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"identifier": {
"type": "dns",
"value": "mail.acer-isu.com"
},
"status": "valid",
"expires": "2022-09-01T19:07:26Z",
"challenges": [
{
"type": "http-01",
"status": "valid",
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/137572949586/ToVhlw",
"token": "",
"validationRecord": [
{
"url": "http://mail.acer-isu.com/.well-known/acme-challenge/jUnqH2SqHhyRwwnHV8VH0gRigDVnaHaWE9ci_vJbniQ",
"hostname": "mail.acer-isu.com",
"port": "80",
"addressesResolved": [
"60.250.195.22"
],
"addressUsed": "60.250.195.22"
}
],
"validated": "2022-08-02T19:07:25Z"
}
]
}
2022-08-19 15:10:34,738:DEBUG:acme.client:Storing nonce: 0101hZW_ubPcWZo5u7j5chWaUYse1ZN1LQN0N_qtcxz-HZE
2022-08-19 15:10:34,739:DEBUG:acme.client:JWS payload:
b''
2022-08-19 15:10:34,741:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/137572949596:
{
"protected": ""
"signature": "",
"payload": ""
}
2022-08-19 15:10:34,926:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/137572949596 HTTP/1.1" 200 763
2022-08-19 15:10:34,927:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 19 Aug 2022 07:10:34 GMT
Content-Type: application/json
Content-Length: 763
Connection: keep-alive
Boulder-Requester: 117921551
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: 0102UQrmoDcR8Ttt-wjB4i42uuuQCh8HZKcuKCUKBVrT_WI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"identifier": {
"type": "dns",
"value": "www.acer-isu.com"
},
"status": "valid",
"expires": "2022-09-01T19:07:26Z",
"challenges": [
{
"type": "http-01",
"status": "valid",
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/137572949596/Xm_g1w",
"token": "nABgNMHcfxLi2R2S7NEgdN1gJHNc_QRJ7hFUFj6GL0o",
"validationRecord": [
{
"url": "http://www.acer-isu.com/.well-known/acme-challenge/nABgNMHcfxLi2R2S7NEgdN1gJHNc_QRJ7hFUFj6GL0o",
"hostname": "www.acer-isu.com",
"port": "80",
"addressesResolved": [
"60.250.195.22"
],
"addressUsed": "60.250.195.22"
}
],
"validated": "2022-08-02T19:07:25Z"
}
]
}
2022-08-19 15:10:34,927:DEBUG:acme.client:Storing nonce: 0102UQrmoDcR8Ttt-wjB4i42uuuQCh8HZKcuKCUKBVrT_WI
2022-08-19 15:10:34,928:DEBUG:certbot._internal.client:CSR: CSR(file='/etc/letsencrypt/csr/0055_csr-certbot.pem', data=b'-----BEGIN CERTIFICATE REQUEST-----\n\n-----END CERTIFICATE REQUEST-----\n', form='pem')
2022-08-19 15:10:34,930:DEBUG:certbot._internal.client:Will poll for certificate issuance until 2022-08-19 15:12:04.930062
2022-08-19 15:10:34,932:DEBUG:acme.client:JWS payload:
b'{\n "csr": ""\n}'
2022-08-19 15:10:34,935:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/finalize/117921551/116333749927:
{
"protected": ""
"signature": "",
"payload": ""
}
2022-08-19 15:10:35,070:ERROR:certbot._internal.renewal:Failed to renew certificate acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2022-08-19 15:10:35,084:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 833, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 590, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 833, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 590, in read
v = self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1441, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 127, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 345, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/usr/lib/python3.6/site-packages/certbot/_internal/client.py", line 440, in obtain_certificate
cert, chain = self.obtain_certificate_from_csr(csr, orderr)
File "/usr/lib/python3.6/site-packages/certbot/_internal/client.py", line 339, in obtain_certificate_from_csr
orderr, deadline, fetch_alternative_chains=self.config.preferred_chain is not None)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 978, in finalize_order
orderr, deadline, fetch_alternative_chains)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 793, in finalize_order
self._post(orderr.body.finalize, wrapped_csr)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 101, in _post
return self.net.post(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1269, in post
return self._post_once(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1282, in _post_once
response = self._send_request('POST', url, data=data, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1174, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

2022-08-19 15:10:35,084:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/mail.acer-isu.com.conf
2022-08-19 15:10:35,112:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-08-19 15:10:35,335:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-08-19 15:10:35,337:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/mail.acer-isu.com/cert6.pem is signed by the certificate's issuer.
2022-08-19 15:10:35,338:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/mail.acer-isu.com/cert6.pem is: OCSPCertStatus.GOOD
2022-08-19 15:10:35,340:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2022-09-01 18:07:44 UTC.
2022-08-19 15:10:35,340:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2022-08-19 15:10:35,341:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer None
2022-08-19 15:10:35,488:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.37
2022-08-19 15:10:35,860:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: Installer, Authenticator, Plugin
Entry point: apache = certbot_apache._internal.entrypoint:ENTRYPOINT
Initialized: <certbot_apache._internal.override_centos.CentOSConfigurator object at 0x7f4cac0c7e80>
Prep: True
2022-08-19 15:10:35,861:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_apache._internal.override_centos.CentOSConfigurator object at 0x7f4cac0c7e80> and installer None
2022-08-19 15:10:35,861:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator apache, Installer None
2022-08-19 15:10:35,869:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/117921551', new_authzr_uri=None, terms_of_service=None), 70d40c13a35806fad44ce45aa94bee6f, Meta(creation_dt=datetime.datetime(2021, 4, 3, 12, 31, 16, tzinfo=), creation_host='mail.acer-isu.com', register_to_eff=None))>
2022-08-19 15:10:35,871:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2022-08-19 15:10:35,873:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2022-08-19 15:10:36,374:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 672
2022-08-19 15:10:36,375:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 19 Aug 2022 07:10:36 GMT
Content-Type: application/json
Content-Length: 672
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"-5wQOiSUvB0": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017-w-v1.3-notice.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2022-08-19 15:10:36,377:DEBUG:certbot._internal.display.obj:Notifying user: Renewing an existing certificate for mail.acer-isu.com
2022-08-19 15:10:36,735:DEBUG:certbot.crypto_util:Generating RSA key (2048 bits): /etc/letsencrypt/keys/0056_key-certbot.pem
2022-08-19 15:10:36,737:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0056_csr-certbot.pem
2022-08-19 15:10:36,737:DEBUG:acme.client:Requesting fresh nonce
2022-08-19 15:10:36,738:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2022-08-19 15:10:36,902:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2022-08-19 15:10:36,903:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 19 Aug 2022 07:10:36 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: 01026MWPMbyo5UQ7pu57uZk0sFj02FBecjeVJR17FiBr3z8
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

2022-08-19 15:10:36,904:DEBUG:acme.client:Storing nonce: 01026MWPMbyo5UQ7pu57uZk0sFj02FBecjeVJR17FiBr3z8
2022-08-19 15:10:36,904:DEBUG:acme.client:JWS payload:
b'{\n "identifiers": [\n {\n "type": "dns",\n "value": "mail.acer-isu.com"\n }\n ]\n}'
2022-08-19 15:10:36,908:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
"protected": ""
"signature": ""
}
2022-08-19 15:10:37,147:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 340
2022-08-19 15:10:37,148:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Fri, 19 Aug 2022 07:10:37 GMT
Content-Type: application/json
Content-Length: 340
Connection: keep-alive
Boulder-Requester: 117921551
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Location: https://acme-v02.api.letsencrypt.org/acme/order/117921551/116333757217
Replay-Nonce: 0101KkjLemQTeXr7C7yu7Mbn_uYRzJKCe66fkMNXwDYiW68
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"status": "ready",
"expires": "2022-08-22T05:19:54Z",
"identifiers": [
{
"type": "dns",
"value": "mail.acer-isu.com"
}
],
"authorizations": [
"https://acme-v02.api.letsencrypt.org/acme/authz-v3/137572949586"
],
"finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/117921551/116333757217"
}
2022-08-19 15:10:37,148:DEBUG:acme.client:Storing nonce: 0101KkjLemQTeXr7C7yu7Mbn_uYRzJKCe66fkMNXwDYiW68
2022-08-19 15:10:37,149:DEBUG:acme.client:JWS payload:
b''
2022-08-19 15:10:37,152:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/137572949586:
{
"protected": ""
"signature": ""
"payload": ""
}
2022-08-19 15:10:37,355:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/137572949586 HTTP/1.1" 200 766
2022-08-19 15:10:37,356:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Fri, 19 Aug 2022 07:10:37 GMT
Content-Type: application/json
Content-Length: 766
Connection: keep-alive
Boulder-Requester: 117921551
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: 010221Z4xUu_c3kNV72b3h2BLNgFLDUX-3duFYClwW0jDUk
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"identifier": {
"type": "dns",
"value": "mail.acer-isu.com"
},
"status": "valid",
"expires": "2022-09-01T19:07:26Z",
"challenges": [
{
"type": "http-01",
"status": "valid",
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/137572949586/ToVhlw",
"token": "jUnqH2SqHhyRwwnHV8VH0gRigDVnaHaWE9ci_vJbniQ",
"validationRecord": [
{
"url": "http://mail.acer-isu.com/.well-known/acme-challenge/jUnqH2SqHhyRwwnHV8VH0gRigDVnaHaWE9ci_vJbniQ",
"hostname": "mail.acer-isu.com",
"port": "80",
"addressesResolved": [
"60.250.195.22"
],
"addressUsed": "60.250.195.22"
}
],
"validated": "2022-08-02T19:07:25Z"
}
]
}
2022-08-19 15:10:37,356:DEBUG:acme.client:Storing nonce: 010221Z4xUu_c3kNV72b3h2BLNgFLDUX-3duFYClwW0jDUk
2022-08-19 15:10:37,357:DEBUG:certbot._internal.client:CSR: CSR(file='/etc/letsencrypt/csr/0056_csr-certbot.pem', data=b'-----BEGIN CERTIFICATE REQUEST-----\n\n-----END CERTIFICATE REQUEST-----\n', form='pem')
2022-08-19 15:10:37,357:DEBUG:certbot._internal.client:Will poll for certificate issuance until 2022-08-19 15:12:07.357880
2022-08-19 15:10:37,358:DEBUG:acme.client:JWS payload:
b'{\n "csr": ""\n}'
2022-08-19 15:10:37,361:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/finalize/117921551/116333757217:
{
"protected": ""
"signature": "",
"payload": ""
}
2022-08-19 15:10:37,499:ERROR:certbot._internal.renewal:Failed to renew certificate mail.acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
2022-08-19 15:10:37,500:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 833, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 590, in read
v = self._sslobj.read(len, buffer)
ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
timeout=timeout
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 638, in urlopen
_stacktrace=sys.exc_info()[2])
File "/usr/lib/python3.6/site-packages/urllib3/util/retry.py", line 368, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/usr/lib/python3.6/site-packages/urllib3/packages/six.py", line 692, in reraise
raise value.with_traceback(tb)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 600, in urlopen
chunked=chunked)
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 384, in _make_request
six.raise_from(e, None)
File "", line 3, in raise_from
File "/usr/lib/python3.6/site-packages/urllib3/connectionpool.py", line 380, in _make_request
httplib_response = conn.getresponse()
File "/usr/lib64/python3.6/http/client.py", line 1365, in getresponse
response.begin()
File "/usr/lib64/python3.6/http/client.py", line 320, in begin
version, status, reason = self._read_status()
File "/usr/lib64/python3.6/http/client.py", line 281, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib64/python3.6/socket.py", line 586, in readinto
return self._sock.recv_into(b)
File "/usr/lib64/python3.6/ssl.py", line 971, in recv_into
return self.read(nbytes, buffer)
File "/usr/lib64/python3.6/ssl.py", line 833, in read
return self._sslobj.read(len, buffer)
File "/usr/lib64/python3.6/ssl.py", line 590, in read
v = self._sslobj.read(len, buffer)
urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1441, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 127, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 345, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/usr/lib/python3.6/site-packages/certbot/_internal/client.py", line 440, in obtain_certificate
cert, chain = self.obtain_certificate_from_csr(csr, orderr)
File "/usr/lib/python3.6/site-packages/certbot/_internal/client.py", line 339, in obtain_certificate_from_csr
orderr, deadline, fetch_alternative_chains=self.config.preferred_chain is not None)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 978, in finalize_order
orderr, deadline, fetch_alternative_chains)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 793, in finalize_order
self._post(orderr.body.finalize, wrapped_csr)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 101, in _post
return self.net.post(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1269, in post
return self._post_once(*args, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1282, in _post_once
response = self._send_request('POST', url, data=data, **kwargs)
File "/usr/lib/python3.6/site-packages/acme/client.py", line 1174, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
resp = self.send(prep, **send_kwargs)
File "/usr/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
r = adapter.send(request, **kwargs)
File "/usr/lib/python3.6/site-packages/requests/adapters.py", line 498, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

2022-08-19 15:10:37,501:DEBUG:certbot._internal.display.obj:Notifying user:

2022-08-19 15:10:37,501:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2022-08-19 15:10:37,501:ERROR:certbot._internal.renewal: /etc/letsencrypt/live/acer-isu.com/fullchain.pem (failure)
/etc/letsencrypt/live/mail.acer-isu.com/fullchain.pem (failure)
2022-08-19 15:10:37,502:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-08-19 15:10:37,502:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in
load_entry_point('certbot==1.22.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3.6/site-packages/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1632, in main
return config.func(config, plugins)
File "/usr/lib/python3.6/site-packages/certbot/_internal/main.py", line 1518, in renew
renewal.handle_renewal_request(config)
File "/usr/lib/python3.6/site-packages/certbot/_internal/renewal.py", line 512, in handle_renewal_request
len(renew_failures), len(parse_failures)))
certbot.errors.Error: 2 renew failure(s), 0 parse failure(s)
2022-08-19 15:10:37,503:ERROR:certbot._internal.log:2 renew failure(s), 0 parse failure(s)

1 Like
7

You don't have to redact pieces of the log. And, it's always handy to have the whole thing. But, I think there is enough. It is strange to me. Log shows:

We can presume authz for acer-isu.com succeeds just before these two but not shown in log fragment
authz for mail.acer-isu.com succeeds
authz for www.acer-isu.com succeeds

Then, the finalize for the order fails with "reset by peer".

That's odd that so many requests to Let's Encrypt worked and then only fail at this finalize request with that specific error.

It would be very interesting to see results of

certbot renew --dry-run

This uses the test system and won't get certs but curious if fails

3 Likes
8

Hello

Unfortunately it is the same result.

certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/acer-isu.com.conf

Simulating renewal of an existing certificate for acer-isu.com and 2 more domains
Failed to renew certificate acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

Processing /etc/letsencrypt/renewal/mail.acer-isu.com.conf

Simulating renewal of an existing certificate for mail.acer-isu.com
Failed to renew certificate mail.acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/acer-isu.com/fullchain.pem (failure)
/etc/letsencrypt/live/mail.acer-isu.com/fullchain.pem (failure)

2 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

9

Can you show output of this? I just want to make sure the "reset by peer" is happening in the same sequence. I am hoping this is enough of the log to see that.

 grep -inE 'authz-v3|finalize|peer|acer-isu' /var/log/letsencrypt/letsencrypt.log

It would be helpful if you place 3 backticks before and after the output to format it better. Like this:
```
grep output
```

3 Likes
10

Hello,

Thank you for all the help and quick response - even on a weekend. Here is the output.

6:2022-08-21 09:30:20,652:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/acer-isu.com.conf
15:2022-08-21 09:30:21,067:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/acer-isu.com/cert8.pem is signed by the certificate's issuer.
16:2022-08-21 09:30:21,073:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/acer-isu.com/cert8.pem is: OCSPCertStatus.GOOD
35:2022-08-21 09:30:21,568:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/18909152', new_authzr_uri=None, terms_of_service=None), 01b7ef07ce883bdfc832c5c2bab181ef, Meta(creation_dt=datetime.datetime(2021, 4, 3, 13, 27, 57, tzinfo=<UTC>), creation_host='mail.acer-isu.com', register_to_eff=None))>
66:2022-08-21 09:30:22,048:DEBUG:certbot._internal.display.obj:Notifying user: Simulating renewal of an existing certificate for acer-isu.com and 2 more domains
84:b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "acer-isu.com"\n    },\n    {\n      "type": "dns",\n      "value": "mail.acer-isu.com"\n    },\n    {\n      "type": "dns",\n      "value": "www.acer-isu.com"\n    }\n  ]\n}'
113:      "value": "acer-isu.com"
117:      "value": "mail.acer-isu.com"
121:      "value": "www.acer-isu.com"
125:    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340934344",
126:    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340935354",
127:    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340934364"
129:  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735504374"
134:2022-08-21 09:30:22,474:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340934344:
140:2022-08-21 09:30:22,632:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3340934344 HTTP/1.1" 200 757
158:    "value": "acer-isu.com"
170:          "url": "http://acer-isu.com/.well-known/acme-challenge/GcJ3cYldpbRb_9dWI4LJfblMMEPzElRjUssp1EokPj0",
171:          "hostname": "acer-isu.com",
186:2022-08-21 09:30:22,637:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340935354:
192:2022-08-21 09:30:22,796:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3340935354 HTTP/1.1" 200 772
210:    "value": "mail.acer-isu.com"
222:          "url": "http://mail.acer-isu.com/.well-known/acme-challenge/-wkgu1LFIkm4hOX5rqNWODPmgXwVvzjZZ3odJH_6EWQ",
223:          "hostname": "mail.acer-isu.com",
238:2022-08-21 09:30:22,802:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340934364:
244:2022-08-21 09:30:22,961:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3340934364 HTTP/1.1" 200 769
262:    "value": "www.acer-isu.com"
274:          "url": "http://www.acer-isu.com/.well-known/acme-challenge/49ydBqG4RMReRv2trMIWsXE843_NW4KIueIQTTU52-E",
275:          "hostname": "www.acer-isu.com",
290:2022-08-21 09:30:22,966:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340934344:
296:2022-08-21 09:30:23,127:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3340934344 HTTP/1.1" 200 763
314:    "value": "acer-isu.com"
326:          "url": "http://acer-isu.com/.well-known/acme-challenge/GcJ3cYldpbRb_9dWI4LJfblMMEPzElRjUssp1EokPj0",
327:          "hostname": "acer-isu.com",
342:2022-08-21 09:30:23,132:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340935354:
348:2022-08-21 09:30:23,292:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3340935354 HTTP/1.1" 200 778
366:    "value": "mail.acer-isu.com"
378:          "url": "http://mail.acer-isu.com/.well-known/acme-challenge/-wkgu1LFIkm4hOX5rqNWODPmgXwVvzjZZ3odJH_6EWQ",
379:          "hostname": "mail.acer-isu.com",
394:2022-08-21 09:30:23,297:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3340934364:
400:2022-08-21 09:30:23,458:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3340934364 HTTP/1.1" 200 775
418:    "value": "www.acer-isu.com"
430:          "url": "http://www.acer-isu.com/.well-known/acme-challenge/49ydBqG4RMReRv2trMIWsXE843_NW4KIueIQTTU52-E",
431:          "hostname": "www.acer-isu.com",
446:b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "acer-isu.com"\n    },\n    {\n      "type": "dns",\n      "value": "mail.acer-isu.com"\n    },\n    {\n      "type": "dns",\n      "value": "www.acer-isu.com"\n    }\n  ]\n}'
475:      "value": "acer-isu.com"
479:      "value": "mail.acer-isu.com"
483:      "value": "www.acer-isu.com"
487:    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827334",
488:    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827344",
489:    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827354"
491:  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735504524"
496:2022-08-21 09:30:23,649:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827334:
502:2022-08-21 09:30:23,807:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394827334 HTTP/1.1" 200 814
520:    "value": "acer-isu.com"
548:2022-08-21 09:30:23,812:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827344:
554:2022-08-21 09:30:23,970:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394827344 HTTP/1.1" 200 819
572:    "value": "mail.acer-isu.com"
600:2022-08-21 09:30:23,976:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827354:
606:2022-08-21 09:30:24,131:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394827354 HTTP/1.1" 200 818
624:    "value": "www.acer-isu.com"
651:2022-08-21 09:30:24,134:INFO:certbot._internal.auth_handler:http-01 challenge for acer-isu.com
652:2022-08-21 09:30:24,134:INFO:certbot._internal.auth_handler:http-01 challenge for mail.acer-isu.com
653:2022-08-21 09:30:24,134:INFO:certbot._internal.auth_handler:http-01 challenge for www.acer-isu.com
654:2022-08-21 09:30:24,148:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: acer-isu.com in: /etc/httpd/conf/httpd.conf
655:2022-08-21 09:30:24,149:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: acer-isu.com in: /etc/httpd/conf/httpd-le-ssl.conf
656:2022-08-21 09:30:24,149:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: mail.acer-isu.com in: /etc/httpd/conf/httpd.conf
691:Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827334>;rel="up"
722:Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827344>;rel="up"
753:Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827354>;rel="up"
769:2022-08-21 09:30:29,162:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827334:
775:2022-08-21 09:30:29,320:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394827334 HTTP/1.1" 200 757
793:    "value": "acer-isu.com"
805:          "url": "http://acer-isu.com/.well-known/acme-challenge/XlnXz1yaZTJPpIi0irVYAkTpqzq3EOXsXHmGxIRPddM",
806:          "hostname": "acer-isu.com",
821:2022-08-21 09:30:29,326:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827344:
827:2022-08-21 09:30:29,483:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394827344 HTTP/1.1" 200 772
845:    "value": "mail.acer-isu.com"
857:          "url": "http://mail.acer-isu.com/.well-known/acme-challenge/azoPpbkXOCyhajV2Wq0OGLIuRT9WgONXpuCy0hiuBQE",
858:          "hostname": "mail.acer-isu.com",
873:2022-08-21 09:30:29,488:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827354:
879:2022-08-21 09:30:29,647:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394827354 HTTP/1.1" 200 769
897:    "value": "www.acer-isu.com"
909:          "url": "http://www.acer-isu.com/.well-known/acme-challenge/bVUGV7FFrAF4qT4qYDwj92sS9rUeCKBc6P-oD9IBpfo",
910:          "hostname": "www.acer-isu.com",
929:2022-08-21 09:30:30,026:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735504524:
935:2022-08-21 09:30:30,167:ERROR:certbot._internal.renewal:Failed to renew certificate acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
959:ConnectionResetError: [Errno 104] Connection reset by peer
993:urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
1010:  File "/usr/lib/python3.6/site-packages/acme/client.py", line 978, in finalize_order
1012:  File "/usr/lib/python3.6/site-packages/acme/client.py", line 793, in finalize_order
1013:    self._post(orderr.body.finalize, wrapped_csr)
1028:requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
1030:2022-08-21 09:30:30,177:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/mail.acer-isu.com.conf
1038:2022-08-21 09:30:30,567:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/mail.acer-isu.com/cert6.pem is signed by the certificate's issuer.
1039:2022-08-21 09:30:30,568:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/mail.acer-isu.com/cert6.pem is: OCSPCertStatus.GOOD
1052:2022-08-21 09:30:31,094:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/18909152', new_authzr_uri=None, terms_of_service=None), 01b7ef07ce883bdfc832c5c2bab181ef, Meta(creation_dt=datetime.datetime(2021, 4, 3, 13, 27, 57, tzinfo=<UTC>), creation_host='mail.acer-isu.com', register_to_eff=None))>
1083:2022-08-21 09:30:31,576:DEBUG:certbot._internal.display.obj:Notifying user: Simulating renewal of an existing certificate for mail.acer-isu.com
1101:b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "mail.acer-isu.com"\n    }\n  ]\n}'
1130:      "value": "mail.acer-isu.com"
1134:    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827344"
1136:  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735505834"
1141:2022-08-21 09:30:32,111:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827344:
1147:2022-08-21 09:30:32,268:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394827344 HTTP/1.1" 200 772
1165:    "value": "mail.acer-isu.com"
1177:          "url": "http://mail.acer-isu.com/.well-known/acme-challenge/azoPpbkXOCyhajV2Wq0OGLIuRT9WgONXpuCy0hiuBQE",
1178:          "hostname": "mail.acer-isu.com",
1193:2022-08-21 09:30:32,273:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394827344:
1199:2022-08-21 09:30:32,435:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394827344 HTTP/1.1" 200 778
1217:    "value": "mail.acer-isu.com"
1229:          "url": "http://mail.acer-isu.com/.well-known/acme-challenge/azoPpbkXOCyhajV2Wq0OGLIuRT9WgONXpuCy0hiuBQE",
1230:          "hostname": "mail.acer-isu.com",
1245:b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "mail.acer-isu.com"\n    }\n  ]\n}'
1274:      "value": "mail.acer-isu.com"
1278:    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394828074"
1280:  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735505864"
1285:2022-08-21 09:30:32,624:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394828074:
1291:2022-08-21 09:30:32,783:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394828074 HTTP/1.1" 200 819
1309:    "value": "mail.acer-isu.com"
1336:2022-08-21 09:30:32,786:INFO:certbot._internal.auth_handler:http-01 challenge for mail.acer-isu.com
1337:2022-08-21 09:30:32,796:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: mail.acer-isu.com in: /etc/httpd/conf/httpd.conf
1338:2022-08-21 09:30:32,797:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: acer-isu.com in: /etc/httpd/conf/httpd.conf
1370:Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394828074>;rel="up"
1386:2022-08-21 09:30:37,414:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/3394828074:
1392:2022-08-21 09:30:37,573:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/3394828074 HTTP/1.1" 200 772
1410:    "value": "mail.acer-isu.com"
1422:          "url": "http://mail.acer-isu.com/.well-known/acme-challenge/yTrcZbAUu_L6rYfycezlKPrsxCKSV1CtvYpRFplAW7o",
1423:          "hostname": "mail.acer-isu.com",
1442:2022-08-21 09:30:37,939:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/finalize/18909152/3735505864:
1448:2022-08-21 09:30:38,078:ERROR:certbot._internal.renewal:Failed to renew certificate mail.acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
1472:ConnectionResetError: [Errno 104] Connection reset by peer
1506:urllib3.exceptions.ProtocolError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
1523:  File "/usr/lib/python3.6/site-packages/acme/client.py", line 978, in finalize_order
1525:  File "/usr/lib/python3.6/site-packages/acme/client.py", line 793, in finalize_order
1526:    self._post(orderr.body.finalize, wrapped_csr)
1541:requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
1546:2022-08-21 09:30:38,081:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/acer-isu.com/fullchain.pem (failure)
1547:  /etc/letsencrypt/live/mail.acer-isu.com/fullchain.pem (failure)
2 Likes
11

Can we have a look at the output of?:
sudo apachectl -t -D DUMP_VHOSTS
or
sudo httpd -t -D DUMP_VHOSTS

and, also, the file:
/etc/httpd/conf/httpd.conf

3 Likes
12

Yeah, the --dry-run looks to also have failed with the "reset by peer" for the acme finalize request. That seems more like an outbound request failure. It is not the same thing we see when the Let's Encrypt servers cannot connect to your server to satisfy the challenge. When those fail the error occurs in the prior authz but your challenges work.

A reset by peer is usually from a firewall.

Hopefully Rudy will have more luck than I did. I'm signing off for the night.

@rg305 Rudy, they have 2 certs. I should have had them use --cert-name to focus on just one but that's why the confusing info. Best of luck to you both. Hope to see it marked solved by morning :slight_smile:

3 Likes
13

Hi Rudy,

httpd -t -D DUMP_HOSTS:

[Sun Aug 21 14:57:17.704215 2022] [core:warn] [pid 145110:tid 140663298734400] AH00117: Ignoring deprecated use of DefaultType in line 440 of /etc/httpd/conf/httpd.conf.
[Sun Aug 21 14:57:17.704303 2022] [alias:warn] [pid 145110:tid 140663298734400] AH00671: The Alias directive in /etc/httpd/conf/httpd.conf at line 555 will probably never match because it overlaps an earlier Alias.
VirtualHost configuration:
*:80                   acer-isu.com (/etc/httpd/conf/httpd.conf:1015)
*:443                  is a NameVirtualHost
         default server mail.acer-isu.com (/etc/httpd/conf.d/ssl.conf:40)
         port 443 namevhost mail.acer-isu.com (/etc/httpd/conf.d/ssl.conf:40)
         port 443 namevhost acer-isu.com (/etc/httpd/conf/httpd-le-ssl.conf:2)
                 alias www.acer-isu.com
*:10025                mail.acer-isu.com (/etc/httpd/conf/httpd.conf:1028)

httpd config

#
# This is the main Apache server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs/2.2/> for detailed information.
# In particular, see
# <URL:http://httpd.apache.org/docs/2.2/mod/directives.html>
# for a discussion of each configuration directive.
#
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.  
#
# The configuration directives are grouped into three basic sections:
#  1. Directives that control the operation of the Apache server process as a
#     whole (the 'global environment').
#  2. Directives that define the parameters of the 'main' or 'default' server,
#     which responds to requests that aren't handled by a virtual host.
#     These directives also provide default values for the settings
#     of all virtual hosts.
#  3. Settings for virtual hosts, which allow Web requests to be sent to
#     different IP addresses or hostnames and have them handled by the
#     same Apache server process.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "logs/foo.log"
# with ServerRoot set to "/etc/httpd" will be interpreted by the
# server as "/etc/httpd/logs/foo.log".
#

### Section 1: Global Environment
#
# The directives in this section affect the overall operation of Apache,
# such as the number of concurrent requests it can handle or where it
# can find its configuration files.
#

#
# Don't give away too much information about all the subcomponents
# we are running.  Comment out this line if you don't mind remote sites
# finding out what major optional modules you are running
ServerTokens OS

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation
# (available at <URL:http://httpd.apache.org/docs/2.2/mod/mpm_common.html#lockfile>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
ServerRoot "/etc/httpd"

#
# PidFile: The file in which the server should record its process
# identification number when it starts.  Note the PIDFILE variable in
# /etc/sysconfig/httpd must be set appropriately if this location is
# changed.
#
PidFile run/httpd.pid

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 600

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15

##
## Server-Pool Size Regulation (MPM specific)
## 

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# ServerLimit: maximum value for MaxClients for the lifetime of the server
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule prefork.c>
StartServers       8
MinSpareServers    5
MaxSpareServers   20
ServerLimit      256
MaxClients       256
MaxRequestsPerChild  4000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule worker.c>
StartServers         4
MaxClients         300
MinSpareThreads     25
MaxSpareThreads     75 
ThreadsPerChild     25
MaxRequestsPerChild  0
</IfModule>

#
# Listen: Allows you to bind Apache to specific IP addresses and/or
# ports, in addition to the default. See also the <VirtualHost>
# directive.
#
# Change this to Listen on specific IP addresses as shown below to 
# prevent Apache from glomming onto all bound IP addresses (0.0.0.0)
#
#Listen 12.34.56.78:80
Listen 80
#Listen 443

#
# Dynamic Shared Object (DSO) Support
#
# To be able to use the functionality of a module which was built as a DSO you
# have to place corresponding `LoadModule' lines at this location so the
# directives contained in it are actually available _before_ they are used.
# Statically compiled modules (those listed by `httpd -l') do not need
# to be loaded here.
#
# Example:
# LoadModule foo_module modules/mod_foo.so
#
Include conf.modules.d/*.conf

#LoadModule auth_basic_module modules/mod_auth_basic.so
#LoadModule auth_digest_module modules/mod_auth_digest.so
#LoadModule authn_file_module modules/mod_authn_file.so
#LoadModule authn_alias_module modules/mod_authn_alias.so
#LoadModule authn_anon_module modules/mod_authn_anon.so
#LoadModule authn_dbm_module modules/mod_authn_dbm.so
#LoadModule authn_default_module modules/mod_authn_default.so
#LoadModule authz_host_module modules/mod_authz_host.so
#LoadModule authz_user_module modules/mod_authz_user.so
#LoadModule authz_owner_module modules/mod_authz_owner.so
#LoadModule authz_groupfile_module modules/mod_authz_groupfile.so
#LoadModule authz_dbm_module modules/mod_authz_dbm.so
#LoadModule authz_default_module modules/mod_authz_default.so
#LoadModule ldap_module modules/mod_ldap.so
#LoadModule authnz_ldap_module modules/mod_authnz_ldap.so
#LoadModule include_module modules/mod_include.so
#LoadModule log_config_module modules/mod_log_config.so
#LoadModule logio_module modules/mod_logio.so
#LoadModule env_module modules/mod_env.so
#LoadModule ext_filter_module modules/mod_ext_filter.so
#LoadModule mime_magic_module modules/mod_mime_magic.so
#LoadModule expires_module modules/mod_expires.so
#LoadModule deflate_module modules/mod_deflate.so
#LoadModule headers_module modules/mod_headers.so
#LoadModule usertrack_module modules/mod_usertrack.so
#LoadModule setenvif_module modules/mod_setenvif.so
#LoadModule mime_module modules/mod_mime.so
#LoadModule dav_module modules/mod_dav.so
#LoadModule status_module modules/mod_status.so
#LoadModule autoindex_module modules/mod_autoindex.so
#LoadModule info_module modules/mod_info.so
#LoadModule dav_fs_module modules/mod_dav_fs.so
#LoadModule vhost_alias_module modules/mod_vhost_alias.so
#LoadModule negotiation_module modules/mod_negotiation.so
#LoadModule dir_module modules/mod_dir.so
#LoadModule actions_module modules/mod_actions.so
#LoadModule speling_module modules/mod_speling.so
#LoadModule userdir_module modules/mod_userdir.so
#LoadModule alias_module modules/mod_alias.so
#LoadModule substitute_module modules/mod_substitute.so
#LoadModule rewrite_module modules/mod_rewrite.so
#LoadModule proxy_module modules/mod_proxy.so
#LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
#LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
#LoadModule proxy_http_module modules/mod_proxy_http.so
#LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
#LoadModule proxy_connect_module modules/mod_proxy_connect.so
#LoadModule cache_module modules/mod_cache.so
#LoadModule suexec_module modules/mod_suexec.so
#LoadModule disk_cache_module modules/mod_disk_cache.so
#LoadModule cgi_module modules/mod_cgi.so
#LoadModule version_module modules/mod_version.so

#
# The following modules are not loaded by default:
#
#LoadModule asis_module modules/mod_asis.so
#LoadModule authn_dbd_module modules/mod_authn_dbd.so
#LoadModule cern_meta_module modules/mod_cern_meta.so
#LoadModule cgid_module modules/mod_cgid.so
#LoadModule dbd_module modules/mod_dbd.so
#LoadModule dumpio_module modules/mod_dumpio.so
#LoadModule filter_module modules/mod_filter.so
#LoadModule ident_module modules/mod_ident.so
#LoadModule log_forensic_module modules/mod_log_forensic.so
#LoadModule unique_id_module modules/mod_unique_id.so
#

#
# Load config files from the config directory "/etc/httpd/conf.d".
#
Include conf.d/*.conf

#
# ExtendedStatus controls whether Apache will generate "full" status
# information (ExtendedStatus On) or just basic information (ExtendedStatus
# Off) when the "server-status" handler is called. The default is Off.
#
#ExtendedStatus On

#
# If you wish httpd to run as a different user or group, you must run
# httpd as root initially and it will switch.  
#
# User/Group: The name (or #number) of the user/group to run httpd as.
#  . On SCO (ODT 3) use "User nouser" and "Group nogroup".
#  . On HPUX you may not be able to use shared memory as nobody, and the
#    suggested workaround is to create a user www and use that user.
#  NOTE that some kernels refuse to setgid(Group) or semctl(IPC_SET)
#  when the value of (unsigned)Group is above 60000; 
#  don't use Group #-1 on these systems!
#
User apache
Group apache

### Section 2: 'Main' server configuration
#
# The directives in this section set up the values used by the 'main'
# server, which responds to any requests that aren't handled by a
# <VirtualHost> definition.  These values also provide defaults for
# any <VirtualHost> containers you may define later in the file.
#
# All of these directives may appear inside <VirtualHost> containers,
# in which case these default settings will be overridden for the
# virtual host being defined.
#

#
# ServerAdmin: Your address, where problems with the server should be
# e-mailed.  This address appears on some server-generated pages, such
# as error documents.  e.g. admin@your-domain.com
#
ServerAdmin root@localhost

#
# ServerName gives the name and port that the server uses to identify itself.
# This can often be determined automatically, but we recommend you specify
# it explicitly to prevent problems during startup.
#
# If this is not set to valid DNS name for your host, server-generated
# redirections will not work.  See also the UseCanonicalName directive.
#
# If your host doesn't have a registered DNS name, enter its IP address here.
# You will have to access it by its address anyway, and this will make 
# redirections work in a sensible way.
#
#ServerName www.example.com:80

#
# UseCanonicalName: Determines how Apache constructs self-referencing 
# URLs and the SERVER_NAME and SERVER_PORT variables.
# When set "Off", Apache will use the Hostname and Port supplied
# by the client.  When set "On", Apache will use the value of the
# ServerName directive.
#
UseCanonicalName Off

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/var/www/html"

#
# Each directory to which Apache has access can be configured with respect
# to which services and features are allowed and/or disabled in that
# directory (and its subdirectories). 
#
# First, we configure the "default" to be a very restrictive set of 
# features.  
#
<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>

#
# Note that from this point forward you must specifically allow
# particular features to be enabled - so if something's not working as
# you might expect, make sure that you have specifically enabled it
# below.
#

#
# This should be changed to whatever you set DocumentRoot to.
#
<Directory "/var/www/html">

#
# Possible values for the Options directive are "None", "All",
# or any combination of:
#   Indexes Includes FollowSymLinks SymLinksifOwnerMatch ExecCGI MultiViews
#
# Note that "MultiViews" must be named *explicitly* --- "Options All"
# doesn't give it to you.
#
# The Options directive is both complicated and important.  Please see
# http://httpd.apache.org/docs/2.2/mod/core.html#options
# for more information.
#
#    Options Indexes FollowSymLinks
     Options FollowSymLinks

#
# AllowOverride controls what directives may be placed in .htaccess files.
# It can be "All", "None", or any combination of the keywords:
#   Options FileInfo AuthConfig Limit
#
    AllowOverride None

#
# Controls who can get stuff from this server.
#
    Order allow,deny
    Allow from all

</Directory>

#
# UserDir: The name of the directory that is appended onto a user's home
# directory if a ~user request is received.
#
# The path to the end user account 'public_html' directory must be
# accessible to the webserver userid.  This usually means that ~userid
# must have permissions of 711, ~userid/public_html must have permissions
# of 755, and documents contained therein must be world-readable.
# Otherwise, the client will only receive a "403 Forbidden" message.
#
# See also: http://httpd.apache.org/docs/misc/FAQ.html#forbidden
#
<IfModule mod_userdir.c>
    #
    # UserDir is disabled by default since it can confirm the presence
    # of a username on the system (depending on home directory
    # permissions).
    #
    UserDir disabled

    #
    # To enable requests to /~user/ to serve the user's public_html
    # directory, remove the "UserDir disabled" line above, and uncomment
    # the following line instead:
    # 
    #UserDir public_html

</IfModule>

#
# Control access to UserDir directories.  The following is an example
# for a site where these directories are restricted to read-only.
#
#<Directory /home/*/public_html>
#    AllowOverride FileInfo AuthConfig Limit
#    Options MultiViews Indexes SymLinksIfOwnerMatch IncludesNoExec
#    <Limit GET POST OPTIONS>
#        Order allow,deny
#        Allow from all
#    </Limit>
#    <LimitExcept GET POST OPTIONS>
#        Order deny,allow
#        Deny from all
#    </LimitExcept>
#</Directory>

#
# DirectoryIndex: sets the file that Apache will serve if a directory
# is requested.
#
# The index.html.var file (a type-map) is used to deliver content-
# negotiated documents.  The MultiViews Option can be used for the 
# same purpose, but it is much slower.
#
DirectoryIndex index.html index.html.var

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#
AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being 
# viewed by Web clients. 
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
    Satisfy All
</Files>

#
# TypesConfig describes where the mime.types file (or equivalent) is
# to be found.
#
TypesConfig /etc/mime.types

#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain

#
# The mod_mime_magic module allows the server to use various hints from the
# contents of the file itself to determine its type.  The MIMEMagicFile
# directive tells the module where the hint definitions are located.
#
<IfModule mod_mime_magic.c>
#   MIMEMagicFile /usr/share/magic.mime
    MIMEMagicFile conf/magic
</IfModule>

#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

#
# EnableMMAP: Control whether memory-mapping is used to deliver
# files (assuming that the underlying OS supports it).
# The default is on; turn this off if you serve from NFS-mounted 
# filesystems.  On some systems, turning it off (regardless of
# filesystem) can improve performance; for details, please see
# http://httpd.apache.org/docs/2.2/mod/core.html#enablemmap
#
#EnableMMAP off

#
# EnableSendfile: Control whether the sendfile kernel support is 
# used to deliver files (assuming that the OS supports it). 
# The default is on; turn this off if you serve from NFS-mounted 
# filesystems.  Please see
# http://httpd.apache.org/docs/2.2/mod/core.html#enablesendfile
#
#EnableSendfile off

#
# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog logs/error_log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

# "combinedio" includes actual counts of actual bytes received (%I) and sent (%O); this
# requires the mod_logio module to be loaded.
#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio

#
# The location and format of the access logfile (Common Logfile Format).
# If you do not define any access logfiles within a <VirtualHost>
# container, they will be logged here.  Contrariwise, if you *do*
# define per-<VirtualHost> access logfiles, transactions will be
# logged therein and *not* in this file.
#
#CustomLog logs/access_log common

#
# If you would like to have separate agent and referer logfiles, uncomment
# the following directives.
#
#CustomLog logs/referer_log referer
#CustomLog logs/agent_log agent

#
# For a single logfile with access, agent, and referer information
# (Combined Logfile Format), use the following directive:
#
CustomLog logs/access_log combined

#
# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of:  On | Off | EMail
#
ServerSignature On

#
# Aliases: Add here as many aliases as you need (with no limit). The format is 
# Alias fakename realname
#
# Note that if you include a trailing / on fakename then the server will
# require it to be present in the URL.  So "/icons" isn't aliased in this
# example, only "/icons/".  If the fakename is slash-terminated, then the 
# realname must also be slash terminated, and if the fakename omits the 
# trailing slash, the realname must also omit it.
#
# We include the /icons/ alias for FancyIndexed directory listings.  If you
# do not use FancyIndexing, you may comment this out.
#
Alias /icons/ "/var/www/icons/"

<Directory "/var/www/icons">
    Options Indexes MultiViews FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all
</Directory>

#
# WebDAV module configuration section.
# 
<IfModule mod_dav_fs.c>
    # Location of the WebDAV lock database.
    DAVLockDB /var/lib/dav/lockdb
</IfModule>

#
# ScriptAlias: This controls which directories contain server scripts.
# ScriptAliases are essentially the same as Aliases, except that
# documents in the realname directory are treated as applications and
# run by the server when requested rather than as documents sent to the client.
# The same rules about trailing "/" apply to ScriptAlias directives as to
# Alias.
#
ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
ScriptAlias /webmail "/var/www/cgi-bin/openwebmail/openwebmail.pl"

#
# "/var/www/cgi-bin" should be changed to whatever your ScriptAliased
# CGI directory exists, if you have that configured.
#
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Order allow,deny
    Allow from all
</Directory>

#
# Redirect allows you to tell clients about documents which used to exist in
# your server's namespace, but do not anymore. This allows you to tell the
# clients where to look for the relocated document.
# Example:
# Redirect permanent /foo http://www.example.com/bar

#
# Directives controlling the display of server-generated directory listings.
#

#
# IndexOptions: Controls the appearance of server-generated directory
# listings.
#
IndexOptions FancyIndexing VersionSort NameWidth=* HTMLTable Charset=UTF-8

#
# AddIcon* directives tell the server which icon to show for different
# files or filename extensions.  These are only displayed for
# FancyIndexed directories.
#
AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip

AddIconByType (TXT,/icons/text.gif) text/*
AddIconByType (IMG,/icons/image2.gif) image/*
AddIconByType (SND,/icons/sound2.gif) audio/*
AddIconByType (VID,/icons/movie.gif) video/*

AddIcon /icons/binary.gif .bin .exe
AddIcon /icons/binhex.gif .hqx
AddIcon /icons/tar.gif .tar
AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
AddIcon /icons/a.gif .ps .ai .eps
AddIcon /icons/layout.gif .html .shtml .htm .pdf
AddIcon /icons/text.gif .txt
AddIcon /icons/c.gif .c
AddIcon /icons/p.gif .pl .py
AddIcon /icons/f.gif .for
AddIcon /icons/dvi.gif .dvi
AddIcon /icons/uuencoded.gif .uu
AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
AddIcon /icons/tex.gif .tex
AddIcon /icons/bomb.gif /core

AddIcon /icons/back.gif ..
AddIcon /icons/hand.right.gif README
AddIcon /icons/folder.gif ^^DIRECTORY^^
AddIcon /icons/blank.gif ^^BLANKICON^^

#
# DefaultIcon is which icon to show for files which do not have an icon
# explicitly set.
#
DefaultIcon /icons/unknown.gif

#
# AddDescription allows you to place a short description after a file in
# server-generated indexes.  These are only displayed for FancyIndexed
# directories.
# Format: AddDescription "description" filename
#
#AddDescription "GZIP compressed document" .gz
#AddDescription "tar archive" .tar
#AddDescription "GZIP compressed tar archive" .tgz

#
# ReadmeName is the name of the README file the server will look for by
# default, and append to directory listings.
#
# HeaderName is the name of a file which should be prepended to
# directory indexes. 
ReadmeName README.html
HeaderName HEADER.html

#
# IndexIgnore is a set of filenames which directory indexing should ignore
# and not include in the listing.  Shell-style wildcarding is permitted.
#
IndexIgnore .??* *~ *# HEADER* README* RCS CVS *,v *,t

#
# DefaultLanguage and AddLanguage allows you to specify the language of 
# a document. You can then use content negotiation to give a browser a 
# file in a language the user can understand.
#
# Specify a default language. This means that all data
# going out without a specific language tag (see below) will 
# be marked with this one. You probably do NOT want to set
# this unless you are sure it is correct for all cases.
#
# * It is generally better to not mark a page as 
# * being a certain language than marking it with the wrong
# * language!
#
# DefaultLanguage nl
#
# Note 1: The suffix does not have to be the same as the language
# keyword --- those with documents in Polish (whose net-standard
# language code is pl) may wish to use "AddLanguage pl .po" to
# avoid the ambiguity with the common suffix for perl scripts.
#
# Note 2: The example entries below illustrate that in some cases 
# the two character 'Language' abbreviation is not identical to 
# the two character 'Country' code for its country,
# E.g. 'Danmark/dk' versus 'Danish/da'.
#
# Note 3: In the case of 'ltz' we violate the RFC by using a three char
# specifier. There is 'work in progress' to fix this and get
# the reference data for rfc1766 cleaned up.
#
# Catalan (ca) - Croatian (hr) - Czech (cs) - Danish (da) - Dutch (nl)
# English (en) - Esperanto (eo) - Estonian (et) - French (fr) - German (de)
# Greek-Modern (el) - Hebrew (he) - Italian (it) - Japanese (ja)
# Korean (ko) - Luxembourgeois* (ltz) - Norwegian Nynorsk (nn)
# Norwegian (no) - Polish (pl) - Portugese (pt)
# Brazilian Portuguese (pt-BR) - Russian (ru) - Swedish (sv)
# Simplified Chinese (zh-CN) - Spanish (es) - Traditional Chinese (zh-TW)
#
AddLanguage ca .ca
AddLanguage cs .cz .cs
AddLanguage da .dk
AddLanguage de .de
AddLanguage el .el
AddLanguage en .en
AddLanguage eo .eo
AddLanguage es .es
AddLanguage et .et
AddLanguage fr .fr
AddLanguage he .he
AddLanguage hr .hr
AddLanguage it .it
AddLanguage ja .ja
AddLanguage ko .ko
AddLanguage ltz .ltz
AddLanguage nl .nl
AddLanguage nn .nn
AddLanguage no .no
AddLanguage pl .po
AddLanguage pt .pt
AddLanguage pt-BR .pt-br
AddLanguage ru .ru
AddLanguage sv .sv
AddLanguage zh-CN .zh-cn
AddLanguage zh-TW .zh-tw

#
# LanguagePriority allows you to give precedence to some languages
# in case of a tie during content negotiation.
#
# Just list the languages in decreasing order of preference. We have
# more or less alphabetized them here. You probably want to change this.
#
LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW

#
# ForceLanguagePriority allows you to serve a result page rather than
# MULTIPLE CHOICES (Prefer) [in case of a tie] or NOT ACCEPTABLE (Fallback)
# [in case no accepted languages matched the available variants]
#
ForceLanguagePriority Prefer Fallback

#
# Specify a default charset for all content served; this enables
# interpretation of all content as UTF-8 by default.  To use the 
# default browser choice (ISO-8859-1), or to allow the META tags
# in HTML content to override this choice, comment out this
# directive:
#
AddDefaultCharset UTF-8

#
# AddType allows you to add to or override the MIME configuration
# file mime.types for specific file types.
#
#AddType application/x-tar .tgz

#
# AddEncoding allows you to have certain browsers uncompress
# information on the fly. Note: Not all browsers support this.
# Despite the name similarity, the following Add* directives have nothing
# to do with the FancyIndexing customization directives above.
#
#AddEncoding x-compress .Z
#AddEncoding x-gzip .gz .tgz

# If the AddEncoding directives above are commented-out, then you
# probably should define those extensions to indicate media types:
#
AddType application/x-compress .Z
AddType application/x-gzip .gz .tgz

#
#   MIME-types for downloading Certificates and CRLs
#
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl    .crl

#
# AddHandler allows you to map certain file extensions to "handlers":
# actions unrelated to filetype. These can be either built into the server
# or added with the Action directive (see below)
#
# To use CGI scripts outside of ScriptAliased directories:
# (You will also need to add "ExecCGI" to the "Options" directive.)
#
#AddHandler cgi-script .cgi

#
# For files that include their own HTTP headers:
#
#AddHandler send-as-is asis

#
# For type maps (negotiated resources):
# (This is enabled by default to allow the Apache "It Worked" page
#  to be distributed in multiple languages.)
#
AddHandler type-map var

#
# Filters allow you to process content before it is sent to the client.
#
# To parse .shtml files for server-side includes (SSI):
# (You will also need to add "Includes" to the "Options" directive.)
#
AddType text/html .shtml
AddOutputFilter INCLUDES .shtml

#
# Action lets you define media types that will execute a script whenever
# a matching file is called. This eliminates the need for repeated URL
# pathnames for oft-used CGI file processors.
# Format: Action media/type /cgi-script/location
# Format: Action handler-name /cgi-script/location
#

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# Putting this all together, we can internationalize error responses.
#
# We use Alias to redirect any /error/HTTP_<error>.html.var response to
# our collection of by-error message multi-language collections.  We use 
# includes to substitute the appropriate text.
#
# You can modify the messages' appearance without changing any of the
# default HTTP_<error>.html.var files by adding the line:
#
#   Alias /error/include/ "/your/include/path/"
#
# which allows you to create your own set of files by starting with the
# /var/www/error/include/ files and
# copying them to /your/include/path/, even on a per-VirtualHost basis.
#

Alias /error/ "/var/www/error/"

<IfModule mod_negotiation.c>
<IfModule mod_include.c>
    <Directory "/var/www/error">
        AllowOverride None
        Options IncludesNoExec
        AddOutputFilter Includes html
        AddHandler type-map var
        Order allow,deny
        Allow from all
        LanguagePriority en es de fr
        ForceLanguagePriority Prefer Fallback
    </Directory>

#    ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
#    ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
#    ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
#    ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var
#    ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var
#    ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var
#    ErrorDocument 410 /error/HTTP_GONE.html.var
#    ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var
#    ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var
#    ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var
#    ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var
#    ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var
#    ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var
#    ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var
#    ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var
#    ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var
#    ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var

</IfModule>
</IfModule>

#
# The following directives modify normal HTTP response behavior to
# handle known problems with browser implementations.
#
BrowserMatch "Mozilla/2" nokeepalive
BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
BrowserMatch "RealPlayer 4\.0" force-response-1.0
BrowserMatch "Java/1\.0" force-response-1.0
BrowserMatch "JDK/1\.0" force-response-1.0

#
# The following directive disables redirects on non-GET requests for
# a directory that does not include the trailing slash.  This fixes a 
# problem with Microsoft WebFolders which does not appropriately handle 
# redirects for folders with DAV methods.
# Same deal with Apple's DAV filesystem and Gnome VFS support for DAV.
#
BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
BrowserMatch "MS FrontPage" redirect-carefully
BrowserMatch "^WebDrive" redirect-carefully
BrowserMatch "^WebDAVFS/1.[0123]" redirect-carefully
BrowserMatch "^gnome-vfs/1.0" redirect-carefully
BrowserMatch "^XML Spy" redirect-carefully
BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully

#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Change the ".example.com" to match your domain to enable.
#
#<Location /server-status>
#    SetHandler server-status
#    Order deny,allow
#    Deny from all
#    Allow from .example.com
#</Location>

#
# Allow remote server configuration reports, with the URL of
#  http://servername/server-info (requires that mod_info.c be loaded).
# Change the ".example.com" to match your domain to enable.
#
#<Location /server-info>
#    SetHandler server-info
#    Order deny,allow
#    Deny from all
#    Allow from .example.com
#</Location>

#
# Proxy Server directives. Uncomment the following lines to
# enable the proxy server:
#
#<IfModule mod_proxy.c>
#ProxyRequests On
#
#<Proxy *>
#    Order deny,allow
#    Deny from all
#    Allow from .example.com
#</Proxy>

#
# Enable/disable the handling of HTTP/1.1 "Via:" headers.
# ("Full" adds the server version; "Block" removes all outgoing Via: headers)
# Set to one of: Off | On | Full | Block
#
#ProxyVia On

#
# To enable a cache of proxied content, uncomment the following lines.
# See http://httpd.apache.org/docs/2.2/mod/mod_cache.html for more details.
#
#<IfModule mod_disk_cache.c>
#   CacheEnable disk /
#   CacheRoot "/var/cache/mod_proxy"
#</IfModule>
#

#</IfModule>
# End of proxy directives.

### Section 3: Virtual Hosts
#
# VirtualHost: If you want to maintain multiple domains/hostnames on your
# machine you can setup VirtualHost containers for them. Most configurations
# use only name-based virtual hosts so the server doesn't need to worry about
# IP addresses. This is indicated by the asterisks in the directives below.
#
# Please see the documentation at 
# <URL:http://httpd.apache.org/docs/2.2/vhosts/>
# for further details before you try to setup virtual hosts.
#
# You may use the command line option '-S' to verify your virtual host
# configuration.

#
# Use name-based virtual hosting.
#
#NameVirtualHost *:80
#
# NOTE: NameVirtualHost cannot be used without a port specifier 
# (e.g. :80) if mod_ssl is being used, due to the nature of the
# SSL protocol.
#

#
# VirtualHost example:
# Almost any Apache directive may go into a VirtualHost container.
# The first VirtualHost section is used for requests without a known
# server name.
#
#<VirtualHost *:80>
#    ServerAdmin webmaster@dummy-host.example.com
#    DocumentRoot /www/docs/dummy-host.example.com
#    ServerName dummy-host.example.com
#    ErrorLog logs/dummy-host.example.com-error_log
#    CustomLog logs/dummy-host.example.com-access_log common
#</VirtualHost>
<VirtualHost *:80>
    ServerAdmin joe.dutka@acer-isu.com
    ServerName acer-isu.com
    ServerAlias www.acer-isu.com
    DocumentRoot /var/www/html
    ErrorLog /var/log/httpd/error.log
    CustomLog /var/log/httpd/access.log combined
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.acer-isu.com [OR]
RewriteCond %{SERVER_NAME} =acer-isu.com
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
Include /etc/httpd/conf/httpd-le-ssl.conf
<VirtualHost *:10025>
     ServerName mail.acer-isu.com
     DocumentRoot /var/www/html
</VirtualHost>

httpd-le-ssl.conf <-- I believe this is automated by certbot:

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerAdmin joe.dutka@acer-isu.com
    ServerName acer-isu.com
    ServerAlias www.acer-isu.com
    DocumentRoot /var/www/html
    ErrorLog /var/log/httpd/error.log
    CustomLog /var/log/httpd/access.log combined

SSLCertificateFile /etc/letsencrypt/live/acer-isu.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/acer-isu.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

This had all worked before. The only changes made to the server was it's location moved resulting in an IP update. There is a direct link from the router to the server so there shouldn't be any firewall in the way - and I have 80/tcp and 443/tcp open with firewalld running on the server. Server itself is in a VM bridged over Ethernet.

14

I just tried the --cert-name suggestion (at least I think I did). Still no dice....

certbot renew --cert-name acer-isu.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/acer-isu.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for acer-isu.com and 2 more domains
Failed to renew certificate acer-isu.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/acer-isu.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Could there be something blocking at the ISP level?

15

If the router is connected directly to your ISP then it most likely has a firewall function built-in. Is there any other network gear between the router and your ISP?

I have posted to an internal group to get wider audience. While we wait for that there is something you could try. The outbound request that fails is much larger than the ones that worked. You redacted the payload info in your log above but I saw the size looking at my own logs.

So, can you show results of these commands?

ping -c 5 -s 2000 acme-v02.api.letsencrypt.org

ping -c 5 -s 100  acme-v02.api.letsencrypt.org
3 Likes
16

Only one of the three names from port 443 is being served via port 80.

And the use of this file: /etc/httpd/conf.d/ssl.conf
makes me question if it was setup properly for the current use.

2 Likes
17

@JoeAcerIsu Adding on to my prior post ... I got a suggestion from a certbot dev. He said when Let's Encrypt used a different provider a Finalize request might fail due to certain MTU size problems. That's something set in your network components. He did not remember seeing one with the current provider (Cloudflare) but it is worth trying. The certbot Finalize request that fails is the largest request body, by far, compared to the other requests that worked (the preliminary requests in the certbot sequence).

He suggested lowering your MTU size dramatically. You will need to consult with a network expert for this to ensure you don't modify it incorrectly and cause other problems.

It might be worth trying the pings I showed. If just one fails it will be easier to test new MTU changes. It is also worth trying ping with even larger request like -s 6000

3 Likes
18

Hi Rudy,

Well I can move the mail.acer-isu.com domain to align with the way the acer-isu.com is set up and restart the Apache server. Not sure if there would be any side effect. The mail.acer-isu.com was set that way manually because certbot --apache will only (obviously) auto generate for the Apache service not the Postfix and Dovecot that we also use.

Again I it has worked and does work now, just the renew is suddenly failing.

Thanks - Joe

2 Likes
19

Hi Mike,

Thanks for the suggestion on the MTU - I'll see if maybe there is a difference somewhere in the pipe.

Here is the output you requested.

[joe.dutka@mail ~]$ ping -c 5 -s 2000 acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 2000(2028) bytes of data.
2008 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=57 time=135 ms
2008 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=2 ttl=57 time=139 ms
2008 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=3 ttl=57 time=137 ms
2008 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=4 ttl=57 time=135 ms
2008 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=5 ttl=57 time=138 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms
rtt min/avg/max/mdev = 135.123/136.899/138.834/1.420 ms
[joe.dutka@mail ~]$ ping -c 5 -s 100  acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 100(128) bytes of data.
108 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=57 time=138 ms
108 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=2 ttl=57 time=137 ms
108 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=3 ttl=57 time=135 ms
108 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=4 ttl=57 time=138 ms
108 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=5 ttl=57 time=137 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 135.277/137.100/138.121/1.071 ms
[joe.dutka@mail ~]$ ping -c 5 -s 9999  acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 9999(10027) bytes of data.
10007 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=57 time=140 ms
10007 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=2 ttl=57 time=138 ms
10007 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=3 ttl=57 time=142 ms
10007 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=4 ttl=57 time=140 ms
10007 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=5 ttl=57 time=139 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 138.350/139.829/142.468/1.446 ms
2 Likes
20

Hi Mike,

I checked the MTU of the host system and the VM and they match at 1500. Not sure what else I can try...?

Do you think it makes sense to simply try to get a new certificate? I think in point of fact, while I incorrectly have 2 certificates, I only use one for the acer-isu.com and mail.acer-isu.com. I can probably drop a certificate, though that doesn't seem to help.

Thanks again for all the help.

-Joe

1 Like