Cannot renew with Certbot "Connection refused by peer"

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: wiki.ceas.wmich.edu

I ran this command: certbot -v renew --dry-run

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/wiki.ceas.wmich.edu.conf

Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Simulating renewal of an existing certificate for wiki.ceas.wmich.edu
Performing the following challenges:
http-01 challenge for wiki.ceas.wmich.edu
Waiting for verification...
Challenge failed for domain wiki.ceas.wmich.edu
http-01 challenge for wiki.ceas.wmich.edu

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: wiki.ceas.wmich.edu
Type: connection
Detail: Fetching http://wiki.ceas.wmich.edu/.well-known/acme-challenge/4gsNLk776etwY1JQjkT_Rvj2mLSSzaSxDYKFL83ILN0: Connection reset by peer

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate wiki.ceas.wmich.edu with error: Some challenges have failed.

All simulated renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/wiki.ceas.wmich.edu/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): Apache 2.4.29

The operating system my web server runs on is (include version):Ubuntu 18.04

My hosting provider, if applicable, is: N/A

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.26.0

This website is a wiki used by a public university. You should be able to access it but not read anything without university credentials. Quick note, I cannot ping or access:
wiki.ceas.wmich.edu/.well-known/acme-challenge
in any way. Not through the browser or curl. Suspicious this might be the issue.

I have tried looking through so many log files and internet posts to figure this whole thing out. I have looked at every single forum post about this that includes the errors: "The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot." and "Connection reset by peer" and cannot find a solution that works.

Would be happy to provide any log file outputs you would need.

Any advice is helpful because this SSL cert expires in 2 days...

2

Check your access logs, your firewall rules, your apache config, your htaccess files...

If you cannot access this link, it will not work (it's fine if it's a 404, but it must allow you to connect): http://wiki.ceas.wmich.edu/.well-known/acme-challenge/4gsNLk776etwY1JQjkT_Rvj2mLSSzaSxDYKFL83ILN0

1 Like
3

Does that folder actually physically exist? I can't find it anywhere in my files? Does it have a default location?

4

No, not with the Apache plug-in which you used. Certbot temporarily changes the Apache config to respond directly to a challenge with that path.

Which means you cannot have a firewall or something block attempts for that before it gets to Apache

UPDATE:
You can look at /var/log/letsencrypt/letsencrypt.log for the temp changes the plug-in makes

2 Likes
5

This is all I get in letsencrypt.log

2022-04-06 14:55:43,092:DEBUG:certbot._internal.display.obj:Notifying user:

2022-04-06 14:55:43,093:ERROR:certbot._internal.renewal:All simulated renewals failed. The following certificates could not be renewed:
2022-04-06 14:55:43,093:ERROR:certbot._internal.renewal: /etc/letsencrypt/live/wiki.ceas.wmich.edu/fullchain.pem (failure)
2022-04-06 14:55:43,093:DEBUG:certbot._internal.display.obj:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2022-04-06 14:55:43,093:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/snap/certbot/1952/bin/certbot", line 8, in
sys.exit(main())
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/main.py", line 1723, in main
return config.func(config, plugins)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/main.py", line 1609, in renew
renewal.handle_renewal_request(config)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 510, in handle_renewal_request
raise errors.Error(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2022-04-06 14:55:43,093:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

6

I think your webserver configuration almost certainly has special configuration set up for /.well-known/acme-challenge/ requests, that we're not aware of. It is causing the request to be misdirected.

Compare:

$ curl -i http://wiki.ceas.wmich.edu/.well-known/acme-challenge/test
curl: (52) Empty reply from server

against:

$ curl -i http://wiki.ceas.wmich.edu/.xwell-known/acme-challenge/test
HTTP/1.1 301 Moved Permanently
Date: Wed, 06 Apr 2022 22:09:51 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https://wiki.ceas.wmich.edu/.xwell-known/acme-challenge/test
Content-Length: 353
Content-Type: text/html; charset=iso-8859-1

Grep around for well-known inside your webserver configuration. Check if you have something like mod_security installed as well.

2 Likes
7

Okay so I checked my apache logs and my firewall. Apache has nothing out of the ordinary and I completely disabled the firewall for the heck of it momentarily and that didn't help either. I don't believe I have any .htaccess files. I couldn't find any of them. Nothing seems to stick out.
Something of note: I cannot access http://wiki.ceas.wmich.edu/.well-known/acme-challenge/4gsNLk776etwY1JQjkT_Rvj2mLSSzaSxDYKFL83ILN0
because whenever I try to visit this link, the link changes to https at port 443 when I feel like it should be pointing at port 80 for apache http. Even erasing https:// completely still adds it back. Is there some sort of setting to turn this off for apache? Any advice on where to go from here?

8

It doesn't redirect to https for me. It just resets the connection.

There is literally a rule somewhere in Apache or some kind of WAF that's blocking the validation.

% curl -IL http://wiki.ceas.wmich.edu/.well-known/acme-challenge
HTTP/1.1 301 Moved Permanently
Date: Wed, 06 Apr 2022 23:22:40 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https://wiki.ceas.wmich.edu/.well-known/acme-challenge
Content-Type: text/html; charset=iso-8859-1

HTTP/1.1 404 Not Found
Date: Wed, 06 Apr 2022 23:22:40 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1

% curl -IL http://wiki.ceas.wmich.edu/.well-known/acme-challenge/
curl: (56) Recv failure: Connection reset by peer
%
2 Likes
9

You can also watch Apache's error_log and access_log to see what happens when you try and request http://wiki.ceas.wmich.edu/.well-known/acme-challenge/test. If the request is being blocked at Apache, some detail may appear there.

If the request is being blocked earlier than Apache, you may see nothing at all, though that's not definitive.

3 Likes
10

I think I found the problem. In letsencrypt.log I found this:

2022-04-07 08:49:59,165:DEBUG:acme.client:Storing nonce: 0001miEH3gh_0nMX7-jAU3LJkEQWW9w_XWHZuk5RXbssTo8
2022-04-07 08:49:59,165:INFO:certbot.auth_handler:Performing the following challenges:
2022-04-07 08:49:59,165:INFO:certbot.auth_handler:http-01 challenge for wiki.ceas.wmich.edu
2022-04-07 08:49:59,215:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: wiki.ceas.wmich.edu in: /etc/apache2/sites-enabled/000-default.conf
2022-04-07 08:49:59,216:DEBUG:certbot_apache.http_01:writing a pre config file with text:
RewriteEngine on
RewriteRule ^/.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

2022-04-07 08:49:59,216:DEBUG:certbot_apache.http_01:writing a post config file with text:
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted

<Location /.well-known/acme-challenge>
Require all granted

Notice the RewriteRule. Is there a way to turn off RewriteRule for lets encrypt or can I find the rewrite rule somewhere and delete it? I do this through apache if that helps.

11

The rewrite rule is purposely put there by the plugin. If it doesn't work, it's either due to some non-standard or buggy Apache configuration or something else interfering with the temporary configuration file.

If you can't find and fix the interfering Apache configuration, you could try the webroot authenticator plugin. This can be combined with the apache installer plugin by using -i apache -a webroot.

3 Likes
13

Could you give me the full command for downloading this plugin? Also can you explain exactly how it works? Does it just force the renewal to work or does it point out better what in Apache is breaking the renewal process or something else?

14

Please note that the snippit of the log above with the rewrite rule should only be temporary. After Certbot has finished, it should be removed from the Apache configuration.

As such it should not be present at the moment and if it is, it should be removed. If it is not currently present, then your issue is with something else.

For using Certbot I will refer to the Certbot documentation:

https://eff-certbot.readthedocs.io/en/stable/using.html

I'm not a big fan of providing complete commands. I encourage users to learn more about stuff so they understand what they're doing instead of copy/pasting complete commands without any understanding what it does.

Also note that, while not documented clearly, the webroot plugin is always available, compared to e.g. the apache plugin which usually requires separate installation.

2 Likes
15

Honestly, I am with you on making people learn, but I have less than 2 days to get this fixed or the university is going to crackdown on our use of self signed certificates which we DO NOT want AT ALL. I really just need some help here. Can you explain to me how webroot works and how to use it and/or install it please? I am in quite a pickle here.

16

It's not rocket science, have you already read the documentation about the webroot plugin?

I cannot give you a complete command, even if I wanted to, as it requires some specifics I don't know yet.

1 Like
17

Yeah I did read it but I do not know what to provide for the webroot path.
I am simply running right now:

certbot run -a webroot -i apache

then it asks for my servers webroot and I don't know what that means

18

From the documentation:

In addition, you’ll need to specify --webroot-path or -w with the top-level directory (“web root”) containing the files served by your webserver. For example, --webroot-path /var/www/html or --webroot-path /usr/share/nginx/html are two common webroot paths.

That doesn't ring any bell? Anything like that in your Apache configuration files present?

2 Likes
19

For Apache it is the folder named by the DocumentRoot directive in effect for the VirtualHost

2 Likes
20

Okay I get what you are saying now. I ran my command and gave it /var/www/html as the webroot and it still did not work or give any new info.

ERROR MESSAGE:
Failed authorization procedure. wiki.ceas.wmich.edu (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://wiki.ceas.wmich.edu/.well-known/acme-challenge/jXErKDTVrN3Bps6fTSAA4-jwpBKNqarxV50ZDYLg5Dc: Connection reset by peer

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: wiki.ceas.wmich.edu
    Type: connection
    Detail: Fetching
    http://wiki.ceas.wmich.edu/.well-known/acme-challenge/jXErKDTVrN3Bps6fTSAA4-jwpBKNqarxV50ZDYLg5Dc:
    Connection reset by peer

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you're using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

Where do I got form here?

21

As long as your Apache configuration is severely broken, the webroot plugin probably won't help unfortunately.

There have been some other suggestion which you haven't followed up yet, such as the error_log and access_log mentioned earlier by _az.

Also searching every Apache configuration file for any mention of .well-known and/or acme-challenge (preferably the combination of both) would help.

2 Likes