"Connection refused" error?

gilgongo · September 10, 2019, 12:14pm

At first I thought this was a problem introduced by a recent firewall change (and putting all port http requests though a proxy). But even if I turn off the firewall I get this problem.

It used to work in the past - but now it doesn’t!

My domain is: alice.hatters.org.uk

I ran this command (daily cronjob /etc/cron.d/certbot):

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew -i apache -a apache/etc/cron.d/certbot

It produced this output:

2019-09-08 09:07:45,267:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7f886d64d080> and installer None
2019-09-08 09:07:45,268:INFO:certbot.plugins.selection:Plugins selected: Authenticator webroot, Installer None
2019-09-08 09:07:45,270:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v01.api.letsencrypt.org/acme/reg/308104', new_authzr_uri=None, terms_of_service=None), 673298e24fb1e70065be14007c4535ca, Meta(creation_dt=datetime.datetime(2016, 1, 24, 15, 4, 50, tzinfo=<UTC>), creation_host='770b6b6a9c37'))>
2019-09-08 09:07:45,271:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-09-08 09:07:45,273:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): [acme-v02.api.letsencrypt.org](http://acme-v02.api.letsencrypt.org/)
2019-09-08 09:07:48,600:WARNING:certbot.renewal:Attempting to renew cert (alice.hatters.org.uk) from /etc/letsencrypt/renewal/alice.hatters.org.uk.conf produced an unexpected error: Requesting [acme-v02.api.letsencrypt.org/directory](http://acme-v02.api.letsencrypt.org/directory): Connection refused. Skipping.
2019-09-08 09:07:48,604:DEBUG:certbot.renewal:Traceback was:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 141, in _new_conn (self.host, self.port), self.timeout, **extra_kw)
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 83, in create_connection raise err
File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 73, in create_connection sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 852, in _validate_conn conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 284, in connect conn = self._new_conn()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7f886d61f978>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 440, in send timeout=timeout
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 639, in urlopen

My web server is (include version):

Apache 2.4.29

The operating system my web server runs on is (include version):

Ubuntu 18.04.3

My hosting provider , if applicable, is:

Jump Networks Ltd

I can login to a root shell on my machine (yes or no, or I don’t know):

Yes

I’m not using a control panel

The version of my client :

0.31.0

mnordhoff · September 10, 2019, 12:31pm

What does “curl -v https://acme-v02.api.letsencrypt.org/directory” output?

gilgongo · September 10, 2019, 12:34pm

It looks like it’s going out through the proxy (localhost 8888) OK…

* Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8888 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to [acme-v02.api.letsencrypt.org:443](http://acme-v02.api.letsencrypt.org:443/)
> CONNECT [acme-v02.api.letsencrypt.org:443](http://acme-v02.api.letsencrypt.org:443/) HTTP/1.1
> Host: [acme-v02.api.letsencrypt.org:443](http://acme-v02.api.letsencrypt.org:443/)
> User-Agent: curl/7.58.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.0 200 Connection established
< Proxy-agent: tinyproxy/1.8.4
<
* Proxy replied 200 to CONNECT request
* CONNECT phase completed!
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* (304) (OUT), TLS handshake, Client hello (1):
* CONNECT phase completed!
* CONNECT phase completed!
* (304) (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: [CN=acme-v02.api.letsencrypt.org](http://cn%3Dacme-v02.api.letsencrypt.org/)
* start date: Jul 19 04:46:54 2019 GMT
* expire date: Oct 17 04:46:54 2019 GMT
* subjectAltName: host "[acme-v02.api.letsencrypt.org](http://acme-v02.api.letsencrypt.org/)" matched cert's "[acme-v02.api.letsencrypt.org](http://acme-v02.api.letsencrypt.org/)"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
> GET /directory HTTP/1.1
> Host: [acme-v02.api.letsencrypt.org](http://acme-v02.api.letsencrypt.org/)
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/json
< Content-Length: 658
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Expires: Tue, 10 Sep 2019 12:32:37 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Tue, 10 Sep 2019 12:32:37 GMT
< Connection: keep-alive
<
{
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"lLfKmPZhOZM": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
"meta": {
"caaIdentities": [
"[letsencrypt.org](http://letsencrypt.org/)"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "[https://letsencrypt.org](https://letsencrypt.org/)"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host localhost left intact

gilgongo · September 10, 2019, 12:37pm

This also supports the idea that certbot isn’t picking up the proxy environment variable perhaps, but in the cron job I’m currently taking the firewall down before the renewal job so in theory the requests should go straight out over port 443/80.

mnordhoff · September 10, 2019, 12:39pm

I have the memory of a goldfish, I forgot about your proxy setup. I don’t know how to debug Python proxying issues.

Since you’re using Ubuntu 18.04, the cron job isn’t running.

When you get that error, is it from Certbot executed by the systemd timer?

gilgongo · September 10, 2019, 12:50pm

Really? So what's doing the renewals?

I believe it's from systemd - but I'm not sure. I assumed it was from the above cron command.

gilgongo · September 10, 2019, 9:49pm

If I put this in a crontab:

* * * * * curl -v https://acme-v02.api.letsencrypt.org/directory 2> /tmp/test.txt

The contents of test.txt looks fine - it's running the request though the proxy as it did above when I ran it manually. And the firewall (which blocks outgoing port 443 and 80) is up.

So is it something to do with systemd, or some other problem perhaps?

gilgongo · September 12, 2019, 6:59pm

OK so out of desperation I renewed manually with just the following and it worked (although one site errored with “client lacks sufficient authorization” :

certbot -q renew -i apache -a apache --preferred-challenges http

So I now have another three months to try to fix the cron job.

What does systemd command in the cronjob do? Does it need to be there? I think it got installed by default from the Ubuntu repos though so I’m reluctant to mess with it.

And Does anyone know what @mnordhoff meant when he said "Since you’re using Ubuntu 18.04, the cron job isn’t running"?

mnordhoff · September 12, 2019, 7:05pm

You need to fix the systemd service. Ignore the cron job.

I haven't worked with changing environment variables in systemd services, but Phil_LE linked to a guide in your previous thread:

It stops the cron job from doing anything. The cron job doesn't execute Certbot. (On systems using systemd.)

Yes.

Yes, it did.

gilgongo · September 12, 2019, 7:13pm

Thanks - the guide Phil_LE linked to didn’t contain anything relevant to my circumstance that I could tell (I’m not running Docker or any containers), but I found some other info about giving global envvars to system units and tried that but it did nothing.

I’m afraid I don’t understand your other comments about systemd stopping the cron job yet needing to be there. But rather than take up any of your time, I think I can continue to research a solution now that at least my sites are up for the next three months.

mnordhoff · September 12, 2019, 7:23pm

I don’t know how else to say it…

Pretend that the cron job does not exist. It does not do anything.

You should not modify the cron job – in the best case, nothing will change. In the worst case, it will start doing things.

You can delete it, but it’s not doing any harm, and it might get recreated later anyway (I’m not sure).

gilgongo · September 12, 2019, 8:56pm

OK I won’t get hung up on why there are things in place that stop other things running that don’t have to run. It is what it is.

Instead, you indicate that the renewal run is performed by systemd. I didn’t know that systemd had that capability, so I’ll investigate that angle in the light of what I currently believe to be true:

That certbot is running twice a day (at 12:00 and 00:00) to do automated renewals via systemd, not cron.
That certbot was auto-renewing before, but something changed to make it stop working. This change was possibly the introduction of a forward proxy.
That certbot, when it’s doing 1, is not using the proxy and is throwing an error about being unable to connect (and there’s no indication of certbot connections in the proxy logs).
That I have attempted to pass the proxy variable to systemd using DefaultEnvironment= in /etc/systemd/system.conf but that this seems to have done nothing.

gilgongo · September 12, 2019, 9:11pm

OK so I see that there is file /lib/systemd/system/certbot.timer which runs the certbot.service. The latter file has this in it:

ExecStart=/usr/bin/certbot -q renew

I note that the crontab renew command was:

certbot -q renew -i apache -a apache --preferred-challenges http

Which I seem to recall is the correct renewal command, so I’ll use that and see. I’ll also try passing the proxy environment to the service as well with EnvironmentFile=/etc/environment for good measure.

schoen · September 12, 2019, 9:29pm

If you used --apache to get your certificate originally, -i apache -a apache --preferred-challenges http is redundant with what certbot -q renew will already do with respect to that certificate!

gilgongo · September 12, 2019, 9:31pm

Ah OK. Thanks!

Passing the proxy environment to the service seems to have done the trick, and I now see certbot working OK except for one domain which is throwing an error. I’ll see if I can investigate that.

schoen · September 12, 2019, 9:31pm

Cool, I’m glad you’re making progress!

system · October 12, 2019, 9:31pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.