Issue a new certificate connection refused

Help
1

Hello,
I got ‘connection refused’ response, then I tried several troubleshoot but doesn’t seem helpful so far.

  1. Tried with --debug-challenges, the challenge URL is accessible from browser, had no idea why the server cannot access it
  2. https://letsdebug.net/jakartatimur.imigrasi.go.id/79943 said
    Get http://jakartatimur.imigrasi.go.id/.well-known/acme-challenge/letsdebug-test: dial tcp 180.178.97.84:80: connect: connection refused
    I created the same filename ‘letsdebug-test’ and opened thru browser, it worked.
  3. https://check-your-website.server-daten.de/?q=jakartatimur.imigrasi.go.id - section 5. Url-Checks said all ok except https request.

My domain is: jakartatimur.imigrasi.go.id

I ran this command:
sudo certbot --debug-challenges --apache -d jakartatimur.imigrasi.go.id -v

It produced this output:
Reporting to user: The following errors were reported by the server:

Domain: jakartatimur.imigrasi.go.id
Type: connection
Detail: Fetching http://jakartatimur.imigrasi.go.id/.well-known/acme-challenge/pH7JjiJiz35MTuMDB6dIyVhxil-e4QKUFkBqDiC2sEI: Connection refused

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA
record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you’re using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
Encountered exception:
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 155, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 226, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. jakartatimur.imigrasi.go.id (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the
domain :: Fetching http://jakartatimur.imigrasi.go.id/.well-known/acme-challenge/pH7JjiJiz35MTuMDB6dIyVhxil-e4QKUFkBqDiC2sEI: Connection refused

Calling registered functions
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.26.1’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1364, in main
return config.func(config, plugins)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1124, in run
certname, lineage)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 120, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 391, in obtain_and_enroll_certificate cert, chain, key, _ = self.obtain_certificate(domains)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 334, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 370, in _get_order_and_authorizations authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 155, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/usr/lib/python3/dist-packages/certbot/auth_handler.py”, line 226, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. jakartatimur.imigrasi.go.id (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the
domain :: Fetching http://jakartatimur.imigrasi.go.id/.well-known/acme-challenge/pH7JjiJiz35MTuMDB6dIyVhxil-e4QKUFkBqDiC2sEI: Connection refused
Failed authorization procedure. jakartatimur.imigrasi.go.id (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://jakartatimur.imigrasi.go.id/.well-known/acme-challenge/pH7JjiJiz35MTuMDB6dIyVhxil-e4QKUFkBqDiC2sEI: Connection refused

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: jakartatimur.imigrasi.go.id
    Type: connection
    Detail: Fetching
    http://jakartatimur.imigrasi.go.id/.well-known/acme-challenge/pH7JjiJiz35MTuMDB6dIyVhxil-e4QKUFkBqDiC2sEI:
    Connection refused

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

My web server is (include version): Apache/2.4.29 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 18.04.1

My hosting provider, if applicable, is: Self hosted

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.26.1

2

The “connection refused” error indicates that your server is not actually listening with Apache on port 80 at the moment. And indeed, that seems to be the case:

$ curl -X GET -I jakartatimur.imigrasi.go.id
curl: (7) Failed to connect to jakartatimur.imigrasi.go.id port 80: Connection refused

Is 180.178.97.84 definitely the correct IP address of your server?

If so, I noticed that you are also listening on port 443 with non-HTTPS: http://jakartatimur.imigrasi.go.id:443 .

If you have changed port 80 to 443 somewhere to cause this, you will need to change it back to port 80 in order to be able to issue a certificate.

Certbot will automatically setup a port 443 HTTPS virtualhost for you afterwards.

1 Like
3

Thanks for the response @_az

Where is your location from? It worked fine from JKT

curl -X GET -I jakartatimur.imigrasi.go.id
HTTP/1.1 200 OK
Date: Thu, 21 Nov 2019 06:53:00 GMT
Server: Apache/2.4.29 (Ubuntu)
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
Set-Cookie: haircki=haircooki; expires=Fri, 22-Nov-2019 06:53:00 GMT; Max-Age=86400
X-Pingback: http://jakartatimur.imigrasi.go.id/wp/xmlrpc.php
Link: <http://jakartatimur.imigrasi.go.id/wp-json/>; rel="https://api.w.org/"      
Link: <http://jakartatimur.imigrasi.go.id/>; rel=shortlink
Vary: Accept-Encoding
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8

Yes

Okay, I have reverted it.

4

I am trying from a couple of different locations - both behave the same way.

After your change, there is a curious symptom.

The first curl request fails with “connection refused”. Subsequent requests then succeed.

But if I want a long enough time between requests - say, more than 10 seconds - it will get “connection refused” again.

I’m pretty sure that Let’s Encrypt is observing the same behavior from your server.

1 Like
5

Here is a way to reproduce the problem from an external perspective:

If we sleep for 10 seconds between requests in a loop, they (almost) all fail:

# for i in `seq 1 5`; do curl -I http://180.178.97.84; sleep 10; done
curl: (7) Failed to connect to 180.178.97.84 port 80: Connection refused
curl: (7) Failed to connect to 180.178.97.84 port 80: Connection refused
curl: (7) Failed to connect to 180.178.97.84 port 80: Connection refused
HTTP/1.1 301 Moved Permanently
Date: Thu, 21 Nov 2019 07:14:04 GMT
Server: Apache/2.4.29 (Ubuntu)
Set-Cookie: haircki=haircooki; expires=Fri, 22-Nov-2019 07:14:04 GMT; Max-Age=86400
X-Pingback: http://jakartatimur.imigrasi.go.id/wp/xmlrpc.php
X-Redirect-By: WordPress
Location: http://jakartatimur.imigrasi.go.id/
Content-Type: text/html; charset=UTF-8

curl: (7) Failed to connect to 180.178.97.84 port 80: Connection refused

Increasing to 20 seconds, they all definitely fail.

But if I reduce the sleep from 10 seconds to 1 second or none at all, all the requests succeed (except the first one of course).

1 Like
6

That’s quite strange. I even tried with 20s sleep, all succeeded. FYI, my internet is using different ISP then my server use.
And as you can see the URL-checks here https://check-your-website.server-daten.de/?q=jakartatimur.imigrasi.go.id, doesn’t seem a problem.

7

Since https://letsdebug.net/jakartatimur.imigrasi.go.id/79956 can reproduce the problem, you might have luck re-running the test a few times via Let’s Debug while running a packet capture from your server, e.g.

sudo tcpdump -i "eth0" "host 172.104.24.29"

You might be able to catch what’s happening with the TCP handshake that results in connection refused. Maybe an upstream ISP is sending the RST and your server’s ACK is not making it to the client.

8

Hi @rasm

“check your website” -> server are in Berlin, Germany.

But if letsdebug doesn’t work, you must have regional settings (firewall or something else), that blocks.

Or it’s a network problem.

9

I believe the captured packet during letsdebug test is the following

17:10:59.741692 IP (tos 0x0, ttl 52, id 43904, offset 0, flags [DF], proto TCP (6), length 160)
    180.178.97.82.52166 > 172.100.1.2.80: Flags [P.], cksum 0xabdc (correct), seq 1:109, ack 1, win 229, options [nop,nop,TS val 1964359664 ecr 3279173835], length 108: HTTP, length: 108
        GET / HTTP/1.1
        Host: jakartatimur.imigrasi.go.id
        User-Agent: Go-http-client/1.1
        Accept-Encoding: gzip

17:10:59.741730 IP (tos 0x0, ttl 64, id 14559, offset 0, flags [DF], proto TCP (6), length 52)
    172.100.1.2.80 > 180.178.97.82.52166: Flags [.], cksum 0xf7e8 (correct), seq 1, ack 109, win 227, options [nop,nop,TS val 3279174083 ecr 1964359664], length 0
17:10:59.900145 IP (tos 0x0, ttl 64, id 14560, offset 0, flags [DF], proto TCP (6), length 1500)
    172.100.1.2.80 > 180.178.97.82.52166: Flags [.], cksum 0xad79 (correct), seq 1:1449, ack 109, win 227, options [nop,nop,TS val 3279174242 ecr 1964359664], length 1448: HTTP, length: 1448
        HTTP/1.1 200 OK
        Date: Thu, 21 Nov 2019 10:10:59 GMT
        Server: Apache/2.4.29 (Ubuntu)
        Set-Cookie: haircki=haircooki; expires=Fri, 22-Nov-2019 10:10:59 GMT; Max-Age=86400
        X-Pingback: http://jakartatimur.imigrasi.go.id/wp/xmlrpc.php
        Link: <http://jakartatimur.imigrasi.go.id/wp-json/>; rel="https://api.w.org/"
        Link: <http://jakartatimur.imigrasi.go.id/>; rel=shortlink
        Vary: Accept-Encoding
        Content-Encoding: gzip
        Content-Length: 22580
        Content-Type: text/html; charset=UTF-8

Most likely 180.178.97.82 is the router that my server is behind and configured to forward the traffic.

10

Since my server is behind NAT, any other checks to pass?

11

I have no idea. You can connect Letsencrypt and you are able to start a new order. But Letsencrypt can’t connect your server.

May be the Letsencrypt log has more informations.

You can - one time - use dns validation and --manual, that should always work.

–

PS: There https://www.uptrends.de/tools/uptime

id
2019-11-21.jakartatimur.imigrasi.go.id.png642×832 62.8 KB

is no problem visible.

–>> Ask your hoster.

12

Thanks. That seems to be a confirmation of the timing theory.

The test sends two requests. The first is to /.well-known/acme-challenge/letsdebug-test, and the second is to /.

If you only observed the second request in the packet capture, it mean the first one was dropped by your NAT gateway, or never made it to your NAT gateway.

This is confirmed by the test result itself, and it matches the timing theory in that the first failed request allows the second request to succeed, as with the shell example.

Do you have any kind of “firewall” or “anti-DDoS” features on your router/NAT gateway? I’ve seen a sort of similar symptom on a previous occasion that was caused by that kind of feature. Alternatively, are you able to run the packet capture from the NAT gateway?

13

Perhaps your site is hacked.

Checked with local tools to see, if I can see the same @_az has seen.

Then:

D:\temp>download http://jakartatimur.imigrasi.go.id/ -h
SystemDefault
X-Pingback: http://jakartatimur.imigrasi.go.id/wp/xmlrpc.php
Link: http://jakartatimur.imigrasi.go.id/wp-json/; rel=“https://api.w.org/”,http://jakartatimur.imigrasi.go.id/; rel=shortlink
Vary: Accept-Encoding
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: text/html; charset=UTF-8
Date: Thu, 21 Nov 2019 20:38:54 GMT
Set-Cookie: haircki=haircooki; expires=Fri, 22-Nov-2019 20:38:54 GMT; Max-Age=86400
Server: Apache/2.4.29 (Ubuntu)

Status: 200 OK

2195,53 milliseconds
2,20 seconds

There is a curious cookie:

haircki=haircooki;

Google has some results:

https://wpplugincheck.com/articles/wordpress-site-hacked-postmortem/

Addendum

One of the more weird parts found in the modified index.php:
setcookie('haircki','haircooki', time()+3600*24);

14

FYI https://tools.pingdom.com and https://webpagetest.org/result/191121_DV_d4ce254f78310b781099f57b7f4a2d10/ both produced the REFUSED errors for me, both of which use real browsers. You can see on the latter website that 2/3 were REFUSED and the 1/3 that succeeded was the middle request.

In the below attached packet capture, you can see that the first and second packets are the connection request being refused (SYN + RST,ACK). Then there is a second attempt that succeeds.

refused.pcap (2.0 KB)

15

No, I don't have the authority.

If I white-listed the IPs of Lets Encrypt, will it help?

@JuergenAuer wow, thanks for pointing this out, let me check thru.

1 Like
16

Unfortunately I don’t have the authority to access the DNS config

17

Hello @_az, even not recommended, will it work if I whitelist the IPs of Lets Encrypt?

18

Whitelisting Let's Encrypt validation addresses is not a supported option; a very recent post mentioning that is at

3 Likes
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.