Help: 98: Address already in use

Help
1

Not sure why certbot didn't auto-renew, not sure why it can't renew manually.
nginx is stopped when I run the below command.
At first, my DNS was missing a v6 address, which is now fixed.
Not sure if this matters, but i'm using Tor Project's Onion Location Header: Tor Project | Onion-Location for upgrading Tor Browser users to my clear-text (port 80) onion address.

My domain is:

I ran this command:

$ sudo certbot renew --force-renewal -v

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/yawnbox.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate for yawnbox.com
Performing the following challenges:
http-01 challenge for yawnbox.com
Cleaning up challenges
Encountered exception during recovery: certbot.errors.MisconfigurationError: nginx restart failed:
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] still could not bind()
Failed to renew certificate yawnbox.com with error: nginx restart failed:
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:80 failed (98: Address already in use)
nginx: [emerg] bind() to [2620:18c:1:128::154]:443 failed (98: Address already in use)
nginx: [emerg] bind() to 103.232.206.154:443 failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] still could not bind()


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/yawnbox.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):

  • nginx/1.21.0

The operating system my web server runs on is (include version):

  • Ubuntu 20.04.2 LTS

My hosting provider, if applicable, is:

  • Self-hosted

I can login to a root shell on my machine (yes or no, or I don't know):

  • Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

  • No

The version of my client is:

  • certbot 1.16.0

Thank you,
-yawnbox

1 Like
2

Assuming the nginx configuration is fine, it looks like maybe nginx has lost track of its pidfile or something like that, causing it to try start two in parallel.

Somewhere to start could be:

  1. Make sure nginx is fully stopped. Verify that after stopping it the usual way (systemctl or service or whatever) you cannot access any of your websites and that there are no nginx processes running.
  2. Restart nginx and verify your sites are up.
  3. Try run Certbot.
1 Like
3

Does it matter that the cert has already expired? Because i can't currently access the site when it's up.

sudo pgrep nginx

This showed results so I did:

sudo pkill -9 nginx

now Chrome says ERR_CONNECTION_REFUSED which is expected.

re-running renew, now:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/yawnbox.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate for yawnbox.com
Performing the following challenges:
http-01 challenge for yawnbox.com
Cleaning up challenges
Encountered exception during recovery: certbot.errors.MisconfigurationError: nginx restart failed:
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] still could not bind()
Failed to renew certificate yawnbox.com with error: nginx restart failed:
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/tor-hs-my-website.sock failed (98: Address already in use)
nginx: [emerg] still could not bind()


- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/yawnbox.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
1 Like
4

Here's the nginx config.

$ sudo cat /etc/nginx/sites-available/default
server {
	listen 103.232.206.154:80 http2;
	listen [2620:18c:1:128::154]:80 http2;
	server_name yawnbox.com;
	location / {
		return 301 https://$host$request_uri;
	}
	server_tokens off;
        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Referrer-Policy "no-referrer";
}

server {
	root /var/www/html;
	index index.html index.htm;
	server_name yawnbox.com;
	location / {
		try_files $uri $uri/ =404;
	}
	listen [2620:18c:1:128::154]:443 ssl http2 ipv6only=on;
	listen 103.232.206.154:443 ssl http2; 
	ssl_certificate /etc/letsencrypt/live/yawnbox.com/fullchain.pem;
	ssl_certificate_key /etc/letsencrypt/live/yawnbox.com/privkey.pem;
	ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;
	ssl_trusted_certificate /etc/letsencrypt/live/yawnbox.com/chain.pem;
	ssl_stapling on;
	ssl_stapling_verify on;
        resolver 1.1.1.1 9.9.9.9;
        resolver_timeout 30s;
	ssl_protocols TLSv1.3 TLSv1.2;
	ssl_ecdh_curve X25519:secp384r1;
        ssl_conf_command Options PrioritizeChaCha;
	ssl_conf_command Ciphersuites TLS_CHACHA20_POLY1305_SHA256;
	ssl_prefer_server_ciphers on;
	ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305;
	ssl_session_cache shared:le_nginx_SSL:10m;
	ssl_session_timeout 10m;
	ssl_session_tickets off;
	server_tokens off;
	add_header Strict-Transport-Security 'max-age=31536000; includeSubDomains; preload';
        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Referrer-Policy "no-referrer";
	add_header Onion-Location http://prtia6ddkrondw6c2tuaw6tzthjfofbpjgkfm6erhmwzwepxv37tikyd.onion$request_uri;
}

server {
        listen unix:/var/run/tor-hs-my-website.sock;
        server_name prtia6ddkrondw6c2tuaw6tzthjfofbpjgkfm6erhmwzwepxv37tikyd.onion;
        access_log /var/log/nginx/hs-my-website.log;
        index index.html;
        root /var/www/html;
	server_tokens off;
        add_header X-Frame-Options DENY;
        add_header X-Content-Type-Options nosniff;
        add_header X-XSS-Protection "1; mode=block";
        add_header Referrer-Policy "no-referrer";
}
1 Like
5

I think what might be happening is an annoying thing about UNIX sockets: if the socket already exists on the filesystem, then a bind call to create it will fail, even if no process is on the other end of the socket.

If nginx died unexpectedly (i.e. got -9'd), the socket wouldn't have been cleaned up, and it would prevent nginx from starting next time.

I'm not sure what the solution is here. I guess I would expect nginx to stat+unlink the socket if it already exists, before trying to bind it. It doesn't seem to do that.

I think, try stop nginx again, remove the socket file, then try again.

If it's possible to get Tor to use a loopback TCP socket instead, that might avoid the problem with the orphaned socket on the filesystem, since TCP sockets get cleaned up automatically.

2 Likes
6

I commented out the Tor onion-location header lines, but now i'm rate-limited with the renewal attempts. I'll have to try again tomorrow. Thanks for the help.

1 Like
7

Please do your testing on the staging environment. Thanks.

2 Likes
8

Hi,

I've removed Tor, the onion header config completely, and still getting an error.

$ sudo certbot renew --force-renewal -v
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/yawnbox.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate for yawnbox.com
Performing the following challenges:
http-01 challenge for yawnbox.com
Waiting for verification...
Challenge failed for domain yawnbox.com
http-01 challenge for yawnbox.com

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: yawnbox.com
  Type:   connection
  Detail: Fetching http://yawnbox.com/.well-known/acme-challenge/-Ih5AKMA58jZeJcgc3iOU1NuPA35BsFqqImH55-zg80: Server is speaking HTTP/2 over HTTP

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate yawnbox.com with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/yawnbox.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

2021-07-16 16:31:42,959:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 93, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 181, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2021-07-16 16:31:42,959:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-07-16 16:31:42,960:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-07-16 16:31:44,313:ERROR:certbot._internal.renewal:Failed to renew certificate yawnbox.com with error: Some challenges have failed.
2021-07-16 16:31:44,317:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 474, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/main.py", line 1387, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/main.py", line 117, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 333, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/client.py", line 375, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/client.py", line 425, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 93, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 181, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2021-07-16 16:31:44,317:DEBUG:certbot.display.util:Notifying user: 
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-07-16 16:31:44,317:ERROR:certbot._internal.renewal:All renewals failed. The following certificates could not be renewed:
2021-07-16 16:31:44,318:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/yawnbox.com/fullchain.pem (failure)
2021-07-16 16:31:44,318:DEBUG:certbot.display.util:Notifying user: - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2021-07-16 16:31:44,319:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/1280/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/main.py", line 1574, in main
    return config.func(config, plugins)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/main.py", line 1461, in renew
    renewal.handle_renewal_request(config)
  File "/snap/certbot/1280/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 499, in handle_renewal_request
    raise errors.Error("{0} renew failure(s), {1} parse failure(s)".format(
certbot.errors.Error: 1 renew failure(s), 0 parse failure(s)
2021-07-16 16:31:44,320:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)
9

Make sure you don't have:

listen 80 http2;

anywhere in your nginx configuration. In nginx, if you enable HTTP/2 over cleartext on any vhost in port 80, it will break all normal port 80 HTTP traffic.

It just needs to be:

listen 80;

Also, as Osiris pointed out, please use --dry-run to test your renewal instead of --force-renewal. It will allow you to avoid hitting rate limits and false-positive successes (due to cached prior authorizations).

3 Likes
10

Hey @_az, this is not likely related to the cause of what's happening, but isn't this kinda contradictory:

{0} renew failure(s), {1} parse failure(s)
1 renew failure(s), 0 parse failure(s)

11

It's a Python format string:

>>> "{0} renew failure(s), {1} parse failure(s)".format(1, 0)
'1 renew failure(s), 0 parse failure(s)'
1 Like
12

Ah... so the first 0 and 1 are indices then and the actual values are indicated in the format() at the end.

https://docs.python.org/3/tutorial/inputoutput.html#the-string-format-method

1 Like
13

Thank you. I've updated the default nginx config by removing http2 from the port 80 server.

I made sure nginx was sopped (no matching services using pgrep). Now a different error:

$ sudo certbot renew --dry-run -v

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/yawnbox.com.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Certificate is due for renewal, auto-renewing...

Plugins selected: Authenticator nginx, Installer nginx

Failed to renew certificate yawnbox.com with error: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All simulated renewals failed. The following certificates could not be renewed:

/etc/letsencrypt/live/yawnbox.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1 renew failure(s), 0 parse failure(s)
1 Like
14

The last issue turned out to be the OCSP stapling configurations being turned off. I've turned them back on and I was able to dry run then force renew the certificate.

https://www.ssllabs.com/ssltest/analyze.html?d=yawnbox.com&s=103.232.206.154&latest

I'll have to work out the Tor onion location header issue in a dev environment going forward. Thanks again for the help.

1 Like
15

Glad you got things working! :partying_face:

Please avoid using --force-renewal like a deadly plague. :skull_and_crossbones: It is a source of many woes and nearly never results in anything beneficial.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.