Connection refused when requesting new cerfificate via script - possible hangover of previous redirects?

Back in February I ran the shell script pasted below to generate a letsencrypt certificate on my server. It worked, and after I had it I set up a docker-compose file with services nginx, certbot, among others, to redirect http://example.com to https://example.com. All was well for 90 days till renewal time. My automated renewal failed. I guess for the same reason my attempt to manually get a new certificate via the init shell file below also fails this time around.

The docker services that handled the redirects from http to https have been stopped yet the redirects appear to remain in place. There's currently no reverse proxy running since I have stopped the nginx container. However visiting http://collabora.alamko.de STILL tries to redirect to https. I suspect this is part of or related to my issue. The browser tells me 'connection refused', the same as the output message below 'Certbot failed'.

It looks like the script which obtains the certificate try's a url at http but this time around it's being redirected to https which is perhaps what's causing the problem? See the output below.

My domain is:
collabora.alamko.de

I ran this command:
See pasted shell script below init-letsencrypt.sh

It produced this output:

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: collabora.alamko.de
  Type:   connection
  Detail: Fetching http://collabora.alamko.de/.well-known/acme-challenge/f7ANvS1aFRNrx5BN7gbeEDGiF5cG3xCLWI-RVm7AOT0: Connection refused

My web server is (include version):
Looking at the sh script mentioned above, init-letsencrypt.sh, it seems to be using nginx via a docker container.

The operating system my web server runs on is (include version):
Ubuntu 20.04.1 LTS

My hosting provider, if applicable, is:
Hetzner

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Certbot via docker. I presume latest'? Details in the init sh script below.

Here is the init-letsencrypt.sh that I've been referencing throughout my post (Not my script, downloaded from a blog out there somewhere). Before, when I ran this script the outcome was a newly generated letsencrypt certificate. This time around I get the 'connection refused' error message:

#!/bin/bash

if ! [ -x "$(command -v docker-compose)" ]; then
  echo 'Error: docker-compose is not installed.' >&2
  exit 1
fi

domains=(collabora.alamko.de)
rsa_key_size=4096
data_path="./data/certbot"
email="myemail@fastmail.com" # Adding a valid address is strongly recommended
staging=0 # Set to 1 if you're testing your setup to avoid hitting request limits

if [ -d "$data_path" ]; then
  read -p "Existing data found for $domains. Continue and replace existing certificate? (y/N) " decision
  if [ "$decision" != "Y" ] && [ "$decision" != "y" ]; then
    exit
  fi
fi


if [ ! -e "$data_path/conf/options-ssl-nginx.conf" ] || [ ! -e "$data_path/conf/ssl-dhparams.pem" ]; then
  echo "### Downloading recommended TLS parameters ..."
  mkdir -p "$data_path/conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
  echo
fi

echo "### Creating dummy certificate for $domains ..."
path="/etc/letsencrypt/live/$domains"
mkdir -p "$data_path/conf/live/$domains"
docker-compose run --rm --entrypoint "\
  openssl req -x509 -nodes -newkey rsa:$rsa_key_size -days 1\
    -keyout '$path/privkey.pem' \
    -out '$path/fullchain.pem' \
    -subj '/CN=localhost'" certbot
echo


echo "### Starting nginx ..."
docker-compose up --force-recreate -d nginx
echo

echo "### Deleting dummy certificate for $domains ..."
docker-compose run --rm --entrypoint "\
  rm -Rf /etc/letsencrypt/live/$domains && \
  rm -Rf /etc/letsencrypt/archive/$domains && \
  rm -Rf /etc/letsencrypt/renewal/$domains.conf" certbot
echo


echo "### Requesting Let's Encrypt certificate for $domains ..."
#Join $domains to -d args
domain_args=""
for domain in "${domains[@]}"; do
  domain_args="$domain_args -d $domain"
done

# Select appropriate email arg
case "$email" in
  "") email_arg="--register-unsafely-without-email" ;;
  *) email_arg="--email $email" ;;
esac

# Enable staging mode if needed
if [ $staging != "0" ]; then staging_arg="--staging"; fi

docker-compose run --rm --entrypoint "\
  certbot certonly --webroot -w /var/www/certbot \
    $staging_arg \
    $email_arg \
    $domain_args \
    --rsa-key-size $rsa_key_size \
    --agree-tos \
    --force-renewal" certbot
echo

echo "### Reloading nginx ..."
docker-compose exec nginx nginx -s reload

Given my server seems to continue to redirect, even though I've switched off the docker services that created and managed the redirects, how can I obtain a new letsencrypt certificate?

1 Like

hi @dougfir and welcome to the LE community forum :slight_smile:

When using --webroot, the -w path provided must match the document root (found in the vhost config) for the expected challenge location (http://your.domain/.well-known/acme-challenge/).

And... who wrote the script?:

1 Like

Hi @rg305 thanks for the welcome!

And who wrote the script

I found the script on this repo.

When using --webroot , the -w path provided must match the document root (found in the vhost config) for the expected challenge location ( http://your.domain/.well-known/acme-challenge/ ).

OK. This is new terminology to me. I'm unsure what to do with this information or how to action your feedback?! Vhost config... is that something that would exist on my server or is it directly related to the script I shared? Requesting a little hand holding here if you will...

2 Likes

That script should be removed from the internet in total. It contains stuff which can lead to hitting rate limits if one does not understand the purpose of the script.

Further more, I assumes a working nginx docker container and it also assumes the document root of the site is /var/www/certbot for some reason. Which is a little bit odd, unless the nginx container is specifically configured to work.

So, a few follow-up questions:

  • Where did you find this script?
  • Is your nginx container custom build by yourself or did you fetched some kind of "pre-build" nginx container and if so, where from?
2 Likes

Hi @Osiris. Thanks for your feedback. Your questions:

The repo where I got the script is here. The script in question is in the root of the repo

I copied what they have on the repo for the nginx container. Here's my docker-compose in case it's useful:

version: "3.5"
networks:
  collabora:

volumes:
  shared_run:
  
services:

  collabora:
    image: collabora/code
    container_name: collabora
    restart: always
    cap_add:
      - MKNOD
    environment:
      - "extra_params=--o:ssl.enable=false --o:ssl.termination=true"
      - domain=${NEXTCLOUD_DOMAIN}
      - dictionaries=en_US
    ports:
      - "9980:9980"
    volumes:
      - ./appdata/collabora:/config 
    networks:
      - collabora

  nginx:
    image: nginx:1.19
    volumes:
      - "shared_run:/run"
      - ./data/nginx/templates:/etc/nginx/templates
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
      - ./reload.sh:/docker-entrypoint.d/reload.sh
    ports:
      - "80:80"
      - "443:443"
    env_file: .env
    networks:
      - collabora

  certbot:
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
    networks:
      - collabora

Please let me know if there's any more info I can provide?

1 Like

Perhaps a dump of the nginx config would be helpful too.
nginx -T

1 Like

Thanks for the pointer. Here's the output of nginx -T from within the nginx container:

nginx -T
2021/06/03 17:49:58 [emerg] 746#746: host not found in upstream "collabora" in /etc/nginx/conf.d/app.conf:30
nginx: [emerg] host not found in upstream "collabora" in /etc/nginx/conf.d/app.conf:30
nginx: configuration file /etc/nginx/nginx.conf test failed

Here is the content of /etc/nginx/conf.d/app.conf in case it's useful info:

cat /etc/nginx/conf.d/app.conf 
ssl_certificate		/etc/letsencrypt/live/collabora.alamko.de/fullchain.pem;
ssl_certificate_key	/etc/letsencrypt/live/collabora.alamko.de/privkey.pem;

server {
	
	listen			80;
	server_name		www.collabora.alamko.de collabora.alamko.de;
	
	location /.well-known/acme-challenge/ {
		root /var/www/certbot;
	}
	
	return 301 https://collabora.alamko.de$request_uri;
}

server {
	
	listen			443 ssl;
	server_name		www.collabora.alamko.de;
	return 301 $scheme://collabora.alamko.de$request_uri;
}


server {
	
	listen			443 ssl;
	server_name		collabora.alamko.de;
    
	location / {
		proxy_pass http://collabora:9980;
		access_log off;
		proxy_set_header Host $host;
	}
    
	# static files
	location ^~ /loleaflet {
		proxy_pass http://collabora:9980;
		proxy_set_header Host $http_host;
	}

	# WOPI discovery URL
		location ^~ /hosting/discovery {
        	proxy_pass http://collabora:9980;
        	proxy_set_header Host $http_host;
        }

	# Capabilities
	location ^~ /hosting/capabilities {
		proxy_pass http://collabora:9980;
		proxy_set_header Host $http_host;
	}

	# main websocket
	location ~ ^/lool/(.*)/ws$ {
		proxy_pass http://collabora:9980;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "Upgrade";
		proxy_set_header Host $http_host;
		proxy_read_timeout 36000s;
	}

	# download, presentation and image upload
	location ~ ^/lool {
		proxy_pass http://collabora:9980;
		proxy_set_header Host $http_host;
	}

	# Admin Console websocket
	location ^~ /lool/adminws {
		proxy_pass http://collabora:9980;
        	proxy_set_header Upgrade $http_upgrade;
        	proxy_set_header Connection "Upgrade";
        	proxy_set_header Host $http_host;
        	proxy_read_timeout 36000s;
        }
}

And here's the nginx.conf file referenced in the message too:

cat /etc/nginx/nginx.conf

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

1 Like

This explains why they used /var/www/certbot :

Try instead:
sudo nginx -T

or also show:
ls -l /etc/nginx/conf.d/*.conf

1 Like

Here is the output of both suggestions:

sudo nginx -T
bash: sudo: command not found
root@123:/# ls -l /etc/nginx/conf.d/*.conf
-rw-r--r-- 1 root root 1729 Jun  3 12:03 /etc/nginx/conf.d/app.conf
-rw-r--r-- 1 root root 1114 Jun  3 12:03 /etc/nginx/conf.d/default.conf

I think because I'm in a container there's limited utilities, in this case no sudo

1 Like

Well you showed app.conf.
Let's also see the default.conf file.

[I still don't see where the certificate files are being used]

1 Like

Nginx doesn't understand what is meant with the hostname collabora in the proxy_pass directives in your configuration. That's why it won't start and that's why a "connection refused" is reported by the LE validation server: there's no webserver running to connect to.

Please fix your proxy_pass directives to use a hostname which nginx can understand and try again.

Also please note that once you've managed to get a certificate issued, DO NOT USE the init-letsencrypt.sh script again, as it will delete perfectly fine certificates if you run it.. If something is not working after you've managed to succesfully get a certificate, you need to figure it out without running the script again.

I'm guessing using ssl in listen 443 ssl; from within a server {} block will work, even if ssl_certificate and ssl_certificate_key are located outside any block at all.

1 Like

And also please show the output of:
certbot certificates

Might be something like:
docker-compose run --rm --entrypoint "certbot certificates" certbot

1 Like

Hi, I updated the proxy_pass directives but I got the same error message:

nginx -T
2021/06/03 18:43:52 [emerg] 80#80: host not found in upstream "collabora" in /etc/nginx/conf.d/oldapp.conf:30
nginx: [emerg] host not found in upstream "collabora" in /etc/nginx/conf.d/oldapp.conf:30
nginx: configuration file /etc/nginx/nginx.conf test failed

Here's how the conf file looks now with this update:

cat /etc/nginx/conf.d/app.conf 
ssl_certificate		/etc/letsencrypt/live/collabora.alamko.de/fullchain.pem;
ssl_certificate_key	/etc/letsencrypt/live/collabora.alamko.de/privkey.pem;

server {
	
	listen			80;
	server_name		www.collabora.alamko.de collabora.alamko.de;
	
	location /.well-known/acme-challenge/ {
		root /var/www/certbot;
	}
	
	return 301 https://collabora.alamko.de$request_uri;
}

server {
	
	listen			443 ssl;
	server_name		www.collabora.alamko.de;
	return 301 $scheme://collabora.alamko.de$request_uri;
}


server {
	
	listen			443 ssl;
	server_name		collabora.alamko.de;
    
	location / {
		proxy_pass http://collabora.alamko.de:9980;
		access_log off;
		proxy_set_header Host $host;
	}
    
	# static files
	location ^~ /loleaflet {
		proxy_pass http://collabora.alamko.de:9980;
		proxy_set_header Host $http_host;
	}

	# WOPI discovery URL
		location ^~ /hosting/discovery {
        	proxy_pass http://collabora.alamko.de:9980;
        	proxy_set_header Host $http_host;
        }

	# Capabilities
	location ^~ /hosting/capabilities {
		proxy_pass http://collabora.alamko.de:9980;
		proxy_set_header Host $http_host;
	}

	# main websocket
	location ~ ^/lool/(.*)/ws$ {
		proxy_pass http://collabora.alamko.de:9980;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "Upgrade";
		proxy_set_header Host $http_host;
		proxy_read_timeout 36000s;
	}

	# download, presentation and image upload
	location ~ ^/lool {
		proxy_pass http://collabora.alamko.de:9980;
		proxy_set_header Host $http_host;
	}

	# Admin Console websocket
	location ^~ /lool/adminws {
		proxy_pass http://collabora.alamko.de:9980;
        	proxy_set_header Upgrade $http_upgrade;
        	proxy_set_header Connection "Upgrade";
        	proxy_set_header Host $http_host;
        	proxy_read_timeout 36000s;
        }
}
1 Like

Hi, I ran your suggested line of code:

docker-compose run --rm --entrypoint "certbot certificates" certbot
Creating collabora_certbot_run ... done
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
No certificates found.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

:confused:

1 Like

Did you update the correct file, i.e., the app.conf used inside the docker container?

I see the docker-compose.yml file from the github repository containing the script you're using has the following at the nginx volume section:

  - ./data/nginx:/etc/nginx/conf.d

This is missing from your docker compose file it seems? So, where does the nginx container get its app.conf from and did you indeed edit the correct one?

1 Like

Here's the default.conf:

But note, since I'm using a version of nginx that uses global variables sourced from .env file I use event substitution which in the nginx image I'm using, now seems to support this out of the box. See answer from Martin further down. i.e. It's the app.conf file that is generated.

cat /etc/nginx/conf.d/default.conf 
server {
    listen       80;
    listen  [::]:80;
    server_name  localhost;

    #charset koi8-r;
    #access_log  /var/log/nginx/host.access.log  main;

    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
    }

    #error_page  404              /404.html;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

    # proxy the PHP scripts to Apache listening on 127.0.0.1:80
    #
    #location ~ \.php$ {
    #    proxy_pass   http://127.0.0.1;
    #}

    # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
    #
    #location ~ \.php$ {
    #    root           html;
    #    fastcgi_pass   127.0.0.1:9000;
    #    fastcgi_index  index.php;
    #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
    #    include        fastcgi_params;
    #}

    # deny access to .htaccess files, if Apache's document root
    # concurs with nginx's one
    #
    #location ~ /\.ht {
    #    deny  all;
    #}
}

Yep, here's some bash output from while inside the nginx container:

ls -l /etc/nginx/conf.d/*.conf
-rw-r--r-- 1 root root 1799 Jun  3 18:31 /etc/nginx/conf.d/app.conf
-rw-r--r-- 1 root root 1114 Jun  3 18:31 /etc/nginx/conf.d/default.conf
-rw-r--r-- 1 root root 1729 Jun  3 18:31 /etc/nginx/conf.d/oldapp.conf
root@3bf8b15abb47:/# cat /etc/nginx/conf.d/app.conf 
ssl_certificate		/etc/letsencrypt/live/collabora.alamko.de/fullchain.pem;
ssl_certificate_key	/etc/letsencrypt/live/collabora.alamko.de/privkey.pem;

server {
	
	listen			80;
	server_name		www.collabora.alamko.de collabora.alamko.de;
	
	location /.well-known/acme-challenge/ {
		root /var/www/certbot;
	}
	
	return 301 https://collabora.alamko.de$request_uri;
}

server {
	
	listen			443 ssl;
	server_name		www.collabora.alamko.de;
	return 301 $scheme://collabora.alamko.de$request_uri;
}


server {
	
	listen			443 ssl;
	server_name		collabora.alamko.de;
    
	location / {
		proxy_pass http://collabora.alamko.de:9980;
		access_log off;
		proxy_set_header Host $host;
	}
    
	# static files
	location ^~ /loleaflet {
		proxy_pass http://collabora.alamko.de:9980;
		proxy_set_header Host $http_host;
	}

	# WOPI discovery URL
		location ^~ /hosting/discovery {
        	proxy_pass http://collabora.alamko.de:9980;
        	proxy_set_header Host $http_host;
        }

	# Capabilities
	location ^~ /hosting/capabilities {
		proxy_pass http://collabora.alamko.de:9980;
		proxy_set_header Host $http_host;
	}

	# main websocket
	location ~ ^/lool/(.*)/ws$ {
		proxy_pass http://collabora.alamko.de:9980;
		proxy_set_header Upgrade $http_upgrade;
		proxy_set_header Connection "Upgrade";
		proxy_set_header Host $http_host;
		proxy_read_timeout 36000s;
	}

	# download, presentation and image upload
	location ~ ^/lool {
		proxy_pass http://collabora.alamko.de:9980;
		proxy_set_header Host $http_host;
	}

	# Admin Console websocket
	location ^~ /lool/adminws {
		proxy_pass http://collabora.alamko.de:9980;
        	proxy_set_header Upgrade $http_upgrade;
        	proxy_set_header Connection "Upgrade";
        	proxy_set_header Host $http_host;
        	proxy_read_timeout 36000s;
        }
}

I think the default.conf comes out of the box. I wonder if I somehow need to remove that one or if nginx is confused by which one to use.

I use variable substitution to read vars from .env which nginx now seems to support. See Martins answer further down here. This generates the app.conf file.

Then I don't understand what's nginx -T still complaining about...

2 Likes

Do you think it's possible nginx is confused by the existence of 2 conf files app.conf and default.conf. I want nginx to use app.conf, I don't know how default.conf get's there presumably it's part of the image :confused:

New info, my 'oldapp.conf' file which I'd left in the same dir seemed to be causing issues. New error message after doing some housekeeping:

nginx -T
2021/06/03 19:07:34 [emerg] 58#58: cannot load certificate "/etc/letsencrypt/live/collabora.alamko.de/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/collabora.alamko.de/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/collabora.alamko.de/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/collabora.alamko.de/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

2 Likes