Certbot failed to authenticate some domains - subdomains

My domain is:
malakan.co.uk.
Trying to setup the subdomain gramps.malakan.co.uk (and others once I know how to do this)

I ran this command:

certonly --webroot -w /var/www/certbot -d gramps.malakan.co.uk --force-renewal --email mattcolbear@gmail.com --agree-tos

It produced this output:

certbot  | Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
certbot  |   Domain: gramps.malakan.co.uk
certbot  |   Type:   unauthorized
certbot  |   Detail: 2606:4700:3031::ac43:9b26: Invalid response from http://gramps.malakan.co.uk/.well-known/acme-challenge/879ddo_yN9c2JEtQcp3CgUBbr_l2zw2J6OTUX6rReLk: 521
certbot  | 
certbot  | Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
certbot  | 
certbot  | Some challenges have failed.
certbot  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
no sure what this means

The operating system my web server runs on is (include version):
Ubuntu 24.04.1 LTS

My hosting provider, if applicable, is:
localhost

I can login to a root shell on my machine (yes or no, or I don't know):yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): docker image: certbot/certbot ( not sure which version.

I am using Docker to get some servers up. I have https setup and functional with a ssl cert for malakan.co.uk. I'm attempting to add subdomains but am having problems with certbot and nginx. I think it's to do with the path the certs are saved in but I'm new to this and am doing it a bit blind.

I see this error in docker compose logs

nginx-proxy  | nginx.1     | nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/gramps.malakan.co.uk/fullchain.pem": BIO_new_file() failed (SSL: error:80000002:system library::No such file or directory:calling fopen(/etc/letsencrypt/live/gramps.malakan.co.uk/fullchain.pem, r) error:10000080:BIO routines::no such file)

What am I doing wrong here?

Files:
nginx.conf

events {
worker_connections 1024;
}

http {
server_tokens off;
charset utf-8;

# always redirect to https
server {
    server_name _;
    listen 80 default_server;
    return 301 https://$host$request_uri;
}

server {
    server_name malakan.co.uk;
    listen 443 ssl;
    http2 on;
    # use the certificates
    ssl_certificate     /etc/letsencrypt/live/malakan.co.uk/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/malakan.co.uk/privkey.pem;
    root /var/www/html;
    index index.php index.html index.htm;

    location / {
        proxy_pass http://helloworld:8000/;
    }
    
    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
}

server {
    server_name gramps.malakan.co.uk;
    listen 443 ssl;
    http2 on;
    # use the certificates
    ssl_certificate     /etc/letsencrypt/live/gramps.malakan.co.uk/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/gramps.malakan.co.uk/privkey.pem;
    
    root /var/www/html;
    index index.php index.html index.htm;

    location / {
        proxy_pass http://helloworld:8000/;
    }
    
    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
}

}

docker-compose.yml

services:

helloworld:
container_name: helloworld
image: crccheck/hello-world
restart: always
environment:
VIRTUAL_PORT: "8000"
VIRTUAL_HOST: gramps.malakan.co.uk
LETSENCRYPT_HOST: gramps.malakan.co.uk
LETSENCRYPT_EMAIL: # your email
networks:
- proxy-tier

proxy:
container_name: nginx-proxy
image: docker.io/nginxproxy/nginx-proxy
restart: always
ports:
- 80:80
- 443:443
environment:
ENABLE_IPV6: "true"
volumes:
- ./nginx_proxy.conf:/etc/nginx/conf.d/my_proxy.conf:ro
- conf:/etc/nginx/conf.d
- dhparam:/etc/nginx/dhparam
- certs:/etc/nginx/certs:ro
- vhost.d:/etc/nginx/vhost.d
- html:/usr/share/nginx/html
- /var/run/docker.sock:/tmp/docker.sock:ro
- ./nginx/nginx.conf:/etc/nginx/nginx.conf
- ./certbot/conf:/etc/letsencrypt
- ./certbot/www:/var/www/certbot
networks:
- proxy-tier

certbot:
container_name: certbot
image: certbot/certbot
volumes:
- ./certbot/conf:/etc/letsencrypt
- ./certbot/www:/var/www/certbot
# command: certonly --webroot -w /var/www/certbot --keep-until-expiring -d malakan.co.uk --agree-tos
command: certonly --webroot -w /var/www/certbot -d gramps.malakan.co.uk --force-renewal --agree-tos
networks:
- proxy-tier

volumes:
certs:
conf:
dhparam:
vhost.d:
html:

networks:
proxy-tier:

When I surf to your website, I'm seeing a Cloudflare " Web server is down" error. With some more details, it says: "The web server is not returning a connection. As a result, the web page is not displaying.".

Please make sure your website is working in general and once that's fixed, try again.

Also, please don't use --force-renewal. In general, don't use options if you don't their meaning. As it suggests you clearly don't know the actual meaning of --force-renewal, as it does not help you in this situation at all.

2 Likes

Apollogies, the website is down due to the errors I posted above.

I have removed the second server block in the nginx confg file which allows https://malakan.co.uk to be up now.

https://gramps.malakan.co.uk is accessible but not not secure, this is what I want to change.

As for the option, I believe --force-renewal will renew the ssl every time the container is loaded up. Yes there is a risk of reaching the ssl limit but I left it there as I just wanted to ensure it was going try to create/renew when it eventually worked and then was going to replace with --keep-until-expiring, like my commented out command has, once functional. What should I be using instead, is there an alternative or should I remove the option?

Edit - may have got it working?
I'm not sure, but certbot eventually said I got a ssl cert for gramps.maklakan.co.uk.
While my browser says it has a ssl cert it also says not secure, what does this mean?

You should just remove the option. You already have persistent volumes configured for Certbot. There's no reason to forcibly renew a perfectly fine production certificate. For testing purposes, the staging environment should be used, but that doesn't issue publicly valid certificates.

You made a typo there. https://gramps.malakan.co.uk is secured by a Google Trust Services certificate issued by Cloudflare.

I have removed the option but when adding both domains I get errors in the logs. What is the correct certbot command?

command: certonly --webroot -w /var/www/certbot -d gramps.malakan.co.uk -d malakan.co.uk --agree-tos

leads to this in the logs:

certbot | You have an existing certificate that contains a portion of the domains you
certbot | requested (ref: /etc/letsencrypt/renewal/gramps.malakan.co.uk.conf)
certbot |
certbot | It contains these names: gramps.malakan.co.uk
certbot |
certbot | You requested these names for the new certificate: gramps.malakan.co.uk,
certbot | malakan.co.uk.
certbot |
certbot | Do you want to expand and replace this existing certificate with the new
certbot | certificate?
certbot | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
certbot | An unexpected error occurred:
certbot | EOFError

It's probably not possible to use Certbot interactively from within Docker (i.e., you can't answer "yes" or "no", which would be required to continue). With --expand you can tell Certbot to expand an existing certificate.

Please note that you might have a redundant certificate stored in Certbot if you expand this existing certificate (e.g., a cert with just a single hostname). If you configure the new certificate with both hostnames and you're not using the certificate with just a single hostname any longer, you should delete it. (Assuming there is such a cert.)

Thank you for the certbot option. That works perfectly!

Now that this is functional, what are the correct steps to add subdomains? I made a hash of it so would like to know how to do this properly.

I did a quick test that didn't work:

  1. Added a new A record for trillium.malakan.co.uk on cloudflaire
  2. I duplicated the gramps server block, changing the server name and the certificates path to have trillium. instead of gramps.
  3. I added -d trillium.malakan.co.uk to the certbot command

Edit: logs

certbot | Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
certbot | Domain: trillium.malakan.co.uk
certbot | Type: connection
certbot | Detail: 86.180.175.242: Fetching http://trillium.malakan.co.uk/.well-known/acme-challenge/WufUznqE49KLUyVM9rRLGuFu_5Y293WT92ddDZd7OmE: Connection refused

1 Like

That's not the IP address I currently see. Maybe an older IP address from before the current IP addresses were propogated?

Thats the current ip address for the server