Renewals started failing

My domain is: lemmy.starbase80.dev

I ran this command: certbot renew --keep-until-expiring -n -v

It produced this output:
(I have over a dozen domains failing with this command, I'm just using Lemmy as an example)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/lemmy.starbase80.dev.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate for lemmy.starbase80.dev
Performing the following challenges:
http-01 challenge for lemmy.starbase80.dev
Using the webroot path /var/www/certbot for all unmatched domains.
Waiting for verification...
Challenge failed for domain lemmy.starbase80.dev
http-01 challenge for lemmy.starbase80.dev
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: lemmy.starbase80.dev
  Type:   unauthorized
  Detail: During secondary validation: 2a06:98c1:3121::1: Invalid response from http://lemmy.starbase80.dev/.well-known/acme-challenge/cuPmCwJBArOr2JciFGm3N_VPHSnNqxmIbZHB8Sz1k_g: 403
Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
Cleaning up challenges
Failed to renew certificate lemmy.starbase80.dev with error: Some challenges have failed.

My web server is (include version): Docker image linuxserver/nginx:latest

The operating system my web server runs on is (include version): Ubuntu 22.04.4 LTS (GNU/Linux 6.5.0-28-generic x86_64)

My hosting provider, if applicable, is: Self hosted, traffic served via Cloudflare Tunnels

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Docker image certbot/certbot:latest


Additional information:

I'm using Proxmox to host this Linux VM. All my websites are served via Docker containers. Public access to sites is delivered through Cloudflare Tunnels. All domains go to localhost:443 (after I set up their initial certificate). HTTP to HTTPS rewriting is on. I have a "switchboard" nginx reverse proxy that serves Docker web content from specific ports.

There is an Authelia instance that puts itself in front of sensitive domains, but Lemmy is fully public. Regardless, the nginx config is such that the certbot challenge is exempted from any Authelia validation, like so:

server {
  listen 443 ssl http2;
  listen [::]:443 ssl http2;
  server_name lemmy.starbase80.dev;

  access_log /var/log/nginx/lemmy.starbase80.dev.access.logs;
  error_log /var/log/nginx/lemmy.starbase80.dev.error.logs;

  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
  ssl_prefer_server_ciphers on;
  ssl_session_cache shared:SSL:10m;
  ssl_session_tickets off;

  ssl_certificate /etc/letsencrypt/live/lemmy.starbase80.dev/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/lemmy.starbase80.dev/privkey.pem;

  include /etc/nginx/snippets/authelia-location.conf;

  location ^~ /.well-known/acme-challenge {
    allow all;
    root /var/www/certbot;
  }

  location / {
    # Exposing Lemmy publicly by commenting out Authelia's validation
    # include /etc/nginx/snippets/proxy.conf;
    # include /etc/nginx/snippets/authelia-authrequest.conf;

    proxy_pass http://lemmy_backend;
  }
}

You can see my entire hosting configuration here: GitHub - notclickable-jordan/docker-symphony: Docker compose files and nginx reverse proxy configs for a multi-application server

The compose.yml section for the certbot renewal looks like this:

  letsencrypt:
    image: certbot/certbot
    container_name: switchboard_letsencrypt
    entrypoint: ""
    command: >
      sh -c "certbot renew --keep-until-expiring -n -v"
    depends_on:
        - nginx
    volumes:
        - letsencrypt:/etc/letsencrypt
        - certbot:/var/www/certbot
    environment:
        - TERM=xterm

I suspect I broke this by trying to get NextCloud working, but I'm not sure how. Thanks in advance for your help!

Hi @jordan-roher, and welcome to the LE community forum :slight_smile:

That implies the primary validation passed the test.
Which, in turn, implies that you may be GeoLocation blocking the secondary validation locations.

See: Unexpected renewal failures during April 2024? Please read this!

5 Likes

Yep, that was it. Thanks for the quick response, I'm glad I'm an easy case.

For those coming here later, I had (a forgotten) rule in Cloudflare that was excluding non-US countries.

It was under (Domain) > Security > WAF. I disabled it and renewal worked.

4 Likes

You might also want to check out this thread. They kept that rule but added an exclusion for /.well-known/acme-challenge requests

The link is to my summary. See their previous post in that thread for more

5 Likes

Even better, thanks!

4 Likes