Certificate renewal fails (403) on reverse proxy server

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domains are: alz.uci.edu, www.alz.uci.edu, adrc.uci.edu, www.adrc.uci.edu, www.mind.uci.edu

I ran this command: certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/adrc.uci.edu.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for adrc.uci.edu and 4 more domains

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: adrc.uci.edu
  Type:   unauthorized
  Detail: During secondary validation: 128.200.55.17: Invalid response from http://adrc.uci.edu/.well-known/acme-challenge/LPMwyHcUeguo3cvBHp8AyCfYWOanXgJWrYCAXw_Xoho: 403

  Domain: alz.uci.edu
  Type:   unauthorized
  Detail: During secondary validation: 128.200.55.17: Invalid response from http://alz.uci.edu/.well-known/acme-challenge/qxSmcqRRkphp_g7eXzbJRHy6d0suP08Q6DEUK90o1eQ: 403

  Domain: www.adrc.uci.edu
  Type:   unauthorized
  Detail: During secondary validation: 128.200.55.17: Invalid response from http://www.adrc.uci.edu/.well-known/acme-challenge/3auqHV4uJCB0LSlTH0Xgqz-JKsfvbsa9IPO7uuEFpYk: 403

  Domain: www.alz.uci.edu
  Type:   unauthorized
  Detail: During secondary validation: 128.200.55.17: Invalid response from http://www.alz.uci.edu/.well-known/acme-challenge/_7zH6biYyVepTbdI8XuQQ_1Moc7hSPZSW0DVQoKdfLM: 403

  Domain: www.mind.uci.edu
  Type:   unauthorized
  Detail: During secondary validation: 128.200.55.17: Invalid response from http://www.mind.uci.edu/.well-known/acme-challenge/pzIdVsjYX1A6Z_USFmNljvm2QoiIFKskLa3ytMRU_DA: 403

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate adrc.uci.edu with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/adrc.uci.edu/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): Apache 2.4.38

The operating system my web server runs on is (include version): Debian 10

My hosting provider, if applicable, is: N/A

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Certbot 1.29.0


Our organization was looking to set up redirects for older URLs that are no longer in use, namely:

to the destination domain https://mind.uci.edu/. After building some basic Apache configuration, we got the plain HTTP versions of the sites above to redirect as intended, but HTTPS requests to any of these sites resulted in a nasty “SSL_ERROR_BAD_CERT_DOMAIN” error (our institutional domain name provider does not automatically register “www.” domains with our requests; our main domain is mind.uci.edu [without the “www.”]). We’re planning on using Let’s Encrypt and Certbot for easier long-term maintenance of certificates for these other domains so that HTTPS requests to these will continue to redirect to our main site without needing to renew through our usual certificate authority.

Currently, we have a single reverse proxy server set up so that our content servers are on a local network. This reverse proxy server is where all our SSL certificates live and is where I started working with certbot. I successfully generated a certificate with certbot (certbot --apache): one that encompasses all 5 domains listed above, since all these domains serve the same purpose.

The certificate was installed and is currently in use. The HTTPS redirects function as expected. I run into difficulties, though, when I attempt to renew them (output above) and the ACME challenge fails. I’ve read Challenge Types - Let's Encrypt and have a good general idea of how the challenge works. I believe the essence of the error message is that the location of the ACME challenge isn’t reachable from the Let’s Encrypt servers, which would make sense because these domains don’t have a dedicated web server or machine. Let’s Encrypt may even be hitting the redirect and encountering https://mind.uci.edu/.well-known/acme-challenge/{long-token-string}, which doesn’t exist (the error code being 403 still confuses me because the location being unavailable should return a 404 error instead).

On our reverse proxy server, we have 2 Apache config files: one that we created ourselves, and one that certbot created. The original config file has nearly all our reverse proxy’s configuration, including the <VirtualHost *:80> block for plain HTTP requests to the sites listed above. Certbot created a config file that only contains a single <VirtualHost *:443> block for the sites above. We can share the contents of these config files if needed.

  • Is there a way to configure an Apache directive (Alias? Location?) to expose a directory or location on our reverse proxy server so the ACME challenge can proceed?
  • There is one more destination domain behind this reverse proxy (in our local network) that we’d like to set up for additional redirects (exactly like we’re doing for https://mind.uci.edu/) – will we be able to use the same /.well-known/acme-challenge framework for this other domain/VirtualHost?

Output of apachectl -S (edited to remove other separately managed domain names that aren’t relevant here):

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server mind.uci.edu (/etc/apache2/sites-enabled/000-default.conf:5)
         port 80 namevhost mind.uci.edu (/etc/apache2/sites-enabled/000-default.conf:5)
         port 80 namevhost alz.uci.edu (/etc/apache2/sites-enabled/000-default.conf:24)
                 alias www.alz.uci.edu
                 alias adrc.uci.edu
                 alias www.adrc.uci.edu
                 alias www.mind.uci.edu
*:443                  is a NameVirtualHost
         default server alz.uci.edu (/etc/apache2/sites-enabled/000-default-le-ssl.conf:2)
         port 443 namevhost alz.uci.edu (/etc/apache2/sites-enabled/000-default-le-ssl.conf:2)
                 alias www.alz.uci.edu
                 alias adrc.uci.edu
                 alias www.adrc.uci.edu
                 alias www.mind.uci.edu
         port 443 namevhost mind.uci.edu (/etc/apache2/sites-enabled/000-default.conf:205)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: MODPERL2
Define: ENABLE_USR_LIB_CGI_BIN
User: name="www-data" id=33
Group: name="www-data" id=33

Any assistance would be appreciated. If there is any additional information needed regarding our webserver/reverse proxy setup, I would be happy to clarify any details. Thank you!

There's a firewall blocking access, at least from my location:

image

Good chance this is also affecting the LE validation.

You should allow access to the path /.well-known/acme-challenge/ globally. Alternatively, you might be able to use the dns-01 challenge instead.

5 Likes

Thanks so much for pointing this out, Osiris! I tested this now and confirmed it was the firewall that was the issue.

It might be a stretch to ask, but despite Let's Encrypt not publishing any IP addresses for us to add to an allowlist, is there any way to automatically allow traffic from Let's Encrypt servers?

EDIT: I have found this article, so I'll look into configuring our firewall and setting rule/filter priorities to always allow port 80. Thanks again!

3 Likes

Certbot has a pre and post hook, which you can use to enable/disable firewall rules as needed.

3 Likes

I can think of a couple of other ways to also deal with that:

  • [accept all HTTP connections] handle the challenge requests and redirect the rest to HTTPS
  • [URL filtering] accept only http://{domain}/.well-known/acme-challenge/* requests
3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.