Apache 2.4 renewal fails

Hello Let's Encrypt Community!

I'm struggeling to renew a certificate on my Apache 2.4.29 server. The original certificate was retrieved and installed using the Apache authenticator and installer plugins.
Now when I run certbot renew --dry-run I always get a message stating that the verification failed.

My domain is: opendrg.org

I ran this command: certbot renew --dry-run

It produced this output:

Challenge failed for domain opendrg.org
http-01 challenge for opendrg.org
Cleaning up challenges
Failed to renew certificate opendrg.org with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/opendrg.org/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: opendrg.org
   Type:   connection
   Detail: Fetching
   https://opendrg.org/coinsync/login.well-known/acme-challenge/JemA08oH13QchrSLsxSwvcHkc56bvXsbmWOR24GCJ4k:
   Error getting validation data

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client.

My web server is (include version): Apache 2.4.29

The operating system my web server runs on is (include version):
Ubuntu Bionic 18.04.5 LTS in LXC container

I can login to a root shell on my machine (yes or no, or I don't know): yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.12.0 (snap)

I also tried different authentication methods - to no avail.
Let's Debug says everything would be okay.

Any help is greatly appreciated!

Best regards
Constantin

2 Likes

What's the output of:

sudo apachectl -t -D DUMP_VHOSTS
2 Likes
root@reverseproxy:~# sudo apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:80                   opendrg.org (/etc/apache2/sites-enabled/000-default.conf:1)
*:443                  opendrg.org (/etc/apache2/sites-enabled/000-default.conf:26)
2 Likes

Let's have a look at the renewal conf file.
/etc/letsencrypt/renewal/{cert.name}.conf

And also the file mentioned in your post.

2 Likes

Do you know what software is generating the HTTP redirect from https://opendrg.org/.well-known/acme-challenge to https://opendrg.org/coinsync/login.well-known/acme-challenge? That is, adding the /coinsync/login to the beginning of the URL, instead of just allowing it to be served by the Apache server?

3 Likes

Hi guys,

thanks for your quick replies! Unfortunately I have no access to the system in question until next Monday.
Then I can provide you with more debugging information.

It basically says authenticator=apache and installer=apache from what I know by heart.

The Apache in my setup acts as reverse proxy for various other services. The redirect is done on Apache reverseproxy level (I think) but for some reason it persists even when I comment it out.

I'll post more info next Monday, thanks for your help so far :slight_smile:

3 Likes

/etc/letsencrypt/renewal/opendrg.org.conf

# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/opendrg.org
cert = /etc/letsencrypt/live/opendrg.org/cert.pem
privkey = /etc/letsencrypt/live/opendrg.org/privkey.pem
chain = /etc/letsencrypt/live/opendrg.org/chain.pem
fullchain = /etc/letsencrypt/live/opendrg.org/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = d86aa55e23cb1804adfa6d62a2fa94fc
#pre_hook=systemctl stop apache2 && sleep 3
#post_hook=systemctl start apache2
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory
2 Likes

For the life of me I can't find out why Apache keeps rewriting requests to /.well-known.
I've tried several ways to stop it from doing so.
This is my config:

<VirtualHost *:80>
	ServerName opendrg.org
        ServerAdmin root@drgportal.org
        ServerSignature Off

        RewriteEngine On

        SSLProxyEngine On

        RewriteCond %{HTTPS} !=on
        # This checks to make sure the connection is not already HTTPS

        #RewriteRule ^/?(.*) https://opendrg.org/coinsync/login$1 [R=301,L]
        # This rule will redirect users from their original location, to the same location but using HTTPS.
        # i.e.  http://www.example.com/foo/ to https://www.example.com/foo/
        # The leading slash is made optional so that this will work either in httpd.conf
        # or .htaccess context

	# Do NOT rewrite Let's Encrypt validation requests
	RewriteRule ^/\.well-known/.+ - [END]
	#Alias /.well-known /var/www/_letsencrypt/.well-known

	RewriteCond %{REQUEST_URI} !^/\.well-known/(.*)
	RewriteCond %{SERVER_NAME} =opendrg.org
	RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

</VirtualHost>


<VirtualHost *:443>
        ServerName opendrg.org
        ServerAdmin root@opendrg.org
        ServerSignature Off

	RewriteEngine On

	# Do NOT rewrite Let's Encrypt validation requests
	RewriteRule ^/\.well-known/.+ - [END]

        RewriteRule ^/?$ https://opendrg.org/coinsync/login [END,R=301,L]

	# keycloak realms-workaround
        RewriteCond %{REQUEST_URI} "^/realms/" [NC]
        RewriteRule "/realms/(.*)" https://%{SERVER_NAME}/keycloak/realms/$1 [R=301]

       # enable HTTP/2, if available
        Protocols h2 http/1.1

        # Websocket
        RewriteMap container_map txt:/etc/apache2/proxyconf-application-containers/container_map.txt
        RewriteCond %{REQUEST_URI} ^/(?<container>(.+))/(?<study>(.+))/ws [NC,OR]
        RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC,OR]
        RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC]
        RewriteRule /(.*)/(.*)/(.*) ws://${container_map:$1}%{REQUEST_URI} [P,QSA,L]
        #LogLevel alert rewrite:trace5

	ErrorDocument 404 'Nothing here'
        ErrorDocument 500 'Nothing here'
        ErrorDocument 502 'Nothing here'
        ErrorDocument 503 'Nothing here'
        ErrorDocument 504 'Nothing here'

        ProxyRequests Off
        RequestHeader set X-Forwarded-Proto "https"
	ProxyPreserveHost On

        IncludeOptional proxyconf-application-containers/*.rewrite_rule
        IncludeOptional proxyconf-application-containers/*.conf

        SSLEngine on

        SSLCertificateFile /etc/letsencrypt/live/opendrg.org/fullchain.pem
        SSLCertificateKeyFile /etc/letsencrypt/live/opendrg.org/privkey.pem
        Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

Files in proxyconf-application-containers/ look like this:

<Location /coinsync/ >
	<IfModule mod_headers.c>
      		Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
	</IfModule>

        ProxyPass                       "http://nextcloud/coinsync/"
        ProxyPassReverseCookiePath      "/"  "/coinsync/"
</Location>

For some reason, requests first get rewritten to /coinsync/login.acme-challenge ... and consequently then get passed through to another server.

Reloading/restarting apache didn't change anything.

2 Likes

This is likely not doing what you expect:

Which is confirmed by the challenge requests being redirected from HTTP to HTTPS.
It missed them in HTTP, then misses them again in HTTPS.

2 Likes

Seems legit :smile: But do you have any idea why?
I'm using Apache 2.4.29 and thought this rule should match.

According to this StackOverflow comment one should use a leading slash when not using DocumentRoot, which I did. Either way (with our without slash) it doesn't work.

2 Likes

Welcome to the Let's Encrypt Community, Constantin :slightly_smiling_face:

Where to begin...


Get rid of this directive in your port 80 VirtualHost:

since it should only be used in an SSL-enabled VirtualHost (like port 443).


Get rid of this condition in your port 80 VirtualHost:

since it will only be applied to the next RewriteRule and is entirely redundant because it's being used in a non-SSL VirtualHost block (for port 80).


This condition in your port 80 VirtualHost:

will prevent redirection to https by this rule:

making this rule:

entirely unnecessary.


You missed this critical part of that comment:

Place this in .htaccess file

You are not (and certainly should not be) using .htaccess files. You are (correctly) using VirtualHosts.

This is the definitive information straight from the apache documentation:

Syntax: RewriteRule Pattern Substitution [flags]

In VirtualHost context, the Pattern will initially be matched against the part of the URL after the hostname and port, and before the query string (e.g. "/app1/index.html"). This is the (%-decoded) URL-path.

URL-path

The part of a url which follows the scheme and hostname as in /path/to/file.html. The url-path represents a web-view of a resource, as opposed to a file-system view.

2 Likes

From where is your port 80 VirtualHost supposed to serve a non-redirected challenge if you don't define a DocumentRoot?

2 Likes

To simplify your life, I highly recommend that you read this:

https://httpd.apache.org/docs/current/rewrite/avoid.html

2 Likes

This is also immensely helpful:

https://cwiki.apache.org/confluence/plugins/servlet/mobile?contentId=115522444#content/view/115522444

2 Likes

Here's a better understanding of ServerSignature:

https://blog.oshim.net/2016/04/turn-off-server-signature-on-apache-web/

2 Likes

Do you have a .htaccess file in your webroot with a redirect in it?

2 Likes

@griffin
Thanks for your input!

I was referring to the comment on the above mentioned SO answer: cweiske is talking of a vhost setup as well:

On an Apache 2.4 vhost without a document root, I had to add a slash after the ^ : RewriteRule ^/\.well-known/.+ - [END]

Anyways, as you pointed out there seem to be lots of misplaced directives in this configuration – now that you mentioned them they all make perfect sense to me. Being confronted with this setup I simply didn't question it because it had worked well up to this point. I will tackle these next week, as I'm only working part time on this project.
Thanks so far, can't wait to try this out!

Best
Constantin

3 Likes

Be sure to check if you have a .htaccess file with rewrites in it. I suspect that may be the primary root of your headaches. We can polish the rest afterwards.

See ya soon! :blush:

2 Likes

Apache only acts as a reverse proxy so I don't have a document root nor a .htaccess file.

1 Like

Yes, it doesn't seem to work, but I don't understand why.

From what I understood from the "When to avoid [mod_rewrite]…" page for reasons of simplicity and safety it is advisable to use Redirect instead of RewriteRule. Also I should probably use a RedirectMatch if I want to rewrite everything but LE requests, right?