Invalid response

My domain is: gnucash.org

I ran this command: certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/gnucash.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.gnucash.org
http-01 challenge for gnucash.org
Enabled Apache rewrite module
Waiting for verification...
Challenge failed for domain www.gnucash.org
http-01 challenge for www.gnucash.org
Cleaning up challenges
Attempting to renew cert (gnucash.org) from /etc/letsencrypt/renewal/gnucash.org.conf produced an unexpected error: Some challenges have failed.. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.gnucash.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.gnucash.org
Enabled Apache rewrite module
Waiting for verification...
Challenge failed for domain www.gnucash.org
http-01 challenge for www.gnucash.org
Cleaning up challenges
Attempting to renew cert (www.gnucash.org) from /etc/letsencrypt/renewal/www.gnucash.org.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gnucash.org/fullchain.pem (failure)
  /etc/letsencrypt/live/www.gnucash.org/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gnucash.org/fullchain.pem (failure)
  /etc/letsencrypt/live/www.gnucash.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
2 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.gnucash.org
   Type:   unauthorized
   Detail: 67.198.37.17: Invalid response from
   http://www.gnucash.org/.well-known/acme-challenge/LFR4SSD7fUAvgZsgRS3I2opfaQQwuThJqpv2fHf1_CU:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
 - The following errors were reported by the server:

   Domain: www.gnucash.org
   Type:   unauthorized
   Detail: 67.198.37.17: Invalid response from
   http://www.gnucash.org/.well-known/acme-challenge/VV_0FqkV1JB6fbI4dQp--v1bldHYFJHBR86EIc9c13Y:
   404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version): Apache2 2.4.41-4ubuntu3.14 amd64

The operating system my web server runs on is (include version): Ubuntu 20.04 LTS - focal

My hosting provider, if applicable, is: none/myself

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.40.0

Some more details:

  • I've been running certbot since 2016, and its worked great (a few hiccups in the early days)
  • It's been auto-renewing ever two months, just fine.
  • I attempted to upgrade to Ubuntu 22.04 LTS (from 20.04 LTS) and everything broke, wildly, in every which way. Completely unrelated to certbot, just other random stuff, a long laundry list of crazy stuff that broke.
  • I downgraded back to 20.04 LTS and everything went back to normal ... except for certbot.
  • Renewals stopped working; I've scoured the net for advice & recommendations, but nothing seems to work. (there are over a dozen pages dealing with my specific error. None cure the problem.)
  • To debug the problem, I created the following Apache2 config file:
NameVirtualHost *:80

<VirtualHost *:80>

  ServerName www.gnucash.org
  ServerAlias gnucash.org
  ServerAdmin webmaster@gnucash.org

#  Redirect / https://www.gnucash.org/

    # Hacks to enable SSL cert renewal
    DocumentRoot /var/lib/letsencrypt/http_challenges

    # Do NOT put trailing slash at the end of these!!
    Alias /.well-known/acme-challenge /var/lib/letsencrypt/http_challenges

    # `Allow from All` is the old apache 2.2 config
    # `Require all granted` is the new apache 2.4 config
    # Do NOT put trailing slash on either Directory or Location!
    <Directory /var/lib/letsencrypt/http_challenges>
        Require all granted
        Satisfy Any
        AllowOverride None
        ForceType text/plain
        Options +Indexes
    </Directory>
    <Location /.well-known/acme-challenge>
        Require all granted
    </Location>

</VirtualHost>
  • I created a file: /var/lib/letsencrypt/http_challenges/foo with contents asdf -- you can view it here:
    Index of /.well-known/acme-challenge

  • and also here:
    http://gnucash.org/

  • These are live, right now; I'm leaving the server open till things get debugged. The actual live website (with expired cert) is here: https://gnucash.org/ -- It still works, as long as you accept the expired-cert menu.

  • The /var/log/apache2 directory contains five logfiles:

access.log  gnucash-access.log  other_vhosts_access.log
error.log   gnucash-error.log
  • I watch these during the renewal. For the two gnucash logfiles, nothing happens (as expected; they are logging the https traffic).
  • The error.log file shows that certbot restarts or reloads apache; it looks fine.
  • The other_vhosts_access.log and the access.log give conflicting messages. The first shows a 200 and the second shows 404 !! For example:
other_vhosts_access.log:
www.gnucash.org:80 18.224.63.216 - - [25/Mar/2023:05:47:17 +0000] "GET /.well-known/acme-challenge/luAcDxDw1uq4GDJDyT5HUt2rfAHCNUPi_Ev5ggYGtMs HTTP/1.1" 200 407 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

but

access.log:
18.224.63.216 - - [25/Mar/2023:05:47:23 +0000] "GET /.well-known/acme-challenge/4-wwAfdgkn1lM6PH8c3YXFKQH7tY8aP3orR4RAUMcCc HTTP/1.1" 404 457 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
  • Note the timestamps are six seconds apart. Note that two different challenges are being used.

  • The first is always 200, the second is always 404.

  • The second is always six seconds after the first.

  • The challenge keys always differ between the first and second.

  • The contents of /var/log/letsencrypt/letsencrypt.log are eye-wateringly long and uninformative. I've read them repeatedly and just can't find anything that seems unexpected.

I'm stumped. I don't know what to do. The strange symptoms suggest that maybe certbot is running twice, even though I invoke it only once? And maybe the second time, the directory is being unexpectedly cleaned up? I'm fresh out of ideas.

I'd guess that you have multiple vhosts that cover www.gnucash.org. Potentially a unnamed vhost which takes its name from your server hostname.

It should be evident in the output from this command, if that's the case:

sudo apachectl -t -D DUMP_VHOSTS
3 Likes

Oh, well, that's an excellent clue! And following the clue does deliver the fix. Sheesh!

So, /etc/apache2/sites-available has everything I expect:

000-default.conf  default-ssl.conf  gnucash.conf  
gnucash-le-ssl.conf  ntp.conf

but golly gee whiz, sites-enabled has a recently minted 000-default.conf -> ../sites-available/000-default.conf which points at the bog-standard and wildly incorrect default. Removing 000-default.conf fixes the problem!

For whatever reason, that bogus 000-default.conf did not affect the main website; it only mangled certbot.

OMG, thanks! I spent about two weeks, on and off, trying to figure this out. Phew. I'm happy that's over!

2 Likes

If you requested a cert for two names (gnucash.org & www.gnucash.org), then two challenges are expected [one for each name].

The first succeeded ("200") because it was able to reach the correct folder.
The second failed ("404") because it failed to reach the correct folder.
Why?
As you have found, Apache does not warn you when one vhost overlaps another.
One of those two names also existed in some other vhost that just happen to be processed (alphabetically) before the one you expected to handle both names.

3 Likes

Thanks. Yes. Well, very specifically, it was the 000-default.conf that gets installed, when one installs apache2 for the first time ever. During unrelated debugging, I had to uninstall and reinstall apache2 and the reinstall failed to notice that there was a website that was already configured. That default got enabled, but it did not cause any issues, in that the normal website continued to work just fine, with no hint of anything wrong ... until certbot discovered it.

So, technically, this is a bug with the Debian and/or Ubuntu apache2 installer script. I guess. And perhaps I should open a bug for that... Any suggestions on how to best do that?

2 Likes

Do you still have the contents of that file? [ifs so, may we see it?]
I suspect it uses the default servername [set in the main config file] by not setting any name at all.
[which should never be set to a name that will be served within any other vhost]

1 Like

Yes, sure. I copy it below, but it is the completely stock unmodified config file that gets installed when you install apache2 (say, on your laptop, or wherever). It works great, if you're just getting started with Apache.

<VirtualHost *:80>
	# The ServerName directive sets the request scheme, hostname and port that
	# the server uses to identify itself. This is used when creating
	# redirection URLs. In the context of virtual hosts, the ServerName
	# specifies what hostname must appear in the request's Host: header to
	# match this virtual host. For the default virtual host (this file) this
	# value is not decisive as it is used as a last resort host regardless.
	# However, you must set it for any further virtual host explicitly.
	#ServerName www.example.com

	ServerAdmin webmaster@localhost
	DocumentRoot /var/www/html

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

Presumably, it's the (bogus) DocumentRoot value that was goofing things up The gnucash website is elsewhere in the filesystem; there is a /var/www/html/index.html but it contains only some Debian/Ubuntu hello-world boilerplate.

As I suspected, there is no set value for servername; So, it defaults to use the one set in the main config file.
Which is likely one of those two names [gnucash.org or www.gnucash.org].
Again, it is bad practice to name the server with one of the FQDNs that it will be serving.

2 Likes

I'm confused by this remark. There are multiple virtual hosts. They're containerized, they're sitting behind firewalls that forward ports, etc. Are you suggesting that if I just comment out ServerName and ServerAlias, everything will continue to work? It's late at night here, and I'm tired, so I'm not about to embark on another adventure, but I guess I can look at this tomorrow.

Not at all.
What I'm saying is the exact opposite.
The servernames should only be in the vhosts.
The default servername in the main config should never overlap/clash with any name used in any vhost.

It's 3:45am here - LOL
Cheers from Miami :beers:

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.