404 on certificate renewal

Help
#1

My domain is: hq.zajc.pl

I ran this command: sudo certbot certonly --apache -d hq.zajc.pl -w /var/www/owncloud --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Cert is due for renewal, auto-renewing...
Simulating renewal of an existing certificate for hq.zajc.pl
Performing the following challenges:
http-01 challenge for hq.zajc.pl
Waiting for verification...
Challenge failed for domain hq.zajc.pl
http-01 challenge for hq.zajc.pl
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

My web server is Apache/2.4.41 (Ubuntu)

The operating system my web server runs on is Ubuntu 20.04.2

I can login to a root shell on my machine: yes.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Certbot 1.13.0

Here are two virtual hosts I run on this machine:

1:

<IfModule mod_ssl.c>
<VirtualHost *:443>
        DocumentRoot /var/www/owncloud
<------>ErrorLog ${APACHE_LOG_DIR}/error.log
<------>CustomLog ${APACHE_LOG_DIR}/access.log combined
SSLCertificateFile /etc/letsencrypt/live/hq.zajc.pl/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/hq.zajc.pl/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
HostNameLookups off
UseCanonicalName off
ServerName hq.zajc.pl
<IfModule mod_headers.c>
  Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
</IfModule>
</VirtualHost>
</IfModule>

The second one:

<VirtualHost *:80>
DocumentRoot /var/www/owncloud
# RewriteEngine on
# RewriteRule ^\.well-known\/acme-challenge\/ - [L]
# RewriteCond %{SERVER_NAME} =hq.zajc.pl
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
HostNameLookups off
ServerName hq.zajc.pl
</VirtualHost>

(please note I turned off redirect to https).

No need to mention that the domain resolves perfectly and there is no AAAA record in the DNS, as we do not run IPv6.

1 Like
#2

Hi @TheWojtek

why should that

in combination with this

work?

1 Like
#3

Hi @JuergenAuer

I just corrected the previous post - obviously you spotted it right, it was actually my previous attempt.

The output is:
wojtek@hq:~$ sudo certbot certonly --apache -d hq.zajc.pl -w /var/www/owncloud --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Cert is due for renewal, auto-renewing...
Simulating renewal of an existing certificate for hq.zajc.pl
Performing the following challenges:
http-01 challenge for hq.zajc.pl
Waiting for verification...
Challenge failed for domain hq.zajc.pl
http-01 challenge for hq.zajc.pl
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: hq.zajc.pl
   Type:   unauthorized
   Detail: Invalid response from
   http://hq.zajc.pl/.well-known/acme-challenge/tgM-YcjjfYSE7CcJk0ZCbOFsV5z64U5rU60aLCX5Ylk
   [91.216.30.50]: 404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
wojtek@hq:~$ sudo mc
1 Like
#4

May be that vHost isn't used.

What says

apachectl -S
1 Like
#5 
**wojtek@hq** : **~** $ apachectl -S

AH00526: Syntax error on line 6 of /etc/apache2/sites-enabled/hq.zajc.pl-secure.conf:

SSLCertificateFile: file '/etc/letsencrypt/live/hq.zajc.pl/cert.pem' does not exist or is empty

Action '-S' failed.

The Apache error log may have more information.

If I commented out the SSLCertificateFile line, Apache doesn't start. And the cert.pem is there.

#6

Combining --apache and -w doesn't make sense. You're using just the apache plugin and the -w option is ignored, as you can see from Plugins selected: Authenticator apache, Installer apache

Also, for the apache plugin to work, you really need a working Apache configuration. Please fix your configuration (i.e., a working apachectl -S) before trying certbot --apache again.

The fact a symbolink link (which the files in /live/ are) doesn't mean it's actually working. It could be pointing to a non-existing destination.

1 Like
#7

I am perfectly aware it is a symlink, it points correctly to the actual file located in /archive (as does the rest of the symlinks in /live) and contains a certificate.

I read through Certbot docs and issued:
chmod 0755 /etc/letsencrypt/{live,archive}

Now the apachectl -S output is:

wojtek@hq:/$ apachectl -S VirtualHost configuration: *:443 hq.zajc.pl (/etc/apache2/sites-enabled/hq.zajc.pl-secure.conf:2) *:80 hq.zajc.pl (/etc/apache2/sites-enabled/hq.zajc.pl.conf:1) ServerRoot: "/etc/apache2" Main DocumentRoot: "/var/www/owncloud" Main ErrorLog: "/var/log/apache2/error.log" Mutex ssl-stapling: using_defaults Mutex ssl-cache: using_defaults Mutex default: dir="/var/run/apache2/" mechanism=default Mutex mpm-accept: using_defaults Mutex watchdog-callback: using_defaults Mutex rewrite-map: using_defaults Mutex ssl-stapling-refresh: using_defaults PidFile: "/var/run/apache2/apache2.pid" Define: DUMP_VHOSTS Define: DUMP_RUN_CFG User: name="www-data" id=33 not_used Group: name="www-data" id=33 not_used

Now for the renew dry-run (I cut down the command by your suggestion, @Osiris):

wojtek@hq:/$ sudo certbot certonly --apache -d hq.zajc.pl --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Cert is due for renewal, auto-renewing...
Simulating renewal of an existing certificate for hq.zajc.pl
Performing the following challenges:
http-01 challenge for hq.zajc.pl
Waiting for verification...
Challenge failed for domain hq.zajc.pl
http-01 challenge for hq.zajc.pl
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: hq.zajc.pl
   Type:   unauthorized
   Detail: Invalid response from
   http://hq.zajc.pl/.well-known/acme-challenge/hqVRFh6WRfvZzK6p1k3RtnIvUs1Tr53DeARqx6cx5S0
   [91.216.30.50]: 404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
#8

Run it as root or sudo.

#9

My last post with logs was "hidden" from the view here, I obviously sudo the certbot command.

#10

Your apachectl -S requires sudo.

2 Likes
#11

I may be dumb, but not that dumb... :slight_smile:

wojtek@hq:/$ sudo apachectl -S
VirtualHost configuration:
*:443                  hq.zajc.pl (/etc/apache2/sites-enabled/hq.zajc.pl-secure.conf:2)
*:80                   hq.zajc.pl (/etc/apache2/sites-enabled/hq.zajc.pl.conf:1)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/owncloud"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex mpm-accept: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
#12

Then use the correct webroot command.

1 Like
#13

I would love to, however:

**wojtek@hq** : **/** $ sudo certbot certonly --webroot -w /var/www/owncloud -d hq.zajc.pl --dry-run

Outputs:

Plugins selected: Authenticator webroot, Installer None
Cert is due for renewal, auto-renewing...
Simulating renewal of an existing certificate for hq.zajc.pl
Performing the following challenges:
http-01 challenge for hq.zajc.pl

Using the webroot path /var/www/owncloud for all unmatched domains.

Waiting for verification...
Challenge failed for domain hq.zajc.pl
http-01 challenge for hq.zajc.pl
Cleaning up challenges

Some challenges have failed.

**IMPORTANT NOTES:**

- The following errors were reported by the server:

Domain: hq.zajc.pl
Type: unauthorized
Detail: Invalid response from
http://hq.zajc.pl/.well-known/acme-challenge/VaL34tr9h8BHUYIFSd3xe-d4m2jovZmp-ENgrD6dadk
[91.216.30.50]: 404

I can not see any issues with webroot, which is correct above.
The same error happens if I use "apache" instead of "webroot" in certbot command. No matter if I point it manually (-w /var/www/owncloud) to the webroot or if I leave the Apache plugin to do it automatically like it should do.

#14

Then you have additional definitions, so /.well-known/acme-challenge is handled by another program. Or additional location definitions.

Find these and remove these.

And what's that?

http://hq.zajc.pl/index.html?_1615646365952

The plugin does not support this browser.

It's not possible to copy that nonsense. Your system want's expired browsers.

PS: It's your system, so you have to fix it. It's not the job of this forum if you have such a configuration.

1 Like
#15

No, Jurgen. I do not have any additional definitions nor location definitions. I actually removed everything certbot-related and reinstalled the snap with the same effect. Whatever configuration was available I have provided it and I am happy to provide whatever else may be helpful to find out the cause.

The renewal does not work even if "standalone" module is selected and launched after Apache is stopped. As I understand it, "standalone" launches it's own webserver for verification:

**wojtek@hq** : **/** $ sudo certbot certonly --standalone -d hq.zajc.pl --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Cert is due for renewal, auto-renewing...
Simulating renewal of an existing certificate for hq.zajc.pl

Performing the following challenges:
http-01 challenge for hq.zajc.pl
Waiting for verification...
Challenge failed for domain hq.zajc.pl
http-01 challenge for hq.zajc.pl
Cleaning up challenges
Some challenges have failed.

**IMPORTANT NOTES:**

- The following errors were reported by the server:
Domain: hq.zajc.pl
Type: unauthorized
Detail: Invalid response from http://hq.zajc.pl/.well-known/acme-challenge/eYpFCUrUN3eA7vPGkPEI3H_znrp-I8cOabnLo3utj6U

[91.216.30.50]: 404

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.

I don't know what you mean by "nonsense", however the page you linked redirects to owncloud login, and indeed the sentence "your system want's expired browsers" really makes no sense in English - I regretfully have no idea what you meant.

Yes, it is my system and I am trying to fix it. Came here for constructive "community support", not for patronizing.

#16

Unfortunately, @JuergenAuer can use some unconstructive language indeed.

That being said, the fact --standalone doesn't error out about NOT being able to bind to port 80 tells me you're not running certbot on the same host as where your webserver is running. Why? Because --standalone tries to bind to port 80 itself. This should be impossible if there is already an Apache webserver running on port 80. Or there is something very strange going on on your server which I'm not understanding. Perhaps multiple IP addresses?

Also, as @JuergenAuer also already provided: your Apache documentroot says "owncloud", but if we surf to your site, we're NOT getting an OwnCloud website. I have absolutely no idea what I'm seeing, other than that earlier quoted error message. The source doesn't tell me what the site is running as software. The <title> says "Web Viewer". There are some <div>s with id "liveSoundLineBox" and "dualtalkSoundLineBox". What am I connecting to?

1 Like
#17

I don't see an owncloud login.

I see

2021-03-13. hq.zajc.pl

There is a short check, I don't see the details, then I see that - Web Viewer

Your root has something like

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml"><head>
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<title></title>
<!--codebase="RSVideoOcx.cab#version=1.0.1.63"--><!--IE package version number, for the board to read, can not delete-->
<script>
	var http = window.location.href;//ip ->ipc-> ip/login.html
	var addInfo = http.split("//")[1].split("?")[1];
	if(typeof addInfo == "undefined" || !(/https?/).test(http)){//local login,eg:file:///E:/.../login.html
		var t=new Date;
		location.replace("index.html?_"+t.getTime());
	}else{  // eg: http://172.18.13.44/?999;eg:http://[ip]:[port]/?username=admin&password=000000
		location.replace("index.html?"+addInfo);
	}
</script>

</head>

</html>

There is a redirect to index.html+time.

Looks like there runs another instance so your "real webserver" is invisible, so Certbot creates the file in the wrong place.

PS: Is there a firewall, a proxy server? As @Osiris wrote:

PPS: I hope nobody uses such old / insecure browsers.

1 Like
#18

I'm guessing "router" or "NAS", as I'm also seeing things like a <div> with class "popupBox" and id "reboot_prompt" in the source. So it apparently is a device which can be rebooted from the web page.

And the firmware of that device is ANCIENT! My IPU, Chrome 44 or lower? What kind of website is that? :laughing:

1 Like
#19

Yep, something, that answers, so the "real webserver" isn't visible.

Looks like you have a very old component installed. We can't know that. But such a component would explain why certificate creation doesn't work. Because it's not your webserver, that answers, it's that old instance.

1 Like
#20

I can access the owncloud login page through http://hq.zajc.pl:443/

So there seems to be an incorrect portmap for port 80? As port 80 obviously shows a very different page..

1 Like