Cannot renew certificate - "Invalid response 400 Bad Request"

I’ve been tearing my hair out over this…
Searching everywhere online, hours spent, still no progress in solving this issue.
All help will be warmly appreciated.

My domain is: arcanevoid.xyz

I ran this command: sudo certbot renew

It produced this output:
Attempting to renew cert (arcanevoid.xyz) from /etc/letsencrypt/renewal/arcanevoid.xyz.conf produced an unexpected error: Failed authorization procedure. www.arcanevoid.xyz (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.arcanevoid.xyz/.well-known/acme-challenge/2VQAX5eA_dSyl1RB5MjfcHr9YinF8T7nw3Z6OxU5Zu4 [163.172.178.77]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1", arcanevoid.xyz (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://arcanevoid.xyz/.well-known/acme-challenge/e4Y1e16A6e3czI1106dJiz6BMqsKjJxz21XaqvrHLZQ [163.172.178.77]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>400 Bad Request</title>\n</head><body>\n<h1>Bad Request</h1". Skipping.

My web server is (include version): Apache/2.4.18

The operating system my web server runs on is (include version): Ubuntu Xenial (16.04 LTS)

My hosting provider, if applicable, is: //

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): 0.31.0

Hi @dorearendil

that's simple. There is a check of your domain, 45 minutes old, there you see the problem ( https://check-your-website.server-daten.de/?q=arcanevoid.xyz ):

Domainname Http-Status redirect Sec. G
http://arcanevoid.xyz/
163.172.178.77 400 0.057 M
Bad Request
http://www.arcanevoid.xyz/
163.172.178.77 400 0.063 M
Bad Request
https://arcanevoid.xyz/
163.172.178.77 200 0.463 N
Certificate error: RemoteCertificateChainErrors
https://www.arcanevoid.xyz/
163.172.178.77 200 0.376 N
Certificate error: RemoteCertificateChainErrors
https://arcanevoid.xyz:80/
163.172.178.77 200 0.624 Q
Certificate error: RemoteCertificateNameMismatch, RemoteCertificateChainErrors
Visible Content: This is a mirage. map of your head dans la chambre rouge random pixels out of europa tlön, uqbar, orbis tertius random brazil san-in yakushima hokkaido touring caerbannoblog pachinko phenomenology another life engram seeker
https://www.arcanevoid.xyz:80/
163.172.178.77 200 0.416 Q
Certificate error: RemoteCertificateChainErrors
Visible Content: This is a mirage. map of your head dans la chambre rouge random pixels out of europa tlön, uqbar, orbis tertius random brazil san-in yakushima hokkaido touring caerbannoblog pachinko phenomenology another life engram seeker
http://arcanevoid.xyz/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
163.172.178.77 400 0.066 M
Bad Request
Visible Content: Bad Request Your browser sent a request that this server could not understand. Reason: You're speaking plain HTTP to an SSL-enabled server port. Instead use the HTTPS scheme to access this URL, please. Apache/2.4.18 (Ubuntu) Server at www.arcanevoid.xyz Port 443
http://www.arcanevoid.xyz/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
163.172.178.77 400 0.060 M
Bad Request
Visible Content: Bad Request Your browser sent a request that this server could not understand. Reason: You're speaking plain HTTP to an SSL-enabled server port. Instead use the HTTPS scheme to access this URL, please. Apache/2.4.18 (Ubuntu) Server at www.arcanevoid.xyz Port 443

Your port 80 - upps, sends https content. And checking port 80, your server answers with

Apache/2.4.18 (Ubuntu) Server at www.arcanevoid.xyz Port 443

You may have

  • a wrong port forwarding port 80 extern -> port 443 intern (and/or)
  • a wrong vHost configuration, so your port 80 is configured as a SSL port

What says

apachectl -S

Many thanks, Juergen. I had viewed Check-your-website for my domain before posting here, but the results were like Greek to me...

To answer your question, this is what I get when typing apachect -S:

 [Mon Jul 01 10:10:32.855912 2019] [so:warn] [pid 24014] AH01574: module cgid_module is already loaded, skipping
    AH00526: Syntax error on line 32 of /etc/apache2/sites-enabled/arcanevoid.xyz.conf:
    SSLCertificateFile: file '/etc/letsencrypt/live/www.arcanevoid.xyz/fullchain.pem' does not exist or is empty
    Action '-S' failed.
    The Apache error log may have more information.

Now, something really weird seems to have happened last night...
This morning, opening my website, I discovered that my certificates had been renewed! But I didn't do it.
Checking the /var/log/letsencrypt/letsencrypt.log file, it seems like some operation began yesterday around 11pm, as I was outside enjoying the cool night air an trying to forget about security certificates.
These are the first relevant lines in the log:

2019-06-30 22:59:37,416:DEBUG:certbot.main:certbot version: 0.31.0
2019-06-30 22:59:37,418:DEBUG:certbot.main:Arguments: ['-q']
2019-06-30 22:59:37,419:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-06-30 22:59:37,441:DEBUG:certbot.log:Root logging level set at 30
2019-06-30 22:59:37,443:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-06-30 22:59:37,466:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7f714fdc3710> and installer <certbot.cli._Default object at 0x7f714fdc3710>
2019-06-30 22:59:37,488:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2019-04-02 21:00:20 UTC.
2019-06-30 22:59:37,488:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2019-06-30 22:59:37,488:INFO:certbot.renewal:Non-interactive renewal: random delay of 49 seconds
2019-06-30 23:00:26,507:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2019-06-30 23:00:26,814:DEBUG:certbot_apache.configurator:Apache version is 2.4.18
2019-06-30 23:00:27,775:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7f714fe20e10>
Prep: True
2019-06-30 23:00:27,781:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7f714fe20e10>
Prep: True
2019-06-30 23:00:27,782:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_debian.DebianConfigurator object at 0x7f714fe20e10> and installer <certbot_apache.override_debian.DebianCo$
2019-06-30 23:00:27,782:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2019-06-30 23:00:27,840:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(only_return_existing=None, key=None, external_account_binding=None, status=None, agreement=None, contac$
2019-06-30 23:00:27,847:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-06-30 23:00:27,856:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2019-06-30 23:00:28,056:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 658
2019-06-30 23:00:28,057:DEBUG:acme.client:Received response:
HTTP 200
Expires: Sun, 30 Jun 2019 23:00:28 GMT
X-Frame-Options: DENY
Content-Type: application/json
Connection: keep-alive
Content-Length: 658
Server: nginx
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Strict-Transport-Security: max-age=604800
Date: Sun, 30 Jun 2019 23:00:28 GMT

What on earth happened? Happy to share the full log file if necessary. Did I just happen, by chance, to change my server settings to the right configuration before I left, thus enabling Certbot to renew my certs automatically??

Just to be clear: is the following a good config for my main vHost file, in order to work smoothly with Certbot? (I changed "VirtualHost *:80" to 443 before ditching the computer yesterday)

<VirtualHost *:443>
        # The ServerName directive sets the request scheme, hostname and port that
        # the server uses to identify itself. This is used when creating
        # redirection URLs. In the context of virtual hosts, the ServerName
        # specifies what hostname must appear in the request's Host: header to
        # match this virtual host. For the default virtual host (this file) this
        # value is not decisive as it is used as a last resort host regardless.
        # However, you must set it for any further virtual host explicitly.
        ServerName www.arcanevoid.xyz
        ServerAlias arcanevoid.xyz

        ServerAdmin don.caviare@gmail.com
        DocumentRoot /home/mycloud/www/html

        # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
        # error, crit, alert, emerg.
        # It is also possible to configure the loglevel for particular
        # modules, e.g.
        #LogLevel info ssl:warn

        ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined

        # For most configuration files from conf-available/, which are
        # enabled or disabled at a global level, it is possible to
        # include a line for only one particular virtual host. For example the
        # following line enables the CGI configuration for this host only
        # after it has been globally disabled with "a2disconf".
        #Include conf-available/serve-cgi-bin.conf

Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/www.arcanevoid.xyz/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.arcanevoid.xyz/privkey.pem
</VirtualHost>

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

That's simple. You have removed your https over port 80, so the normal renew has worked.

But the result is partial wrong.

You have created two new certificates ( https://check-your-website.server-daten.de/?q=arcanevoid.xyz#ct-logs ):

Issuer not before not after Domain names LE-Duplicate next LE
Let's Encrypt Authority X3 2019-06-30 2019-09-28 www.arcanevoid.xyz - 1 entries duplicate nr. 1
Let's Encrypt Authority X3 2019-06-30 2019-09-28 arcanevoid.xyz, www.arcanevoid.xyz - 2 entries duplicate nr. 1

But you use the wrong certificate with one domain name:

CN=www.arcanevoid.xyz
	01.07.2019
	29.09.2019
expires in 90 days	www.arcanevoid.xyz - 1 entry

Instead, you should use the certificate with both domain names.

Result: Your non-www is insecure.

Domainname Http-Status redirect Sec. G
http://arcanevoid.xyz/
163.172.178.77 200 0.066 H
http://www.arcanevoid.xyz/
163.172.178.77 200 0.060 H
https://arcanevoid.xyz/
163.172.178.77 200 0.600 N
Certificate error: RemoteCertificateNameMismatch
https://www.arcanevoid.xyz/
163.172.178.77 200 0.390 B

Try

certbot -reinstall -i apache -d arcanevoid.xyz -d www.arcanevoid.xyz

then certbot should find the certificate with both domain names and should ask, if you want to reinstall that certificate.

If that had worked:

  • make a backup of your certificates
  • delete the certificate with one domain name certbot delete certname

so you don't have two certificates, one with a wrong name.

Many, many thanks for your detailed explanations, Juergen.
Glad to know that I was at least on the right track with modifying my vHost file!

As regards the duplicate, I tried the command you suggested, but Certbot didn't ask me if I wanted to reinstall the certificate:

Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Cert not yet due for renewal
Keeping the existing certificate
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/arcanevoid.xyz.conf
Deploying Certificate to VirtualHost /etc/apache2/sites-enabled/arcanevoid.xyz.conf

I've actually enabled an automatic redirection from arcanevoid.xyz to www.arcanevoid.xyz, so I guess the security risk isn't that big anyway... Do you think I should still run certbot delete --cert-name arcanevoid.xyz just to be sure?

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.