Invalid response: unauthorized (404) with certbot certificate generation (all details provided)

We have been having issues with certbot and the creation/renewal of SSL certificates for a while now on one of our development server.

Seeing as we ran into issues every time we had to renew in the past few months, and that this time I simply couldn’t get it to work, I decided to remove every certificate we had, clean a few things up and generate a new one to hopefully get it to work flawlessly once and for all. Unfortunately, as you might’ve guessed from my presence in these forums, things didn’t go according to plans.

I am running into the same issues as we were having previously, which is that every certificate creation or renewal ends results in the “unauthorized” error. I’ve scoured the Internet looking for answers or clues, but alas, I still cannot get it to work.

Here are some of the things I have tried:

  • Creating a file in the different /.well-known/acme-challenge directories and checking if it can be reached from a browser - it works, as you can see here: http://demo.bpdl.eckidev.com/.well-known/acme-challenge/test
  • Shutting off Apache and trying to run the commands with --standalone
  • Running certbot for one domain at a time (one certificate each). Ran into the same issue, obviously.

Here is the classic information form; with a --dry-run to avoid running into the limits overtime with all of my testing. I’ve also added an Apache configuration at the bottom; every domain has the same config, with only the domain, directories and user names changing.


My domain is: bpdl.eckidev.com (as well as the following: david.bpdl.eckidev.com, demo.bpdl.eckidev.com, emile.bpdl.eckidev.com, jd.bpdl.eckidev.com, pa.bpdl.eckidev.com)

I ran this command: certbot certonly --dry-run --apache -d bpdl.eckidev.com -d david.bpdl.eckidev.com -d demo.bpdl.eckidev.com -d emile.bpdl.eckidev.com -d jd.bpdl.eckidev.com -d pa.bpdl.eckidev.com

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for bpdl.eckidev.com
http-01 challenge for david.bpdl.eckidev.com
http-01 challenge for jd.bpdl.eckidev.com
http-01 challenge for pa.bpdl.eckidev.com
http-01 challenge for demo.bpdl.eckidev.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. demo.bpdl.eckidev.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://demo.bpdl.eckidev.com/.well-known/acme-challenge/H59PQEdc2ASbknzDz0x4p41szNu1sB_0kPjKQPdxjcU [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p", bpdl.eckidev.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://bpdl.eckidev.com/.well-known/acme-challenge/Pc4MQOzm09KjVcV6S6B-t6aV_qecYYkfcXMcWOuad0A [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p", jd.bpdl.eckidev.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://jd.bpdl.eckidev.com/.well-known/acme-challenge/1irJPXAnysDSBqFTD1SgWB6ApHZwYYUM4PgM_wOBA5I [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p", david.bpdl.eckidev.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://david.bpdl.eckidev.com/.well-known/acme-challenge/OzyJI2jfrQiZK44r5Xsw4JNWMaZCcoP9p-X_J2EKAAc [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p", pa.bpdl.eckidev.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://pa.bpdl.eckidev.com/.well-known/acme-challenge/7suTBGYDsDUcHW_maNiW_EpSaioMDp4_C6XQHkoZ5Mc [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>404 Not Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: demo.bpdl.eckidev.com
   Type:   unauthorized
   Detail: Invalid response from
   http://demo.bpdl.eckidev.com/.well-known/acme-challenge/H59PQEdc2ASbknzDz0x4p41szNu1sB_0kPjKQPdxjcU
   [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: bpdl.eckidev.com
   Type:   unauthorized
   Detail: Invalid response from
   http://bpdl.eckidev.com/.well-known/acme-challenge/Pc4MQOzm09KjVcV6S6B-t6aV_qecYYkfcXMcWOuad0A
   [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: jd.bpdl.eckidev.com
   Type:   unauthorized
   Detail: Invalid response from
   http://jd.bpdl.eckidev.com/.well-known/acme-challenge/1irJPXAnysDSBqFTD1SgWB6ApHZwYYUM4PgM_wOBA5I
   [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: david.bpdl.eckidev.com
   Type:   unauthorized
   Detail: Invalid response from
   http://david.bpdl.eckidev.com/.well-known/acme-challenge/OzyJI2jfrQiZK44r5Xsw4JNWMaZCcoP9p-X_J2EKAAc
   [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   Domain: pa.bpdl.eckidev.com
   Type:   unauthorized
   Detail: Invalid response from
   http://pa.bpdl.eckidev.com/.well-known/acme-challenge/7suTBGYDsDUcHW_maNiW_EpSaioMDp4_C6XQHkoZ5Mc
   [165.227.41.202]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>404 Not
   Found</title>\n</head><body>\n<h1>Not Found</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is: Apache/2.4.18

The operating system my web server runs on is: Ubuntu 16.04.4

My hosting provider, if applicable, is: Digital Ocean droplet

I can login to a root shell on my machine: yes

I’m using a control panel to manage my site: no

The version of my client is: certbot 0.31.0


Here is an Apache VirtualHost configuration (others follow the same pattern, as stated above):

<VirtualHost *:80>
    DocumentRoot /home/emile/emile.bpdl.eckidev.com/public_html/
</VirtualHost>

<VirtualHost *:80 *:443>
    ServerName emile.bpdl.eckidev.com

    AssignUserId emile emile

    ServerAdmin dev@eckinox.ca
    Define PROJECT_PATH "/home/emile/emile.bpdl.eckidev.com/"

    LogLevel warn ssl:warn
    Include conf-available/website.conf

    # Commented those lines when removing the old certificates
    #Include /etc/letsencrypt/options-ssl-apache.conf
    #SSLCertificateFile /etc/letsencrypt/live/emile.bpdl.eckidev.com/fullchain.pem
    #SSLCertificateKeyFile /etc/letsencrypt/live/emile.bpdl.eckidev.com/privkey.pem
</VirtualHost>

And here is the content of the website.conf, which is included in the VirtualHost:

RewriteEngine On

ErrorLog ${PROJECT_PATH}/logs/apache.error.log
CustomLog ${PROJECT_PATH}/logs/apache.access.log combined
DirectoryIndex index.html index.php
DocumentRoot "${PROJECT_PATH}/public_html/"

<Directory "${PROJECT_PATH}/public_html/">
     Options Indexes FollowSymLinks
     Options -MultiViews -Indexes
     AllowOverride All
     Require all granted
</Directory>

<IfModule mod_deflate.c>
    AddOutputFilterByType DEFLATE application/javascript
    AddOutputFilterByType DEFLATE application/rss+xml application/rdf+xml
    AddOutputFilterByType DEFLATE application/vnd.ms-fontobject
    AddOutputFilterByType DEFLATE application/x-font
    AddOutputFilterByType DEFLATE application/x-font-opentype
    AddOutputFilterByType DEFLATE application/x-font-otf
    AddOutputFilterByType DEFLATE application/x-font-truetype
    AddOutputFilterByType DEFLATE application/x-font-ttf
    AddOutputFilterByType DEFLATE application/x-javascript
    AddOutputFilterByType DEFLATE application/xhtml+xml application/xml
    AddOutputFilterByType DEFLATE font/otf font/ttf font/eot font/opentype
    AddOutputFilterByType DEFLATE image/svg+xml image/x-icon
    AddOutputFilterByType DEFLATE application/json
    AddOutputFilterByType DEFLATE text/css text/html text/javascript text/plain text/xml text/cache-manifest
</IfModule>

<ifModule mod_headers.c>
  Header always set X-Content-Type-Options nosniff 

   # Cache control
    <FilesMatch "\.(jpg|jpeg|png|gif|css|js|swf|ico|bmp|eot|woff|woff2|ttf|svg|ogv|mp4|webm)$">
        Header set Cache-Control "max-age=1209600, public"
    </FilesMatch>

    # Allows font from CDN to load flawlessly
    <FilesMatch "\.(eot|ttf|otf|woff|woff2|svg)$">
        Header set Access-Control-Allow-Origin "*"
    </FilesMatch>
</ifModule>

Although I’m not the one who originally setup this server and those configurations, I’ve looked over everything and cannot for the life of me find the source of the problems. I’ve set up multiple websites and web apps in the past without any issues.

Thanks in advance for your help,
Émile

@bmw, is it bad to have both <VirtualHost *:80> and <VirtualHost *:80 *:443>? That doesn’t feel like a familiar setup to me and perhaps it’s confusing to Certbot.

If I am not mistaken, I believe the first VirtualHost (<VirtualHost *:80>) had been added by the original developer specifically to help Certbot detect the right document path for the domain’s configuration (unless the website.conf is parsed and included correctly by Certbot, in which case I can simply remove it).

I will try removing the <VirtualHost *:80>, leaving only the other one, and let you know how it goes.

Leaving only the <VirtualHost *:80 *:443> caused the follow issue with Certbot:
Unable to find a virtual host listening on port 80 which is currently needed for Certbot to prove to the CA that you control your domain. Please add a virtual host for port 80.)

I have therefore updated the VirtualHost configs to the following template:

<VirtualHost *:80>
    ServerName emile.bpdl.eckidev.com

    AssignUserId emile emile

    ServerAdmin dev@eckinox.ca
    DocumentRoot "/home/emile/emile.bpdl.eckidev.com/public_html/"

    Define PROJECT_PATH "/home/emile/emile.bpdl.eckidev.com/"
    Include conf-available/website.conf
</VirtualHost>

<VirtualHost *:443>
    ServerName emile.bpdl.eckidev.com

    AssignUserId emile emile

    ServerAdmin dev@eckinox.ca
    DocumentRoot "/home/emile/emile.bpdl.eckidev.com/public_html/"
    Define PROJECT_PATH "/home/emile/emile.bpdl.eckidev.com/"

    LogLevel warn ssl:warn
    Include conf-available/website.conf
</VirtualHost>

and removed the DocumentRoot that was originally in the included website.conf.

However, the original issue still remains:
Invalid response from http://bpdl.eckidev.com/.well-known/acme-challenge/ft3GbbFUE6Sg_Vz13s6Ihr9NW57UsaiaSGYr3q0pBCA (and many others)

The response is still a 404 Not Found error.

I have tried removing the <VirtualHost *:443>, but no luck either,

Hi @EmileP

a

definition is always bad. Checking your first domain there is the expected result - Grade Q, http sent over port 443 ( https://check-your-website.server-daten.de/?q=bpdl.eckidev.com ):

Domainname Http-Status redirect Sec. G
http://bpdl.eckidev.com/
165.227.41.202 200 0.240 H
https://bpdl.eckidev.com/
165.227.41.202 -4 0.437 W
SendFailure - The underlying connection was closed: An unexpected error occurred on a send. The handshake failed due to an unexpected packet format.
http://bpdl.eckidev.com:443/
165.227.41.202 400 0.226 Q
Bad Request
Visible Content: Bad Request Your browser sent a request that this server could not understand. Reason: You’re speaking plain HTTP to an SSL-enabled server port. Instead use the HTTPS scheme to access this URL, please. Apache/2.4.18 (Ubuntu) Server at bpdl.eckidev.com Port 443
http://bpdl.eckidev.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
165.227.41.202 404 0.223 A
Not Found
Visible Content: Not Found The requested URL /.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de was not found on this server. Apache/2.4.18 (Ubuntu) Server at bpdl.eckidev.com Port 80

But /.well-known/acme-challenge answers as expected with http status 404 - Not Found.

Looks like you have a lot of multiple definitions.

What says

apachectl -S

PS: Now checked your emile.bpdl.eckidev.com - same problem, http over port 443, Grade Q.

1 Like

PS: Your setup is really bad. Looks like you have two wrong configured ports per domain.

Normally, the answer

An unexpected error occurred on a send. The handshake failed due to an unexpected packet format.

of a https port says: “Hey, I am a http port, I don’t send the small handshake, instead the complete content -> result is too long”.

So checking https + port 443, the tool tests http + port 443. Typically, a regular http status 200 is returned.

But there is the typical Q-answer of a http request - Bad Request:

Bad Request Your browser sent a request that this server could not understand. Reason: You’re speaking plain HTTP to an SSL-enabled server port. Instead use the HTTPS scheme to access this URL, please. Apache/2.4.18 (Ubuntu) Server at bpdl.eckidev.com Port 443

So it looks that you have two wrong configured ports, one answers port 80, one answers port 443.

But the curious thing: Your /.well-known/acme-challenge works, looks like you have additional definitions in that directory.

So it’s a “configuration never seen”.

Thanks for the help and information @JuergenAuer,

I’ve removed the <VirtualHost *443> for the time being (until the certificates are created and configured correctly in those entries).

This leaves me with each apache configuration looking like the following:

<VirtualHost *:80>
    ServerName emile.bpdl.eckidev.com

    AssignUserId emile emile

    ServerAdmin dev@eckinox.ca
    DocumentRoot "/home/emile/emile.bpdl.eckidev.com/public_html/"

    Define PROJECT_PATH "/home/emile/emile.bpdl.eckidev.com"
    Include conf-available/website.conf
</VirtualHost>

Now at this point, here is the result of apachectl -S:

VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server bpdl.eckidev.com (/etc/apache2/sites-enabled/bpdl.eckidev.com.conf:1)
         port 80 namevhost bpdl.eckidev.com (/etc/apache2/sites-enabled/bpdl.eckidev.com.conf:1)
         port 80 namevhost david.bpdl.eckidev.com (/etc/apache2/sites-enabled/david.bpdl.eckidev.com.conf:1)
         port 80 namevhost demo.bpdl.eckidev.com (/etc/apache2/sites-enabled/demo.bpdl.eckidev.com.conf:1)
         port 80 namevhost emile.bpdl.eckidev.com (/etc/apache2/sites-enabled/emile.bpdl.eckidev.com.conf:1)
         port 80 namevhost jd.bpdl.eckidev.com (/etc/apache2/sites-enabled/jd.bpdl.eckidev.com.conf:1)
         port 80 namevhost pa.bpdl.eckidev.com (/etc/apache2/sites-enabled/pa.bpdl.eckidev.com.conf:1)

One vhost per domain, all on port 80, and all of which look like the above snippet.

However, I still get the same “unauthorized” response with a 404 error.

Well, as it turns out, after all of this… the issue wasn’t the configurations themselves. Running a netstat -peanut lead me to discover the following:

Somehow, there was a second apache2 process running on the server; one that was not controlled by service. Therefore, whenever we changed the configurations and reloaded or restarted the web server with service apache2 reload or service apache2 restart, well, the other apache2 was still up and running with the old configurations.

I simply stopped the “real” apache2 process with service apache2 stop, killed the remaining apache2 processes with a killall -9 apache2, and then started the web server back up with the service apache2 start. Obviously, everything worked wonders after that.

Sorry for wasting your time with this issue, and thanks a lot for your help - without it, I probably would not have found the issue!

2 Likes

Yep, sometimes there are such orphaned processes. Then a kill or a reboot is required.

And now

your configuration is clean :+1:

Happy to read that it had worked.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.