Certbot error - failed authorization procedure

yaandy · January 13, 2020, 7:54pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: invystasafe.com

I ran this command: certbot --apache

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
No names were found in your configuration files. Please enter in your domain
name(s) (comma and/or space separated) (Enter ‘c’ to cancel): invystasafe.com
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for invystasafe.com
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. invystasafe.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://invystasafe.com/.well-known/acme-challenge/OYyHDH1mlsaGcX3Ru_q5pqcE1TnCWsHRK6lYOONyUjk [54.219.172.188]: “\n\n404 Not Found\n\n

Not Found

\n<p”

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: invystasafe.com
Type: unauthorized
Detail: Invalid response from
http://invystasafe.com/.well-known/acme-challenge/OYyHDH1mlsaGcX3Ru_q5pqcE1TnCWsHRK6lYOONyUjk
[54.219.172.188]: “\n\n404 Not
Found\n\n

Not Found
\n<p”
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.My web server is (include version):

The operating system my web server runs on is (include version): Ubuntu 18.04 LTS

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): Not sure what a ‘control panel’ is in this context. I use AWS to manage the server instance, and ssh into the instance to start/stop the Apache server

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0

LuB · January 13, 2020, 10:58pm

What SSH client do you use?

For instance I use PuTTY.

Let us know mate!

yaandy · January 13, 2020, 11:28pm

I use the standard Ubuntu ssh client (literally called ‘ssh’). Also, with an AWS EC2 instance, only certain IP’s (as configured by me) can ssh from a remote IP to my AWS instance.

I am curious why are the question regarding ssh client; can you elaborate?

Thank you

LuB · January 14, 2020, 1:40am

Oh its just I had issues(not the exact same, but similar) until I used 'sudo -s' in linux.

I have absolutely no idea what that command does to be honest, but it kick started my ability to be able to interact with certbot. (my linux skills are white belt levels, i am but a humble php developer grappling with this stuff lol)

Do any other commands work? Can you run:

certbot certificates

Cheers

yaandy · January 14, 2020, 3:31am

% certbot certificate
Saving debug log to /var/log/letsencrypt/letsencrypt.log

No certs found.

As a new user, It appears I cannot upload the letsencrypt.log file, so I have it on a google drive that you can access - https://drive.google.com/file/d/1S-BJRaxCTtiZA4RTl0soMYa89yJXMnk2/view?usp=sharing

Thank you for your help in advance.

LuB · January 14, 2020, 3:40am

You have setup the domain names in the config file correct?
If so, its worth trying sudo.

So instead of

certbot --apache

It would be

sudo certbot --apache

As per this document:
https://certbot.eff.org/lets-encrypt/ubuntubionic-apache.html

Just a thought, not 100% sure. Keep us in the loop and we shall get to this bottom of this!

LuB

rg305 · January 14, 2020, 4:11am

The failure indicates that certbot is unable to use the apache plugin to successful authenticate the request.

Please show the output of:
apachectl -S
and the vhost config file for that domain.

yaandy · January 14, 2020, 4:41pm

Here’s the output of apachectl -S.

VirtualHost configuration:
*:80 gary-ThinkPad (/etc/apache2/sites-enabled/000-default.conf:1)
ServerRoot: “/usr”
Main DocumentRoot: “/usr/share/httpdocs/htdocs”
Main ErrorLog: “/var/logs/apache2/error_log”
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex rewrite-map: using_defaults
Mutex authdigest-client: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex authn-socache: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/logs/" mechanism=default
Mutex cache-socache: using_defaults
Mutex authdigest-opaque: using_defaults
PidFile: “/var/logs/httpd.pid”
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name=“daemon” id=1
Group: name=“daemon” id=1

Also, Looking at the letsencrypt.log file I noticed the following:

certbot looks at /etc/apache2 directory. I had built the Apache server to use /etc/apache. So I rebuilt the server to use /etc/apache2
certbot adds an entry to /etc/apache2sites-enabled/000-default.conf. My Apache server was not Including this file. This is now included. The apachtcl -S output above is AFTER I made this change.
Presently, I am running into an status 429 - too many failed authorizations. So, after reading letsencrypt.org/docs/rate-limits, I will wait a few hours and try again.

Thank you for your support.

yaandy · January 14, 2020, 9:59pm

(several hours later)
root:/etc/apache2# certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
No names were found in your configuration files. Please enter in your domain
name(s) (comma and/or space separated) (Enter ‘c’ to cancel): invystasafe.com
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for invystasafe.com
Cleaning up challenges
An unexpected error occurred:
ValueError: Unable to insert label!
Please see the logfiles in /var/log/letsencrypt for more details.

— Here’s a link to the letsencrypt log file —

Thank you in advance

yaandy · January 14, 2020, 10:07pm

No change by using sudo in front

rg305 · January 15, 2020, 1:39am

The output of apachectl -S does NOT show a virtual host for domain: invystasafe.com
You need to first make http://invystasafe.com work (correctly) before trying to make it https://

yaandy · January 15, 2020, 5:06pm

http://invystasafe.com does work (try it)… Since your post I have added a virtual host, but I am still having problems. Below is output from: (1) apache2ctl -S, (2) certbot --apache, (3) dig invystasafe.com (to show you the domain & ip address are registered), and (4) a link to the latest LetsEncrypt log file. I truly appreciate your support.

Output from apache2ctl -S
VirtualHost configuration:
*:80 invystasafe.com (/etc/apache2/sites-enabled/000-default.conf:1)
ServerRoot: “/usr”
Main DocumentRoot: “/usr/share/httpdocs/htdocs”
Main ErrorLog: “/var/logs/apache2/error_log”
Mutex default: dir="/var/logs/" mechanism=default
Mutex cache-socache: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex rewrite-map: using_defaults
Mutex authdigest-client: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex authn-socache: using_defaults
Mutex ssl-cache: using_defaults
PidFile: “/var/logs/httpd.pid”
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name=“daemon” id=1
Group: name=“daemon” id=1

Output from --certbot --apache
certbot --apache
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?

1: invystasafe.com

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel):
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for [invystasafe.com](http://invystasafe.com/)
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. [invystasafe.com](http://invystasafe.com/) (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://invystasafe.com/.well-known/acme-challenge/Yh7s-wDy0oVvAAzgz3pUFrvNu_PTL82KNfhvaENdgUo [54.219.172.188]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>401
Unauthorized</title>\n</head><body>\n<h1>Unauthorized</"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: [invystasafe.com](http://invystasafe.com/)
   Type:   unauthorized
   Detail: Invalid response from
http://invystasafe.com/.well-known/acme-challenge/Yh7s-wDy0oVvAAzgz3pUFrvNu_PTL82KNfhvaENdgUo
   [54.219.172.188]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>401
   Unauthorized</title>\n</head><body>\n<h1>Unauthorized</"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

Here is the output of dig:
$ dig invystasafe.com

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> invystasafe.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16082
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;invystasafe.com. IN A

;; ANSWER SECTION:
invystasafe.com. 60 IN A 54.219.172.188

;; Query time: 22 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Jan 15 16:39:19 UTC 2020
;; MSG SIZE  rcvd: 60

Here is a google-drive link to the LetsEncrypt log file - https://drive.google.com/file/d/0B5FonhzKX5NhTEpyLUVSUkhuRkc0WXNRMk5aaHpsSDE4Slk4/view?usp=sharing

rg305 · January 15, 2020, 11:42pm

I didn’t just say “it doesn’t work”. What I meant with “make it may work correctly” is that just because “it works” doesn’t mean it was done correctly.
You must understand that Apache will do it’s utmost to run (at any/all cost). It will ignore duplicate names, overlapping names, etc. So just because it runs doesn’t mean it is “correct”.
Why does it need to be “correct”?
Because certbot it NOT as forgiving and doesn’t understand “Apache logic” [it’s more like NGINX].

So, can we see the file:
/etc/apache2/sites-enabled/000-default.conf
[in hopes of making it “correct” - from all perspectives]

yaandy · January 15, 2020, 11:59pm

cat /etc/apache2/sites-enabled/000-default.conf

<VirtualHost *:80>
# The ServerName directive sets the request scheme, hostname and port that
# the server uses to identify itself. This is used when creating
# redirection URLs. In the context of virtual hosts, the ServerName
# specifies what hostname must appear in the request's Host: header to
# match this virtual host. For the default virtual host (this file) this
# value is not decisive as it is used as a last resort host regardless.
# However, you must set it for any further virtual host explicitly.
ServerName invystasafe.com
DocumentRoot "/usr/share/httpdocs/htdocs/login"
#ServerAdmin webmaster@localhost
#DocumentRoot /var/www/html

# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the loglevel for particular
# modules, e.g.
#LogLevel info ssl:warn

#ErrorLog ${APACHE_LOG_DIR}/error.log
#CustomLog ${APACHE_LOG_DIR}/access.log combined

# For most configuration files from conf-available/, which are
# enabled or disabled at a global level, it is possible to
# include a line for only one particular virtual host. For example the
# following line enables the CGI configuration for this host only
# after it has been globally disabled with "a2disconf".
#Include conf-available/serve-cgi-bin.conf
</VirtualHost>

Here is a relevant section from httpd.conf

Listen 80
DocumentRoot "/usr/share/httpdocs/htdocs"
    >     <Directory "/usr/share/httpdocs/htdocs/login">
    >         AuthType None
    >         Require all granted
    >     </Directory>
    >

<Directory />
           Options none
           Order deny,allow
           Allow from all
           AuthType Basic
           AuthUserFile "/usr/local/apache/passwords"
           AuthName "Restricted Files"
           Require valid-user
           ErrorDocument 404 "File Not Found"
    </Directory>

rg305 · January 16, 2020, 12:19am

It seems that certbot gets caught by your "Directory /" authentication requirement.
As seen in the 1600 lines of the LE logs:

2020-01-15 16:36:02,065:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: invytasafe.com in: /etc/apache2/sites-enabled/000-default.conf
2020-01-15 16:36:02,065:DEBUG:certbot_apache.http_01:writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]
    
2020-01-15 16:36:02,065:DEBUG:certbot_apache.http_01:writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>

it doesn't match up with what I assume you expected:

and logs the failure as:

Domain: invystasafe.com
Type:   unauthorized
Detail: Invalid response from http://invystasafe.com/.well-known/acme-challenge/aJzp4aGhfD5kfqxUZg7kWf7Lm6_x0FbZy1AzOKjcudo [54.219.172.188]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\">\n<html><head>\n<title>401 Unauthorized</title>\n</head><body>\n<h1>Unauthorized</"

rg305 · January 16, 2020, 12:22am

In order to “fix” this, I would recommend adding your own permanent version of:

RewriteEngine on
RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

to the config file.

Ensure that a “TEST” text file placed in that challenge folder can be reached from the Internet.
And then retry certbot.

yaandy · January 17, 2020, 5:17pm

Thanks for all you help (& patience)… its working now.

system · February 16, 2020, 5:28pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.