403 Unauthorized with uWSGI

My domain is: s.irminger.org

I ran this command: sudo certbot certonly --apache --dry-run -d s.irminger.org -d www.s.irminger.org

It produced this output:

sudo certbot certonly --apache --dry-run -d s.irminger.org -d www.s.irminger.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for s.irminger.org
http-01 challenge for www.s.irminger.org
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.s.irminger.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.s.irminger.org/.well-known/acme-challenge/s8ORreKaocDWn0sSnvjwXdT-j_xubN5ftW062QQ2gkM: "\n<html xmlns="http://www.w3.org/1999/xhtml\" lang="en" xml:lang="en">\n\n <meta charset="UTF-8" />\n <met", s.irminger.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://s.irminger.org/.well-known/acme-challenge/UgfjNJsZgifQgP1Ap2vEyGow8bo-SLo2DkJcSkTW6XI: "\n<html xmlns="http://www.w3.org/1999/xhtml\" lang="en" xml:lang="en">\n\n <meta charset="UTF-8" />\n <met"

IMPORTANT NOTES:

My web server is (include version): Apache/2.4.29 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 18.04

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.28.0

Hi there, thank you for your help with this. I've been struggling to figure this out. I'm self-hosting a Searx instance and cannot get Certbot to work past uWSGI. I can't get s.irminger.org/.well-known/acme-challenges/***** to show, it's captured by searx. When I disable the uWSGI instance by commenting it out in the virtualhost conf for this subdomain the dry run for certbot goes through.

Does someone have an idea to suggest? I've spent quite a bit of time searing and trying things on this and am bummed I haven't been able to figure it out. I've seen work around out there but haven't been able to get them to work for me.

This is the uWSGI section in the vhost

Options FollowSymLinks Indexes SetHandler uwsgi-handler uWSGISocket /run/uwsgi/app/searx/socket

Thank you

Please show that config file (to include the portion you commented out).



       ServerAdmin @protonmail.com
        ServerName s.irminger.org
        ServerAlias www.s.irminger.org

DocumentRoot /var/www/s.irminger.org/


 <Directory /var/www/s.irminger.org/searx>
            Options Indexes FollowSymLinks
           AllowOverride All
            Require all granted
        </Directory>

 <Location />
            Options FollowSymLinks Indexes
            SetHandler uwsgi-handler
            uWSGISocket /run/uwsgi/app/searx/socket
        </Location>

 ErrorLog ${APACHE_LOG_DIR}/error.log
        CustomLog ${APACHE_LOG_DIR}/access.log combined
</VirtualHost>```

Sorry, do this here.
So we can read it clearly (not on your site)
Edit that post and place three backticks above and three beneath all of it.

```

OK.
You need to exclude the challenge requests from that location block.
I’m not sure whether Location or Directory should be used to accomplish this.
Try LOCATION first; above the current location place this one:

<location /.well-known/acme-challenge>
    Options FollowSymLinks Includes
    AllowOverride None 
    Require all granted
</location>
1 Like

Thank you for the help rg305

It liked Location and didn't like Directory. Once I did that and ran the dry run, s.irminger.org passed and www.s.irminger.org did not pass with same error. When I ran certbot without --dry-run for just s.irminger.org it failed with same error. I ran both without --dry-run and will post that below. I did check the domain dns again to verify that the a records are correct.

sudo certbot certonly --apache -d s.irminger.org -d www.s.irminger.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for s.irminger.org
http-01 challenge for www.s.irminger.org
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.s.irminger.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.s.irminger.org/.well-known/acme-challenge/D8kMGAQMi87TSIGF-5n5HEaD_-IZyrSu-qCYM4FOLUA: "\n<html xmlns="http://www.w3.org/1999/xhtml\" lang="en" xml:lang="en">\n\n <meta charset="UTF-8" />\n <met", s.irminger.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://s.irminger.org/.well-known/acme-challenge/wlMAm6ikX5IcPW1cqvitc5EqC6PdloAIE0rOIFH84sw: "\n<html xmlns="http://www.w3.org/1999/xhtml\" lang="en" xml:lang="en">\n\n <meta charset="UTF-8" />\n <met"

IMPORTANT NOTES:

1 Like

There must be some conflicting/overlapping server names.
Check with:
apache2ctl -S
apachectl -t -D DUMP_VHOSTS
grep -Eri 'servername|serveralias|virtualhost' /etc/apache2

[edit - also place a file in the expected challenge folder for testing; as follows:]
mkdir /var/www/s.irminger.org/.well-known
mkdir /var/www/s.irminger.org/.well-known/acme-challenge
echo "just a test" > /var/www/s.irminger.org/.well-known/acme-challenge/1234

I have a couple other subdomains on this VPS already and a vhost for irminger.org set up, but didn't see any duplicates for s.irminger.org.conf. There was a s.irminger.org.conf.save in sites-available so I removed that but no changes with certbot. What would constitute an overlap? On another VPS last year I was able to get an ssl for s.irminger.org and several other subdomains so I haven't been able to do what I've done before.. :frowning:

I tried the second suggestion, and it says permission denied on the echo command. I did place a file there earlier at that is accessible at http://s.irminger.org/.well-known/acme-challenge/index.txt.

echo "just a test" > /var/www/s.irminger.org/.well-known/acme-challenge/1234
-bash: /var/www/s.irminger.org/.well-known/acme-challenge/1234: Permission denied

same with sudo. What do you think?

The same name being used as an alias elsewhere or a wildcard that overlaps with it.
Can you post some of that output?
Like: grep -Eri 'servername|serveralias|virtualhost' /etc/apache2

And did you create the test file 1234?

Here's the output from the grep line:

sudo grep -Eri 'servername|serveralias|virtualhost' /etc/apache2
/etc/apache2/apache2.conf:# If you do not specify an ErrorLog directive within a <VirtualHost>
/etc/apache2/apache2.conf:# logged here.  If you *do* define an error logfile for a <VirtualHost>
/etc/apache2/conf-available/other-vhosts-access-log.conf:# Define an access log for VirtualHosts that don't define their own logfile
/etc/apache2/conf-available/localized-error-pages.conf:# even on a per-VirtualHost basis.  If you include the Alias in the global server
/etc/apache2/ports.conf:# have to change the VirtualHost statement in
/etc/apache2/mods-available/info.conf:	#  http://servername/server-info (requires that mod_info.c be loaded).
/etc/apache2/mods-available/status.conf:	# with the URL of http://servername/server-status
/etc/apache2/sites-available/go.irminger.org-le-ssl.conf:<VirtualHost *:443>
/etc/apache2/sites-available/go.irminger.org-le-ssl.conf:        ServerName go.irminger.org
/etc/apache2/sites-available/go.irminger.org-le-ssl.conf:	ServerAlias www.go.irminger.org
/etc/apache2/sites-available/go.irminger.org-le-ssl.conf:</VirtualHost>
/etc/apache2/sites-available/go.irminger.org.conf:<VirtualHost *:80>
/etc/apache2/sites-available/go.irminger.org.conf:        ServerName go.irminger.org
/etc/apache2/sites-available/go.irminger.org.conf:	ServerAlias www.go.irminger.org
/etc/apache2/sites-available/go.irminger.org.conf:</VirtualHost>
/etc/apache2/sites-available/default-ssl.conf:	<VirtualHost _default_:443>
/etc/apache2/sites-available/default-ssl.conf:	</VirtualHost>
/etc/apache2/sites-available/000-default.conf:<VirtualHost *:80>
/etc/apache2/sites-available/000-default.conf:</VirtualHost>
/etc/apache2/sites-available/000-default.conf.dpkg-dist:<VirtualHost *:80>
/etc/apache2/sites-available/000-default.conf.dpkg-dist:	# The ServerName directive sets the request scheme, hostname and port that
/etc/apache2/sites-available/000-default.conf.dpkg-dist:	# redirection URLs. In the context of virtual hosts, the ServerName
/etc/apache2/sites-available/000-default.conf.dpkg-dist:	#ServerName www.example.com
/etc/apache2/sites-available/000-default.conf.dpkg-dist:</VirtualHost>
/etc/apache2/sites-available/s.irminger.org.conf:<VirtualHost *:80>
/etc/apache2/sites-available/s.irminger.org.conf:        ServerName s.irminger.org
/etc/apache2/sites-available/s.irminger.org.conf:	ServerAlias www.s.irminger.org
/etc/apache2/sites-available/s.irminger.org.conf:</VirtualHost>
/etc/apache2/sites-available/cloud.irminger.org-le-ssl.conf:<VirtualHost *:443>
/etc/apache2/sites-available/cloud.irminger.org-le-ssl.conf:        ServerName cloud.irminger.org
/etc/apache2/sites-available/cloud.irminger.org-le-ssl.conf:	ServerAlias www.cloud.irminger.org
/etc/apache2/sites-available/cloud.irminger.org-le-ssl.conf:</VirtualHost>
/etc/apache2/sites-available/cloud.irminger.org.conf:<VirtualHost *:80>
/etc/apache2/sites-available/cloud.irminger.org.conf:        ServerName cloud.irminger.org
/etc/apache2/sites-available/cloud.irminger.org.conf:	ServerAlias www.cloud.irminger.org
/etc/apache2/sites-available/cloud.irminger.org.conf:</VirtualHost>
/etc/apache2/sites-available/irminger.org-le-ssl.conf:<VirtualHost *:443>
/etc/apache2/sites-available/irminger.org-le-ssl.conf:        ServerName irminger.org
/etc/apache2/sites-available/irminger.org-le-ssl.conf:	ServerAlias www.irminger.org
/etc/apache2/sites-available/irminger.org-le-ssl.conf:</VirtualHost>
/etc/apache2/sites-available/irminger.org.conf:<VirtualHost *:80>
/etc/apache2/sites-available/irminger.org.conf:        ServerName irminger.org
/etc/apache2/sites-available/irminger.org.conf:	ServerAlias www.irminger.org
/etc/apache2/sites-available/irminger.org.conf:</VirtualHost>

I had already created the .well-known/acme-challenge folders and when I ran the echo line the output is

-bash: /var/www/s.irminger.org/.well-known/acme-challenge/1234: Permission denied

thank you for your help with this

edit:

drwxr-xr-x 7 www-data www-data 4096 Jan 23 04:42 s.irminger.org
drwxr-xr-x 3 www-data www-data 4096 Jan 21 14:54 .well-known
drwxr-xr-x 3 www-data www-data 4096 Jan 23 06:30 acme-challenge
I have a file index.text in acme-challenge and it will display when I comment out the uwsgi section but it gets captured by searx now http://s.irminger.org/.well-known/acme-challenge/index.txt

I can't reach that file it returns error 404

Make sure you have created the folders first.
please show:
ls -l /var/www/s.irminger.org/.well-known

[use sudo if needed]
sudo mkdir /var/www/s.irminger.org/.well-known
sudo mkdir /var/www/s.irminger.org/.well-known/acme-challenge
sudo echo "testing" > /var/www/s.irminger.org/.well-known/acme-challenge/1234

In review of the grep output.
I'd like to see a bit more, please include root in the search and focus on the relevant file only:
sudo grep -Ei 'servername|serveralias|virtualhost|root' /etc/apache2/sites-available/s.irminger.org.conf
[or just upload that entire file - the previously posted version got a bit "diffused"]

Also, please show:
ls -l /etc/apache2/sites-enabled/

I think I may have pulled up an old tab last night and thought that index.txt was still accessible. Please see my comment in the edit above.

  • ls -l /var/www/s.irminger.org/.well-known
 total 8
 drwxr-xr-x 3 www-data www-data 4096 Jan 23 06:30 acme-challenge
 -rw-r--r-- 1 www-data www-data    8 Jan 21 14:53 blah.txt```

* sudo echo "testing" > /var/www/s.irminger.org/.well-known/acme-challenge/1234

``` /var/www/s.irminger.org/.well-known/acme-challenge$ sudo echo "testing" > /var/www/s.irminger.org/.well-known/acme-challenge/1234
 -bash: /var/www/s.irminger.org/.well-known/acme-challenge/1234: Permission denied```

* sudo grep -Ei 'servername|serveralias|virtualhost|root' /etc/apache2/sites-available/s.irminger.org.conf

```sudo grep -Ei 'servername|serveralias|virtualhost|root' /etc/apache2/sites-available/s.irminger.org.conf
<VirtualHost *:80>
        ServerName s.irminger.org
	ServerAlias www.s.irminger.org
DocumentRoot /var/www/s.irminger.org/	
</VirtualHost>```

* ls -l /etc/apache2/sites-enabled/

```total 0
lrwxrwxrwx 1 root root 59 Jan 15 22:14 cloud.irminger.org-le-ssl.conf -> /etc/apache2/sites-available/cloud.irminger.org-le-ssl.conf
lrwxrwxrwx 1 root root 42 Jan 16 13:31 cloud.irminger.org.conf -> ../sites-available/cloud.irminger.org.conf
lrwxrwxrwx 1 root root 56 Jan 18 06:16 go.irminger.org-le-ssl.conf -> /etc/apache2/sites-available/go.irminger.org-le-ssl.conf
lrwxrwxrwx 1 root root 39 Jan 18 05:49 go.irminger.org.conf -> ../sites-available/go.irminger.org.conf
lrwxrwxrwx 1 root root 53 Jan 15 22:10 irminger.org-le-ssl.conf -> /etc/apache2/sites-available/irminger.org-le-ssl.conf
lrwxrwxrwx 1 root root 36 Jan 15 21:58 irminger.org.conf -> ../sites-available/irminger.org.conf
lrwxrwxrwx 1 root root 38 Jan 22 03:39 s.irminger.org.conf -> ../sites-available/s.irminger.org.conf```

I think the three backticks work better when on a separate line (by themselves).

Well the user is "www-data" but that doesn't explain why sudo/root get permission denied...

Try creating an empty file first:
sudo touch /var/www/s.irminger.org/.well-known/acme-challenge/1234
then append to it with:
sudo echo "testing" >> /var/www/s.irminger.org/.well-known/acme-challenge/1234

This failure may explain why certbot fails as it may also be unable to create a file in that folder (even when run as root)

Ahh, I haven’t used backticks before. Learning a lot here.

  • sudo touch /var/www/s.irminger.org/.well-known/acme-challenge/1234

  • sudo echo “testing” >> /var/www/s.irminger.org/.well-known/acme-challenge/1234
    -bash: /var/www/s.irminger.org/.well-known/acme-challenge/1234: Permission denied

  • ls -l
    total 8
    -rw-r–r-- 1 root root 0 Jan 24 02:30 1234
    drwxr-xr-x 2 www-data www-data 4096 Jan 23 05:12 blah
    -rwxr-xr-x 1 www-data www-data 6 Jan 21 16:58 index.txt

OMG!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Where is that?
show both:
pwd
sudo find / -name 1234

  • pwd
    /var/www/s.irminger.org/.well-known/acme-challenge

  • sudo find / -name 1234
    /var/www/s.irminger.org/.well-known/acme-challenge/1234
    /proc/916/task/1234

I don’t know if I possibly changed something in a main configuration file for apache or something that is causing all of this. It’s just so particular to this subdomain that I’ve made for searx, since I created another subdomain (go.irminger.org) for another deployment after setting up searx and certbot worked for that.

I can only assume the directories are very locked down…

So we may have to put the challenge folder somewhere else…
Somewhere new and only for the challenges.
Like:
mkdir /etc/ACMEchallenges
and we use that path instead.

Yes?

1 Like

So the files does now exist:

But it is empty:

...and can't be reached form the Internet:
wget http://s.irminger.org/.well-known/acme-challenge/1234
--2019-01-24 02:43:40-- http://s.irminger.org/.well-known/acme-challenge/1234
Resolving s.irminger.org (s.irminger.org)... 142.93.232.124
Connecting to s.irminger.org (s.irminger.org)|142.93.232.124|:80... connected.
HTTP request sent, awaiting response... 404 NOT FOUND
2019-01-24 02:43:40 ERROR 404: NOT FOUND.

I say we go PLAN B: another (dedicated) folder path.

1 Like

Ok, I will try that now.

I did try a challenge folder a couple days ago at /var/www/html/.well-known/acme-challenges referencing other forum posts and tried to do the webroot option in certbot but couldn't make it happen.

Webroot option has to match the document root in use - you just can't point it anywhere you want :frowning:

1 Like