Converting from certbot-auto to certbot - challenge not accepted

Help
#1

I was running fine with certbot-auto, but since it is depreciated, I am trying to get back up and running. I deleted the existing certs (well, as far as I can tell) using "sudo certobt delete". I commented out the
SSLCertificateFile and SSLCertificateKeyFile entries in the domain conf files (I have sub domains). I also removed my automagic from my crontab. I deleted the certbot-auto file and purged and re-apt'ed certbot. I also commented out the http to https redirect in the config files. The challenge referenced below has the same error for all domains.

My domain is: happykathi.com (also have sub domains)

I ran this command: sudo certbot

It produced this output:
Some challenges have failed.

IMPORTANT NOTES:

IMPORTANT NOTES:

My web server is (include version): apache 2.4.29

The operating system my web server runs on is (include version): ubuntu 18.04 64bit

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.11.0

1 Like
#2

It seems your problem lies within the Apache configuration.
Please start by showing the output of:
apachectl -S

1 Like
#3

And a really big FYI:
To change from certbot-auto to certbot, you didn't need to delete/replace any of the certs.

2 Likes
#4 
$ sudo apachectl -S
VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server davidstoll.net (/etc/apache2/sites-enabled/15_davidstoll.net.conf:360)
         port 80 namevhost davidstoll.net (/etc/apache2/sites-enabled/15_davidstoll.net.conf:360)
                 wild alias *.davidstoll.net
         port 80 namevhost happykathi.com (/etc/apache2/sites-enabled/20_happykathi.com.conf:6)
                 wild alias *.happykathi.com
         port 80 namevhost davidstoll.com (/etc/apache2/sites-enabled/30_davidstoll.com.conf:6)
                 wild alias *.davidstoll.com
*:443                  is a NameVirtualHost
         default server davidstoll.net (/etc/apache2/sites-enabled/15_davidstoll.net.conf:27)
         port 443 namevhost davidstoll.net (/etc/apache2/sites-enabled/15_davidstoll.net.conf:27)
         port 443 namevhost happykathi.com (/etc/apache2/sites-enabled/20_happykathi.com.conf:15)
         port 443 namevhost davidstoll.com (/etc/apache2/sites-enabled/30_davidstoll.com.conf:14)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/home/webdocs"
Main ErrorLog: "/var/log/apache_error.log"
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex watchdog-callback: using_defaults
Mutex ssl-stapling-refresh: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
1 Like
#5

Can you show the choices made during the sudo certbot run?
Please show the file:
/etc/apache2/sites-enabled/20_happykathi.com.conf

1 Like
#6

Hi @davidstoll

that's bad:

Two port 80 vHosts with the same set of domain names. Merge both in one, delete the other. First, make a backup.

Every combination of port and domain name must be unique.

2 Likes
#7

I chose "1" for Apache plugin. Then left the next option blank (to choose all domains).

$ cat /etc/apache2/sites-enabled/20_happykathi.com.conf

<directory /data/happykathi.com/>
    allow from all
    Require all granted
</directory>

<VirtualHost *:80>
    ServerName happykathi.com
    ServerAlias *.happykathi.com
    ServerAdmin none@happykathi.com
    DocumentRoot /data/happykathi.com
    #Redirect / https://happykathi.com/
</VirtualHost>

<VirtualHost *:443>
    ServerName "happykathi.com"
    DocumentRoot /data/happykathi.com/
</VirtualHost>
1 Like
#9

I'm not sure I follow you. It runs fine and has been running fine with sub domains in this way for years. Maybe it's just bad form?

Are you saying that they need to be on (for instance) 81 and 82?

1 Like
#10

You should remove this section and let certbot create it properly for you:

2 Likes
#11

I don't see the name overlap.
There are two very similar names; but one ends with .com and the other .net.

The added wildcard entries seems rather unnecessary and possibly confusing.
I would change them to just www.{same.domains}.

1 Like
#12

Sorry, was too late - ignore the post.

2 Likes
#13

ok, cool, that worked on 2 of them (doing the same 443 section removal), but one is still not working. Same error saying that the challenge failed.

Here is the conf:

<directory /data/davidstoll.com/>
    allow from all
    Require all granted
</directory>

<VirtualHost *:80>
    ServerName davidstoll.com
    ServerAlias *.davidstoll.com
    ServerAdmin none@davidstoll.com
    DocumentRoot /data/davidstoll.com
    #Redirect / https://davidstoll.com/
</VirtualHost>

So I don't see the same issue with this one. By the way, thank you for helping me this far!
Also, I noticed that it created a new file with the name 20_happykathi.com-le-ssl.conf
Is it no longer ok to keep everything 80 and 443 entries in one file?
Thanks

2 Likes
#14

You could keep the entire config in just one file.
But more is actually better here; as it can simplify things.
Like when you need to add or remove a section (it's in a separate file).

So what is the latest error message?

1 Like
#15

Same error saying that the challenge failed, but just for the one domain.

Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: davidstoll.com
   Type:   unauthorized
   Detail: Invalid response from
   http://davidstoll.com/.well-known/acme-challenge/wb6cMiBCJP_2HdKQi8YsUYcJR83z-x1buKgCc5s0uHs
   [69.174.176.167]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>403
   Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
1 Like
#16

I take nothing for granted...
Let's confirm the files in use with:
apachectl -S

1 Like
#17 
$ sudo apachectl -S
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server davidstoll.net (/etc/apache2/sites-enabled/15_davidstoll.net-le-ssl.conf:2)
         port 443 namevhost davidstoll.net (/etc/apache2/sites-enabled/15_davidstoll.net-le-ssl.conf:2)
                 wild alias *.davidstoll.net
         port 443 namevhost happykathi.com (/etc/apache2/sites-enabled/20_happykathi.com-le-ssl.conf:2)
                 wild alias *.happykathi.com
*:80                   is a NameVirtualHost
         default server davidstoll.net (/etc/apache2/sites-enabled/15_davidstoll.net.conf:360)
         port 80 namevhost davidstoll.net (/etc/apache2/sites-enabled/15_davidstoll.net.conf:360)
                 wild alias *.davidstoll.net
         port 80 namevhost happykathi.com (/etc/apache2/sites-enabled/20_happykathi.com.conf:6)
                 wild alias *.happykathi.com
         port 80 namevhost davidstoll.com (/etc/apache2/sites-enabled/30_davidstoll.com.conf:6)
                 wild alias *.davidstoll.com
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/home/webdocs"
Main ErrorLog: "/var/log/apache_error.log"
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex watchdog-callback: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
1 Like
#19

Alright, let's get this straightened out.

What are the current outputs of these commands?

sudo ls -l /etc/apache2/sites-available
sudo ls -l /etc/apache2/sites-enabled

What are the current contents of these files?

/etc/apache2/sites-enabled/15_davidstoll.net.conf
/etc/apache2/sites-enabled/30_davidstoll.com.conf
/etc/apache2/sites-enabled/20_happykathi.com.conf
/etc/apache2/sites-enabled/15_davidstoll.net-le-ssl.conf
/etc/apache2/sites-enabled/20_happykathi.com-le-ssl.conf
1 Like
#20

$sudo ls -l /etc/apache2/sites-available
-rw-r--r-- 1 root root 1417 Feb 6 2020 000-default.conf
-rw-r--r-- 1 root root 6338 Jul 16 2019 default-ssl.conf

$sudo ls -l /etc/apache2/sites-enabled
-rw-r--r-- 1 root root 373 Jan 23 2020 00_default.conf
-rw-r--r-- 1 root root 577 Jan 23 2020 01_default_directories.conf
-rw-r----- 1 root root 15332 Jan 19 08:08 15_davidstoll.net.conf
-rw-r--r-- 1 root root 4121 Jan 19 08:08 15_davidstoll.net-le-ssl.conf
-rw-r----- 1 root root 818 Jan 19 08:08 20_happykathi.com.conf
-rw-r--r-- 1 root root 497 Jan 19 08:08 20_happykathi.com-le-ssl.conf
-rw-r----- 1 root root 575 Jan 19 08:09 30_davidstoll.com.conf

The contents of the happykathi was above, now it's just duplicated in the ssl.conf version it created
The davidstoll.com (the problem domain) is essentially the same as the happykathi one, but I'll put it here anyway:
$sudo cat 30_davidstoll.net.conf

<directory /data/davidstoll.com/>
    allow from all
    Require all granted
</directory>

<VirtualHost *:80>
    ServerName davidstoll.com
    ServerAlias *.davidstoll.com
    ServerAdmin none@davidstoll.com
    DocumentRoot /data/davidstoll.com
    #Redirect / https://davidstoll.com/
</VirtualHost>

And the .net one is the same as the .com except it just has .net in all the appropriate places.

1 Like
#21

The files in sites-enabled are supposed to be symbolic links to files in sites-available.

To enable a site:

a2ensite {config filename without .conf}

To disable site:

a2dissite {config filename without .conf}

1 Like
#22

In sites-enabled...

these files are enormous:

15_davidstoll.net.conf - 15332 bytes
15_davidstoll.net-le-ssl.conf - 4121 bytes

compared to these:

20_happykathi.com.conf - 818 bytes
20_happykathi.com-le-ssl.conf - 497 bytes
30_davidstoll.com.conf - 575 bytes

The .net clearly has more contents. I have a feeling a lot of it is default comments and settings.

You have a 000-default.conf (correct default) in sites-available, but a 00_default.conf (what?) in sites-enabled.

What's in default-ssl.conf in sites-available?

1 Like