Why can't I certbot -d toastmastervoice.com and create a new VirtualHost and certificate?

It does look like a problem with the Certbot Apache plugin. During certificate installation, it seems unable to generate an HTTPS version of your virtualhost.

I don’t think the other errors are related.

Could you paste the contents of:

/etc/apache2/sites-enabled/089-toastmasters.conf

Hopefully that will be enough to diagnose it. If not, we might need your entire Apache configuration…

3 Likes

They are for unrelated certificates. I don’t see how they could have any impact on the domain in question.

2 Likes

Certbot goes through all of the existing lineages to determine whether there is one that already covers toastmastervoice.com, or whether it needs to create a new lineage.

While it is going through the lineages, it finds that some of them are broken (config file is bad, or symlinks are messed up, or whatever). When Certbot encounters a broken lineage, it logs the error, skips it and moves onto the next one.

Finally, it reaches the end, and it hasn’t found a matching lineage, so it decides to create a new one for toastmastervoice.com. In this log, it does find an existing one, and asks the user whether they want to re-issue the existing certificate or re-install it (the latter choice is made).

At this point, the problems with the other lineages are probably quite irrelevant. We’re not using them, we don’t care about them. One broken lineage isn’t going to (at least, doesn’t IME) make the rest of Certbot blow up.

3 Likes

@_az

Ah. Thanks for the great explanation. Makes sense. This is why I’m a big fan of divorcing the various aspects of the process from one another. I guess in my eyes I’d rather deal with a different machine (with a known-good configuration) handling the transaction process and clerical work. I’ve seen too many cases lately of configuration changes made by certbot conflicting with particular setups. Not that that’s necessarily what has happened in this topic. Guess I feel like there’s a difference between getting certificates properly issued and configuring a system to properly use https.

1 Like

/etc/apache2/sites-enabled/089-toastmasters.conf is:

<VirtualHost *:80>
	ServerAdmin christos.hayward@gmail.com
    ServerName toastmastervoice.com
	ServerAlias www.toastmastervoice.com toastmastersvoice.com www.toastmastersvoice.com
    #ServerAlias www.orthodoxchurchfathers.com fathers.jonathanscorner.com

	DocumentRoot /home/christos/toastmasters/
	<Directory />
		Options FollowSymLinks
		AllowOverride None
	</Directory>
	<Directory /home/christos/toastmasters>
		Options ExecCGI FollowSymLinks Indexes MultiViews
		AllowOverride All
		Order allow,deny
		allow from all
	</Directory>

	DirectoryIndex index.php index.cgi index.html

	ErrorLog ${APACHE_LOG_DIR}/toastmasters-error.log

	# Possible values include: debug, info, notice, warn, error, crit,
	# alert, emerg.
	LogLevel warn

	CustomLog ${APACHE_LOG_DIR}/toastmasters-access.log combined

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>

#RewriteEngine on
#RewriteCond %{SERVER_NAME} =orthodoxchurchfathers.cjshayward.com [OR]
#RewriteCond %{SERVER_NAME} =orthodoxchurchfathers.com
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

    #<VirtualHost *:80>
	#ServerAdmin CJSHayward@POBox.com
	#ServerName www.orthodoxchurchfathers.com
	#ServerAlias fathers.jonathanscorner.com
	#DocumentRoot /home/christos/oldmirror
	#RewriteEngine On
	#RewriteRule ^(.*)$ http://orthodoxchurchfathers.com$1 [R=301,L]
#RewriteCond %{SERVER_NAME} =fathers.jonathanscorner.com [OR]
#RewriteCond %{SERVER_NAME} =www.orthodoxchurchfathers.com
#RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
    #</VirtualHost>
2 Likes

Things don’t work out in the following certbot function:

I don’t actually fully understand the code exactly, but perhaps (and that’s a big “perhaps”) it’s because the VirtualHost includes three ServerAlias options which are not included in the command you’ve used? You only specified toastmastervoice.com while the VirtualHost also specifies www.toastmastervoice.com, toastmastersvoice.com and www.toastmastersvoice.com.

You could try including all four hostnames (with four times the -d option).

2 Likes

I haven’t had ServerAlias cause difficulties before, and I don’t really suspect it now.

I tried, for a sanity check, to comment out all the ServerAliases leaving only the ServerName entry. The behavior when I tried to reinstall the existing certificate (instead of the #2 option of Request and Renew) seemed to be identical.

I might be able to get the desired behavior by adapting and installing the certificate according to the pattern of the other sites-enabled entries, but that’s a losing solution.

2 Likes

Yeah, it was a wild guess… :roll_eyes:

Debugging Augeas (the library responsible for parsing the Apache configuration files) is certainly possible, but would entail putting all kinds of debug statements in the Python code and see what’s going on. Without an Apache configuration which can reproduce this error, that’s hard to do…

I can try to trick certbot by generating a self signed certificate for your domains, so it won’t try to get new ones, but will try to generate a new TLS VirtualHost if it sees it hasn’t been installed yet. Not sure if certbot gets tricked by self signed certs though :stuck_out_tongue:

2 Likes

I can post a tarball of sites-enabled/, but that’s quite a lot of generosity for a reading assignment.

It seems to get a certificate, it just doesn’t know where to install it. The certificate is there and is seen by runs of certbot-auto; it’s just not fully installed enough to serve the site via HTTPS.

2 Likes

Quite.

That’s why I’m hoping certbot accepts a self signed cert so I can test the installation part. Hopefully I’ll get an error with your VirtualHost too and see where it goes wrong.

2 Likes

Ha! Success! At least, I’m getting the same error as you :smiley: Had to hack my certbot just a tiny little bit (comment out some check I hope I can find back and un-comment it…)

Now time for some debugging…

2 Likes

OK, I’ve figured it out:

The Augeas parser doesn’t like indented comments at the end of the configuration file…

For example, this is fine (for clarity, I left out the top most of the file):

(...)
</VirtualHost>
#Foo

An extra blank line doesn’t matter, this is also fine:

(...)
</VirtualHost>

#Foo

However, when you try adding spaces or tabs, it doesn’t like that! These configuration files return the error!

(...)
</VirtualHost>
 #Foo

(Notice the subtle space before the comment.)

(...)
</VirtualHost>

     #Foo

Not good too!

So: remove the trailing comments (not required anyway) and you’re good to go!

4 Likes

That’s some awesome debugging. :smiley:

1 Like

No, this is actually very, VERY simple debugging… :stuck_out_tongue: Just add breakpoint() after every line in the function, do a print(variable) statement on the debugger command line (where variable is the variable you want to check, such as new_matches from the Python code) and you compare the output of those variable prints to a configuration you know is working.

I noticed new_matches returns a single Augeas reference to the newly generated VirtualHost with a working Apache configuration, but returned empty with this VirtualHost. So I just removed something from the erroneous configuration file and checked if it returned a “match” that time. And kept removing stuff, until I finally DID get a match! That was after I removed all the trailing comments… Then I added the comments back again and tried to remove only pieces of it… But it kept failing, until I removed all the indentation. Then I tried a few scenario’s with #Foo and #Bar and stuff and noticed the indentation was key to the failing Augeas library.

As you can see, fairly simple and the timestamps of the posts say it can be done within 20 minutes :smiley:

The “tricky” part was to keep it as “real” as possible, I wanted certbot to accept a self signed certificate with @CJSHayward s host, so I could use his configuration file without editing it, but that was easily done by commenting out this line. :stuck_out_tongue:

I’ve opened an issue on the certbot Github.

3 Likes

Looks like a good start...

...then something horrifying. :nauseated_face: Glad it worked though. I've found when editing PHP files that I sometimes have to backspace and recreate newlines when copying things to get rid of the extra characters. Sometimes native nbsp and other whitespace gobbledygook interferes. My historical favorite was forgetting to remove an unneeded parameter to a printf statement that had no corresponding place in the format string. I spent four hours commenting out chunks of code while assuming "nah, couldn't be the printf" until finally I showed it to a colleague. As soon as he pulled it up on his screen, my eyes immediately fell on the printf and I slapped myself in the forehead.

:face_with_raised_eyebrow:

1 Like

@Osiris

That wasn’t supposed to be an insult to you. :smile: Your approach was very effective and discovered one of the more elusive types of bugs. My :nauseated_face: face was because what you went through reminded me of many of my own experiences of commenting and if(0)'ing chunks of code like a mad surgeon looking for the cause of something crazy. I found it quite funny. :joy: I suppose python can be pretty picky about its indentation. I’ve only used it to any extent for network flow / optimization problems.

By the by, on another topic you mentioned my time being an issue with a problem I was identifying. I mentioned Samsung Internet and you suggested it might have been updated. You may very well be right. I explored it further and found that Samsung internet now suggests changing your time when it encounters an outdated certificate. I assumed that the cert in question was a fake that was set to start in the future, which I’ve seen for a few placeholders. Previously that browser simply stated that the cert was outdated (or had invalid dates if it starts after it ends). Its behavior now is really deceptive and annoying.

1 Like

Wow. Thanks!

This resolved my issue completely.

3 Likes

Glad everything worked out. :slightly_smiling_face: Props to @Osiris for his diligent digging and debugging.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.