Can't renew, getting errors with multiple domains


#1

Hey there, I’m up for my first renewal of 11 domains on my CentOS 7 machine. The initial deployment was painless, but now I’m having some trouble.

After getting 20 emails from LetsEncrypt this morning alone, I figure I better look into this.

The cron for /usr/bin/certbot renew fails, and when I run /usr/bin/certbot renew manually I get the below errors. I’ve truncated it a little.

It seems that when it’s processing a domain conf file (domain.com and domain2.net in my example below), it ends up looking at only 1 Apache file (myserver.tld in my example below).

Each domain has it’s own dedicated file in /etc/httpd/sites-enabled/domain.com-le-ssl.conf where the VirtualHost is defined.

As a result all are failing. Any insight on how to make the renew work?

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/domain.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/myserver.tld-le-ssl.conf', u'Syntax error'),)
Attempting to renew cert from /etc/letsencrypt/renewal/domain.com.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/myserver.tld-le-ssl.conf', u'Syntax error'),). Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/domain2.net.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/myserver.tld-le-ssl.conf', u'Syntax error'),)
Attempting to renew cert from /etc/letsencrypt/renewal/domain2.net.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/myserver.tld-le-ssl.conf', u'Syntax error'),). Skipping.


#2

Without the contents of the conf-file mentioned in the error, it’ll be guess work…


#4

Well, let me post all 3. I’m removing (what I think is) sensitive data. All 11 domains look similar to the below.

/etc/letsencrypt/renewal/domain.com.conf:

# renew_before_expiry = 30 days
version = 0.8.1
cert = /etc/letsencrypt/live/domain.com/cert.pem
privkey = /etc/letsencrypt/live/domain.com/privkey.pem
chain = /etc/letsencrypt/live/domain.com/chain.pem
fullchain = /etc/letsencrypt/live/domain.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = apache
installer = apache
account = <removed>

/etc/letsencrypt/renewal/domain2.net.conf:

# renew_before_expiry = 30 days
version = 0.8.1
cert = /etc/letsencrypt/live/domain2.net/cert.pem
privkey = /etc/letsencrypt/live/domain2.net/privkey.pem
chain = /etc/letsencrypt/live/domain2.net/chain.pem
fullchain = /etc/letsencrypt/live/domain2.net/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = apache
installer = apache
account = removed

/etc/httpd/sites-enabled/myserver.tld-le-ssl.conf:

<VirtualHost *:80>
    ServerName myserver.tld
    DocumentRoot /var/www/html
    ErrorLog logs/myserver.tld-error_log
    CustomLog logs/myserver.tld-access_log combined
RewriteEngine on
RewriteCond %{SERVER_NAME} =myserver.tld [OR]
RewriteCond %{SERVER_NAME} =1.2.3.4 [OR]
RewriteCond %{SERVER_NAME} =localhost [OR]
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
</VirtualHost>

I’m not sure why they’re all going after the myserver.tld when I think they should be going after their own individual conf and VirtualHost files?


#5

certbot checks all Apache configuration files. I’m no mod_rewrite expert, but I think it might be something with those RewriteCond (mostly because all other stuff looks OK and I’m not sure you can use those = like that.)


#6

In addition the /etc/httpd/sites-enabled/myserver.tld-le-ssl.conf does not have an SSL info / port 443 connection ( which is slightly odd if it’s the “le-ssl.conf” file.

Can you paste the original file ( without modification ) at somewhere like pastebin.com ?


#7

Oh, sorry. I have 2 files for mydomain.tld, and I pasted the http version facepalm.

Here’s the https version with “le-ssl.conf”. Let’s try this again:

/etc/httpd/sites-enabled/mydomain.tld-le-ssl.conf:

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName mydomain.tld
    DocumentRoot /var/www/html
    ErrorLog logs/mydomain.tld-error_log
    CustomLog logs/mydomain.tld-access_log combined
SSLCertificateFile /etc/letsencrypt/live/mydomain.tld/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mydomain.tld/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/mydomain.tld/chain.pem
</VirtualHost>

/etc/letsencrypt/renewal/domain.com-le-ssl.conf:

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerName domain.com
    ServerAlias www.domain.com
    DocumentRoot /var/www/html/domain
    ServerAdmin not@available.com
    UseCanonicalName Off
    ErrorLog logs/domain.com-error_log
    CustomLog logs/domain.com-access_log combined
    UserDir disabled
    UserDir enabled drsprite
RewriteEngine on
RewriteCond %{SERVER_NAME} =domain.com [OR]
RewriteCond %{SERVER_NAME} =www.domain.com
# Some rewrite rules in this file were were disabled on your HTTPS site,
# because they have the potential to create redirection loops.
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
SSLCertificateFile /etc/letsencrypt/live/domain.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/domain.com/chain.pem
</VirtualHost>

<Directory /var/www/html/domain>
    AllowOverride All
</Directory>

</IfModule>

/etc/letsencrypt/renewal/domain2.net-le-ssl.conf:

<IfModule mod_ssl.c>
<Directory /var/www/html/domain2>
    AllowOverride all
</Directory>

<VirtualHost *:443>
    ServerName domain2.net
    ServerAlias www.domain2.net
    DocumentRoot /var/www/html/domain2
    ServerAdmin not@available.com
    UseCanonicalName Off
    ErrorLog logs/domain2.net-error_log
    CustomLog logs/domain2.net-access_log combined
    UserDir disabled
    UserDir enabled drsprite
RewriteEngine on
RewriteCond %{SERVER_NAME} =www.domain2.net [OR]
RewriteCond %{SERVER_NAME} =domain2.net
# Some rewrite rules in this file were were disabled on your HTTPS site,
# because they have the potential to create redirection loops.
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]
SSLCertificateFile /etc/letsencrypt/live/domain2.net/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain2.net/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/domain2.net/chain.pem
</VirtualHost>


</IfModule>


#8

It would be more helpful if you didn’t edit the files - your domain name is public anyway :wink:

The lines

RewriteEngine on
RewriteCond %{SERVER_NAME} =www.domain2.net [OR]
RewriteCond %{SERVER_NAME} =domain2.net
# Some rewrite rules in this file were were disabled on your HTTPS site,
# because they have the potential to create redirection loops.
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

don’t make sense here - and should all be removed / commented out ( assuming you don’t have other lines you have just not shown )


#9

Yeah I thought about that after I submitted it. :thinking:

I didn’t add those lines, certbot took care of the whole thing. I’ll comment them out. Should I re-run the certbot after?


#10

I’m not hiding anything else. Those are the full configs.

Commenting that out and trying to renew the certs did not help.


#11

I’m not sure that certbot follows the “include” - what’s in that file ?


#12

Include /etc/letsencrypt/options-ssl-apache.conf contains this (this is a full copy/paste (nothing omitted)):

# Baseline setting to Include for SSL sites

SSLEngine on

# Intermediate configuration, tweak to your needs
SSLProtocol             all -SSLv2 -SSLv3
SSLCipherSuite          ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA
SSLHonorCipherOrder     on

SSLOptions +StrictRequire

# Add vhost name to log entries:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common

#CustomLog /var/log/apache2/access.log vhost_combined
#LogLevel warn
#ErrorLog /var/log/apache2/error.log

# Always ensure Cookies have "Secure" set (JAH 2012/1)
#Header edit Set-Cookie (?i)^(.*)(;\s*secure)??((\s*;)?(.*)) "$1; Secure$3$4"

#13

There is nothing in there that certbot should object to.

I’m not sure if certbot actually reads the include though ( I dont’ use certbot myself - hopefully one of the other users is around who does.) In the meantime, you could test with the first domain by adding the “include” into the file itself.


#14

Yeah I’m not sure certbot is objecting to it either.

It seems it’s getting stuck on the wrong conf file?

Here’s the output of 2 of the 11 domains (they all repeat this exact message) when I try to renew. This time it’s not altered :wink:

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/belchertownweather.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),)
Attempting to renew cert from /etc/letsencrypt/renewal/belchertownweather.com.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),). Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/obrienlabs.net.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),)
Attempting to renew cert from /etc/letsencrypt/renewal/obrienlabs.net.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),). Skipping.

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/atlas.obrienlabs.net.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),)
Attempting to renew cert from /etc/letsencrypt/renewal/atlas.obrienlabs.net.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),). Skipping.

So it’s trying to process belchertownweather.com, but for some reason it’s looking at the VirtualHost for atlas.obrienlabs.net? Seems there’s a mismatch and I’m not sure where that lives.

atlas.obrienlabs.net is a valid domain that needs to be renewed as well, but they all seem to be stuck on it. atlas.obrienlabs.net is also the hostname of the server. Unsure if that’s a hint as to whats going on?


#15

I suspect it reads all the files in /etc/httpd/sites-enabled/ … and “atlas” is simply the first it reads with an error in it. Does that one have the “include” in it ? if you swap it ( just in that file) from an “include” to in-line - does it then complain about the next file rather than “atlas” ?


#16

I removed the Include from the atlas.obrienlabs.net-le-ssl.conf file and tried a renew, and am getting the same errors as above. No change


#17

OK, and still referring to “atlas” file as the error ?

re=reading the configs above to see if I can see any other potential errors ( although I think we got them all )


#18

Yeah, each /etc/letsencrypt/renewal/*.conf is referring to /etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf for some reason


#19

Unsure if this is helpful, hopefully it is. Here is the log output

cat /var/log/letsencrypt/letsencrypt.log shows a lot of this below which repeats.

2016-12-27 02:25:03,944:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2017-01-05 14:12:00 UTC.
2016-12-27 02:25:03,944:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2016-12-27 02:25:03,948:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2016-12-27 02:25:04,344:DEBUG:certbot.plugins.disco:Other error:(PluginEntryPoint#apache): ('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error')
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/plugins/disco.py", line 106, in prepare
    self._initialized.prepare()
  File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 190, in prepare
    self.check_parsing_errors("httpd.aug")
  File "/usr/lib/python2.7/site-packages/certbot_apache/augeas_configurator.py", line 74, in check_parsing_errors
    raise errors.PluginError(msg)
PluginError: ('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error')
2016-12-27 02:25:04,345:DEBUG:certbot.plugins.selection:No candidate plugin
2016-12-27 02:25:04,345:DEBUG:certbot.plugins.selection:No candidate plugin
2016-12-27 02:25:04,345:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
2016-12-27 02:25:04,345:INFO:certbot.main:Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),)
2016-12-27 02:25:04,346:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/obrienlabs.net.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),). Skipping.
2016-12-27 02:25:04,346:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 348, in renew_all_lineages
    main.obtain_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 554, in obtain_cert
    installer, auth = plug_sel.choose_configurator_plugins(config, plugins, "certonly")
  File "/usr/lib/python2.7/site-packages/certbot/plugins/selection.py", line 197, in choose_configurator_plugins
    diagnose_configurator_problem("authenticator", req_auth, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/plugins/selection.py", line 272, in diagnose_configurator_problem
    raise errors.PluginSelectionError(msg)
PluginSelectionError: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(('There has been an error in parsing the file (%s): %s', u'/etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf', u'Syntax error'),)

2016-12-27 02:25:04,346:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
  File "/bin/letsencrypt", line 9, in <module>
    load_entry_point('certbot==0.9.3', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 776, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 592, in renew
    renewal.renew_all_lineages(config)
  File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 365, in renew_all_lineages
    len(renew_failures), len(parse_failures)))
Error: 11 renew failure(s), 0 parse failure(s)


#20

Well, deleting the contents of the /etc/httpd/sites-enabled/atlas.obrienlabs.net-le-ssl.conf file and running a renew was successful. I added the contents back so my site would work.

I have no idea what the syntax error was, but I think I’m ok for the next 3 months…


#21

Shot in the dark: The mydomain.tld-le-ssl.conf file pasted above starts with <IfModule mod_ssl.c> but doesn’t end with </IfModule>. Is that a copy and paste error, or an actual issue with the file?