Error during install and renewal failure

The output of certbot --version is now certbot 0.14.2 correct?

Can you provide the file containing the Apache virtual host that Certbot is trying to modify? Feel free to redact values as you feel is appropriate.

Yes, certbot --version shows 0.14.2.
Here is the Apache config (I would attach it, but am not allowed):
#
# Global server name:
#
ServerName www.usgwh.com

#
# Insecure public site. Redirects to the secure site.
#
<virtualHost *:80>
ServerName www.usgwh.com
Serveralias wwui.usgwh.com

Change this to www.usgwh.com after WUI V1 is retired.

Redirect permanent / https://wwui.usgwh.com

#
# Ths secure public site.
#
<VirtualHost *:443>
ServerName www.usgwh.com
ServerAlias wwui.usgwh.com
DocumentRoot /var/www/wwui/${WWUI_ENV}/web
DirectoryIndex index.php
ErrorLog ${APACHE_LOG_DIR}/wwui-error.log
LogLevel warn rewrite:warn
CustomLog ${APACHE_LOG_DIR}/wwui-access.log combined
AcceptPathInfo on
RewriteEngine on
SetEnv symfonyEnvironment ${WWUI_ENV}

SSLCertificateFile /etc/letsencrypt/live/wwui.usgwh.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/wwui.usgwh.com/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf

#
# The preview document root directory.
#
# ${WUI_ENV} is defined as preview in /etc/apache2/
<Directory /var/www/wwui/${WWUI_ENV}/web>
AllowOverride All
Require all granted

If the request is not for a real file, then rewrite it to index.php.

RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule .? %{ENV:BASE}/index.php [L]

#
# Set up the Apache status page.
#
ExtendedStatus On
<Location /server-status>
SetHandler server-status
Require all granted

#
# Apache status.
#
<VirtualHost *:80>
ServerName wwui.usgwh.com
SetHandler server-status

Thanks for the Apache config @dpatterson. I’m struggling to reproduce the problem though.

I want to figure out what’s going on here so we can fix it, but if you just need to renew your cert and want to work around the problem, you should be able to run something like:

certbot certonly --cert-name wwui.usgwh.com --domains $(sudo openssl x509 -in /etc/letsencrypt/live/wwui.usgwh.com/cert.pem -noout -text | grep DNS | sed 's/ DNS://g') --authenticator webroot --installer apache --webroot-path /var/www/wwui/preview/web && apachectl -k graceful

Based on your comments in your Apache config, I’m assuming /var/www/wwui/preview/web is where Apache is serving the files for the domains you’re requesting a cert for. If this isn’t correct, change the path in the command above to point to that directory. Also, while you manually have to reload Apache in the command above, this will be handled for you during renewal.

With that out of the way, I have some more questions for you so I can try to reproduce the problem you’re having:

  1. Is the Apache config you provided above all in a single file?
  2. Can you provide a full log showing the problem? By default these are saved in /var/log/letsencrypt. Feel free to redact values such as domains, IP addresses, and email addresses as you want.
  3. How are you setting WWUI_ENV in your Apache config?

@bmw,

  1. Yes, that is in a single file.

  2. See below. Also, I noticed in line 11 of the log that it is attempting to use the command apache2ctl. On all of the Ubuntu systems I have installed the command is apachectl (no 2).

  3. A line in /etc/apache2/envvars: export WWUI_ENV=preview
    letsencrypt.log:
    2017-06-28 11:13:02,421:DEBUG:certbot.main:certbot version: 0.14.2
    2017-06-28 11:13:02,421:DEBUG:certbot.main:Arguments: [’–quiet’]
    2017-06-28 11:13:02,421:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
    2017-06-28 11:13:02,440:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7fe8b05220d0> and installer <certbot.cli._Default object at 0x7fe8b05220d0>
    2017-06-28 11:13:02,440:DEBUG:certbot.cli:Default Detector is Namespace(account=<certbot.cli._Default object at 0x7fe8b053b510>, agree_dev_preview=None, allow_subset_of_names=<certbot.cli._Default object at 0x7fe8b053b2d0>, apache=<certbot.cli._Default object at 0x7fe8b0522850>, apache_challenge_location=<certbot.cli._Default object at 0x7fe8b04c6090>, apache_ctl=<certbot.cli._Default object at 0x7fe8b04c6490>, apache_dismod=<certbot.cli._Default object at 0x7fe8b04bf6d0>, apache_enmod=<certbot.cli._Default object at 0x7fe8b04bf8d0>, apache_handle_modules=<certbot.cli._Default object at 0x7fe8b04c6210>, apache_handle_sites=<certbot.cli._Default object at 0x7fe8b04c6390>, apache_init_script=<certbot.cli._Default object at 0x7fe8b04c65d0>, apache_le_vhost_ext=<certbot.cli._Default object at 0x7fe8b04bf510>, apache_logs_root=<certbot.cli._Default object at 0x7fe8b04bff10>, apache_server_root=<certbot.cli._Default object at 0x7fe8b04bf1d0>, apache_vhost_root=<certbot.cli._Default object at 0x7fe8b04bfe10>, authenticator=<certbot.cli._Default object at 0x7fe8b05220d0>, break_my_certs=<certbot.cli._Default object at 0x7fe8b04bf150>, cert_path=<certbot.cli._Default object at 0x7fe8b0511390>, certname=<certbot.cli._Default object at 0x7fe8b052c110>, chain_path=<certbot.cli._Default object at 0x7fe8b051c150>, checkpoints=<certbot.cli._Default object at 0x7fe8b052cc90>, config_dir=<certbot.cli._Default object at 0x7fe8b051c490>, config_file=None, configurator=<certbot.cli._Default object at 0x7fe8b05220d0>, csr=<certbot.cli._Default object at 0x7fe8b053b090>, debug=<certbot.cli._Default object at 0x7fe8b053bc10>, debug_challenges=<certbot.cli._Default object at 0x7fe8b053bd10>, dialog=None, domains=<certbot.cli._Default object at 0x7fe8b052c290>, dry_run=<certbot.cli._Default object at 0x7fe8b052c7d0>, duplicate=<certbot.cli._Default object at 0x7fe8b053b610>, eff_email=<certbot.cli._Default object at 0x7fe8b052cbd0>, email=<certbot.cli._Default object at 0x7fe8b052cad0>, expand=<certbot.cli._Default object at 0x7fe8b052ced0>, force_interactive=<certbot.cli._Default object at 0x7fe8b052c410>, fullchain_path=<certbot.cli._Default object at 0x7fe8b0511e90>, func=<function renew at 0x7fe8aa4d2398>, hsts=<certbot.cli._Default object at 0x7fe8b04bf650>, http01_port=<certbot.cli._Default object at 0x7fe8b04bf050>, ifaces=<certbot.cli._Default object at 0x7fe8b052c0d0>, init=<certbot.cli._Default object at 0x7fe8b052ca90>, installer=<certbot.cli._Default object at 0x7fe8b05220d0>, key_path=<certbot.cli._Default object at 0x7fe8b05115d0>, logs_dir=<certbot.cli._Default object at 0x7fe8b051c9d0>, manual=<certbot.cli._Default object at 0x7fe8b0522f50>, manual_auth_hook=<certbot.cli._Default object at 0x7fe8b04bfa10>, manual_cleanup_hook=<certbot.cli._Default object at 0x7fe8b04c6810>, manual_public_ip_logging_ok=<certbot.cli._Default object at 0x7fe8b04c6910>, must_staple=<certbot.cli._Default object at 0x7fe8b04bf350>, nginx=<certbot.cli._Default object at 0x7fe8b0522a90>, no_bootstrap=<certbot.cli._Default object at 0x7fe8b053b910>, no_self_upgrade=<certbot.cli._Default object at 0x7fe8b053b810>, no_verify_ssl=<certbot.cli._Default object at 0x7fe8b053be10>, noninteractive_mode=<certbot.cli._Default object at 0x7fe8b052c590>, num=<certbot.cli._Default object at 0x7fe8b053b4d0>, os_packages_only=<certbot.cli._Default object at 0x7fe8b053b710>, post_hook=<certbot.cli._Default object at 0x7fe8b053bdd0>, pre_hook=<certbot.cli._Default object at 0x7fe8b053bfd0>, pref_challs=<certbot.cli._Default object at 0x7fe8b04bfd50>, prepare=<certbot.cli._Default object at 0x7fe8b052c890>, quiet=True, reason=<certbot.cli._Default object at 0x7fe8b052ce90>, redirect=<certbot.cli._Default object at 0x7fe8b04bf450>, register_unsafely_without_email=<certbot.cli._Default object at 0x7fe8b052c8d0>, reinstall=<certbot.cli._Default object at 0x7fe8b052cdd0>, renew_by_default=<certbot.cli._Default object at 0x7fe8b053b0d0>, renew_hook=<certbot.cli._Default object at 0x7fe8b053bbd0>, renew_with_new_domains=<certbot.cli._Default object at 0x7fe8b053b1d0>, rsa_key_size=<certbot.cli._Default object at 0x7fe8b04bf250>, server=<certbot.cli._Default object at 0x7fe8b051cc90>, staging=<certbot.cli._Default object at 0x7fe8b053bb10>, standalone=<certbot.cli._Default object at 0x7fe8b0522d10>, standalone_supported_challenges=<certbot.cli._Default object at 0x7fe8b04c6a50>, staple=<certbot.cli._Default object at 0x7fe8b04bfa50>, strict_permissions=<certbot.cli._Default object at 0x7fe8b04bfc50>, text_mode=<certbot.cli._Default object at 0x7fe8b052c710>, tls_sni_01_port=<certbot.cli._Default object at 0x7fe8b053bf10>, tos=<certbot.cli._Default object at 0x7fe8b053b410>, uir=<certbot.cli._Default object at 0x7fe8b04bf850>, update_registration=<certbot.cli._Default object at 0x7fe8b052c9d0>, user_agent=<certbot.cli._Default object at 0x7fe8b053b290>, validate_hooks=<certbot.cli._Default object at 0x7fe8b053b9d0>, verb=‘renew’, verbose_count=<certbot.cli._Default object at 0x7fe8b0511550>, webroot=<certbot.cli._Default object at 0x7fe8b04bfc10>, webroot_map=<certbot.cli._Default object at 0x7fe8b04c6c50>, webroot_path=<certbot.cli._Default object at 0x7fe8b04c66d0>, work_dir=<certbot.cli._Default object at 0x7fe8b051c690>)
    2017-06-28 11:13:02,455:DEBUG:certbot.log:Root logging level set at 30
    2017-06-28 11:13:02,456:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
    2017-06-28 11:13:02,472:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2017-07-23 18:50:00 UTC.
    2017-06-28 11:13:02,472:INFO:certbot.renewal:Cert is due for renewal, auto-renewing…
    2017-06-28 11:13:02,473:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
    2017-06-28 11:13:02,480:DEBUG:certbot.plugins.util:Can’t find apache2ctl, attempting PATH mitigation by adding /usr/sbin:/usr/local/bin:/usr/local/sbin
    2017-06-28 11:13:02,554:DEBUG:certbot_apache.configurator:Apache version is 2.4.18
    2017-06-28 11:13:02,821:DEBUG:certbot.plugins.disco:Other error:(PluginEntryPoint#apache): Error Parsing variable: ${WWUI_ENV}
    Traceback (most recent call last):
    File “/usr/lib/python2.7/dist-packages/certbot/plugins/disco.py”, line 120, in prepare
    self._initialized.prepare()
    File “/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py”, line 196, in prepare
    self.vhosts = self.get_virtual_hosts()
    File “/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py”, line 615, in get_virtual_hosts
    new_vhost = self._create_vhost(path)
    File “/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py”, line 591, in _create_vhost
    self._add_servernames(vhost)
    File “/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py”, line 539, in _add_servernames
    servername, serveraliases = self._get_vhost_names(host.path)
    File “/usr/lib/python2.7/dist-packages/certbot_apache/configurator.py”, line 527, in _get_vhost_names
    servername = self.parser.get_arg(servername_match[-1])
    File “/usr/lib/python2.7/dist-packages/certbot_apache/parser.py”, line 351, in get_arg
    raise errors.PluginError(“Error Parsing variable: %s” % var)
    PluginError: Error Parsing variable: ${WWUI_ENV}
    2017-06-28 11:13:02,823:DEBUG:certbot.plugins.selection:No candidate plugin
    2017-06-28 11:13:02,823:DEBUG:certbot.plugins.selection:No candidate plugin
    2017-06-28 11:13:02,823:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
    2017-06-28 11:13:02,823:INFO:certbot.main:Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
    The error was: PluginError(u’Error Parsing variable: ${WWUI_ENV}’,)
    2017-06-28 11:13:02,824:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/wwui.usgwh.com.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
    The error was: PluginError(u’Error Parsing variable: ${WWUI_ENV}’,). Skipping.
    2017-06-28 11:13:02,825:DEBUG:certbot.renewal:Traceback was:
    Traceback (most recent call last):
    File “/usr/lib/python2.7/dist-packages/certbot/renewal.py”, line 418, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
    File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 634, in renew_cert
    installer, auth = plug_sel.choose_configurator_plugins(config, plugins, “certonly”)
    File “/usr/lib/python2.7/dist-packages/certbot/plugins/selection.py”, line 196, in choose_configurator_plugins
    diagnose_configurator_problem(“authenticator”, req_auth, plugins)
    File “/usr/lib/python2.7/dist-packages/certbot/plugins/selection.py”, line 271, in diagnose_configurator_problem
    raise errors.PluginSelectionError(msg)
    PluginSelectionError: The apache plugin is not working; there may be problems with your existing configuration.
    The error was: PluginError(u’Error Parsing variable: ${WWUI_ENV}’,)

    2017-06-28 11:13:02,825:DEBUG:certbot.log:Exiting abnormally:
    Traceback (most recent call last):
    File “/usr/bin/certbot”, line 11, in
    load_entry_point(‘certbot==0.14.2’, ‘console_scripts’, ‘certbot’)()
    File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 742, in main
    return config.func(config, plugins)
    File “/usr/lib/python2.7/dist-packages/certbot/main.py”, line 692, in renew
    renewal.handle_renewal_request(config)
    File “/usr/lib/python2.7/dist-packages/certbot/renewal.py”, line 435, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
    Error: 1 renew failure(s), 0 parse failure(s)

Thanks

Also, I noticed in line 11 of the log that it is attempting to use the command apache2ctl. On all of the Ubuntu systems I have installed the command is apachectl (no 2).

You should have both. On my Ubuntu 16.04 system, /usr/sbin/apachectl is a symlink to /usr/sbin/apache2ctl.

Sorry for the continued stream of questions, but do you have another file that uses WWUI_ENV? Unfortunately, there's no information in the logs about which file Certbot is erroring on, but after adding the file you gave me as is to the default Apache installation on Ubuntu 16.04, enabling the necessary modules, adding WWUI_ENV to envvars, changing domain names, and setting up the files in /etc/letsencrypt, I still can't reproduce.

@bmw, Hmm. Maybe we should back up a bit.
You may have noticed that title of my original post includes the phrase “Error during install”.
When I originally installed certbot, the installation failed toward the end. I posted a question here about it, but I don’t think that I ever got an answer. You can find it here: New install: Wrong document root. Perhaps a peek at that might yield something useful.

As to having another file using WUI_ENV, no, I don’t. That is currently the only Apache virtual host on that Ubuntu host.

I'm sorry you never got a response on your other post. To respond now:

I obviously did something wrong...

I don't think you did. Before Certbot 0.14.0, it ignored Apache configuration files that contain more than one virtual host. I think by doing this, the vhost in 000-default.conf was the only one that made sense for Certbot to modify so it picked that one.

  1. How do I back this out?

Certbot creates checkpoints of your Apache configuration which you can restore using the command certbot rollback. You can specify how many checkpoints to rollback with the --checkpoints flag (e.g. to rollback two checkpoints, run certbot rollback --checkpoints 2). This will revert any files that Certbot modified to the state they were in before Certbot made the changes for that checkpoint. Keep in mind this means that any changes you have made yourself since that checkpoint was made will also be reverted.

  1. How do I run certbot and tell it what configuration file to modify?

If it's not clear which virtual host to modify, Certbot will ask you. I suspect this probably should be fixed for you now that you have Certbot 0.14.0+.

So another thing Certbot does is parse all files in sites-available, not just your enabled sites. With this in mind, do any of the other files in sites-available use WWUI_ENV? If you run grep -r WWUI_ENV /etc/apache2, do you see any files other than envvars and the file you included above? If so, can you provide the contents of that file?

Grepping for WUI_ENV shows that it shows up in two files: wwui-preview.conf and #wwui-preview.conf#.
What generates ## files?

Emacs. It’s actually possible that’s the problem. If you move or delete that file, do you still have the problem?

@bmw, sorry for the delay. You know how priorities shift. :wink:
Okay. Cleaned up that directory a bit. Here are its contents now:
dpatterson@WWUI:/etc/apache2/sites-available$ ll
total 36
drwxr-xr-x 2 root root 4096 Jul 5 08:45 ./
drwxr-xr-x 8 root root 4096 Jun 27 05:32 …/
-rw-r–r-- 1 root root 1465 Apr 24 12:57 000-default.conf
-rw-r–r-- 1 root root 1582 Apr 24 14:48 000-default-le-ssl.conf
-rw-r–r-- 1 root root 1582 Apr 24 12:49 000-default-le-ssl.conf~
-rw-r–r-- 1 root root 6338 Apr 5 2016 default-ssl.conf
-rwxr-xr-- 1 root root 1734 Jan 16 08:49 wwui-offline.conf*
-rwxr-xr-- 1 root root 2624 Apr 25 11:10 wwui-preview.conf*

And here is the result of a renew dry run:

dpatterson@WWUI:/var/www/wwui/preview$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/wwui.usgwh.com.conf

Cert is due for renewal, auto-renewing…
Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(u’Error Parsing variable: ${WWUI_ENV}’,)
Attempting to renew cert from /etc/letsencrypt/renewal/wwui.usgwh.com.conf produced an unexpected error: The apache plugin is not working; there may be problems with your existing configuration.
The error was: PluginError(u’Error Parsing variable: ${WWUI_ENV}’,). Skipping.
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/wwui.usgwh.com/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)
1 renew failure(s), 0 parse failure(s)

Thanks.

Sorry to do this again, but to verify the output of grep -lr WWUI_ENV /etc/apache2 is exactly:

/etc/apache2/sites-available/wwui-preview.conf
/etc/apache2/envvars

Any other files, regardless of extension or location in /etc/apache2 may be relevant. Additionally, are you including any files outside of /etc/apache2 in your Apache config?

If this is correct and you’re not including any other files, can you provide a copy of your other virtual hosts? I’m still unable to reproduce the problem using the file you provided above so I’d like to more closely mimic your Apache configuration. I’d like a copy of all files in sites-available. I’d also like to see the output of ls -l /etc/apache2/sites-enabled/. If any file there is not a symlink, can you give me a copy of that file as well? Feel free to redact values as you deem appropriate and/or email the files to me rather than pasting them here. My email address is my username @eff.org.

No apology necessary. I appreciate the assistance.
Here is the command and output:

dpatterson@WWUI:/var/www/wwui/preview$ grep -lr WWUI_ENV /etc/apache2/
/etc/apache2/sites-available/wwui-preview.conf
/etc/apache2/envvars

Thanks for double checking.

If you haven't manually included any files outside of /etc/apache2 yourself, can you send me more info about your vhost configuration:

can you provide a copy of your other virtual hosts? I'm still unable to reproduce the problem using the file you provided above so I'd like to more closely mimic your Apache configuration. I'd like a copy of all files in sites-available. I'd also like to see the output of ls -l /etc/apache2/sites-enabled/. If any file there is not a symlink, can you give me a copy of that file as well? Feel free to redact values as you deem appropriate and/or email the files to me rather than pasting them here. My email address is my username @eff.org.

Brad,

Attached is a tgz with the entire /etc/apache2 directory.
I con’t think that there is any sensitive information in there.

Thanks,
Dave

@dpatterson, if you only sent your e-mail to the forum, the attachment didn’t make it through. (If you sent a copy of the e-mail to @bmw as well, then he probably did receive the attachment.)

May I ask which Apache version and OS you using? Is their a How-To for Nginx .conf on newer versions? Thank you in advance.

Apache 2.4.18, 2017-06-26
Ubuntu 16.04.2 LTS

D.

Thank you. I had trouble with Debian 9.

I also didn’t get an message with the attachment at my @eff.org email address.

@bmw, I’ve tried sending it again.