Renew: Cron job fails but works on the command line

The standard form is attached at the end.

I have multiple VIRTUAL HOST domains on a Apache 2…2.15 based Centos 6.9 host (all updates installed) that I have root/ssh access to. This is a DELL blade server with 32GB of memory with equal amount of swap space.

When running the command
/usr/local/bin/certbot-auto renew --no-bootstrap --apache
from the command line it does what it supposed to do without error message albeit the 2.6 DeprecationWarning:

/root/.local/share/letsencrypt/lib/python2.6/site-packages/cryptography/init.py:26: DeprecationWarning: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of cryptography will drop support for Python 2.6
DeprecationWarning
Saving debug log to /var/log/letsencrypt/letsencrypt.log

and for every domain cert I have it shows
Cert not yet due for renewal
which is correct as I only installed them recently.

However, running the CRON job through CRONTAB
23 3 * * * root /usr/local/bin/certbot-auto renew --no-bootstrap
is a different thing:

Creating virtual environment…
Installing Python packages…
Had a problem while installing Python packages.

pip prints the following errors:

DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Requirement already satisfied (use --upgrade to upgrade): argparse==1.4.0 in /.local/share/letsencrypt/lib/python2.6/site-packages (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 11))
Collecting pycparser==2.14 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 17))
//.local/share/letsencrypt/lib/python2.6/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform.
+This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
//.local/share/letsencrypt/lib/python2.6/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause
+certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
Downloading pycparser-2.14.tar.gz (223kB)



You should consider upgrading via the ‘pip install --upgrade pip’ command.

Certbot has problem setting up the virtual environment.

We were not be able to guess the right solution from your pip
output.

NORMAL FORM:


Please fill out the fields below so we can help you better.

My domain is: multiple
CRON ran this command: /usr/local/bin/certbot-auto renew --no-bootstrap
It produced this output: Certbot has problem setting up the virtual environment.
My web server is (include version): Apache/2.2.15 (Unix)
The operating system my web server runs on is (include version): CentOS release 6.9 (Final)
My hosting provider, if applicable: myself
I can login to a root shell on my machine (yes or no, or I don't know): ssh/root access
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): NO

I have tried every possible combo of flags in the crontab, the last one has this

/PATH/certbot-auto --non-interactive --no-bootstrap – no-self-upgrade --post-hook “apachectl graceful”

but nothing works, it always complains about NOT being able to install python packages.
The above command on the command line has NO problems at all.

Few thoughts. First, are you sure it’s being run under the same user? root directly is different than normal user via sudo, and this changes the paths of the application. Second, have you tried calling the installed certbot directly in cron?

I do not use sudo. If I have to install anything or have to do any system administration work (like updates/installs) I use “su -”, install/update/whatever and get out - root in crontab is perfectly correct.
The machine in question only has a very limited number of accounts … the editors/developers of webpages and the su account for root.
I just saw that the files are in the private folder for root and not in /usr/share …
Why not giving an option of installing it globally?

certbot-auto runs sudo internally, so if you run it from a normal user account, it won’t work properly when run directly as root from cron. If you run certbot-auto while running as root (and HOME is set to /root), then it should be okay.

Supported systems have packages for them that you can install globally. Certbot-auto doesn’t use system packages for some things, and thus can’t easily be installed globally. Unfortunately, Certbot isn’t packaged for EL6 (it is for EL7 as part of EPEL).

@bmw, could you also look at this question in regard to certbot-auto's attempt to update Python packages from cron?

@jobst, I’m assuming the space that you have between -- and no-self-upgrade in your example command above wasn’t actually present in your crontab?

Why not giving an option of installing it globally?

We’re talking about changing the default behavior to install Certbot globally for a number of reasons, but in the meantime you can control where certbot-auto places its files using the environment variable VENV_PATH.

What’s going on here is certbot-auto is trying to create a new installation (or upgrade an old one if you ran without --no-self-upgrade and there was an upgrade available), but this process failed. If using VENV_PATH doesn’t fix the problem for you, can you provide the full error output from certbot-auto? It looks like the relevant output for debugging this problem was excluded from the original post.

@schoen its a typing error, sorry - not existing in the CRON tab.

@bmw I fixed it doing it another way. I installed a id_rsa and id_rsa.pub local for root on that machine with no password.

If you now set the command in CRONTAB to

    ssh localhost /PATH/certbot-auto --non-interactive --no-bootstrap --no-self-upgrade --post-hook "apachectl graceful"

as it will now get the full environment and it works …

So if anyone else has the problem - this is a way out.

Jobst

@bmw here is the full length error email.

Sorry cannot use attach “new users are not allowed” … so lines below might be broken although I made sure they are not.


Creating virtual environment…
Installing Python packages…
Had a problem while installing Python packages.

pip prints the following errors:

DEPRECATION: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of pip will drop support for Python 2.6
Requirement already satisfied (use --upgrade to upgrade): argparse==1.4.0 in /.local/share/letsencryptlibpython2.6/site-packages (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 11))
Collecting pycparser==2.14 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 17))
//.local/share/letsencrypt/lib/python2.6/site-packages/pip/vendor/requests/packages/urllib3util/ssl.py:315: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#snimissingwarning.
SNIMissingWarning
//.local/share/letsencrypt/lib/python2.6/site-packages/pip/vendor/requests/packages/urllib3util/ssl.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning
.
InsecurePlatformWarning

Downloading pycparser-2.14.tar.gz (223kB)
Collecting asn1crypto==0.22.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 21))
Downloading asn1crypto-0.22.0-py2.py3-none-any.whl (97kB)
Collecting cffi==1.4.2 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 24))
Downloading cffi-1.4.2.tar.gz (365kB)
Collecting ConfigArgParse==0.10.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 41))
Downloading ConfigArgParse-0.10.0.tar.gz
Collecting configobj==5.0.6 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 43))
Downloading configobj-5.0.6.tar.gz
Collecting cryptography==1.8.2 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 45))
Downloading cryptography-1.8.2.tar.gz (423kB)
Collecting enum34==1.1.2 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 72))
Downloading enum34-1.1.2.tar.gz (46kB)
Collecting funcsigs==1.0.2 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 75))
Downloading funcsigs-1.0.2-py2.py3-none-any.whl
Collecting idna==2.5 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 78))
Downloading idna-2.5-py2.py3-none-any.whl (55kB)
Collecting ipaddress==1.0.16 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 81))
Downloading ipaddress-1.0.16.tar.gz
Collecting linecache2==1.0.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 84))
Downloading linecache2-1.0.0-py2.py3-none-any.whl
Collecting ordereddict==1.1 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 87))
Downloading ordereddict-1.1.tar.gz
Collecting packaging==16.8 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 89))
Downloading packaging-16.8-py2.py3-none-any.whl
Collecting parsedatetime==2.1 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 92))
Downloading parsedatetime-2.1-py2-none-any.whl
Collecting pbr==1.8.1 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 95))
Downloading pbr-1.8.1-py2.py3-none-any.whl (89kB)
Collecting pyOpenSSL==16.2.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 98))
Downloading pyOpenSSL-16.2.0-py2.py3-none-any.whl (43kB)
Collecting pyparsing==2.1.8 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 101))
Downloading pyparsing-2.1.8-py2.py3-none-any.whl (54kB)
Collecting pyRFC3339==1.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 110))
Downloading pyRFC3339-1.0-py2.py3-none-any.whl
Collecting python-augeas==0.5.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 113))
Downloading python-augeas-0.5.0.tar.gz (90kB)
Collecting pytz==2015.7 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 115))
Downloading pytz-2015.7-py2.py3-none-any.whl (476kB)
Collecting requests==2.12.1 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 129))
Downloading requests-2.12.1-py2.py3-none-any.whl (574kB)
Collecting six==1.10.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 132))
Downloading six-1.10.0-py2.py3-none-any.whl
Collecting traceback2==1.4.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 135))
Downloading traceback2-1.4.0-py2.py3-none-any.whl
Collecting unittest2==1.1.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 138))
Downloading unittest2-1.1.0-py2.py3-none-any.whl (96kB)
Collecting zope.component==4.2.2 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 141))
Downloading zope.component-4.2.2.tar.gz (546kB)
Collecting zope.event==4.1.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 143))
Downloading zope.event-4.1.0.tar.gz (476kB)
Collecting zope.interface==4.1.3 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 145))
Downloading zope.interface-4.1.3.tar.gz (141kB)
Collecting mock==2.0.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 163))
Downloading mock-2.0.0-py2.py3-none-any.whl (56kB)
Collecting letsencrypt==0.7.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 174))
Downloading letsencrypt-0.7.0-py2-none-any.whl
Collecting certbot==0.15.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 178))
Downloading certbot-0.15.0-py2.py3-none-any.whl (266kB)
Collecting acme==0.15.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 181))
Downloading acme-0.15.0-py2.py3-none-any.whl (98kB)
Collecting certbot-apache==0.15.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 184))
Downloading certbot_apache-0.15.0-py2.py3-none-any.whl (135kB)
Collecting certbot-nginx==0.15.0 (from -r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 187))
Downloading certbot_nginx-0.15.0-py2.py3-none-any.whl (65kB)
Collecting setuptools>=11.3 (from cryptography==1.8.2->-r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 45))
In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
setuptools>=11.3 from https://pypi.python.org/packages/fc/cb748dcabb152fbc7eceaf585d052b8bc11cf028b917a7d2343cf1e0c092c3/setuptools-36.0.1-py2.py3-none-any.whl#md5=7a52500dcfd7c4f37f5d20e462c93560 (from cryptography==1.8.2->-r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 45))
//.local/share/letsencrypt/lib/python2.6/site-packages/pip/vendor/requests/packages/urllib3/util/ssl.py:120: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. For more information, see https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning.
InsecurePlatformWarning
You are using pip version 1.4.1, however version 9.0.1 is available.
You should consider upgrading via the ‘pip install --upgrade pip’ command.

Certbot has problem setting up the virtual environment.

We were not be able to guess the right solution from your pip
output.

Consult https://certbot.eff.org/docs/install.html#problems-with-python-virtual-environment
for possible solutions.
You may also find some support resources at https://certbot.eff.org/support/ .

The error there is an instance of #4640 which has a fix in our development branch that will be included in our release next week. Sorry for the trouble but I’m glad you got it working.

@bmw - but this is different as it works from the command line and NOT when initiated by cron. So when initiated by cron it misses some vital info (environment, path, ???) it should have and then barfs. Starting the cron job with a "ssh localhost " in front of it will give it that missing info (which I do not know what it is) and then executes it properly.

I am not sure what changes in the environment on your system are causing this problem to manifest. Executing certbot-auto using cron on CentOS 6.9 works for me. My best guess is Python environment variables or the Python installation or virtualenv executable available in your PATH.

Regardless of what that change is though, I believe the problem that occurs due to these environment changes will be fixed in the next release. The error that’s happening is:

In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
setuptools>=11.3 from https://pypi.python.org/packages/fc/cb748dcabb152fbc7eceaf585d052b8bc11cf028b917a7d2343cf1e0c092c3/setuptools-36.0.1-py2.py3-none-any.whl#md5=7a52500dcfd7c4f37f5d20e462c93560 (from cryptography==1.8.2->-r /tmp/tmp.fhMh8I2wGC/letsencrypt-auto-requirements.txt (line 45))

This problem has been fixed and should not occur using the version of certbot-auto we are releasing with Certbot 0.16.0 late next week.

If you’re interested in continuing to help debug this, I encourage you to try running certbot-auto in cron without using SSH after this release comes out. If you still have the problem then, feel free to @name me directly.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.