Help test Certbot Apache and Nginx fixes for TLS-SNI-01 outage

@ndesktop, so it hung running either sudo certbot --apache or sudo certbot --nginx? Can you paste the log of the issue found in /var/log/letsencrypt? Feel free to redact domains, emails, and IPs as you deem appropriate.

@untuned, ah, that’s because . venv/bin/activate only affects your current shell. If you close and reopen it, you have to run that command again or you can run Certbot directly without putting it in your PATH from venv/bin/certbot.

@bmw Since I wrote and before you responded, the whitelist came through for our server, so that satisfied my immediate need.

But I thought I’d let you know that I’ve since tried your improved instructions. Again, didn’t work for me. I think it may be a Centos/RHEL 6 issue of having python 2.6. I ran both certbot renew and certbot --apache and both croaked on the following:
File “/space/home/dmsmith/certbot/certbot/venv/lib/python2.6/site-packages/lexicon/common/options_handler.py”, line 23
super(SafeOptions, self).update({k:v for k,v in update_options.items() if v})

I can provide more of the output if needed.

DM

Some notes:

  • Due to the default sudo secure_path setting on Debian Stretch, I had to run sudo ./venv/bin/certbot to use the virtualenv certbot even with the virtualenv activated.

  • Let’s Encrypt appears to have re-enabled tls-sni-01 for existing account keys so I had to pass --preferred-challenges=http to be able to test http-01.

Unfortunately, I was ultimately unable test http-01 issuance due to my server configuration. I have a single http virtual host that redirects to the https form regardless of domain. (The server hosts a number of pure-HTTPS domains.) certbot certonly -a nginx failed with Could not automatically find a matching server block. Set the server_name directive to use the Nginx installer.

I understand the nginx installer’s reluctance to proceed in this circumstance, but perhaps the authenticator could fall back to the default virtual host in this situation, especially considering tls-sni-01 works in this configuration.

1 Like

@DMSmith, I’m glad you were able to obtain the cert you needed. You’re right that the problem you saw was due to Python 2.6. tools/venv.sh installs all of Certbot’s plugins and some of them do not work on this version of Python. Unfortunately, I forgot about this when I edited the instructions. Thanks for pointing this out. I’ve updated the original instructions again to work around this problem if you feel like trying again.

@Patches, thanks for the helpful notes. I’ve updated the instructions in the original post to deal with both of those problems. Unfortunately, getting that error message about matching server blocks is currently expected if you don’t have a server block listening on port 80 containing a server_name directive matching the name you’re requesting a certificate for. This is one of things we’re planning on fixing before the release though.

1 Like

Hi @bmw, I just got this issue when running the last command:

tools/_venv_common.sh -e acme -e . -e cerbot-apache -e certbot-nginx

  • VENV_NAME=venv
  • rm -rf *.egg-info
  • date +%s
  • mv venv venv.1515882035.bak
  • virtualenv --no-site-packages --setuptools venv --python /usr/bin/python2
    Running virtualenv with interpreter /usr/bin/python2
    New python executable in /home//certbot/certbot/venv/bin/python2
    Also creating executable in /home//certbot/certbot/venv/bin/python
    Installing setuptools, pkg_resources, pip, wheel…
    Complete output from command /home//certbo…bot/venv/bin/python2 - setuptools pkg_resources pip wheel:
    Traceback (most recent call last):
    File “”, line 24, in
    File “/usr/share/python-wheels/pip-8.1.1-py2.py3-none-any.whl/pip/init.py”, line 215, in main
    File “/home//certbot/certbot/venv/lib/python2.7/locale.py”, line 581, in setlocale
    ** return _setlocale(category, locale)**
    locale.Error: unsupported locale setting

…Installing setuptools, pkg_resources, pip, wheel…done.
Traceback (most recent call last):
File “/usr/lib/python3/dist-packages/virtualenv.py”, line 2363, in
main()
File “/usr/lib/python3/dist-packages/virtualenv.py”, line 719, in main
symlink=options.symlink)
File “/usr/lib/python3/dist-packages/virtualenv.py”, line 988, in create_environment
download=download,
File “/usr/lib/python3/dist-packages/virtualenv.py”, line 918, in install_wheel
call_subprocess(cmd, show_stdout=False, extra_env=env, stdin=SCRIPT)
File “/usr/lib/python3/dist-packages/virtualenv.py”, line 812, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /home//certbo…bot/venv/bin/python2 - setuptools pkg_resources pip wheel failed with error code 1

Later edit: made it work with this: https://stackoverflow.com/questions/14547631/python-locale-error-unsupported-locale-setting

I think SystemD’s PrivateTmp feature on CentOS 7 and RHEL 7 might be an issue. I added a GitHub issue for it. After disabling PrivateTmp it works for me using CentOs 7.3.

Once this feature lands, will I need to do anything to update existing sites so they renew correctly or will certbot switch to http-01 automatically?

1 Like

Somehow it’s not working for me.
https://pieterjan.pro/LetsEncrypt.docx
I use Apache2 on Raspbian Stretch

Not sure this is the right thread in which to post this suggestion, and of course I may not have spotted some glaring issue, or an easy fix, but it seems that for anyone who has multiple sub-domains fronted by a reverse proxy, both http and dns might be complex to make work consistently. I think there might be a secure and relatively easy way to automate TLS-SNI whitelisting on a domain name basis.

What if the domain ownership is validated via a required sub-domain e.g. le-89729572.mydomain.com with either a DNS or HTTP validation method, and then SNI is whitelisted for any sub domains using the same IP address(es)? That seems highly automatable and verifiable, and if abused easily revoked on a domain basis.

@ryanjaeb, again, thank you so much for the detailed issue. A fix for this has landed in master. Once the feature is released, as long as you’re getting an updated version of Certbot on your system, it will switch to HTTP-01 automatically.

@PieterjanDeClippel, I think you may have had the same problem as @ryanjaeb. From the certbot directory, try running git pull and then run Certbot again to see if you still have the issue.

@bmw Inside a docker container that runs nginx - this is the error we are getting:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for test1.aicial.io
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. test1.aicial.io (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://test1.aicial.io/.well-known/acme-challenge/HfntYogksMO0sqVwj0GyvJAiYbh47bamCGj453LQON4: Timeout

Can use really/nginx-certbot:testing which has certbot built per the instructions in this post and then linked.

Does anybody know if the TLS-SNI-01 outage affects the Let’s Encrypt for cPanel plugin? I’ve been able to successfully issue certificates using that plugin since the start of the outage, which I assume is because it uses the HTTP validation method.

However, I just wanted to make sure that the hosting provider I work for (and thus get hosting through) hasn’t had any exception made, and I can therefor continue issuing certificates on other hosting providers. I obviously don’t maintain certificates on any VPS machines or anything, so this issue may not apply to me at all.

@troykelly: the validation servers timed out trying to connect to the your server to retrieve the validation file. It doesn’t appear to be related to the new functionality. Your website works from my connection, so please try again and if the error persists open a new top-level thread so a Let’s Encrypt engineer can see if there are any issues with their network.

@sulliops: There are a few different CPanel plugins for Let’s Encrypt. The CPanel-authored AutoSSL plug-in uses http-01.

In case you are referring to a certain plugin formerly known as Let's Encrypt for cPanel, I have updated the front page of the website with a clarifying statement.

1 Like

@Patches It doesn’t work when the below is in the config for the server

if ($scheme != "https") {
    return 301 https://$host$request_uri;
}

The above doesn’t cause an issue before - just with the new code - hence was raising at an issue here.

1 Like
  • certbot-auto, with apache installed
  • /var/log/letsencrypt - cannot find anything (except the error regarding currently selected authenticator, which is the one disabled).

Ok, updates for me:

  1. # sudo ./certbot-auto --os-packages-only
    All seems ok, two warning lines:
    W: Ignoring Provides line with DepCompareOp for package python-cffi-backend-api-max
    W: Ignoring Provides line with DepCompareOp for package python-cffi-backend-api-min
    which looks benign.

  2. # export VENV_ARGS="--python $(command -v python2 || command -v python 2.7)"
    No output, ok.

  3. # tools/_venv_common.sh -e acme -e . -e certbot-apache -e certbot-nginx
    Some warnings, but finished ok:
    ++ /home/ca/certbot/tools/merge_requirements.py /home/ca/certbot/tools/dev_constraints.txt /tmp/tmp.1AVj9Qv3rM
    /home/ca/certbot/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:318: SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name Indication) extension to TLS is not available on this platform. This may cause the server to present an incorrect TLS certificate, which can cause validation failures. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning. SNIMissingWarning
    /home/ca/certbot/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning
    /home/ca/certbot/venv/local/lib/python2.7/site-packages/pip/_vendor/requests/packages/urllib3/util/ssl_.py:122: InsecurePlatformWarning: A true SSLContext object is not available. This prevents urllib3 from configuring SSL appropriately and may cause certain SSL connections to fail. You can upgrade to a newer version of Python to solve this. For more information, see https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning. InsecurePlatformWarning

then final message
Please run the following command to activate developer environment:
source venv/bin/activate

  1. # venv/bin/certbot --apache --preferred-challenges http
    Saving debug log to /var/log/letsencrypt/letsencrypt.log
    Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
1: [redacted].ro
2: airtime.[redacted].ro
3: fb.[redacted].ro
4: listen.[redacted].ro
5: www.[redacted].ro
-------------------------------------------------------------------------------
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1,2,3,4,5
Cert not yet due for renewal
`You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.` `(ref: /etc/letsencrypt/renewal/airtime.[redacted].ro.conf)`
What would you like to do?
-------------------------------------------------------------------------------
1: Attempt to reinstall this existing certificate
2: Renew & replace the cert (limit ~5 per 7 days)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for [redacted].ro
http-01 challenge for airtime.[redacted].ro
http-01 challenge for fb.[redacted].ro
http-01 challenge for listen.[redacted].ro
http-01 challenge for www.[redacted].ro
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. fb.[redacted].ro (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://fb.[redacted].ro/.well-known/acme-challenge/nKsabs49FVQQgPheCTm4vEghYcNhGo9GNJ2I4XgRE6w: "
<!DOCTYPE html>
<html lang="en" id="facebook" class="no_js">
<head><meta charset="utf-8" /><meta name="referrer" content="defaul", listen.[redacted].ro (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://listen.[redacted].ro/.well-known/acme-challenge/1yiF4phYb-KDZbPZ4whjYxI6Wtac-nLClyD3wbb6LlE [xxx.xxx.xxx.xxx]: 404

IMPORTANT NOTES:
- The following errors were reported by the server:
` Domain: fb.[redacted].ro` ` Type: unauthorized` ` Detail: Invalid response from` ` http://fb.[redacted].ro/.well-known/acme-challenge/nKsabs49FVQQgPheCTm4vEghYcNhGo9GNJ2I4XgRE6w:` ` "<!DOCTYPE html>` ` <html lang="en" id="facebook" class="no_js">` ` <head><meta charset="utf-8" /><meta name="referrer"` ` content="defaul"`
Domain: listen.[redacted].ro
Type: unauthorized
Detail: Invalid response from
http://listen.[redacted].ro/.well-known/acme-challenge/1yiF4phYb-KDZbPZ4whjYxI6Wtac-nLClyD3wbb6LlE
[xxx.xxx.xxx.xxx]: 404
``
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

For the fb.[redacted].ro and listen.[redacted].ro subdomains it might be normal to fail.

  • fb.[redacted].ro is a redirect to a Facebook application page
  • listen[redacted].ro is an internal redirect to an internal Icecast stream (via Airtime)

Below are the Apache conf files for those two subdomains failing (http and https):

fb.[redacted].ro.conf
<VirtualHost *:80>
ServerName fb.[redacted].ro
ProxyPreserveHost On

Redirect permanent / https://www.facebook.com/[redacted]/
RewriteEngine On
RewriteCond %{SERVER_NAME} =airtime.[redacted].ro
RewriteRule ^ https://www.facebook.com/[redacted]/%{REQUEST_URI} [END,NE,R=permanent]
RewriteCond %{SERVER_NAME} =fb.[redacted].ro
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

fb.[redacted].ro-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName fb.[redacted].ro
ProxyPreserveHost On

Redirect permanent / https://www.facebook.com/[redacted]/
RewriteEngine On
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

SSLCertificateFile /etc/letsencrypt/live/airtime.[redacted].ro/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/airtime.[redacted].ro/privkey.pem Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/airtime.[redacted].ro/chain.pem
</VirtualHost>
</IfModule>

listen.[redacted].ro.conf
<VirtualHost *:80>
ServerName listen.[redacted].ro
ProxyPreserveHost On
ProxyPass / http://localhost:8000/airtime_128
ProxyPassReverse / http://localhost:8000/airtime_128
RewriteEngine on
RewriteCond %{SERVER_NAME} =listen.[redacted].ro
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>

listen.[redacted].ro-le-ssl.conf
<IfModule mod_ssl.c>
<VirtualHost *:443>
ServerName listen.[redacted].ro
ProxyPreserveHost On
ProxyPass / http://localhost:8000/airtime_128
ProxyPassReverse / http://localhost:8000/airtime_128
SSLCertificateFile /etc/letsencrypt/live/airtime.[redacted].ro/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/airtime.[redacted].ro/privkey.pem Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateChainFile /etc/letsencrypt/live/airtime.[redacted].ro/chain.pem </VirtualHost>
</IfModule>

I cannot upload the log file (Sorry, new users cannot upload attachments), but here are the logged things for fb. and listen. subdomains:

Domain: fb.[redacted].ro
Type: unauthorized
Detail: Invalid response from http://fb.[redacted].ro/.well-known/acme-challenge/nKsabs49FVQQgPheCTm4vEghYcNhGo9GNJ2I4XgRE6w: "<!DOCTYPE html>
<html lang="en" id="facebook" class="no_js">
<head><meta charset="utf-8" /><meta name="referrer" content="defaul"

Domain: listen.[redacted].ro
Type: unauthorized
Detail: Invalid response from http://listen.[redacted].ro/.well-known/acme-challenge/1yiF4phYb-KDZbPZ4whjYxI6Wtac-nLClyD3wbb6LlE [xxx.xxx.xxx.xxx]: 404

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2018-01-15 07:11:41,985:INFO:certbot.auth_handler:Cleaning up challenges
2018-01-15 07:11:43,481:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File "venv/bin/certbot", line 9, in <module>
load_entry_point('certbot', 'console_scripts', 'certbot')()
File "/home/ca/certbot/certbot/main.py", line 1240, in main
return config.func(config, plugins)
File "/home/ca/certbot/certbot/main.py", line 994, in run
certname, lineage)
File "/home/ca/certbot/certbot/main.py", line 113, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/home/ca/certbot/certbot/renewal.py", line 297, in renew_cert
new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File "/home/ca/certbot/certbot/client.py", line 318, in obtain_certificate
self.config.allow_subset_of_names)
File "/home/ca/certbot/certbot/auth_handler.py", line 81, in get_authorizations
self._respond(resp, best_effort)
File "/home/ca/certbot/certbot/auth_handler.py", line 138, in _respond
self._poll_challenges(chall_update, best_effort)
File "/home/ca/certbot/certbot/auth_handler.py", line 202, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. fb.[redacted].ro (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://fb.[redacted].ro/.well-known/acme-challenge/nKsabs49FVQQgPheCTm4vEghYcNhGo9GNJ2I4XgRE6w: "<!DOCTYPE html>
<html lang="en" id="facebook" class="no_js">
<head><meta charset="utf-8" /><meta name="referrer" content="defaul", listen.[redacted].ro (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://listen.[redacted].ro/.well-known/acme-challenge/1yiF4phYb-KDZbPZ4whjYxI6Wtac-nLClyD3wbb6LlE [xxx.xxx.xxx.xxx]: 404

I can confirm that the latest version (https://github.com/certbot/certbot/commit/60dd67a60e4f17b5d46718d3ff941a2e292398c7) now works on Raspbian Stretch with Apache2. Linux Kernel:
Linux version 4.9.73-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611) ) #1072 SMP Sun Dec 31 19:37:41 GMT 2017

3 Likes

After doing the setup described above, running this command:

./certbot-auto --apache -d www.patchvault.org --preferred-challenges http-01

results in:

/opt/eff.org/certbot/venv/lib/python2.6/site-packages/cryptography/__init__.py:26: DeprecationWarning: Python 2.6 is no longer supported by the Python core team, please upgrade your Python. A future version of cryptography will drop support for Python 2.6
  DeprecationWarning
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
/opt/eff.org/certbot/venv/lib/python2.6/site-packages/acme/jose/jwa.py:110: DeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
  signer = key.signer(self.padding, self.hash)
Performing the following challenges:
None of the preferred challenges are supported by the selected plugin

Running off github master as of 15 Jan 1430hrs EST (commit 60dd67a60e4f17b5d46718d3ff941a2e292398c7)

Strike that. I had not run

sudo venv/bin/certbot --apache --preferred-challenges http

correctly. I was able to renew two certs by adding “renew” to the statement above.

Hi,

I did not manage to make it work. I am currently using an EC2 instance from AWS, with nginx and Node.js

In my nginx config file, I’ve tried both:

server {
    listen       80;
    server_name  my.domain.com;
}

and

server {
    listen       80;
    server_name  my.domain.com;
    location / {
        proxy_pass  http://nodejs;
        proxy_set_header   Connection "";
        proxy_http_version 1.1;
        proxy_set_header        Host            $host;
        proxy_set_header        X-Real-IP       $remote_addr;
        proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

And here is the answer:

Obtaining a new certificate
Performing the following challenges:
http-01 challenge for my.domain.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure.

my.domain.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://my.domain.com/.well-known/acme-challenge/hVz7w9UOn-wCkRib_Xb1hDTrIIbqn63xbUQlkTcPT1M [XXX.XXX.XXX.XXX]: 404

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: my.domain.com
   Type:   unauthorized
   Detail: Invalid response from
   http://my.domain.com/.well-known/acme-challenge/hVz7w9UOn-wCkRib_Xb1hDTrIIbqn63xbUQlkTcPT1M
   [XXX.XXX.XXX.XXX]: 404

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

It does not manage to find /.well-known/

Thanks for your help!