Importing main function from pip version of certbot doesn't work

Help
1

Hey there,

I'm writing a python project, "Onepass" which is a password manager, and I want to setup https for the web-ui. This is logical as usernames and passwords are passed through the api to- and from the web-ui so I want to make those secure.

This is the folder structure of the project:

Onepass
| > backend/
| > frontend/
| > libs/
\ > Onepass.py

All the dependencies are installed in the libs folder using the following:

python3 -m pip install [dep] --target=./libs

Then in Onepass.py, I insert the folder path in the sys path so that the project can import dependencies from this libs folder. This way the user doesn't have to manually install all the dependencies and instead I already provide them ("Dependabot" will be used in the repo to keep the dependencies updated).

I discovered the python version of certbot and made a plan on how to integrate it into my project so that it's as easy as possible for the user:

  1. certbot and certbot-apache are installed in the libs folder
  2. Make user run sudo apt-get install apache2 libaugeas0 -y
  3. Then install Onepass from github
  4. On first run, import the main function from certbot, using from certbot.main import main, and run main('--apache') to setup https.

However, doing from certbot.main import main doesn't work: I get the error

ModuleNotFoundError: No module named 'zope.component'

Is there some way to fix this? For testing purposes, I've done what the documentation says (setup venv and bind /opt/cerbot/bin/certbot to /usr/bin/certbot) and that works. I can even go inside the venv and do the import myself (from certbot.main import main) and that works.

However, when I want to import the main function, where certbot is not installed in a venv but in the libs folder, it fails as seen above. Keep in mind that inside /opt/certbot/bin/certbot, they also just import the main function exactly as I want to do above.

My conclusion is that you can import the main function of certbot when it's inside a venv but not when it's just in a folder installed.

Is there any way to make this work? Otherwise I would need to setup a venv for my complete project just for this or let the user manually install certbot using the venv and then do some weird stuff inside my project to interact with it.

Anyhow, thanks in advance.

2

It looks like zope is only installed inside that venv. Have you installed certbot properly in your app's venv?

1 Like
3

My Onepass project doesn't have a venv. I just have the libs folder which I add to the sys path and all my functions then import the dependencies as normal from the libs folder (so that the user doesn't have to install them themselves). If I install certbot and certbot-apache in the libs folder and do ls, is see the following:

acme                                          cffi.libs                            idna-3.3.dist-info             pytz                               _version.py
acme-1.25.0.dist-info                         charset_normalizer                   josepy                         pytz-2022.1.dist-info              zope
augeas                                        charset_normalizer-2.0.12.dist-info  josepy-1.13.0.dist-info        requests                           zope.component-5.0.1.dist-info
augeas.py                                     ConfigArgParse-1.5.3.dist-info       OpenSSL                        requests-2.27.1.dist-info          zope.component-5.0.1-py3.8-nspkg.pth
bin                                           configargparse.py                    parsedatetime                  requests_toolbelt                  zope.event-4.5.0.dist-info
certbot                                       configobj-5.0.6.dist-info            parsedatetime-2.6.dist-info    requests_toolbelt-0.9.1.dist-info  zope.event-4.5.0-py3.6-nspkg.pth
certbot-1.25.0.dist-info                      configobj.py                         pkg_resources                  setuptools                         zope.hookable-5.1.0.dist-info
certbot_apache                                cryptography                         __pycache__                    setuptools-61.3.1.dist-info        zope.hookable-5.1.0-py3.8-nspkg.pth
certbot_apache-1.25.0.dist-info               cryptography-36.0.2.dist-info        pycparser                      six-1.16.0.dist-info               zope.interface-5.4.0.dist-info
certifi                                       distro                               pycparser-2.21.dist-info       six.py                             zope.interface-5.4.0-py3.8-nspkg.pth
certifi-2021.10.8.dist-info                   distro-1.7.0.dist-info               pyOpenSSL-22.0.0.dist-info     test
cffi                                          _distutils_hack                      pyrfc3339                      urllib3
cffi-1.15.0.dist-info                         distutils-precedence.pth             pyRFC3339-1.1.dist-info        urllib3-1.26.9.dist-info
_cffi_backend.cpython-38-x86_64-linux-gnu.so  idna                                 python_augeas-1.1.0.dist-info  validate.py

If I insert from certbot.main import main into my Onepass.py script, which I directly run, it gives the error.

However, when I install certbot and certbot-apache inside a testing venv (completely separate from Onepass), it works fine and I can even go into a python shell inside the venv and import main. But when I install them in the libs folder and try to import it in my script, it gives the error.

So again, my project doesn't have a venv.

When I install certbot and certbot-apache in the libs folder, I see zope in there. You can look for yourself in the ls output above.

4

You do realize that each venv has their own libs, right?

If you're not using a venv for your application, you should probably use subprocess.run().

1 Like
5

Two (other) things:

  • Is Apache only used for the challenge for the cert? If so, you could also (better) use the build in webserver using the standalone plugin using --standalone. No Apache required
  • Is Certbot the best option for you? I believe there are smaller Python ACME clients/libraries available. Check ACME Client Implementations - Let's Encrypt and look for the Python section.

Edit: It seems none of Python implementations are actually libraries, but that might not be an issue. Because ACME clients need to be run frequently to check for renewing using cron or systemd timers anyway.

4 Likes
6

I do. I don't want to use venv's for anything if possible. I want to use certbot without a venv. I want to use Onepass without a venv. I just want to install it in my libs folder, import it's main function and interact with/use certbot that way. However the import part is going wrong.

Inside a testing venv it works fine, but when I import it without certbot being in a venv (instead just being in the libs folder), I get the error.

7

I have a flask framework and host it using waitress (both python libraries). So yeah this is great because that means I (and all the users) don't need to 1) install apache2 themselves and 2) don't need to use it. Thanks for this.

The attractive thing about certbot is that I can do --apache (or --standalone as I've just learned) and bam https. That's what I want. And run --renew q every few hours and bam up-to-date certificates. The user doesn't even have to think about it. It. just. works.
If any of the python ACME clients/libraries can do this, I'm fine with using those. Described above is all I want: a simple, but mostly user-interaction free/light, solution for https. And something that I can interact with from inside my project.

Doesn't matter. As long as one of them is in my libs folder, I can import functions from it and interact with it to setup/sustain https. Onepass is a service (it always runs) so I can do the renewing part myself every few hours from within the project. Avoids the user having to setup cronjobs.

8

An acme client, like certbot, helps with getting a cert. But, you also need to configure the server running your webui for https between it and some client (like a browser).

If you configure that server then any acme client will suffice.

Are each of your "users" running their own server and webui? What two systems are you trying to get HTTPS for?

2 Likes
9

Ehm, acme-tiny looks like a library -- but you have to make the CSR before.

And I think you can just import acme and get this.

1 Like
10

Yes. It's an opensource (but not yet released) self hosted software.

This is the thing. I don't know if something like waitress can do this, or that I need something like nginx or apache for this. I don't know what I would need to configure. That's why I chose certbot + apache at the beginning because the apache plugin handles all the configuration because I don't know how to. However @Osiris said that the --standalone will suffice and that I don't need apache. But where would I then configure? You get what I mean? It's starting to get confusing a bit.

Edit: Also, everyone is always talking about their domain (example.com). However, this is a self hosted software, so you just reach the web-ui using an ip, like http://192.168.2.15:8080/admin. I hope it doesn't matter that the url is an ip adress and not a nice domain?

11

I do not understand what you are trying to achieve.

What service are you serving over TLS? A webserver? Some other socket?

There are an absolute shitload of acme clients that can be better than certbot at this job. There are even webservers that include acme clients, like Caddy, mod_md for apache, and nginx plugins.

It looks like you are purposefully making it harder on yourself. :smiley:

Is it like that for development or is it supposed to be like that in production as well? You cannot get a certificate for a private IP (anywhere) or an IP at all (from Let's Encrypt).

1 Like
12

Yes, that's another option indeed, just use the acme-python library instead Certbot+acme. The API documentation for acme-python is here: API Documentation — acme-python 0 documentation (the name acme was already used for a different project.. :stuck_out_tongue:)

3 Likes
13

That's in fact a big deal: almost none of the free CAs offer certificates for IP addresses. Currently, only the new CA on the block, Googles CA, lets you get a cert for an IP address, but only if you own the entire IP block. Also, a cert for a private IP address can never be issued by a publicly trusted CA!

You really should learn more about ACME, Let's Encrypt and how it works. Start for example here:

3 Likes
14

It looks like that changed: acme · PyPI

1 Like
15

Offtopic: I'm referring to the "readthedocs" site, as that's where the documentation is hosted :slight_smile: See the acme readthedocs project here: acme | Read the Docs Which is not ACME protocol related at all :wink:

3 Likes
16

I'm sorry for all this. I thought I understood https.

I don't

2 Likes
17

There's a big difference between non-publicly trusted HTTPS (which can be done by anyone by just generating a self-signed certificate with a few simple OpenSSL commands) and publicly trusted HTTPS :slight_smile: The latter comes with a lot of rules. Rules which are required to make those CAs trusted in the first place and to keep the trust system in tact.

2 Likes
18

Onepass is a self hosted server software that has a web-ui. Using the web-ui, a user can log in and manage their passwords. That means that GET requests are made by the browser to get the passwords of the user and POST requests are made by the browser to remove/add/edit passwords of the user. That means that web requests are made, that contain usernames and passwords (that being from logging in, from getting the current passwords, from adding a password, etc.).

It's dangerous to make these requests over http, as anyone on the network can inspect them and can get to know login credentials or even get the cookie and make authorized requests for themselves. So to "fix" this, the network requests would need to be secure so that anyone else on the network can't look at them. As far as I know, the solution is https.

Is this something that I could do instead then? I mean... all I want is that those api requests containing usernames and passwords are secure (that is the whole thing that I'm trying to achieve). I just want the requests to look weird until they reach the server where they are decrypted, so that nobody can steal the passwords while their underway. I don't need publicly trusted stuff if I understand it correctly.

Let me first ask: if this is something that I could do (taking into account the ip-address hosting)? Would this require setup for every client (would a user have to add a cert to every browser for example)? And my second question would then be if there are any guides for this (of course there are but like... ones that you suggest).

Thanks, either way, for all the help and mostly your patience haha

1 Like
19

It's possible to set up your own small, private CA, yes.

Yes. Browsers are only supplied with publicly trusted root certificates. Browsers can be "fed" other trusted root certificates by manual adding them or e.g. in a company setting by using a certain "policy" in a managed system such as Windows Active Directory. Or sometimes a virus scanner inserts its own root certificate at installation, so it can detect mallware by being a Man in the Middle.

So if you're setting up your own, private CA for this, the user would somehow need to trust the certificate of that private CA.

There are indeed numerous guides. I myself have used OpenSSL Certificate Authority — Jamie Nguyen but that's just for manual certificate creation for max. 5 certs in total. Entirely NOT your setup. There are also software systems such as smallstep designed to set up a private CA. But I think that's overkill for your system, where you have a relatively small Python application which just requires a single certificate. So I'm afraid you have to look and see if there's something out there what happens to be what you need. If there is any.

Also, think about it from the perspective of your user: "Would I be fine with installing a fairly random root certificate into my browser, just so I can use that software? What consequences will that have? Can that software developer now perhaps also listen in on other HTTPS streams I make to the internet? Is this safe?"

Personally, if certain software asks me to install a root certificate, I won't use it.

Also, 'self hosted', is it suppose to be running somewhere on a server in a network with multiple users from other locations? Or is it supposed to be running locally on the computer of the user itself? As if it's the latter, the user can just surf to http://localhost:8080/admin. Localhost is safe even with HTTP.

2 Likes
20

Alright. So it's a bit of a "yes but don't do it". It would work, but a user would have to add a cert to every browser client for it to work.

It would also be a lot of work when you, for example, have a phone, desktop and laptop from which you'd want to access your passwords. I mean, the software could be used by just one user, but it could also be that someone uses it for their whole family and then it starts to get hard to get it installed for every device of every user.

So basically: "yes but no".

It's supposed to be run on something like a server or vm, and then users can access it on their client devices over the network (browsers, chrome extensions, etc.) or, when it is portforwarded, even over the internet. So: http://[server-ip]:8080/admin. Or better said:

"locations" could be just lan but maybe also internet if it's portforwarded and the users want to use it at their work too or something.

So is the only solution the following:?

A) The users accepts that it goes over http and accepts the risks (e.g. they only use it on lan)
B) The user port forwards the web-ui, sets up a (free) domain name and then I make sure that if the domain name is given, I setup https for it using a python module from the list given a few comments earlier.