I'm trying to use the certbot acme client inside of an AWS lambda function, and so am building all the dependencies into a zip archive using something very similar to poetry export -f requirements.txt requirements.txt then pip install -r requirements.txt.
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
setuptools from https://files.pythonhosted.org/packages/6d/38/c21ef5034684ffc0412deefbb07d66678332290c14bb5269c85145fbd55e/setuptools-50.3.2-py3-none-any.whl#sha256=2c242a0856fbad7efbe560df4a7add9324f340cf48df43651e9604924466794a (from acme==1.9.0->-r requirements.txt (line 1))
Having dug around in the code I believe the issue is caused by setuptools not being pinned to a particular version in in the certbot acme client.
I was going to create an issue as a possible bug report, but noticed a few issues to do with setuptools already and so wanted to check this wasn't something intentional and that I may be missing a more obvious solution?
I'm sorry I don't understand your reply. Please could you elaborate? If your reply is related to the the Ubuntu snap system, I would appreciate some direction on how this is related. The environment I am experiencing this error in is a CircleCI python:3.8.6 container.
This is the first time I have used the certbot library, and so appreciate the effort and respect shown by all those assisting.
Ah I see - ha yes, humour is always so hard crossing borders and with the limitations of technical forms! Thank-you for taking the time to look at my issue and offer assistance.
No, I did not use snapd to install certbot. I am installing certbot through a standard pip install based on a requirements.txt file exported by poetry, in a standard python docker container pulled from Docker Hub. The pip install is failing because the calculated dependency graph does not pin a version of setuptools, and so installing from pip with hashes fails because I don't have hashes for setuptools, unless I --allow-unsafe. Should I install certbot some other way and if so which C-libs etc. do I need to copy into my lambda environment? I just want to use the acme client in a python script in a small lambda function, so minimal install is preferable and I don't want to think too much about shims and so on.
I've noticed pytz is also not pinned (again, realise with the 3.9 release this may be a deliberate decision/moving towards obsolescence), so realise the error may be something deeper/different (or that pytz will be the next error after solving this one).
I can't say that I know with certainty how that should best be installed within CircleCI.
And it seems we are not alone on this, and I can see why you came looking for help here; when the only thing returned for certbot on their help forum is more than three years old and completely un-helpful: Search results for 'certbot' - CircleCI Discuss
That said, there are some very clever people in this community (which will undoubtedly chime in on this given adequate time - it is Sunday afternoon in the states now - where most of them live) like @_az, @certbot-devs
I'm thinking 1.9.0 may not be the most adequate version for that system, but I'll leave the advice for the experts (thou exact CircleCI proficiency may hard to find it might be similar enough to, or based on, some other .well-known system).
READERS: Get involved. Be heard. Do your part, it starts with: If you read something you like; then like it !
I think that the setuptools situation is a bit complicated, and to make things worse, it's currently undergoing changes. I think bmw or adferrand should be able to answer in more detail.
As a guide, you can look at how Certbot can be deployed from pip by looking at certbot-auto.
In particular, the embedded pipstrap snippet is responsible for installing a pinned version of setuptools.
certbot-auto is also responsible for installing relevant C libraries like libssl, libffi, libaugeas etc.
In general, if I were to look at running something ACME-related in Lambda, I would probably choose something with less dependencies (or a zero dependency static binary, such as lego). It would also probably save you a decent bit of money, as the memory usage and runtime duration would be a lot lower.
Thanks for everybody's help. @_az - I would definitely prefer to use an acme client with fewer dependencies, do you know if lego has a python wrapper? The only thin ACME client I've found it acme-tiny which looks relatively specific to a few use cases, and makes me think it would be easier to write my own client, which I am very keen not to do.
I could also --allow-unsafe inside my CI script, but fear this would just lead to the deployment breaking again in the future with new releases (because I can't pin to particular versions etc.).
_az would definitely know better than me, but I believe that acme.sh is written purely in script/shell and should have much less in the way of dependencies. I'm currently modifying my own thin client that is pure PHP, but I think your circumstances might be beyond that.
Perhaps this is of value about acme.sh?
Purely written in Shell with no dependencies on python or the official Let's Encrypt client
Ah, I think I misunderstood your original post: You are using the acme module in your Python Lambda function, rather than using the Certbot application proper. So you wouldn't be requiring the certbot module at all.
FWIW poetry add acme && poetry export && pip install --require-hashes works alright for me in that environment:
I'm not too sure why pip would be complaining about setuptools in this instance, but like I said, one of the senior maintainers should be able to help out tomorrow(ish).
Thanks for your help - I've switched my dependency to acme = "^1.9.0" rather than certbot. However, I'm still getting the error in Circle. Please could you share the poetry.lock and pyproject.toml generated in the container above to assist me debugging this (probably tomorrow morning - it's getting late in the UK!).
Again, really can't thank everyone enough for their help.
It's in Node.js rather than in Python, but I put together an ACME renewal process in Lambda as a hobby project if you're interested in getting some ideas from it that at least worked for me: acme-lambda-renewal | Peter & Jessi Cooper & Family
Thanks for your help everyone - I intend to investigate the issue with hashes as a bug in my build process, however to check the feasibility of what I'm doing, I've added --without-hashes to my export, and am testing the ACME API process.
However, I'm now getting:
[ERROR] KeyError: 'Directory field "new-reg" not found'
Traceback (most recent call last):
File "/var/task/tls_manager/certgen/main.py", line 50, in handler
reg = acme.register()
File "/var/task/acme/client.py", line 279, in register
response = self._post(self.directory[new_reg], new_reg)
File "/var/task/acme/messages.py", line 255, in __getitem__
raise KeyError('Directory field "' + self._canon_key(name) + '" not found')
I'm using the ACME v2 endpoint, but this looks like an issue with a v1 client trying to use a v2 endpoint. Am I right to assume that acme==1.9.0 is an ACME v2 RFC 8555 client? Am I missing something?
!
!