I maintain a client that connects to LE’s acme server
I want to request a feature but don’t know the right forum. Would that be here, or on the boulder repo?
The request is to have the staging server offer guaranteed responses based on:
dedicated endpoints
specialized headers
custom domain names
The would allow clients to perform unit tests against the staging server instead of only integrated tests.
In order to test certificate generation and error handling, I must run tests run on a machine + domain the ACME staging server can reach for validation. While I must still perform that test before release, an acceptable test-suite only needs to make sure that I can send a properly formatted response to ACME and receive a properly formatted response in return. Stated differently, my Integrated Tests ensure that I can get a certificate for a domain that ACME can verify… but my Unit Tests only want to ensure that I send a proper response to ACME and receive a certain answer.
For example, these could be “special” domains that ensure a particular response:
Requesting a certificate for success.. would always return a valid certificate, bypassing the domain check
Requesting a certificate for error_1 would guarantee a validation error
Currently I either have to toggle my firewall or deploy my codebase onto the public internet in order to run tests. A feature like this would make it much easier for developers to catch errors locally, then run the integrated tests against the API and real domains with a reasonable expectation they will pass.
This would mean that integrated tests are hitting a local version of boulder and not what is running on LetsEncrypt. The point of the tests are to ensure that the client is hitting LE correctly over the internet. That means testing that one can reach LE and that LE is running a comparable API.
Aside from that:
Continuous Integration tests would become overly complicated. One would have to constantly update the environment to reflect the currently deployed version of Boulder.
That would also require anyone who is forking to build their own environment.
I'm more than happy to use rsync and run the integrated test suites against the LE staging server when needed. I already set up a server to handle that. People who fork my library - or others - won't.
Having some sort of featureset on the LE staging environment that can allow developers to simulate a properly networked environment would greatly bring down the barrier of entry to contributors working on clients and client tests across the board.
I have to say I sympathise more with jvanasco’s position on this. The reality is that Let’s Encrypt is not just a generic install of Boulder, its CA function is tied very closely to lots of (expensive) infrastructure and personnel requirements, so it’s not enough for a serious client to check that their code works with their own copy of Boulder (though I’d certainly advise them to do that) but they must also be sure it works with the actual Let’s Encrypt system.
I’d liken this to Payment Gateways. It’s useful to test code against your own implementation of the Payment Gateway APIs, where you can fake the responses timing out, or receiving unusual errors. But it’s very valuable if the Gateway provides in their own pre-production environment a way to see their system produce some errors for real e.g. a credit card number that appears valid but always fails verification. I would always prefer a provider that offers me this over one that says I should just test my own systems and then cross my fingers.
It may be that this is a lot of work for Let’s Encrypt, but I’d hope that if they’re amenable to this feature existing but just don’t have the manpower to deliver it they’d be open to client maintainers like jvanasco pitching in to do the initial development and testing in Boulder code.
One thing though javanasco, I don’t think your Unit tests ought to rely on the Internet if possible. To me a Unit test ought to be very self-contained, end-to-end tests are valuable, but should be the last step in testing, finding only subtle problems that occur when the whole system is assembled and used, typically as a result of an design mistake not a typo or numerical error.
On one hand, wanting to test with a configuration that matches production as closely as possible makes sense (for things like TLS/cipher compatibility, etc.)
On the other hand, any code that is staging-only (even if it’s flag-gated) would lead to a mismatch between staging and production, and you definitely want staging to match production as closely as possible to expose all bugs prior to them going live.
Certbot opted for mocking in unit tests and using a local boulder instance for integration tests. Maybe a good middle ground would be to use staging for a “connectivity” integration test, while using a local boulder to test everything else. Boulder could be pulled by your build process (some clients have existing scripts you could reuse), and you could even match the exact commit currently used in staging/production (the /build endpoint exposes this).
The question isn’t about what I (or most people in this forum) can handle, it’s what is of most benefit to moderate or junior developers who may want to fork and submit patches to various projects.
The “barrier to entry” to running tests is incredibly high, because a developer must set up proper DNS and networking that can allow the LetsEncrypt staging ACME server to handle the validation lookups. This setup requires a lot of a developer - which may not be easy or possible in some business environments - and can take several hours for DNS records to propagate.
[quote=“tialaramex, post:4, topic:17529”]
One thing though jvanasco, I don’t think your Unit tests ought to rely on the Internet if possible. To me a Unit test ought to be very self-contained, end-to-end tests are valuable, but should be the last step in testing, finding only subtle problems that occur when the whole system is assembled and used, typically as a result of an design mistake not a typo or numerical error[/quote]
You’re right. Maybe this is, name wise, more of a functional test.
Right now our test suite runs like this:
Default-
Unit tests do not require internet. Tests all the various methods, parsing, etc.
Integrated tests will hit the LetsEncrypt staging system for various tasks, do not expect reverse validation
Environment flag set
Integrated tests hit the LetsEncrypt staging system for various tasks, does expect reverse validation
As a package maintainer, I will always have to run that last set of tests. However it is hard for contributors to do that. So my wishlist request is that a mechanism can be made so that contributors to projects can run a test suite that does not requires the DNS/networking setup, but does communicate with letsencrypt. Requiring the dns/networking is too much. Requiring a boulder setup is too much. Using only unittests that parse a local text copy of the expected response leaves too much room for error – as bugs could be in the http integration layer.
@pfg For the most part, people aren’t building clients to a “boulder server”, they’re building to the “LetsEncrypt service”. For now they are one and the same, but testing against the Master branch of boulder is not necessarily the same as testing against the deployed instance of whatever at LetsEncrypt.
To be clear, what I was suggesting was testing against the same commit of boulder that's running on staging or production. It's still not a perfect match, but very close, and there's still the possibility to have a limited set of tests that run against staging to ensure there's almost nothing that could even in theory cause issues.
I got that. My point is that most people don't think of projects as "boulder clients" (open source server), but as "letsencrypt clients" (non-commercial service) -- and letsencrypt just happens to run boulder. A junior developer with no server-side or networking experience is more than capable of forking a client for a fix or feature, but not necessarily capable of setting up all the DNS and networking required for accurate testing against the staging server or coupling that with the dev/ops skills needed to run a local version of boulder.
The current situation is that a high-barrier exists for innovating new clients or features on existing clients -- and that means development is limited to a much smaller and self-selecting group of developers (or people just make stuff that may or may not work, because it's too hard to test... so package maintainers are reluctant to go through Pull Requests).
That’s an interesting idea! I definitely support the idea of making an easier-to-use test harness to help people create, test, and improve clients. Right now I’ve been thinking in the direction of providing a ready-to-use Docker image that is kept up to date with the latest released version of Boulder and “just works” out of the box. This has the advantage that it can use the same testing flags we use in our own integration tests, for instance to override all DNS results to localhost or some other fixed IP.
However, even a Docker image may be a bit too complex to set up, and doesn’t provide what you’d really want from a test harness - for instance, the ability to reliably produce a 500 or a timeout on a given URL. Rather than creating specialized responses in the staging environment, maybe it would be better to provide a standalone Mock-ACME server that is easy to run on its own.
There are a few reasons I prefer “local Boulder” or “local Mock-ACME:”
Makes it easier for people to write tests that don’t depend on the Internet, which makes them more reliable, faster, and easier to iterate on.
We don’t want to encourage people to depend on our staging environment in our automated builds. So far staging has been pretty reliable, but we do intend to prototype more radical changes there, do load tests, and potentially take it down for periods of time. We don’t want client developers’ automated tests to fail when we do this.
Perhaps a single-file python executable, or something distributed with certbot (ie, certbot-fakeserver)
From my perspective, the MVP would do this:
have responses/endpoint structure that is pegged to the current version of the LE server
generate predictable fake responses. i.e. it doesn't need to actually generate a new cert, but simply accept the input and generate an output of a cert request via http
then developers could distribute or reference running the current certbot-fakeserver script, which would serve requests on a configurable local port