I have a requirement for the following flow that allows us to automate adding customer websites to our load balancers
Customer Signs places an order with our sales team.
Certbot runs using DNS challenge and sends them the required TXT key. This would happen in our backend services as an automation.
When the customer has managed to add the required key we need to rerun the challenge to validate it.
This is a bit of odd flow because typically our customers are web creatives who won't typically manage the DNS for the company they work for - currently, we have to get the person on the phone at exactly the right time we're running certbot, sit there with a shell open the load balancer, wait for propagation and then hit enter.
I notice if we rerun certbot in the same shell session over and over it always produces the same key, if we exit the shell session and SSH back into the load balancer it creates a new one.
So I guess I'm assuming there must be a session somewhere we can backup/restore to allow for possible long delays between the generation of code and it actually being added?
Note this wouldn't have to happen on the load balancer, the ideal situation would be we'd being doing this via a separate API and then having an automation that securely transfers the cert to the load balancer we the sites are ready to go live.
Hi there, welcome. I have a few remarks regarding the issue you're facing.
Let me start with a question: do you actually require the DNS challenge?
This should not be the case. The order (and with that challenges) re-use is handled at the Let's Encrypt server, not locally by Certbot. Pending orders and authorizations/challenges are valid for 7 days and they're coupled to the ACME account in use. Using one shell or another should not matter, unless for some reason Certbot would use a different account when you log in again.
Ideally your company would not require the user to place an order with your sales team (why a sales team to begin with? Do you charge your users for free Let's Encrypt certificates?), but your systems would automatically issue a certificate without any hassle.
So I'm using this command as a test, if I CTRL+C out of it, run it again I get the same challenge:
sudo certbot certonly -v --manual --preferred-challenges dns -d www.ourdomain.co.uk
that's odd so you're saying that it should always return the same challenge while it's valid?
No we don't charge for SSL certs, our load balancers form part an offering that allows us to append applications to client domains without them needing to host them, but of course they need SSL certs to route an application on the client domain.
Hope that clears up the use case, let me know if it doesn't.
Basically we need to be able to send a challenge to a customer via certbot, then complete the challenge in a different session.
And it should do that for the next 7 days regardless of if you logged out and in again, as long as it's the same server with the same Certbot (or at least the same ACME account).
Well, it wouldn't be called "valid" as that's the state when the challenge actually has been validated. But yes, the order and challenges should be available for a while, unless they get triggered and fail.
That's good to know While it wouldn't be illegal or something to charge a fee for free Let's Encrypt certs, it's certainly something that's frowned upon by many volunteers of this Community
That should be possible. I don't know why logging out and in again to the shell would result in a new challenge token. Unless your Certbot data isn't stored in between sessions.
Also still interested to hear why you require the dns-01 challenge and can't use the much simpler http-01 challenge.
Me to. Also, if need the DNS Challenge I would think it would be better for your customer to create a CNAME once pointing to a DNS Server you control. Once that is setup you could automate the flow.
As it is, you will need your customer to add a TXT record for each fresh cert (a renewal) so like every 60 days or so. Sounds unwieldly.
That is a good point, although rerunning the same certbot command should just renew though no?
The problem with HTTP-01 challenge is it requires more technical knowledge from the customer point of view, but I suppose it could be an option depending on the person.
For example many customers are on wordpress, which depending on version and hosting it can be a real pain to add pages like that.
Thanks for your help on this guys, we're looking to automate this before new year as it's almost a full time job doing it the way we are, so any advice you can offer would be much appreciated.
Sorry I'm reading this on the train and missed that - the problem is applications need to sit on the root domain. for example yourdomain.com/marketing/ needs to get routed to our landing page solution, root needs to go to your own website, so a subdomain isn't really an option.
I think I've found an issue, I don't understand why the account isn't getting used on this load balancer, but exiting a shell from another server with a properly setup certbot does give the same TXT key back, so great catch there thank you!
Regarding renewal, is it the case that a new TXT key will need adding every time or can they leave the existing challenge in place and we can run shell scripts to autorenew installed certs?
Valid authorizations (and thus TXT RRs if the dns-01 challenge is used) are only valid (currently, might be WAY shorter in the future) for 30 days. And Let's Encrypt certificates are valid for 90 days with the recommendation to renew after 60 days (leaving 30 days as room for problem fixing, if necessary).
So: yes, renewals require new TXT records.
Perhaps we need to look at your setup in a more global way instead of focussing on a single issue: how are the customers servers and your servers related? Are there customer servers? Or only servers hosted by you? If there are multiple, how are they related to each other? Which server is terminating the TLS connection (i.e.: where is the certificate installed)?
And if you need to include the root domain example.com, you'd need the CNAME _acme-challenge.example.com pointing to a DNS server under your controle.
Ok here's an example, lets say you own a website called somedomain.com and you want our landing pages.
We'd want to issue a cert for somedomain.com
You're saying we get them to add a cname for _acme-challenge.somedomain.com pointing at our TXT record so that we be somedomain.com.digitalmiracle.co.uk?
I didn't realise you can redirect it via cname to our own DNS, we're on Route53 so we can easily automate challenges for renewals then!
Then you can write/run a short script to pre-generate a few hundred subdomains in acme-dns
Doing this will generate a bunch of subdomains that you can assign to customers as needed. They can then point the _acme_challenge record to that domain.
Second: you can improve on the above by the following:
You can do this on-demand or against pre-determined domains (i.e. you just want to onboard every client at once)
Doing this all together, this utility is that you can compute what a customer's acme_challenge domain will be in advance. You can have your site's control panel/docs tell them to CNAME the _acme_challenge record to point at customer1.auth.example.com. Once they do that, they can just email/call you - then you confirm DNS, ensure you have the domain registered into acme-dns, and then run the challenge.
If you don't use my script/trick to rename the uuids to domains, you can just pre-generate them and send them out to clients as needed. Using my script however, you don't need to pre-generate the domains.
Yeah it's a tough one to explain, think enterprise tech where you can multiple applications sitting on the URL, /chat is an application sitting a completely different server to the actual website. That's we're doing for SME clients with things like landing pages, eCommerce all kinds of stuff, they can add whole applications to their website with no technical requirements their side. We just route traffic through our load balancers to the correct application, one of which is their existing site.
I think you guys have nailed it though, I didn't realise you could CNAME the challenge and route53 automation for certbot is easy enough, I'm just writing some code now to test it.
I don't think the utmost details are necessary, but things like /all/kind/of/urls aren't that important. What is important is what server is hosting which hostname and for which hostname do you want a certificate.
If example.com is hosted by you and you want a certificate for that hostname, why wouldn't the http-01 challenge work?
Internal shizzle for separate /paths aren't that interesting. What is interesting is which hostname is pointing to what server et cetera.
I still don't understand why the http-01 challenge wouldn't suffice here.
It's hosted on the client existing server and our servers both of which using the same domain.
Regarding the http-01 challenge, it's not convenient in all cases for clients to add these pages because of plethora of hosting environments(managed wordpress on godaddy for example) along with half the time the guys the made these customers websites aren't available to change them.
That doesn't make sense to me. So you're telling me the hostname has multiple IP addresses associated with it? One for your clients server and one for yours?
I'm just a simple medical doctor, not an IT guy with corporate experience. For me, it all comes down to very simple things: a hostname has an A and/or AAAA resource record pointing to one or more IP addresses. Those IP addresses and only those IP addresses will answer to HTTP or HTTPS requests for that hostname.
If YOU require a certificate for a certain hostname, but you're NOT actually hosting the IP addresses associated with that hostname, why would you require a certificate to begin with?
I don't either. In cases like this, you can almost always use an "auto-ssl" feature - if you're getting traffic on port 80 and don't have a certificate for it, provision it on the fly. Several webservers and load balancers offer this.
DNS-01 is useful when you're dealing with high traffic or numerous customers and need to avoid dogpiles on certificate ordering or want to be generating backup certificates in the background.
While it wouldn't be illegal or something to charge a fee for free Let's Encrypt certs, it's certainly something that's frowned upon by many volunteers of this Community 