Receiving Expiration Emails for Dozens of Domains

Good point about hosting but couldn't it be "some hosting provider"? Like with an accidental typo of email address?

3 Likes

I've used this email for Let's Encrypt with Digital Ocean and Cloudron configs. It all started October 2nd, 2021. The email address in question colson at nabx dot com Very strange!!

1 Like

The bottom hostnames are hosted by DigitalOcean and (most of) the top domains are non-existent any longer it seems. (Makes sense that the certificates would be expiring for those hostnames.)

Possibly a DigitalOcean issue?

4 Likes

I also saw Cloudflare and googleusercontent among the first few so not exclusively DigitalOcean

4 Likes

@lestaff Any suggestions here? No obvious pattern. Started about a week or so after the fixes to the expiry emailing problems in late Sept. This not apparently related to those issues either.

We are stumped :slight_smile: Thanks

3 Likes

Well, I've seen similar reports here in the past. I think usually it's because there's a large-scale integrator that doesn't really understand what the ACME account email address is used for and they end up having many customers' domains under one account, which happens to have one of the customer's emails (like, whichever customer happened to have the first cert issued or the most recent cert issued or something like that). The integration guide tries to be clear about who the subscriber is supposed to be, but I think that not everybody who needs to do so actually reads it or understands it.

@cdolson, have you always configured your own certificates for your own VMs with something like certbot, or have you ever had a hosting provider just creating the certificate for you using their own system?

Unless this is related somehow to the problems they had with expiration notices a few months ago. But I tend to doubt it.

For reference, some past threads with some similar issues:

5 Likes

So basically, it seems, new certificates are being issued under the first found (or created) LE account.
Unfortunately for you (probably for being the first in that line) you are now receiving all this noise.

2 Likes

Yeah, it stinks. Is there any way to track the guilty party :slight_smile:

Could LE start putting the requester IP in the expiry email to help track down such cases? At least it might be a clue. As @petercooperjr shows this is not the first one.

3 Likes

In this case, the IP would likely be that very same shared host.
The "culprit" is the ACME client (OR cPanel implementation).

3 Likes

yeah, but which one exactly? Would be helpful to know - wouldn't it?

3 Likes

We're looking into it! But no, we will not include the requester's IP address in these emails.

6 Likes

@cdolson We've confirmed that there is an Subscriber account with your email address set in the contact field which has issued approximately 500 certificates for a wide variety of identifiers (domain names), starting in mid-August of this year.

I personally suggest using the "unsubscribe" link to suppress these emails. Unfortunately sometimes other subscribers make typos.

3 Likes

Another interesting thing: Cloudron appears to have its own user-agent, acme-cloudron. I haven't seen that one before. That suggests they may have their own ACME client, which could be doing surprising things.

Could you post in the Cloudron support forums (https://forum.cloudron.io/) describing your problem and see if they can shed any light on it?

When you use Cloudron, is it on a VPS fully dedicated to you, or are you using a VPS that is shared with other people?

4 Likes

Based on the ~500 certificates... my money is on a very very shared system.

4 Likes

An update: I checked our API logs, and there are two accounts using the email address you shared. One of them has a lot of different issuances, and also has a lot of requests to update the account with a new email address. All those email addresses seem unrelated. Also, that one account has requests from a variety of IP addresses.

I suspect the Cloudron images may have a bug where either they commonly generate the same account key pair (which could result in unrelated instances acting as the same account), or where they ship with an already registered ACME account, which winds up shared by many users. I started a Cloudron instance of my own, but wasn't able to reproduce. But it's possible that only some VMs have this issue, or that an older VM image had the problem and it has since been fixed.

@cdolson, can you please report the issue to Cloudron support? If that turns out to be the case, Cloudron should remediate by deactivating the account.

7 Likes

Thanks @aarongable, done! Unfortunately that means I won't get notifications for the few certificates I generated with that email address. No matter, I'll figure it out.

@jsha, I've just started a thread in their support forum linking here with a brief description and a screenshot of your last post.

Also, regarding your VPS question, I primarily had spun up dedicated Cloudron instances on DigitalOcean and had not shared those installs with any other users.

4 Likes

I am one of the co-founders of Cloudron. Thanks for reporting this issue @cdolson .

Just looking into the issue, I think the issue is that the DigitalOcean 1-click image on Cloudron has an issue that it is reusing the acme account key. Looks like this regression was introduced in our previous release which indeed happened a couple of months ago.

I am looking into a fix and will report back here. Sorry for the spam @cdolson !

8 Likes

@jsha thanks for investigating.

An update on what I found: Every Cloudron install gets it's own account key. They all start with the same email id (webmaster@cloudron.io) but it then gets later updated to the email of the cloudron admin. The issue only happens in the DO 1-click image because we have mistakenly taken the VM "snapshot" along with an account key.

We pushed a new image 2 weeks ago, so I think @cdolson should already not receive any more mails. IIUC, @cdolson was probably the last person who used the last version of the 1-click image and this is why he gets all the emails.

9 Likes

@girish: So, I think that one would consider the account key as being compromised (as many of your customers have access to it), so as @jsha said you should deactivate the ACME account. (Using certbot it's the unregister command, for example, but it looks like you're using your own client.)

Are these clients with a shared account key still likely running out "in the wild"? If so, I hope that they're configured that if the account key they're trying to use gets deactivated that they'll try to create a new one. I think I've read somewhere that that was the recommended approach for people developing their own ACME clients, though I'm not sure where (as I don't see it in the Integration Guide).

7 Likes

@petercooperjr yes, the account key in the wild is already deactivated now (thanks to @JamesLE). I think there is another key in the wild but it's part of our old DO 1-click image. I am in touch with the DO team to get access to the older image, so I can revoke it as well (since they only make the latest image available even to the publisher).

We are pushing out an update later today that will automatically generate new acme account keys for cloudrons using the compromised keys.

6 Likes