Implementing ARI / POST issue

WouterTinus · March 27, 2023, 8:56pm

So I've been trying to implement ARI for win-acme according to the RFC but I can't get the POST endpoint working.

The GET endpoint works fine so I'm pretty sure there is no problem with my calculation of certID, yet I always get the error "Certificate not found" on post. Reading the Boulder source, that's caused by this call failing:

metadata, err := wfe.sa.GetSerialMetadata(ctx, &sapb.Serial{Serial: serial})

Is it possible that something is not entirely deployed or implemented yet? Same thing happens both in test and prod, even on brand-new certificates.

Recent request payload on test:

{"certID":"MGgwDQYJYIZIAWUDBAIBBQAEIOpkIqHI_Ls_OhlCdhE04lJbuQeWAJOnNXYBVVd5McY8BCDSVL7W2ZsvDB2XvGCxFyqANyOEGpzMnWghk6IzVt0J4QITAPq2fJCRPuOPeIQPmgxgZpHUYw","replaced":true}

Any pointers would be appreciated

Osiris · March 27, 2023, 9:18pm

Could you perhaps provide the entire JWS object causing the error?

WouterTinus · March 27, 2023, 9:41pm

Sure. So this is a simple test case where I order a new certificate and then immediately post the update request (on the test endpoint)

{"protected":"eyJhbGciOiJFUzI1NiIsInVybCI6Imh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2dldC9kcmFmdC1pZXRmLWFjbWUtYXJpLTAwL3JlbmV3YWxJbmZvLyIsIm5vbmNlIjoiOEYwNWtQUll6T2FDYThfYS1oSHBFMGtKOWFSaWpOQ0ZnQUFNSUk0YkRwdlEyWEEiLCJraWQiOiJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvOTM4NjY2MTQifQ","payload":"eyJjZXJ0SUQiOiJNR2d3RFFZSllJWklBV1VEQkFJQkJRQUVJT3BrSXFISV9Mc19PaGxDZGhFMDRsSmJ1UWVXQUpPbk5YWUJWVmQ1TWNZOEJDRFNWTDdXMlpzdkRCMlh2R0N4RnlxQU55T0VHcHpNbldnaGs2SXpWdDBKNFFJVEFQcTY3YTRvYnR3RGpWeXdrUjdDV1VfVmNBIiwicmVwbGFjZWQiOnRydWV9","signature":"ZieLUfE3NsSo107WK0IjK2L02amcEcDYpwUyXqGFl8dBY0b4c4VxSvoavw9Y8qhxGkfunQGJBoC_gNMcZduTFw"}

Nummer378 · March 27, 2023, 10:17pm

Your ASN.1 certid object doesn't look valid to me:

The example from draft-ietf-acme-ari-01 uses this payload:

MFswCwYJYIZIAWUDBAIBBCCeWLRusNLb++vmWOkxm34qDjTMWkc3utIhOMoMwKDqbgQg2iiKWySZrD+6c88HMZ6vhIHZPamChLlzGHeZ7pTS8jYCCD6jRWhlRB8c

which decodes to:

  0  91: SEQUENCE {
  2  11:   SEQUENCE {
  4   9:     OBJECT IDENTIFIER sha-256 (2 16 840 1 101 3 4 2 1)
       :     }
 15  32:   OCTET STRING
       :     9E 58 B4 6E B0 D2 DB FB EB E6 58 E9 31 9B 7E 2A
       :     0E 34 CC 5A 47 37 BA D2 21 38 CA 0C C0 A0 EA 6E
 49  32:   OCTET STRING
       :     DA 28 8A 5B 24 99 AC 3F BA 73 CF 07 31 9E AF 84
       :     81 D9 3D A9 82 84 B9 73 18 77 99 EE 94 D2 F2 36
 83   8:   INTEGER 3E A3 45 68 65 44 1F 1C
       :   }

Your payload

MGgwDQYJYIZIAWUDBAIBBQAEIOpkIqHI/Ls/OhlCdhE04lJbuQeWAJOnNXYBVVd5McY8BCDSVL7W2ZsvDB2XvGCxFyqANyOEGpzMnWghk6IzVt0J4QITAPq2fJCRPuOPeIQPmgxgZpHUYw==

decodes to:



  0 104: SEQUENCE {
  2  13:   SEQUENCE {
  4   9:     OBJECT IDENTIFIER sha-256 (2 16 840 1 101 3 4 2 1)
 15   0:     NULL
       :     }
 17  32:   OCTET STRING
       :     EA 64 22 A1 C8 FC BB 3F 3A 19 42 76 11 34 E2 52
       :     5B B9 07 96 00 93 A7 35 76 01 55 57 79 31 C6 3C
 51  32:   OCTET STRING
       :     D2 54 BE D6 D9 9B 2F 0C 1D 97 BC 60 B1 17 2A 80
       :     37 23 84 1A 9C CC 9D 68 21 93 A2 33 56 DD 09 E1
 85  19:   INTEGER 00 FA B6 7C 90 91 3E E3 8F 78 84 0F 9A 0C 60 66 91 D4 63
       :   }

Which looks like there's an extra null byte there.

rg305 · March 28, 2023, 1:15am

83   8:   INTEGER 3E A3 45 68 65 44 1F 1C
85  19:   INTEGER 00 FA B6 7C 90 91 3E E3 8F 78 84 0F 9A 0C 60 66 91 D4 63

And are both Integer type?

WouterTinus · March 28, 2023, 5:22am

Thanks for your feedback. Based on this and the source shared by @webprofusion I found two issues with my previous code which looked promising to be the "smoking guns", but unfortunately it's still not working.

Changed certificateId.ToAsn1Object().GetEncoded(); to certificateId.ToAsn1Object().GetDerEncoded();
I now use the leaf certificate as input to construct the certificateId, rather than the issuer of the leaf (the parameter naming in the BouncyCastle library threw me off)

Yet the behaviour remains the entirely the same. I can happily GET the renewalInfo but POSTing using the same identifier immediately afterwards (the id generator function is shared between the two) leads to the "Certificate not found" error.

Now the payload is:

{"protected":"eyJhbGciOiJFUzI1NiIsInVybCI6Imh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2dldC9kcmFmdC1pZXRmLWFjbWUtYXJpLTAwL3JlbmV3YWxJbmZvLyIsIm5vbmNlIjoiQjM3QzF6TnRVWk9HYWlQOTZ6eGhad18wVU5Eel9ud0pMSXdCMlRBMldiOTlza3MiLCJraWQiOiJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvOTM4NjY2MTQifQ","payload":"eyJjZXJ0SUQiOiJNR2d3RFFZSllJWklBV1VEQkFJQkJRQUVJSEMxQ01idDVodGlOXzM4VDVvcXA5Ny1VRDR6VkxmNWpOMWlLZ2RmdWt3dkJDQTdIY0FxOXBEUUV2d3h3NjlJVVZBUHFmTmlRX09GRXZlV0t5ZlpPM2MxUmdJVEFQcURiUG9qU1JQbW5id2owX1ZlcXNWVkFnIiwicmVwbGFjZWQiOnRydWV9","signature":"oefmcP-E0rhyxgOv1bAZzXEIBTbio9MiOJ7lMImEDBAe8N5jc0RC_kfUTUHd9caZWBtEjrsFmNJVSCH7cokJ-g"}

Nummer378 · March 28, 2023, 7:48am

That's supposed to be the certs serial. I haven't looked at them in greater detail, but I guess the example uses 64 bit numbers and LE staging/prod uses 128 bit serials

Nummer378 · March 28, 2023, 4:49pm

There's actually a difference in how Boulder selects certificates on GET vs POST requests.

On GET requests, it basically boils down to:

github.com

letsencrypt/boulder/blob/235d177a20f020e3dbd016dd33ca201f4e880ead/sa/model.go#L77


      
          		args...,
          	)
          	return &model, err
          }
          
          
const certFields = "registrationID, serial, digest, der, issued, expires"
          
          
// SelectCertificate selects all fields of one certificate object identified by
          // a serial. If more than one row contains the same serial only the first is
          // returned.
          func SelectCertificate(s db.OneSelector, serial string) (core.Certificate, error) {
          	var model core.Certificate
          	err := s.SelectOne(
          		&model,
          		"SELECT "+certFields+" FROM certificates WHERE serial = ? LIMIT 1",
          		serial,
          	)
          	return model, err
          }
          
          
const precertFields = "registrationID, serial, der, issued, expires"

On POST requests, it's:

github.com

letsencrypt/boulder/blob/235d177a20f020e3dbd016dd33ca201f4e880ead/sa/precertificates.go#L38


      
          	})
          	if err != nil {
          		return nil, err
          	}
          	return &emptypb.Empty{}, nil
          }
          
          
// GetSerialMetadata returns metadata stored alongside the serial number,
          // such as the RegID whose certificate request created that serial, and when
          // the certificate with that serial will expire.
          func (ssa *SQLStorageAuthority) GetSerialMetadata(ctx context.Context, req *sapb.Serial) (*sapb.SerialMetadata, error) {
          	if req == nil || req.Serial == "" {
          		return nil, errIncompleteRequest
          	}
          
          
	if !core.ValidSerial(req.Serial) {
          		return nil, fmt.Errorf("invalid serial %q", req.Serial)
          	}
          
          
	recordedSerial := recordedSerialModel{}
          	err := ssa.dbReadOnlyMap.WithContext(ctx).SelectOne(

So there appears to be a definite difference in behaviour, though I'm not sure why either relates to your problem. Your serials look correct to me, though you might want to post the full certificate here as well so that we can cross-check. Looking up staging certs from CT is painful.

The NULL in your ASN.1 is still there by the way, ~~so that may still be the issue~~. Though the parsing code looks identical enough for both POST and GET.*

(Also: You are actually using an ECDSA account key, correct?)

*The OpenSSL OCSP tool also produces these NULL stop-byte in it's OCSP requests, so it seems valid to do that.

PPS: Your issuerKeyHash looks wrong (it doesn't match staging's R3 nor E1), but I don't think boulder cares much about these values at all.

WouterTinus · March 28, 2023, 5:44pm

Thanks for investigating, I did another run and captured the entire certificate chain as received from the staging endpoint. BouncyCastle reads the serial as 21790042814121155191743235894085329638597059.

Then I GET to https://acme-staging-v02.api.letsencrypt.org/get/draft-ietf-acme-ari-00/renewalInfo/MGgwDQYJYIZIAWUDBAIBBQAEINbwv9Ii7lJ7Rl4MK_UC8owq_M2InTeCZYXsMrnbPuPwBCDNzUtT3QwFEFhoaK3_FfGJEtnTw644tq24sfDn3seIAQITAPojLjqFA3zKfrlqpLFiaishww, which succeeds and suggests renewal between 2023-05-26 and 2023-05-28.

Then I POST to https://acme-staging-v02.api.letsencrypt.org/get/draft-ietf-acme-ari-00/renewalInfo/ with this as payload:

{"protected":"eyJhbGciOiJFUzI1NiIsInVybCI6Imh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2dldC9kcmFmdC1pZXRmLWFjbWUtYXJpLTAwL3JlbmV3YWxJbmZvLyIsIm5vbmNlIjoiQTI3MkExX1JnbHl0T25DWkhxSFpMNkRtNGs0d254aEhQdlBzSjM1eWZQckViREkiLCJraWQiOiJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvOTM4NjY2MTQifQ","payload":"eyJjZXJ0SUQiOiJNR2d3RFFZSllJWklBV1VEQkFJQkJRQUVJTmJ3djlJaTdsSjdSbDRNS19VQzhvd3FfTTJJblRlQ1pZWHNNcm5iUHVQd0JDRE56VXRUM1F3RkVGaG9hSzNfRmZHSkV0blR3NjQ0dHEyNHNmRG4zc2VJQVFJVEFQb2pManFGQTN6S2ZybHFwTEZpYWlzaHd3IiwicmVwbGFjZWQiOnRydWV9","signature":"bW1SR6uuG2dysdnLy6F_0MMim_sy4iGzDXG1dWWZUXipNcb6c7WVwztbp4oJlWs3NpX9nu17f5XB5B0_aI1zKA"}

Which boils down to:

{"certID":"MGgwDQYJYIZIAWUDBAIBBQAEINbwv9Ii7lJ7Rl4MK_UC8owq_M2InTeCZYXsMrnbPuPwBCDNzUtT3QwFEFhoaK3_FfGJEtnTw644tq24sfDn3seIAQITAPojLjqFA3zKfrlqpLFiaishww","replaced":true}

(note that the certID is identical between the POST payload and the GET url)
And the response looks like

 {
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Certificate not found",
  "status": 404
}

Here is the full certificate chain for the certificate that I got for this run, straight from Boulder:

(Also: You are actually using an ECDSA account key, correct?)

Yes, that's the default for win-acme. The account used for this run was https://acme-staging-v02.api.letsencrypt.org/acme/acct/93866614, if that helps

WouterTinus · March 28, 2023, 6:53pm

Aha. So I guess my initial implementation was correct after all.

I'm using the BouncyCastle library to compute the certID bytes, and its constructor takes three parameters: hash algorithm identifier, a certificate, and a serial number. Originally as the "certificate" I used the intermediate (i.e. R3) because that's what made more sense to me, but then after reading the source for Certify The Web I noticed that they put the leaf there, and I thought that this may have been the reason why my POST was not working.

But after reading your comment, I guess my original intuition was right after all and it's a bug in Certify The Web. This probably doesn't get us closer to a solution for the POST, but fixing this might prevent future issues when Let's Encrypt or others tighten their validation.

I have here another certificate with serial 21817192943007918687248147744066584879172227

Where I get the certID using the intermediate as "certificate" instead of the leaf:

MGgwDQYJYIZIAWUDBAIBBQAEIOpkIqHI_Ls_OhlCdhE04lJbuQeWAJOnNXYBVVd5McY8BCDSVL7W2ZsvDB2XvGCxFyqANyOEGpzMnWghk6IzVt0J4QITAPpy97ckKyhiDoYs0WtPaWIKgw

And all other symptoms are the same.

webprofusion · March 29, 2023, 6:31am

Yeah we don't do the POST to update ARI yet in Certify The Web , so I haven't tried it. I'll have a look at the certbot code because I assume that works.

@aarongable is the expert here I think!

webprofusion · March 29, 2023, 8:33am

Just as a follow up, posting with the same CertID for get fails for me too, which seems like a Boulder bug.

I'm not seeing an ARI implementation in certbot to prove that it works.

@mholt did you get your version working with POST or have you not tried that yet?

mholt · March 29, 2023, 6:13pm

I also get HTTP 404 urn:ietf:params:acme:error:malformed - Certificate not found with POST to the renewalInfo endpoint.

GET (with the CertID sequence appended to the renewalInfo URI) works for the same cert.

Can it only be found if it's within the renewal window?

I was hoping to test with Pebble but it looks like it hasn't been implemented there yet, either.

WouterTinus · March 29, 2023, 6:30pm

Reading the source from boulder/wfe.go at main · letsencrypt/boulder · GitHub I don't think it's anything sophisticated like that. In fact it still very much a work in progress, seeing this comment:

// TODO(#6732): Write the replaced status to persistent storage.

Should we raise an issue in Github?

EDIT: commented at ARI: store whether a certificate has been marked as replaced · Issue #6732 · letsencrypt/boulder · GitHub

mholt · March 29, 2023, 6:36pm

There is already this:

github.com/letsencrypt/boulder

ARI: store whether a certificate has been marked as replaced

opened 11:15PM - 08 Mar 23 UTC

aarongable

The [current ARI draft spec](https://www.ietf.org/archive/id/draft-ietf-acme-ari…-01.html) includes a POST endpoint that allows the client to inform the server that a certificate has been replaced. This allows the server to cease considering the certificate when computing renewal windows for other certificates, and to revoke the certificate if it is part of an ongoing incident. We should add a storage layer and implement logic so make this endpoint meaningful to our operations.

But I think a 404 is a separate bug. Opened an issue:

github.com/letsencrypt/boulder

ARI: POST returns 404 where GET succeeds

opened 06:36PM - 29 Mar 23 UTC

mholt

**Summary:** See this forum thread for context: https://community.letsencr…ypt.org/t/implementing-ari-post-issue/195334/9?u=mholt Basically, the `GET` endpoint for ARI works, but `POST` does not, always returning 404 even when the GET succeeded. **Steps to reproduce:** Make `POST` request as described by https://datatracker.ietf.org/doc/draft-ietf-acme-ari/ for a valid certificate. (Verify it is the correct CertID by making the associated GET request successfully.) **Expected result:** 200 OK **Actual result:** 404 **Additional details:** 3 independent developers so far can confirm this is not working. We might all be reading it wrong, so let us know if there's something we need to do differently. Here's my attempt in Go: ```go func (c *Client) UpdateRenewalStatus(ctx context.Context, account Account, b64CertIDSeq string) error { if err := c.provision(ctx); err != nil { return err } payload := struct { CertID string `json:"certID"` Replaced bool `json:"replaced"` }{ CertID: b64CertIDSeq, Replaced: true, } resp, err := c.httpPostJWS(ctx, account.PrivateKey, account.Location, c.dir.RenewalInfo, payload, nil) if err != nil { return err } if resp.StatusCode != http.StatusOK { return fmt.Errorf("updating renewal status: HTTP %d", resp.StatusCode) } return nil } ``` Most of the rest of the code can be found at https://github.com/mholt/acmez Note that this code, using the same b64CertIDSeq input value, works: ```go func (c *Client) GetRenewalInfo(ctx context.Context, b64CertIDSeq string) (RenewalInfo, error) { if err := c.provision(ctx); err != nil { return RenewalInfo{}, err } endpoint := c.dir.RenewalInfo + b64CertIDSeq var ari RenewalInfo _, err := c.httpReq(ctx, http.MethodGet, endpoint, nil, &ari) if err != nil { return RenewalInfo{}, err } return ari, nil } ```

aarongable · March 30, 2023, 7:36pm

I've landed a change which should fix this. Thanks to all of you for helping report and investigate this! Assuming nothing untoward happens, expect the fix to be deployed a week from today.

mholt · March 30, 2023, 7:39pm

Awesome! Thanks!

mcpherrinm · April 4, 2023, 3:24pm

The fix is being deployed to staging now.

webprofusion · April 5, 2023, 3:23am

Thanks! Can confirm that posting the update to the endpoint works for me now (200 OK) using same CertID used for the query. I don't notice any other change tothe renewal info after posting but I imagine it remains static when checking the same certId, as a renewal will have a new certId. I noticed the endpoint used for the post is currently confusingly prefixed /get/ but I'm sure that's temporary anyway.

mcpherrinm · April 7, 2023, 7:15pm

The fix was deployed to production yesterday