I am marking this as a feature request since it is actually a request to add a few lines of code that would fix this issue. I realize that this is an extreme edge case that is only causing issues with very specific types of setup, but I nevertheless wanted to make you aware of the issue.
I am currently testing a setup of three BoulderCAs that are connected to the same database cluster and use the same certificate for signing.
To test this setup, I created an ansible script that requests a certificate, renews it, revokes it, and checks the OCSP response after each of those steps. After running the script multiple times, at one point I received an error when the OCSP response came back as “good” even after the certificate was revoked.
A look at certificateStatus in the database revealed that the status of the certificate was indeed set to revoked, but the ocspResponse blob still contained the “good” response. I was able to determine that this happened because the OCSP update after renewing the certificate was handled by a different server than the certificate revocation, and those servers’ clocks were slightly out of sync. This resulted in certificateRevokedDate actually being one second earlier than ocspLastUpdated from the OCSP update after the certificate renewal, which means that after the revocation, the ocspUpdater was not triggered again.
Below is what I extracted from the database logs of the three Boulder instances:
After certificate is issued (on Boulder 1), CA writes the serial number, the status good and the expiration date to the database
### INSERT INTO `boulder_sa_integration`.`certificateStatus`
### SET
### @1='ff2ff3a2a24ce296dcd0752738ad15f974bb'
### @2=0
### @3='good'
### @4='0000-00-00 00:00:00'
### @5='0000-00-00 00:00:00'
### @6=0
### @7='0000-00-00 00:00:00'
### @8=1
### @9=''
### @10='2017-08-12 15:39:00'
### @11=0
The OCSP updater gets triggered (on Boulder 2) and stores the OCSP response (containing “good”) and the ocspLastUpdated time.
### UPDATE `boulder_sa_integration`.`certificateStatus`
### WHERE
### @1='ff2ff3a2a24ce296dcd0752738ad15f974bb'
### @2=0
### @3='good'
### @4='0000-00-00 00:00:00'
### @5='0000-00-00 00:00:00'
### @6=0
### @7='0000-00-00 00:00:00'
### @8=1
### @9=''
### @10='2017-08-12 15:39:00'
### @11=0
### SET
### @1='ff2ff3a2a24ce296dcd0752738ad15f974bb'
### @2=0
### @3='good'
### @4='2017-08-08 16:38:33'
### @5='0000-00-00 00:00:00'
### @6=0
### @7='0000-00-00 00:00:00'
### @8=1
### @9='0<82>\x (…)
### @10='2017-08-12 15:39:00'
### @11=0
When certificate is revoked (on Boulder 3), status is updated to revoked and the revokedDate is set.
### UPDATE `boulder_sa_integration`.`certificateStatus`
### WHERE
### @1='ff2ff3a2a24ce296dcd0752738ad15f974bb'
### @2=0
### @3='good'
### @4='28017-08-0 16:38:33' (ocspLastUpdated)
### @5='0000-00-00 00:00:00' (revokedDate)
### @6=0
### @7='0000-00-00 00:00:00'
### @8=1
### @9='0<82>\x03\x1d\ (…)
### @10='2017-08-12 15:39:00'
### @11=0
### SET
### @1='ff2ff3a2a24ce296dcd0752738ad15f974bb'
### @2=0
### @3='revoked'
### @4='2017-08-08 16:38:33' (ocspLastUpdated)
### @5='2017-08-08 16:38:32' (revokedDate)
### @6=0
### @7='0000-00-00 00:00:00'
### @8=2
### @9='0<82>\x03\x1d\x (…)
### @10='2017-08-12 15:39:00'
### @11=0
Due to the clock on Boulder 3 lagging a little bit behind Boulder 2, ocspLastUpdated ends up being later than revokedDate, and therefore the OCSP response is never updated.
I have set up NTP functionality to sync the clocks between the servers, however, since 100% sync on miliseconds level can not be achieved, I imagine it is theoretically still possible for this to happen again.
I discovered that the ocspUpdater, when writing a new OCSP response to the database, actually checks the status of the certificate to make sure that it doesn’t for example write a good response to a certificate that has been revoked in the meantime, so I suggest adding a similar check when writing the revoked status to the database, to make sure that revokedDate is actually later than ocspLastUpdated.
Thanks, and keep up the good work!
