I ran this command:
It produced this output:
E030449 boulder-ra [AUDIT] Could not persist order error: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
The operating system my web server runs on is (include version):
Ubuntu 16.04 LTS
My hosting provider, if applicable, is:
Amazon AWS EC2
I can login to a root shell on my machine (yes or no, or I don’t know):
I am running Boulder (without Docker) on Amazon AWS EC2, with certificates being saved to AWS RDS (MariaDB).
We are signing certificates using AWS CloudHSM which has single-threaded performance of 330 op/sec.
I have been investigating why Boulder can only sign 20 certs/sec (and also return 500 errors due to timeout errors).
The machine running Boulder has 8 cores and when I run
top, I can see all the boulder components sharing the workload fairly evenly, except
boulder-sa seems overwhelmed, even at just 20 certs/sec.
I am skeptical that this is expected behaviour because the amount of data being read/written to the database is small and queries are simple and used indices.
How can I tell if
boulder-sa (or the database) are misconfigured?
Has anyone else who runs Boulder seen this kind of sluggish storage performance?
boulder-sa chewing up so much CPU time for a handful of certs/sec ?
Any help appreciated, … including tips about how to investigate further, or config settings to adjust.
Screen capture shows
260%. Timeout errors start occurring as it gets closer to
300% (which I think means it has 3 CPU cores working hard). All the other components are similar around
15%, barely breaking a sweat.)