This may seem like somewhat of a strange question, but bear with me.
I am a learning cryptographer developing a "library" that is a port of numerous cryptographic implementations such as AES and RSA to a somewhat low-perf. embedded system. It is my plan to support signature authentication for certificates issued within this library by including the public key used for this purpose. By default I plan to provide, packaged with the library, the public key used to sign the certificates for the sites/servers I run, and simply allow end users to import their own. To that end:
What public key(s) do I need to use to support authenticating these sites' certificates, and what procedure should I use to authenticate these certificates? A simple application of RSA_Encrypt(certificate, e = 65537, pubkey)? Or something else? I've spend some time exploring how encryption/decryption work, but signature verification is something I have yet to entirely learn how to do.
Thanks in advance for any assistance.
This is not supposed to discourage you, just consider this as a fair warning:
In my university, the very first thing they tell you about cryptography is: "Unless you really, really know what you're doing, never roll your own crypto". Beginners make (significant) mistakes, this is just human nature. But in actual security critical applications, these mistakes can be risky.
That said, I'm not entirely sure what you're trying to do here: The certificates on your webserver(s) are already cryptographically signed - by Let's Encrypt. Verifying these signatures is part of what a TLS library is supposed to do. These libraries receive the certificates in the TLS handshake and verify the so-called certificate chain, up to a known root certificate. These root certificates are usually bundled either with the TLS library itself, or with the operating system.
If you want to do Public Key Pinning for some reason, then what you want is to ship the public key of your certificates with your application and verify if the certificate's public key and the shipped public keys are identical. This has lots of edges and traps though and is in pratice never that simple.
That is why I have undertaken this project with great care to "port" and not "redesign" the library. AES and SHA are both sourced from another public domain repository that is the most compatible I could find with the ez80 architecture. Where I needed to implement things myself, such as in the PRNG, I did significant research on how to do it and on the best practices with people who understand the specifics of the hardware. I have also posted the code itself to the community I work with that develops for that system, and thus far the only possible vulnerabilities we can see are side-channel-type attacks, but that's more due to the nature of the hardware rather than the library. So perhaps I should take the same care to extensively research TLS and signature verification before proceeding. Also, perhaps I should elaborate instead of cryptically saying "embedded system"--the platform in question here is a TI-84+ CE graphing calculator. The library in question is called HASHLIB.
What I am trying to do here is implement the TLS protocol such that when the embedded system (calculator) connects to the server(s) I am hosting, the server can send its SSL certificate, and the calculator can verify that the SSL certificate is authentic.
That's quite a task you've got at hand! I've actually written my own TLS implementation (client-side only) a while ago, altough just for experimental testing and not for production use.
The verification of X.509 certificates is just one of many tasks a TLS stack commonly does, altough sometimes this process is delegated to another component, e.g the OS. I'm not sure if I can recommend good literature, but RFC 5280 is always a good starting point to learn more about the internals of X.509. When working with X.509 at a low level, knowledge of ASN.1/DER encoding is also required: Let's Encrypt has written a nice introduction into this topic: A Warm Welcome to ASN.1 and DER - Let's Encrypt
Other than this, I'm not sure this community can give you what you seek. General crypto questions (not related to Let's Encrypt) are suited better for a different forum, for example the Cryptography StackExchange.
In fact, I'm pretty sure this thread is also quite out of the scope of this Community in general. While some threads about securing a server are at least a little bit related with Let's Encrypt, I'm afraid the topic of this thread has almost no relation with Let's Encrypt at all, except for the fact a Let's Encrypt certificate also uses public/private key encryption.
Fair enough. Well, thanks at least for getting me pointed in the right direction; I'll take this question to the recommended StackExchange forum. Thanks.