I will only point out a couple things. Need to be away soon
To answer a question on your git: crt.sh only shows issued production certs. You are using the staging server, which is great while testing, but even if issued they do not show in crt.sh. The test certs should appear on your machine though.
Also, I see your DNS is CNAME'd to an AWS Elastic Load Balancer. This is likely complicating things as all servers that could process an http challenge must respond identically. This usually takes some care on constructing your requests and managing certs. Using a DNS challenge might simplify the challenge part but an ELB still has challenges managing the certs.
Can you use an AWS issued cert (ACM) in the Load Balancer?
Thanks for the reply.
I tried, for the sake of it, with the letsencrypt production issuer too - no luck (so if it were, crt.sh would have shown something, I guess).
I wouldn't be surprised if the AWS LB setup (or the DNS itself) is influencing things somehow. However, I previously managed with a similar (if not same) cert-manager setup to successfully get a certificate for applebanana.stag.aws.worksome.net (don't judge, just a test service and domain name).
I'll look into ACM - and will consider DNS01 challenge instead.
But the dilemma here is still open, even if I come across some workaround.
So what is different? Probably easier for you to identify than any of us. You have more insight to your (somewhat complex) config.
Often we see trouble behind Load Balancers with people doing http challenge. When the ACME client is run from, say, Server1 it results in an http challenge request from the Let's Encrypt server to the LB. If the LB sends that request to Server1 it will work. If it gets sent to Server2 (or other) then it won't unless you have done some magic to get the http challenge files distributed to each server.
I don't know what is failing in your setup. I am just describing some basic mechanics in hopes it helps you understand what happened.