It is looking like ECDSA cert may be an issue. But, you got an RSA cert earlier today, and an RSA cert yesterday, and several more in days just prior to that. All the R10 and R11 certs are RSA. The older ones with R3 could be either but probably were RSA given this pattern.
So, a little puzzled why the only one that works is nearly expired.
We tested on another server with another domain and certificate.
This server has the same configuration:
Apache Tomcat/8.5.34
Ubuntu 22.04.3 LTS
Certboot v 3.1.0
Domain: staging.aulanexo.com
And the same thing happens when I renew the certificate. The renewal is successful but Tomcat gives an error:
org.apache.catalina.LifecycleException: Failed to initialize component [Connector[HTTP/1.1-443]]
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:112)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:552)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:875)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
at org.apache.catalina.startup.Catalina.load(Catalina.java:632)
at org.apache.catalina.startup.Catalina.load(Catalina.java:655)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
Caused by: org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:995)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:107)
...12 more
Caused by: java.lang.IllegalArgumentException: Cannot store non-PrivateKeys
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:115)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:86)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:244)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1087)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.init(AbstractJsseEndpoint.java:265)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:581)
at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:68)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:993)
... 13 more
Caused by: java.security.KeyStoreException: Cannot store non-PrivateKeys
at sun.security.provider.JavaKeyStore.engineSetKeyEntry(JavaKeyStore.java:258)
at sun.security.provider.JavaKeyStore$JKS.engineSetKeyEntry(JavaKeyStore.java:56)
at sun.security.provider.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:117)
at sun.security.provider.JavaKeyStore$DualFormatJKS.engineSetKeyEntry(JavaKeyStore.java:70)
at java.security.KeyStore.setKeyEntry(KeyStore.java:1140)
at org.apache.tomcat.util.net.jsse.JSSEUtil.getKeyManagers(JSSEUtil.java:225)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:113)
...20 more
Both servers have many automatic certificate renewals without problems. It's something that started happening now.
I don't know either. The ECDSA is probably Not related. This latest domain has been using ECDSA certs since 2024 June. (The E5 / E6 are ECDSA intermediates).
So, do you get that error when you have TomCat pointed directly at the .pem files in ../live/ folder?
Is there any other messages shown by TomCat about why it does not like the .pem files? I have only seen messages related to the KeyStore. Pretty sure that's related to older method using pfx
Can you show the TomCat config related to the cert? Might look something like
It should have changed. At least it should if you use the Certbot files directly. As Certbot places the latest cert in /archive/ with the next number in sequence. Then the /live/ symlink is changed to point to this latest cert set. You normally don't point directly at the specific iteration to avoid needing to change it each time.
I don't know what that "protocol" means either. I think I'm too far outside my scope of expertise. I'll leave it to others. Or, try a TomCat forum.
I realize you say nothing changed but clearly something has changed. You don't seem to have any trouble getting certs. For some reason your configuration is now complaining about something. Maybe a TomCat forum will provide more info to work with. Best I can offer. Sorry
Maybe ask a Tomcat specific support channel, as the error message isn't helping me at all either. I have NO clue what so ever what Tomcat is complaining about exactly.
The only thing I can think of is the earlier mentioned RSA vs ECDSA, but from the posts above it doesn't seem like that is it.. So I'm at a loss regarding the certificate
Maybe Tomcat doesn't like the unintended regression in Certbot 3.1.0? Perhaps you can downgrade to Certbot 3.0.1? See snap list certbot --all to see if you have 3.0.1 still laying around on your server and use sudo snap revert certbot --revision XXXXXX to revert to 3.0.1 if possible. (Where XXXXXX is of course the revision number from the list from the earlier command.)
Still the above wouldn't count as an issue due to a change from Let's Encrypt
Great catch! Looks like a likely explanation for the errors in post #1
And, I guess that means TomCat must take the .pem files and convert them for some KeyStore itself? Or maybe just a holdover message from before they supported reading .pem files directly?
Also sounds like a work-around is reverting to Certbot prior to 3.1. Although hopefully the Certbot team push that fix soon.
I reached the renewal limit for that domain. I have tried changing the domain and it seems that the work-around certboot regression works. A new certificate was obtained and Tomcat did not give an error.
Tomorrow the possibility of requesting a new certificate with the initial domain that generated the incident will be renewed. I will test it and confirm if it solves the problem.
Well, depends if one expects the software to accept PKCS8 and PKCS1
By the way, I expect you to be able to convert the PKCS1 privkey into PKCS8, so you can use one of the recently issued certs. Shouldn't be too hard with some OpenSSL commands.