Minimal, manual, config with Tomcat 9?

SimonHGR · March 24, 2023, 10:06pm

My domain is: dancingcloudservices.net
I ran this command: sudo certbot certonly --standalone
It produced this output: four files (and a lot more): cert.pem, chain.pem, fullchain.pem, privkey.pem (all under /etc/letsencrypt/live/dancingcloudservices.net
My web server is (include version): Tomcat 9
The operating system my web server runs on is (include version): Ubuntu 22.04.1 LTS
My hosting provider, if applicable, is: (not sure if this is applicable, I'm effectively self-hosted, just the underlying machine is "hosted") AWS
I can login to a root shell on my machine: Yes
I'm using a control panel to manage my site: No
The version of my client is: certbot 1.21.0

EDIT: I have a feeling I might have a permission problem (which would, of course, make it a 100% tomcat problem, less appropriate to have asked here). I'm going to see if I can make any progress on this line of reasoning (Of course, I still very much welcome any suggestions anyone might have).

EDIT 2: I found an entry in the catalina.out log file:

java.io.FileNotFoundException: Configured file [/etc/letsencrypt/archive/dancingcloudservices.net/cert1.pem] does not exist

However, that's not actually true (Notes: I went with the archive file in case the soft link was causing confusion and I also set the ownership to "tomcat" after first noticing this complaint and realizing that the owner was root). ls -l ... produces:

`-rw-r--r-- 1 tomcat tomcat 1866 Mar 24 21:23 /etc/letsencrypt/archive/dancingcloudservices.net/cert1.pem`

EDIT 3: The installation of tomcat is provided by AWS, and I'm very unclear about it's provenance. In particular, I think I might be missing the "Apache Portable Runtime" which seems like it's needed in order to use the connector that permits direct use of openssl and thereby these standard format key/certificate files.

I will build tomcat from source, build the APR, and see where that goes. Might take a while though...

Original follows:

I'm trying to get a trivial tomcat application to run under https. It works under http, and I configured it successfully to run on port 80 (i.e. I'm pretty certain I have edited the right config file--there are two, and one is not used). Unfortunately, I'm getting complaints from my browser to the effect of:

dancingcloudservices.net sent an invalid response.
ERR_SSL_PROTOCOL_ERROR

I admit I'm very new at this (this is a learning exercise at this stage) so I might easily have overlooked the obvious.

I ran:
sudo certbot certonly --standalone
which populated a bunch of stuff under /etc/letsencrypt...
And I configured tomcat's server.xml file with changes in these two blocks (this is the current version)

    <Connector port="80" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="443" />

and:

    <Connector port="443" protocol="org.apache.coyote.http11.Http11AprProtocol"
               maxThreads="50" SSLEnabled="true" >
        <UpgradeProtocol className="org.apache.coyote.http2.Http2Protocol" />
        <SSLHostConfig>
                <Certificate
                         certificateKeyFile="/etc/letsencrypt/live/dancingcloudservices.net/chain.pem"
                         certificateFile="/etc/letsencrypt/live/dancingcloudservices.net/cert.pem"
                         certificateChainFile="/etc/letsencrypt/live/dancingcloudservices.net/chain.pem"
                         type="RSA" />
        </SSLHostConfig>
    </Connector>

Can anyone suggest where I should start with investigating/fixing this?

MikeMcQ · March 24, 2023, 10:30pm

From an earlier thread on Tomcat I think you can use the .pem files directly contrary to the Tomcat docs. In your case I think this would work

certificateKeyFile="/etc/letsencrypt/live/dancingcloudservices.net/privkey.pem"
certificateFile="/etc/letsencrypt/live/dancingcloudservices.net/cert.pem"
certificateChainFile="/etc/letsencrypt/live/dancingcloudservices.net/chain.pem"

Also see post #2 here

SimonHGR · March 24, 2023, 10:47pm

Many thanks for the input Mike. Yes, I think I found that earlier thread which was why I was trying in the first place. I will pursue the stackoverflow link and see if that adds anything. I also think I've moved a bit forward, edits in the main question...

Much appreciate your taking time to comment, thank you

MikeMcQ · March 25, 2023, 2:38am

Don't use the .pem files in the /archive/ folder, use the ones in /live/

The /live/ are symlinks to the most recent /archive cert whose name will change each time you renew (every 60 days usually).

Also, please don't repeatedly update your first post. It makes it difficult to follow the current status. Also, it changes how the alerts work for volunteers trying to follow the thread.

SimonHGR · March 25, 2023, 3:28am

The /archive/ vs /live/ thing was an attempt to see if the soft link was somehow causing trouble. It made no difference, it doesn't work with either location.

As to updates, you prefer to read the entire chain from top to bottom then--is that what you mean? Or is there something I'm missing? I'm going to assume that's what you're asking for as I make the next updates...

SimonHGR · March 25, 2023, 3:31am

Next updates. I tried to build from source, which I thought would give me access to the the APR, but it didn't.

Then I discovered that I can install the components of APR using "apt install" so I did that. This did change things somewhat; now, instead of getting a Java error on the file, it seems fails in a new way with "error(1)"

I get the impression there's something not working in OpenSSL, or the connection to it, but after 8 hours banging my head, I have to go to be for now.

MikeMcQ · March 25, 2023, 3:42am

Mostly, the problem in your first post was you used chain.pem for the certificateKeyFile. The KeyFile should be the private key (privkey.pem)

As to the cert and intermediates, I believe you could omit ChainFile and use fullchain.pem for certificateFile. Or, use the cert.pem for File and chain.pem for ChainFile.

That's just the cert stuff. I am not able to reach your site (right now) with HTTP or HTTPS so I think you have more fundamental problems.

This test site will help verify that at least HTTP is working from the public internet

SimonHGR · March 25, 2023, 12:23pm

Ah, exellent, many thanks. I had no idea how to allocate the files, but took that assignment directly from an older question on this forum.

The site is down because it's a test site-I have no desire to leave a default install of Tomcat running with no protection at all when it's not doing anything (nor, frankly, to pay AWS 12 cents a day while it does that nothing ) But thanks for the observation.

I will build a new deployment with the file assignment you describe and check back later.

Again, much appreciated.

MikeMcQ · March 25, 2023, 12:42pm

I don't have first-hand experience with Tomcat but looking at the docs and other threads they usually have all of these in their connector for TLS. You only have one of them.

    secure="true"
    scheme="https"
    SSLEnabled="true"

The Tomcat docs say this

SSLEnabled
Use this attribute to enable SSL traffic on a connector. To turn on SSL handshake/encryption/decryption on a connector set this value to true. The default value is false. When turning this value true you will want to set the scheme and the secure attributes as well to pass the correct request.getScheme() and request.isSecure() values to the servlets See SSL Support for more information.

I've already gone beyond this forum's scope so follow up probably best done on a Tomcat forum or similar

https://tomcat.apache.org/tomcat-9.0-doc/config/http.html

SimonHGR · March 25, 2023, 12:59pm

Yes, thank you. As simple as that.

Greatly appreciate your help. And if I were able to edit my own work, I really should have seen that I had used the same file twice, which had to be a red flag. Unfortunately, blind copying without understanding is even more error prone than anything else.

Thank you again.

SimonHGR · March 25, 2023, 2:15pm

Certainly I have more to do, this was my first attempt to make something do HTTPS and now that's working I shall look at improvements and fine tuning (the webapp itself it ridiculously basic too, and could use many hours of work, but it functions for the task at hand at least). I should probably make the server force-redirect to https too (though in this situation I might simply shut off port 80 anyway and distribute the https link to those that need to use this)

Anyway, your help, and the extra guidance is much appreciated, I will be checking into the attributes you mentioned and likely more besides. In the meantime, to ensure that this can be as helpful as possible to anyone else in the future, I will add below the key takeaways in a single place.

SimonHGR · March 25, 2023, 2:29pm

Key takeaways for configuring HTTPS using pem files rather than Java keystore (this is on AWS EC2, but presumably all apt based distros too?):

You'll need the following apt packages:
tomcat9 apache2-dev libapr1-dev libaprutil1-dev

"tomcat9" is obvious, but the extras are needed to support the "Apache Portable Runtime" which is needed for the connector that can use the pem files.

At a minimum (and as noted by MikeMcQ earlier, there seems likely to be more that might be beneficial) the files can be configured in the connector block as below. The file that contains these lines is /etc/tomcat9/server.xml, which is pretty non-standard for Tomcat installations (see final note below).

I changed the port from 8443 to 443, and highlighted the key correspondences between config entries and the files created by certbot:

Also note MikeMcQ's suggestion that the chain file might not be needed if the certificateFile entry points at the full chain pem file. I haven't tried this yet but have no reason to doubt him.

(A side note is that the apt installed version of Tomcat9 seems to have a bit of a misconfiguration; if you try to shut it down, it fails/refuses stating that the conf/server.xml file isn't found. That file is indeed not in the place it's looking, it's in /etc/tomcat9/server.xml, as noted above)

MikeMcQ · March 25, 2023, 3:45pm

That won't work for Let's Encrypt cert renewals using HTTP Challenge (which you are with --standalone)

You probably want to look at using --webroot rather than --standalone now that Tomcat is running. Although, I suppose maybe you could have Tomcat listening on port 443 only and continue to use --standalone on port 80 - maybe.

SimonHGR · March 25, 2023, 8:03pm

That sounds like something I should investigate for the general case (and should surely be mentioned in this thread for posterity Mike.)

In my case, I'll be building a server from scratch every time I do this, it will run for three days, then be destroyed. I was thinking I would simply run the certbot command non interactively before even having the configuration script install tomcat.

But I will certainly be doing more discovery over time, as future requirements might well differ from this immediate one.

Thanks again!

Osiris · March 25, 2023, 8:08pm

I hope you're not getting the exact same certificate over and over again? Or if you are, you'll store the certificate on a permanent storage which doesn't get destroyed every time?

If not, that sounds like a very good way to hit rate limits. And also a waste of resources on the Let's Encrypt side. Even if the certs are free of charge for you, they aren't for Let's Encrypt. It costs lots of money to operate a CA.

SimonHGR · March 25, 2023, 8:27pm

Ah, well, yes, that was exactly what I was planning to do--probably between three and five times per year, so not much more frequently than normal refresh would be. (though I've done it, I guess three times, to this point during the learning process).

I can store it, obviously, in an S3 bucket or similar, though it'll make the configuration script yet more complex (my problem not yours, of course, but that's why I wasn't planning to do this) but given the low rate of usage, much of the time it will be useless and I'll have to get a fresh one anyway.

I certainly don't want to waste resources or behave in an ungrateful manner; how do you feel about this scenario?

SimonHGR · March 25, 2023, 8:31pm

I guess the plan would be:

Store the files
Copy them into place in the config
Then just run an update--certbot itself will work out if it needs a refresh.

I guess I'll do some reading and try to find out how this works--in particular how I avoid the certificates happening to time out in the middle of my three day operational window.

OK, I'll try to pursue that.

rg305 · March 25, 2023, 8:56pm

If your expected reuse time is larger than 60 days, it really makes little difference which way you go - I would choose the least complicated method.
[and if it exceeds 90 days, the stored cert would already be expired!]

rg305 · March 25, 2023, 8:59pm

That said...

There is a entire staging environment for that very purpose - to learn from [while NOT consuming production certs].

SimonHGR · March 25, 2023, 9:09pm

Ah yes, thanks rg305, in discovering the document on rate limits, I noticed something about staging environment. Sigh... the joys of learning things just after you needed them Hopefully this thread will help someone else in the future do better than I have.