Using with Ubuntu 18.04/Jetty

Help
1

I am new to server/website configuration. I successfully installed Certbot (step 3) from the instructions generated by the choices: running "none of the above" on "Ubuntu 18.04 LTS". Step 4 also went smoothly--via SSH (putty) I was able to temporarily stop the web service, run "sudo certbot certonly --standalone" and restart the web server.

Then came across instructions for step 5:

You'll need to install your new certificate in the configuration file for your webserver.

Yikes!
I'm going through the manual provided by Jetty.org, but am a bit boggled as their instructions are strongly tied to other certifying programs. It doesn't help knowing that in some cases Ubuntu, in creating a repository-based process for loading Jetty, has configured things differently (e.g., using "serviced") than Jetty's documents, which assume one has loaded via wget and followed their configuration suggestions.

I'm wondering if anyone else has Jetty as their webserver with Certbot, and especially if loaded via the Ubuntu repository. If so, any suggestions on documentation or how to proceed?

I'm currently ping-ponging between Certbot docs an Jetty pages on SSL. With each reread, a few more terms make sense than before.

Thanks.

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: adonax.com

I ran this command: none yet

It produced this output: none yet

My web server is (include version): Jetty installed via Ubuntu repository, listed as Jetty 9, 9.4.15-1~18.04.1ubuntu1

The operating system my web server runs on is (include version): Ubuntu 18.04 LTS

My hosting provider, if applicable, is: Linode

I can login to a root shell on my machine (yes or no, or I don't know): YES

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.31.0

2 Likes
2

I don't have specifics for your exact config.
But in general:
Since you used --standalone, you will have to make manual configuration changes (every 60~90 days).
Primarily, inserting the use of the certificate files into the https block of the web server.
[not entirely sure where that is in jetty but that step will also need to be done on each renewal - but a quick google on adding a cert to jetty should do]
The certificate files needs (and their location) can be found with:
certbot certificates

If there is anyway to determine the webroot path, you should switch from standalone to webroot; as that would NOT require a webserver stop & start.

EDIT: review for cert into jetty howto How to use Letsencrypt certificate & private key with Jetty · GitHub

1 Like
3

You might find that terminating SSL at Jetty is way more trouble than it’s worth. Having to restart Jetty[1] every time your certificate is renewed or the SSL runtime used by Jetty receives a security update, is very slow and expensive because of how stateful Java applications are and how heavy the startup cost is.

I’m stuck doing operations for a few Java applications running on Wildfly and Tomcat, and without exception I stick the Java part behind nginx or haproxy - much better web servers in their own right and both of which Certbot can deal with very easily.

IMO as a rule, Java applications are a bad place to terminate SSL because all of the complete lack of graceful reload in any of the web app containers/servers.

This is a bit of a non-answer but I hope I can save you from trying.

1. You can write a class that invokves SSLContextFactory.reload, but I wouldn’t count that as first party support. See https://danielflower.github.io/2017/04/08/Lets-Encrypt-Certs-with-embedded-Jetty.html for example

3 Likes
4

The statement about stopping and restarting jetty being trouble puzzles me. So far, it’s as simple as the following two statements: sudo systemctl stop jetty9, sudo systemctl start jetty9 and each takes executes close to immediately.

Have you had problems with Jetty before, or are you extrapolating from your experiences with other Java applications, like Tomcat and Apache?. (I’m running Jetty independently of Apache. I chose Jetty specifically to avoid combos like Tomcat + Apache or Tomcat + nginx.)

5

systemctl stop/start/restart has the appearance of being instantaneous because it is forking. Your web application still takes additional time to begin servicing requests again. It’s up to you of course, but the webapp becoming inaccessible or dropping requests, even for 5-10 seconds, isn’t something I’d be happy with. Especially if the deployed web app gets killed half way through performing some kind of work.

Maybe I am mistaken and your webapp has a crazy low startup time, no state and does no asynchronous work - I just never see it :laughing:.

1 Like
6

I have to agree with @_az and add that putting a well behaved proxy in front of anything improves the situation.
That said, if you are confortable with the stop+start turnaround time you can continue with that process.
However, you will still have to deal with the PEM to PFX into KETSTORE (preferrably in an automated fashion) part of that method.
[which can be overted by the use of a well behaved inline proxy]

7

I got the impression from the certbot instructions that if it was easy to start and stop the web service, the stand-alone option was preferred. Seems like I could have/should have used the --webroot option instead? Can one redo?

Ah, yes, many hits on “adding a cert to jetty”! Thanks for that!

I’m not 100% I know what you mean by “webroot path”. With this Ubuntu/Jetty install, there is a folder called /var/lib/jetty9/webapps/root in which one would puts the site files (.war or static html). I’m thinking webroot refers to this directory. Yes?

EDIT: Also, I am going slowly here, my brain is tired, and didn’t realize there were two pending replies when I first posted this.

Yes, I understand that systemctl allows forking and that this probably explains the appearance of the command executing instantly. Prior when starting and stopping jetty, it did take several seconds, the amount depending on the applications (static html a quicker start than .war). So that is a good point. Thank you.

So changing over to --webroot instead of --standalone seems like an important step.
DONE, via certbot delete command followed by a new certbot certonly command.

8

Technically --webroot must only match the root where requests to /.well-known/acme-challenge/ are served. [acme clients are not concerned with any other part of the site]
That can, and most often is, redirected to a dedicated path specifically for such use.
In NGINX, it’s done easily with a location block containing a root statement.
In Apache, it’s usually done with an alias statement [can even be global].

9

I have changed from --stand-alone to --webroot. I think this went okay. CertBot has a delete command that allowed me a do-over.

Pointing out the command "certbot certificates" was helpful for the process of installing the key and cert for Jetty.

At this point, starting the Jetty service fails with errors in its logs file.

keystore password was incorrect

java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

I'm having trouble conceptualizing at what stage the problem arises. Could there have been a faulty aspect of how the "certbot certonly command" was executed? Or is this more likely arising from the stage when the private key and certificate are brought into Jetty? I received no errors when bringing the key and cert to Jetty.

There are so many passwords and permissions at different stages!

Here is a more complete log of the error, for what it is worth.

2019-11-05 12:20:38.768:INFO:oejs.Server:main: jetty-9.4.15.v20190215; built: unknown; git: unknown; jvm 11.0.4+11-post-Ubuntu-1ubuntu218.04.3
2019-11-05 12:20:38.847:INFO:oejdp.ScanningAppProvider:main: Deployment monitor [file:///var/lib/jetty9/webapps/] at interval 1
2019-11-05 12:20:39.452:INFO:oeja.AnnotationConfiguration:main: Scanning elapsed time=0ms
2019-11-05 12:20:39.798:INFO:oejshC.root:main: Warning: No org.apache.tomcat.JarScanner set in ServletContext. Falling back to default JarScanner implementation.
2019-11-05 12:20:40.169:INFO:oajs.TldScanner:main: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
2019-11-05 12:20:40.203:INFO:oejs.session:main: DefaultSessionIdManager workerName=node0
2019-11-05 12:20:40.203:INFO:oejs.session:main: No SessionScavenger set, using defaults
2019-11-05 12:20:40.204:INFO:oejs.session:main: node0 Scavenging every 660000ms
2019-11-05 12:20:40.310:INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@28975c28{root,/,file:///var/lib/jetty9/webapps/root/,AVAILABLE}{/root}
2019-11-05 12:20:40.351:INFO:oejs.AbstractConnector:main: Started ServerConnector@1cdade1b{HTTP/1.1,[http/1.1]}{0.0.0.0:80}
2019-11-05 12:20:40.750:WARN:oejx.XmlConfiguration:main:
java.security.PrivilegedActionException: java.io.IOException: keystore password was incorrect
        at java.base/java.security.AccessController.doPrivileged(Native Method)

... and

Caused by:
java.io.IOException: keystore password was incorrect
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2108)
  
... and

Caused by:
java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
        at java.base/sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2108)
10

You skipped some detail…

Did you get a new cert?
Were you able to create a PFX file from the cert files?
Did you use a password when creating the PFX file?
Were you able to load the PFX file into the keystore?

11

Did you get a new cert?

AFAIK yes. When running the -webroot option, after having done the -delete, I went through the following dialogue.

fgphil@adonax:/$ sudo certbot certonly --webroot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
to cancel): adonax.com
Obtaining a new certificate

IMPORTANT NOTES:
 - Congratulations! Your certificate and chain have been saved at:
   /etc/letsencrypt/live/adonax.com/fullchain.pem
   Your key file has been saved at:
   /etc/letsencrypt/live/adonax.com/privkey.pem

Were you able to create a PFX file from the cert files?

I didn't attempt to do this. The current Jetty instructions are to combine the private key and the public key into a PKCS12 file, and then to make a keystore file from this.

A password was used when creating the PKCS12 file. The password was also used when loading the PKCS12 file into the keystore.

IDK if this matters: I did not specify the exact file location of the target keystore when creating the keystore file, and it was made in my current directory. I then copied the keystore file to the location where Jetty expects it to be. Are there relative addresses or something else that might be broken by a copy/move of the file?

EDIT: just searched PFX and I see that it is another name for PKCS12!

1 Like
12

Perhaps Jetty needs it to be loaded into an already existing keystore...

13

I’ll give it a go tomorrow and report back.

14

Still not working.
I tried loading the keystore directly into its file location. Getting same error message.

java.security.PrivilegedActionException: java.security.UnrecoverableKeyException: Get Key failed: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.

Three more ideas:

  1. inspect letsencrypt log files
  2. inspect ssl.ini (jetty's configuration file for ssl operations)
  3. inspect the keystore doc using Jetty's tool

= = = =

  1. letsencrypt log, most recent entry:

    2019-11-06 05:34:33,808:DEBUG:certbot.main:certbot version: 0.31.0
    2019-11-06 05:34:33,809:DEBUG:certbot.main:Arguments: ['-q']
    2019-11-06 05:34:33,810:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
    2019-11-06 05:34:33,822:DEBUG:certbot.log:Root logging level set at 30
    2019-11-06 05:34:33,822:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
    2019-11-06 05:34:33,856:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7f9b981aae48> and installer <certbot.cli._Default object at 0x7f9b981aae48>
    2019-11-06 05:34:33,867:INFO:certbot.renewal:Cert not yet due for renewal
    2019-11-06 05:34:33,868:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
    2019-11-06 05:34:33,868:DEBUG:certbot.renewal:no renewal failures

As far as I can tell, this pertains to a daily automatic check of whether the cert requires renewal, and has nothing to do with attempts to make use of key or cert.

Is there a way to configure a log entry for any attempts to use these files? I'm guessing there isn't.

  1. jetty's ssl.ini configuration file:
> # ---------------------------------------
> # Module: ssl
> # Enables a TLS(SSL) Connector on the server.
> # This may be used for HTTPS and/or HTTP2 by enabling
> # the associated support modules.
> # ---------------------------------------
> --module=ssl
> 
> ### TLS(SSL) Connector Configuration
> 
> ## Connector host/address to bind to
> # jetty.ssl.host=0.0.0.0
> 
> ## Connector port to listen on
> # jetty.ssl.port=8443
> 
> ## Connector idle timeout in milliseconds
> # jetty.ssl.idleTimeout=30000
> 
> ## Number of acceptors (-1 picks default based on number of cores)
> # jetty.ssl.acceptors=-1
> 
> ## Number of selectors (-1 picks default based on number of cores)
> # jetty.ssl.selectors=-1
> 
> ## ServerSocketChannel backlog (0 picks platform default)
> # jetty.ssl.acceptorQueueSize=0
> 
> ## Thread priority delta to give to acceptor threads
> # jetty.ssl.acceptorPriorityDelta=0
> 
> ## Connect Timeout in milliseconds
> # jetty.ssl.connectTimeout=15000
> 
> ## Whether request host names are checked to match any SNI names
> # jetty.ssl.sniHostCheck=true
> 
> ## max age in seconds for a Strict-Transport-Security response header (default -1)
> # jetty.ssl.stsMaxAgeSeconds=31536000
> 
> ## include subdomain property in any Strict-Transport-Security header (default false)
> # jetty.ssl.stsIncludeSubdomains=true
> 
> ### SslContextFactory Configuration
> ## Note that OBF passwords are not secure, just protected from casual observation
> ## See http://www.eclipse.org/jetty/documentation/current/configuring-security-secure-passwords.html
> 
> ## The Endpoint Identification Algorithm
> ## Same as javax.net.ssl.SSLParameters#setEndpointIdentificationAlgorithm(String)
> #jetty.sslContext.endpointIdentificationAlgorithm=HTTPS
> 
> ## SSL JSSE Provider
> # jetty.sslContext.provider=
> 
> ## Keystore file path (relative to $jetty.base)
> # jetty.sslContext.keyStorePath=etc/keystore
> 
> ## Truststore file path (relative to $jetty.base)
> # jetty.sslContext.trustStorePath=etc/keystore
> 
> ## Keystore password
> # jetty.sslContext.keyStorePassword=OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4
> 
> ## Keystore type and provider
> # jetty.sslContext.keyStoreType=JKS
> # jetty.sslContext.keyStoreProvider=
> 
> ## KeyManager password
> # jetty.sslContext.keyManagerPassword=OBF:1u2u1wml1z7s1z7a1wnl1u2g
> 
> ## Truststore password
> # jetty.sslContext.trustStorePassword=OBF:1vny1zlo1x8e1vnw1vn61x8g1zlu1vn4
> 
> ## Truststore type and provider
> # jetty.sslContext.trustStoreType=JKS
> # jetty.sslContext.trustStoreProvider=
> 
> ## whether client certificate authentication is required
> # jetty.sslContext.needClientAuth=false
> 
> ## Whether client certificate authentication is desired
> # jetty.sslContext.wantClientAuth=false
> 
> ## Whether cipher order is significant (since java 8 only)
> # jetty.sslContext.useCipherSuitesOrder=true
> 
> ## To configure Includes / Excludes for Cipher Suites or Protocols see tweak-ssl.xml example at
> ## https://www.eclipse.org/jetty/documentation/current/configuring-ssl.html#configuring-sslcontextfactory-cipherSuites
> 
> ## Set the size of the SslSession cache
> # jetty.sslContext.sslSessionCacheSize=-1
> 
> ## Set the timeout (in seconds) of the SslSession cache timeout
> # jetty.sslContext.sslSessionTimeout=-1
> 
> ## Allow SSL renegotiation
> # jetty.sslContext.renegotiationAllowed=true
> # jetty.sslContext.renegotiationLimit=5

It seems interesting that there is a commented out line for the keystore password in this file. I tried replacing the "keystore" password with both a literal copy and an OBS encrypted copy of the password I used when configuring the keystore file. Neither did anything to change the error message received.

  1. Inspecting the keystore file using Jetty's tool
    Command to do so follows:
$ cd $JETTY_BASE
$ keytool -list -keystore etc/keystore -storetype jks -storepass '' -v

resulting listing (*with "..." for deletions):

fgphil@adonax:/usr/share/jetty9$ keytool -list -keystore etc/keystore -storetype jks -storepass '[PSWD DELETED FROM LISTING]' -v
Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: 1
Creation date: Nov 6, 2019
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=adonax.com
Issuer: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Serial number: 3aa...
Valid from: Thu Oct 31 16:14:59 PDT 2019 until: Wed Jan 29 15:14:59 PST 2020
Certificate fingerprints:
         SHA1: 45:72:...
         SHA256: 27:7C:18:...
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 1.3.6... Criticality=false
0000: 04 81 ... 
multiple lines deleted
9 "ObjectId" records total

Certificate[2]:
Owner: CN=Let's Encrypt Authority X3, O=Let's Encrypt, C=US
Issuer: CN=DST Root CA X3, O=Digital Signature Trust Co.
Serial number: a014 ...
Valid from: Thu Mar 17 09:40:46 PDT 2016 until: Wed Mar 17 09:40:46 PDT 2021
Certificate fingerprints:
         SHA1: E6:A3:B4:...
         SHA256: 25:84:7D:..
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3

Extensions:

#1: ObjectId: 1.3... Criticality=false
AuthorityInfoAccess [
  [
   accessMethod: ocsp
   accessLocation: URIName: http://isrg.trustid.ocsp.identrust.com
,
   accessMethod: caIssuers
   accessLocation: URIName: http://apps.identrust.com/...
]
]

7 "ObjectId" records which I deleted in this listing, in total


*******************************************
*******************************************

An important thing, perhaps, is that the password used to allow this listing works. So I wonder at what stage of decryption the jetty program is objecting.

= = = = =

IDK what to do next. Maybe delete the cert and key and get another, paying extra attention to what passwords are used when. But I don't think I did anything unusual, and there were no objections when loading the keystore.

15

Progress!

Editing the ssl.ini to include the keystore password and the directive “needClientAuth=true” has gotten me to where the service loads once again and port 80 displays correctly.

But https://mywebsite.com remains a problem, as it elicits the error “Secure connection failed…ssl peer cannot verify your certificate”.

There are probably still some steps I have overlooked/omitted, pertaining to decrypting, is my bet.

Continuing to research.

16

This error is related to client authentication, where the user's browser has to have a certificate accepted by the server—did you really mean to enable that?

17

I am still struggling with the learning curve, am still slow to connect terms with their concepts in real time!

So you are saying that I can’t browse my site via https because MY browser isn’t acceptable?

It seems to me that I’m browsing https sites all the time. Does one only get into sites securely via some sort of login process? For example, I can go to https://community.letsencrypt.org without having signed in.

There are two configuration lines that seem pertinent here (shown as their defaults when the ssl.ini file is first generated):

# jetty.sslContext.needClientAuth=false
# jetty.sslContext.wantClientAuth=false

I had changed the “need” to true and uncommented it (per blindly following instructions), and this prevented my browser from accessing the site via port 443.

Setting “need” to false, and either having “want” set to either true or false DOES allow my browser to use https. I currently have it as “want” set to “true”.

So, then, everything is actually fine and I didn’t realize it?!
(SMH)

I don’t think I need to restrict clients to only those with certificates. Even e-commerce (which I hope to eventually implement) doesn’t really require client certs, does it? I know I have used shopping carts and made payments without having a cert.

Thank you for helping me out! This has been quite a learning curve for me.

1 Like
18

Requiring Client Authentication is like putting a sign on the entrance to your web site that says: “Members ONLY” or “Private Property” or “No Public Access” or “Keep OUT” …

19

Right, currently client certs are used almost exclusively by corporate intranet applications (where the intranet administrator or another kind of IT administrator issues client certificates to all of the individual browsers that are supposed to be able to connect to the application). They're rarely used on sites intended for the general public. Most Internet users don't have any client certs in their browsers at all.

20

Thanks for the replies, explanation and support! It was very helpful as well as encouraging to have you all replying.

I don’t know how to designate a “solution” as it involved many steps. I think what I will do is try to put together some sort of step-by-step guide for Jetty on Ubuntu and post it somewhere. I do try to “pay forward” where I can, helping others.

Best wishes!

2 Likes