Deploying Boulder in Production

c-eshalev · July 11, 2018, 3:04pm

Hello,
We are planning on using boulder to run our own CA server.
I have successfully followed your guide on: https://github.com/letsencrypt/boulder/blob/master/README.md to start my boulder server on localhost.

However, now I want to deploy a production server (to be used without localhost). I am trying to follow the same guide but I have encountered several knowledge gaps.
The main one being: I would like to use certbot against this instance from another server. However, by default boulder’s own https cerificate is not validated. I would be happy to set boulder up to get a certificate from letsencrypt, but am not sure how I would go about doing that.

Here is a sample curl output describing the error. (I get a similar error from certbot, but curl is more readable):

curl https://my_public_boulder_instance.com:4430
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html

curl performs SSL certificate verification by default, using a "bundle"
 of Certificate Authority (CA) public keys (CA certs). If the default
 bundle file isn't adequate, you can specify an alternate file
 using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
 the bundle, the certificate verification probably failed due to a
 problem with the certificate (it might be expired, or the name might
 not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
 the -k (or --insecure) option.

When I use -k the server answers just fine. I assume it’s just a matter of having boulder use the right HTTPS certificate to get this working, but I still haven’t figured out how to do that. Since this is a public server, I would gladly sign this with certbot + letsencrypt.org

Also, would be happy to know if there is “production guide” I could follow instead of the “developer’s guide”.

JuergenAuer · July 11, 2018, 3:34pm

Hi @c-eshalev

calling https://acme-v02.api.letsencrypt.org/ - the api-endpoint: This is a normal website.

Looking the headers:

Server: nginx

Looks like Boulder uses a nginx - frontend as webserver with the regular nginx - SSL-Support.

Osiris · July 11, 2018, 4:29pm

I guess just like any other Let's Encrypt certificate: choose a client, get yourself a certificate and somehow install it into Boulder

schoen · July 11, 2018, 4:31pm

Boulder doesn't have ACME client functionality. The public Boulder instance that operates the Let's Encrypt service is behind infrastructure where HTTPS was already set up elsewhere. So if you want to have your own Boulder instance with a publicly-trusted certificate, you have to get that certificate separately.

In this case you may want to run a reverse proxy like a nginx proxy that has the certificate and forwards requests to your own Boulder instance on localhost.

Osiris · July 11, 2018, 5:02pm

Let's Encrypts Boulder sits behind a CDN, so the webserver reported in the headers doesn't really have to mean Boulder also uses nginx internally.

jared.m · July 11, 2018, 5:07pm

Let's Encrypt's Boulder server sits behind a CDN. Boulder itself does not have to.

Osiris · July 11, 2018, 5:19pm

Very true obviously, my bad, I've edited my post to be more clear.

c-eshalev · July 12, 2018, 9:03am

I’ve added an nginx frontend on myboulderserver.com, but this is not enough…
I tried forwarding ports 4000 and 4001 through port 443
CURL now works with HTTPS.

But now requesting a certificate from myboulderserver.com with cerbot running on mywebapp.com does not work:
2018-07-12 08:48:40,386:DEBUG:certbot.main:certbot version: 0.25.0
2018-07-12 08:48:40,386:DEBUG:certbot.main:Arguments: [’–server’, ‘https://myboulderserver.com’, ‘–agree-tos’, ‘–nginx’, ‘–no-eff-email’, ‘–no-redirect’, ‘–email’, ‘eshalev@cisco.com’, ‘-d’, ‘https://myboulderserver.com’]
2018-07-12 08:48:40,387:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-07-12 08:48:40,395:DEBUG:certbot.log:Root logging level set at 20
2018-07-12 08:48:40,395:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-07-12 08:48:40,396:DEBUG:certbot.plugins.selection:Requested authenticator nginx and installer nginx
2018-07-12 08:48:40,501:DEBUG:certbot.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin - Alpha
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator
Initialized: <certbot_nginx.configurator.NginxConfigurator object at 0x7f63f72c9710>
Prep: True
2018-07-12 08:48:40,502:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_nginx.configurator.NginxConfigurator object at 0x7f63f72c9710> and installer <certbot_nginx.configurator.NginxConfigurator object at 0x7f63f72c9710>
2018-07-12 08:48:40,503:INFO:certbot.plugins.selection:Plugins selected: Authenticator nginx, Installer nginx
2018-07-12 08:48:40,601:DEBUG:acme.client:Sending GET request to https://myboulderserver.com.
2018-07-12 08:48:40,604:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): myboulderserver.com
2018-07-12 08:48:40,622:DEBUG:urllib3.connectionpool:https://myboulderserver.com “GET / HTTP/1.1” 200 None
2018-07-12 08:48:40,623:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx/1.10.3 (Ubuntu)
Date: Thu, 12 Jul 2018 08:48:40 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Content-Encoding: gzip

<html>
		<body>
			This is an <a href="https://github.com/ietf-wg-acme/acme/">ACME</a>
			Certificate Authority running <a href="https://github.com/letsencrypt/boulder">Boulder</a>.
			JSON directory is available at <a href="/directory">/directory</a>.
		</body>
	</html>
	
2018-07-12 08:48:40,623:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.25.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1323, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1078, in run
    le_client = _init_le_client(config, authenticator, installer)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 642, in _init_le_client
    acc, acme = _determine_account(config)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 521, in _determine_account
    config, account_storage, tos_cb=_tos_cb)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 172, in register
    acme = acme_from_config_key(config, key)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 50, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 721, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1054, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 971, in _check_response
    'Unexpected response Content-Type: {0}'.format(response_ct))
acme.errors.ClientError: Unexpected response Content-Type: text/html
2018-07-12 08:48:40,624:ERROR:certbot.log:An unexpected error occurred:

Finally:
I can clearly see that there is an existing HTTPS interface on ports 44301 and 44300. This is serving non-authenticated https certificates. Wouldn’t it be better to just get this to work somehow? Is there another dockerfile which uses them by default?

When I look at what I get when I just add query my nginx frontent I see that it is serving dev-configuration json that is comming from inside the docker images (not served from my source). See all the references for localhost. This has me questioning if I should be using the provided dockerfiles at all…

curl https://myboulderserver.com/directory

{
  "key-change": "http://localhost:4000/acme/key-change",
  "meta": {
    "caaIdentities": [
      "happy-hacker-ca.invalid"
    ],
    "terms-of-service": "http://boulder:4000/terms/v1",
    "website": "https://github.com/letsencrypt/boulder"
  },
  "new-authz": "http://localhost:4000/acme/new-authz",
  "new-cert": "http://localhost:4000/acme/new-cert",
  "new-reg": "http://localhost:4000/acme/new-reg",
  "revoke-cert": "http://localhost:4000/acme/revoke-cert",
  "vBALMlU07hw": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"

JuergenAuer · July 12, 2018, 9:09am

Looks like your nginx - Server sends the wrong Content-Type. Must be

application/jose+json

So your CertBot doesn't want to parse the result.

c-eshalev · July 12, 2018, 9:11am

But the response is not json.
The /directory link is json
but the myserver.com/ link is text/html

JuergenAuer · July 12, 2018, 9:15am

myserver.com isn't used, only informational. But /directory is used

Your client sends a request, check_response finds the wrong Content-Type.

File "/usr/lib/python3/dist-packages/acme/client.py", line 1054, in get
self._send_request('GET', url, **kwargs), content_type=content_type)
File "/usr/lib/python3/dist-packages/acme/client.py", line 971, in _check_response
'Unexpected response Content-Type: {0}'.format(response_ct))
acme.errors.ClientError: Unexpected response Content-Type: text/html

PS: Is there a website to test?

c-eshalev · July 12, 2018, 9:35am

I don't understand what you are asking here.

One thing I can say is that the exact same certbot command on my mywebapp.com works with the default letsencrypt.org.
It fails only when I run it with: --server https://myboulderserver.com

Aside from this the encoding headers for the /directory look correct (see below). I don't think NGINX is meddling with anything here. You will also note that the error message I posted looked like it was relating to the site root, as that was the text that apears in log right before the exception.

But putting all of this aside, the fact that the directory is serving links "localhost" instead of myboulderserver.com makes me think that approach of using this dockerfile might be wrong. (this json looks like it's baked into the image)

$ curl -v https://myboulderserver.com/directory

ALPN, offering http/1.1
SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
server certificate verification OK
server certificate status verification SKIPPED
common name: CENSORED
server certificate expiration date OK
server certificate activation date OK
certificate public key: RSA
certificate version: #3
subject: CN=boulder.ctao6.net
start date: Thu, 12 Jul 2018 07:16:14 GMT
expire date: Wed, 10 Oct 2018 07:16:14 GMT
issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
compression: NULL
ALPN, server accepted to use http/1.1

GET /directory HTTP/1.1
Host: CENSORED
User-Agent: curl/7.47.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx/1.10.3 (Ubuntu)
< Date: Thu, 12 Jul 2018 09:27:43 GMT
< Content-Type: application/json
< Content-Length: 569
< Connection: keep-alive
< Cache-Control: public, max-age=0, no-cache
< Replay-Nonce: 272zwtipRlbQMUvabUsfJTkZQ0CIxx8ACCJ_ZPnax7s
<
{
"a_OR_XmKTcQ": "Adding random entries to the directory",
"key-change": "http://localhost:4000/acme/key-change",
"meta": {
"caaIdentities": [
"happy-hacker-ca.invalid"
],
"terms-of-service": "http://boulder:4000/terms/v1",
"website": "GitHub - letsencrypt/boulder: An ACME-based certificate authority, written in Go."
},
"new-authz": "http://localhost:4000/acme/new-authz",
"new-cert": "http://localhost:4000/acme/new-cert",
"new-reg": "http://localhost:4000/acme/new-reg",
"revoke-cert": "http://localhost:4000/acme/revoke-cert"

Connection #0 to host mybouldersever.com left intact

JuergenAuer · July 12, 2018, 9:57am

Now it looks ok - https://boulder.ctao6.net/ has a new Letsencrypt-certificate.

But your directory entries may be wrong - http, localhost and a special port. Should work with https://boulder.ctao6.net/ instead.

c-eshalev · July 12, 2018, 11:03am

Agreed…
But why am I getting the text/html error?

_az · July 12, 2018, 11:13am

Hey,

You need to point Certbot at your directory, not at the home page of Boulder .

For example,

certbot register --server https://boulder.ctao6.net/directory --register-unsafely-without-email

Now, your other problem, as already pointed out, is that the absolute URLs generated by Boulder point to localhost:4000.

To solve this, in your nginx proxy_pass configuration, you need to pass two headers:

Host: boulder.ctao6.net
X-Forwarded-Proto: https

I’m quite sure that the Docker environments included with Boulder are not intended for production use. It’s been raised before, but there really isn’t any public “production ops manual” available for Boulder.

c-eshalev · July 12, 2018, 12:14pm

Thanks!
That gets me to the next stage.
Now it claiming that my domain is not registered...
(I removed the real domain name fro the printout because I'm not sure I want it on a public forum)
I can include it if it is important.

When I run certbot I get the error below:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Performing the following challenges:
tls-sni-01 challenge for censored_server_name.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. censored_server_name.com (tls-sni-01): urn:acme:error:unknownHost :: The server could not resolve a domain name :: No valid IP addresses found for censored_server_name.com

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: censored_server_name.com
Type: unknownHost
Detail: No valid IP addresses found for censored_server_name.com

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.

All of my servers and containers can resolve this address.
However, when I login to the boulder docker container, it is not able to resolve any DNS (not even google.com)

docker exec -i -t boulder_boulder_run_1 curl www.google.com
curl: (6) Could not resolve host: www.google.com

Is this somehow related to the FAKE_DNS parameter mentioned in GitHub - letsencrypt/boulder: An ACME-based certificate authority, written in Go.? I initialized this to boulder.ctao6.net, but that was probably a bad choice.

_az · July 12, 2018, 12:18pm

So I think that the Docker environment for Boulder does not provide real DNS resolvers, and instead points to something called challtestsrv (which does not do real DNS).

In production, Let’s Encrypt runs https://www.unbound.net/ for its resolvers.

I think you need to setup some resolvers and then configure the validation authority (test/config/va.json) to use them:

    "dnsResolvers": [
      "127.0.0.1:8053",
      "127.0.0.1:8054"
    ],

You will need to run your own recursive nameservers (rather than using 1.1.1.1 or 8.8.8.8) because you need to make queries without any caching happening whatsoever.

You can configure Unbound with https://github.com/jsha/unboundtest/blob/master/unbound.conf as a template that vaguely mimics what they use in production.

I really have no idea what FAKE_DNS is about - it appears to still involve challtestsrv. Maybe @cpu can comment … or provide better advice on how to approach setting Boulder up for real use.

**Dlsclaimer: this is all gleaned info, I’m not sure any of it is current or accurate.

c-eshalev · July 12, 2018, 2:20pm

If I had access to the dockerfile I could try to reverse engineer and debug it.
However the docker-compose.yml only downloads a binary docker image: letsencrypt/boulder-tools-go${TRAVIS_GO_VERSION:-1.10.2}:2018-06-12
Any idea where I can see the dockerfile for that?
The answer to my dns problems might be in there…

Patches · July 12, 2018, 2:58pm

jsha · July 13, 2018, 9:56pm

I can confirm what @_az says is correct. You’ll need to change the dnsResolvers settings to point at a DNS server that can resolve the hostnames you want to issue for. If those hostnames are only resolvable from inside your internal network, you’ll need to point at your internal nameserver.

Also, just to make sure: You understand that certificates from your own Boulder instance will not be trusted by most browsers, right? Only certificates from a publicly trusted CA will be trusted by default in browsers.