Wildcard certificate issuer problem

yann-l · 2018-04-11 12:44:38 UTC

I run letsencrypt certificates for years, and they work perfectly. Today I try to create a wildcard certificate. It works. I did it this way:
certbot certonly --manual -d *.domain.org -d domain.org --preferred-challenges dns-01 --server https://acme-staging-v02.api.letsencrypt.org/directory

I get a wildcard certificate, but the issuer is Issuer: CN=Fake LE Root X1, which is not recognized by firefox.
Previous certificates had another issuer: Issuer: CN=Let’s Encrypt Authority X3, O=Let’s Encrypt, C=US

Maybe I did something wrong?

stevenzhu · 2018-04-11 12:47:06 UTC

Hi,

According to your command, you went to the staging server instead of the production one.
certbot certonly --manual -d *.domain.org -d domain.org --preferred-challenges dns-01 --server https://acme-v02.api.letsencrypt.org/directory

Thank you

jared.m · 2018-04-11 15:33:34 UTC

Yup, as @stevenzhu mentioned, this is pointing to the staging server. Did you follow a tutorial that told you to use this command?

yann-l · 2018-04-11 17:04:15 UTC

Arg yes, that was obvious … I followed a howto and blindly copy pasted the server URL. Thanks for the hint, it works now. I just need to find a way to automat that.

I host my DNS on the same machine using bind, Is there a dns plugin I can use without external service? Or maybe write pre/post script to add the TXT record?

schoen · 2018-04-11 17:06:27 UTC

That’s quite possible. That script is then called a “manual-auth-hook”.

yann-l · 2018-04-11 17:13:34 UTC

Perfect, that’s exactly what I need. Thanks.

yann-l · 2018-04-11 22:39:36 UTC

I started playing with those script, but I have problems. My command is:
certbot certonly --manual -d *.domain.org -d domain.org --preferred-challenges dns-01 --server https://acme-v02.api.letsencrypt.org/directory --manual-auth-hook /path/to/certbot_auth_hook.sh

My script is called and correctly adds the correct line in the bind zone file.
First problem: it is executed only once (but there are 2 domains, so it should be called twice to add the 2 lines )
Second problem: after script is run nothing happens, and certbot seems stuck
Third problem: I still have a question asking me if I want my IP to be logged (First question will probably not appear when I’ll run the script only once every 2 months) => --manual-public-ip-logging-ok Fixes this question

# certbot certonly --manual -d *.domain.org -d domain.org --preferred-challenges dns-01 --server https://acme-staging-v02.api.letsencrypt.org/directory --manual-auth-hook /path/to/certbot_auth_hook.sh 
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/domain.org.conf)

What would you like to do?
-------------------------------------------------------------------------------
1: Keep the existing certificate for now
2: Renew & replace the cert (limit ~5 per 7 days)
-------------------------------------------------------------------------------
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for domain.org
dns-01 challenge for domain.org

-------------------------------------------------------------------------------
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
-------------------------------------------------------------------------------
(Y)es/(N)o: Y

[STUCK here forever, nothing in log file]

schoen · 2018-04-11 22:52:32 UTC

It should be called twice. Are you sure it’s not just replacing the previous value when it’s called the second time? Maybe you could make your script also run something like this to be sure?

echo "$0 $@" >> /tmp/$$
env >> /tmp/$$

You can address these questions with --force-renewal and --manual-public-ip-logging-ok (although if you just run certbot renew then you can run it much more frequently and it will only renew when necessary).

yann-l · 2018-04-11 22:56:47 UTC

Yes I’m sure. I already added echo in /tmp and it is called only once.
Maybe it’s because of point 2? After it is called once, nothing happen. certbot is stuck. I know my script is over because I added a echo at the end of the script and it’s correctly called.

schoen · 2018-04-11 22:57:51 UTC

OK, could you post your log file from /var/log/letsencrypt?

yann-l · 2018-04-11 23:00:55 UTC

sure:

2018-04-11 22:58:18,965:DEBUG:certbot.main:certbot version: 0.22.2
2018-04-11 22:58:18,966:DEBUG:certbot.main:Arguments: ['--manual', '-d', '*.domain.org', '-d', 'domain.org', '--preferred-challenges', 'dns-01', '--server', 'https://acme-staging-v02.api.letsencrypt.org/directory', '--manual-auth-hook', '/root/bin/certbot_auth_hook.sh', '--break-my-certs', '--manual-public-ip-logging-ok', '--force-renewal']
2018-04-11 22:58:18,967:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#dns-cloudflare,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-04-11 22:58:18,975:DEBUG:certbot.log:Root logging level set at 20
2018-04-11 22:58:18,976:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-04-11 22:58:18,976:DEBUG:certbot.plugins.selection:Requested authenticator manual and installer None
2018-04-11 22:58:18,977:DEBUG:certbot.plugins.selection:Single candidate plugin: * manual
Description: Manual configuration or run your own shell scripts
Interfaces: IAuthenticator, IPlugin
Entry point: manual = certbot.plugins.manual:Authenticator
Initialized: <certbot.plugins.manual.Authenticator object at 0x7f753ac436d8>
Prep: True
2018-04-11 22:58:18,977:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.manual.Authenticator object at 0x7f753ac436d8> and installer None
2018-04-11 22:58:18,977:INFO:certbot.plugins.selection:Plugins selected: Authenticator manual, Installer None
2018-04-11 22:58:18,981:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7f753ab1fa20>)>), contact=('mailto:yann@domain.org',), agreement=None, status='valid', terms_of_service_agreed=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/5901852', new_authzr_uri=None, terms_of_service='https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf'), 9a73d827d763407b01d1b1f92c63de73, Meta(creation_dt=datetime.datetime(2018, 4, 11, 11, 48, 16, tzinfo=<UTC>), creation_host='panoramix.domain.org'))>
2018-04-11 22:58:18,983:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
2018-04-11 22:58:18,985:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
2018-04-11 22:58:19,230:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 724
2018-04-11 22:58:19,231:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 724
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Wed, 11 Apr 2018 22:58:19 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 11 Apr 2018 22:58:19 GMT
Connection: keep-alive

b'{\n  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",\n  "meta": {\n    "caaIdentities": [\n      "letsencrypt.org"\n    ],\n    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",\n    "website": "https://letsencrypt.org/docs/staging-environment/"\n  },\n  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",\n  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",\n  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",\n  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert",\n  "v5OLDGmt8z0": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"\n}'
2018-04-11 22:58:19,245:DEBUG:certbot.renewal:Auto-renewal forced with --force-renewal...
2018-04-11 22:58:19,245:INFO:certbot.main:Renewing an existing certificate
2018-04-11 22:58:19,351:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0082_key-certbot.pem
2018-04-11 22:58:19,354:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0082_csr-certbot.pem
2018-04-11 22:58:19,355:DEBUG:acme.client:Requesting fresh nonce
2018-04-11 22:58:19,355:DEBUG:acme.client:Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order.
2018-04-11 22:58:19,527:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-order HTTP/1.1" 405 0
2018-04-11 22:58:19,528:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 103
Allow: POST
Replay-Nonce: RfNeeD1drobOt-wwKGC8-jBfAisGVkDB86IH9mV5vJg
Expires: Wed, 11 Apr 2018 22:58:19 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 11 Apr 2018 22:58:19 GMT
Connection: keep-alive

b''
2018-04-11 22:58:19,528:DEBUG:acme.client:Storing nonce: RfNeeD1drobOt-wwKGC8-jBfAisGVkDB86IH9mV5vJg
2018-04-11 22:58:19,528:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "*.domain.org"\n    },\n    {\n      "type": "dns",\n      "value": "domain.org"\n    }\n  ],\n  "status": "pending",\n  "resource": "new-order"\n}'
2018-04-11 22:58:19,531:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC81OTAxODUyIiwgIm5vbmNlIjogIlJmTmVlRDFkcm9iT3Qtd3dLR0M4LWpCZkFpc0dWa0RCODZJSDltVjV2SmciLCAidXJsIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvbmV3LW9yZGVyIn0",
  "signature": "VSPBAW9jYoLg9kVAm55z-j2tTdgwOJTgo5PCJMNO6YipBsAj6hc9JbsNNShSOW9Wjnj2rlwQgVp7PXVY8yqRN-lgGfRgRshSEdzDv4hotH4xhUlh3BwEwrKDP-Ks5ijnMl5ZbwWXTFLiY5NgKsC5o_kujBe-mKmoSR8cc25B3LW_lRHkNu2aJZZe51bIrRFSMTW48dYgBpLqaSCDqBYa8vkFofEOJEPdd-S5N8pirGqDeGITNZyy1Zh1TfvrqzwyQizbpifEcRy1WKV5e_a3ohmVbvBb0NymwiwPHHnbuIhLQ9kx5yCBr5BxW13rz1cQ00s97n0kOs_YNeizW-K1AQ",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogIioubGVib3VsYW5nZXIub3JnIgogICAgfSwKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImxlYm91bGFuZ2VyLm9yZyIKICAgIH0KICBdLAogICJzdGF0dXMiOiAicGVuZGluZyIsCiAgInJlc291cmNlIjogIm5ldy1vcmRlciIKfQ"
}
2018-04-11 22:58:19,723:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 562
2018-04-11 22:58:19,724:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 562
Boulder-Requester: 5901852
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/5901852/359482
Replay-Nonce: MZUcH43fWaBZqQL0HmTurdf0cS4m91DAnhkhiRKuI58
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Wed, 11 Apr 2018 22:58:19 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 11 Apr 2018 22:58:19 GMT
Connection: keep-alive

b'{\n  "status": "pending",\n  "expires": "2018-04-18T22:58:19.605019669Z",\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "*.domain.org"\n    },\n    {\n      "type": "dns",\n      "value": "domain.org"\n    }\n  ],\n  "authorizations": [\n    "https://acme-staging-v02.api.letsencrypt.org/acme/authz/NKsNPxkLBatZhgxnBXl6UuallREZXOoXQKl6meICiXs",\n    "https://acme-staging-v02.api.letsencrypt.org/acme/authz/He-W-wtCyMWR3tjcy4CzXMGlw2G3TBMsmnMkj5oaS6o"\n  ],\n  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/5901852/359482"\n}'
2018-04-11 22:58:19,724:DEBUG:acme.client:Storing nonce: MZUcH43fWaBZqQL0HmTurdf0cS4m91DAnhkhiRKuI58
2018-04-11 22:58:19,725:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/acme/authz/NKsNPxkLBatZhgxnBXl6UuallREZXOoXQKl6meICiXs.
2018-04-11 22:58:19,939:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /acme/authz/NKsNPxkLBatZhgxnBXl6UuallREZXOoXQKl6meICiXs HTTP/1.1" 200 525
2018-04-11 22:58:19,940:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 525
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Wed, 11 Apr 2018 22:58:19 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 11 Apr 2018 22:58:19 GMT
Connection: keep-alive

b'{\n  "identifier": {\n    "type": "dns",\n    "value": "domain.org"\n  },\n  "status": "valid",\n  "expires": "2018-05-11T11:55:23Z",\n  "challenges": [\n    {\n      "type": "dns-01",\n      "status": "valid",\n      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/NKsNPxkLBatZhgxnBXl6UuallREZXOoXQKl6meICiXs/116869777",\n      "token": "4GrzIodAvXAp77L4tUmkBHUqVEpkK58MIN461YEt5Xs",\n      "validationRecord": [\n        {\n          "hostname": "domain.org"\n        }\n      ]\n    }\n  ],\n  "wildcard": true\n}'
2018-04-11 22:58:19,941:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/acme/authz/He-W-wtCyMWR3tjcy4CzXMGlw2G3TBMsmnMkj5oaS6o.
2018-04-11 22:58:20,152:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /acme/authz/He-W-wtCyMWR3tjcy4CzXMGlw2G3TBMsmnMkj5oaS6o HTTP/1.1" 200 1411
2018-04-11 22:58:20,152:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 1411
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Wed, 11 Apr 2018 22:58:20 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Wed, 11 Apr 2018 22:58:20 GMT
Connection: keep-alive

b'{\n  "identifier": {\n    "type": "dns",\n    "value": "domain.org"\n  },\n  "status": "valid",\n  "expires": "2018-05-11T12:09:05Z",\n  "challenges": [\n    {\n      "type": "dns-01",\n      "status": "pending",\n      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/He-W-wtCyMWR3tjcy4CzXMGlw2G3TBMsmnMkj5oaS6o/116873428",\n      "token": "4QDEJPNAN46Q3EwmbJu7S643PBrJi1T89zHgzcsTJrc"\n    },\n    {\n      "type": "http-01",\n      "status": "valid",\n      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/He-W-wtCyMWR3tjcy4CzXMGlw2G3TBMsmnMkj5oaS6o/116873429",\n      "token": "dRvLJZUO68_CXLznvPMbG499t9EA61ZMWaR7FkSonCQ",\n      "validationRecord": [\n        {\n          "url": "https://domain.org/.well-known/acme-challenge/dRvLJZUO68_CXLznvPMbG499t9EA61ZMWaR7FkSonCQ",\n          "hostname": "domain.org",\n          "port": "443",\n          "addressesResolved": [\n            "212.129.3.231",\n            "2001:bc8:342f:101::1"\n          ],\n          "addressUsed": "212.129.3.231"\n        },\n        {\n          "url": "http://domain.org/.well-known/acme-challenge/dRvLJZUO68_CXLznvPMbG499t9EA61ZMWaR7FkSonCQ",\n          "hostname": "domain.org",\n          "port": "80",\n          "addressesResolved": [\n            "212.129.3.231",\n            "2001:bc8:342f:101::1"\n          ],\n          "addressUsed": "2001:bc8:342f:101::1"\n        }\n      ]\n    }\n  ]\n}'
2018-04-11 22:58:20,153:INFO:certbot.auth_handler:Performing the following challenges:
2018-04-11 22:58:20,154:INFO:certbot.auth_handler:dns-01 challenge for domain.org
2018-04-11 22:58:20,154:INFO:certbot.auth_handler:dns-01 challenge for domain.org
111

schoen · 2018-04-11 23:16:59 UTC

@bmw, could this be a bug with cached authorizations in ACMEv2? Or do you have any other theory about why Certbot hangs here?

@yann-l, could you maybe also try running with -vvv to get more verbosity in your regular output? (I’m not sure that will actually give you any information that didn’t get saved into this log, though.)

bmw · 2018-04-11 23:41:02 UTC

This would happen if the manual auth hook never exited. Try printing something to stdout/stderr in your script. Certbot captures this output and puts it in its logs. If it never shows up, your script probably isn’t returning.

yann-l · 2018-04-12 09:39:14 UTC

Ok I found what’s causing the hang of certbot. My script print about 12 lines to stderr and about 55 lines to stdout. Adding a &>/dev/null to those commands and now it doesn’t hang anymore.

schoen · 2018-04-12 16:46:38 UTC

It seems like Certbot should allow the hook scripts to give output. It uses this method to call the hooks (with Python’s subprocess.Popen):

    cmd = Popen(shell_cmd, shell=True, stdout=PIPE,
                stderr=PIPE, universal_newlines=True)
    out, err = cmd.communicate()

So I don’t see a reason that it should hang if the hook script provides output.

jvanasco · 2018-04-12 18:17:36 UTC

it would also help to see the shell script certbot_auth_hook.sh. there are so many reasons why it could cause a hang in there, such as if it is calling something that fails, and not doing a foo || exit 255, python can hang on that.

yann-l · 2018-04-12 19:15:02 UTC

The exact line that makes certbot hang it this one:
/etc/init.d/rollerd start

Commenting this line makes things work. or adding &>/dev/null

Running rollerd from command line prints nothing

my script only adds a line in the zone file, stop rollerd, resign zone, restart bind and restart rollerd.

jvanasco · 2018-04-12 20:28:05 UTC

i would dig into this further, because what you disclosed shouldn’t really happen. maybe it’s a bad init.d script? have you tried running that start command when rollerd is both running and not running?

rollerd is designed to run as a daemon and not exit - but that init.d script should exit.

You can also use rollctl to control a running version of rollerd without stopping/starting directly.

yann-l · 2018-04-12 20:47:28 UTC

using rollctl instead of init script works. I’m not an expert of init scripts, I can’t really says what’s wrong there

system · 2018-05-12 20:47:34 UTC

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.