Sporadic failures on ipv6-only host

adrianimboden · June 3, 2017, 10:20am

Hi all

I have ipv6-only hosts for my test environment. I sporadically see that certificates cannot be requested using this configuration. The ipv4 hosts have no problem with the same procedure (the codebase is the same).

The error message says, that the host is not available, but I tried from multiple remote hosts that have ipv6 connectivity and the connection to the webserver works perfectly.

What maybe special in my case is:
The certbot itself does not have an ipv6 address, but the webserver that has the ipv6 address is a reverse-proxy that forwards all requests to /.well-known/acme-challenge/* to the certbot-instance.

My domain is:
demo.cust.thingdust-dev.io

I ran this command:
certbot certonly --non-interactive -vv --standalone --agree-tos --non-interactive --email adi@thingdust.com -d demo.cust.thingdust-dev.io --standalone-supported-challenges http-01

It produced this output:
2017-06-03 09:52:05.894448: 2017-06-03 09:51:58,822:DEBUG:certbot.main:Root logging level set at 10
2017-06-03 09:51:58,822:INFO:certbot.main:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-06-03 09:51:58,823:DEBUG:certbot.main:certbot version: 0.7.0
2017-06-03 09:51:58,823:DEBUG:certbot.main:Arguments: [’–non-interactive’, ‘-vv’, ‘–standalone’, ‘–agree-tos’, ‘–non-interactive’, ‘–email’, ‘adi@thingdust.com’, ‘-d’, ‘demo.cust.thingdust-dev.io’, ‘–standalone-supported-challenges’, ‘http-01’]
2017-06-03 09:51:58,823:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#webroot,PluginEntryPoint#null,PluginEntryPoint#manual,PluginEntryPoint#standalone)
2017-06-03 09:51:58,824:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
2017-06-03 09:51:58,932:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
Description: Automatically use a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7faab9981990>
Prep: True
2017-06-03 09:51:58,932:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7faab9981990> and installer None
2017-06-03 09:51:58,948:DEBUG:certbot.main:Picked account: <Account(f0ae0d145dc94039119d28f958728f53)>
2017-06-03 09:51:58,949:DEBUG:root:Sending GET request to https://acme-v01.api.letsencrypt.org/directory. args: (), kwargs: {}
2017-06-03 09:51:58,952:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2017-06-03 09:51:59,202:DEBUG:requests.packages.urllib3.connectionpool:“GET /directory HTTP/1.1” 200 352
2017-06-03 09:51:59,203:DEBUG:root:Received <Response [200]>. Headers: {‘Content-Length’: ‘352’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Boulder-Request-Id’: ‘NfhlN_5HNeTSc_bC4dz0PFDSvCSqltCk5t0wjA6ybpU’, ‘Strict-Transport-Security’: ‘max-age=604800’, ‘Server’: ‘nginx’, ‘Connection’: ‘keep-alive’, ‘Pragma’: ‘no-cache’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘X-Frame-Options’: ‘DENY’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘oLhlffrN-p975lg4N_4Kr9l64A2rdRBYGvHLabi892s’}. Content: ‘{\n “key-change”: “https://acme-v01.api.letsencrypt.org/acme/key-change”,\n “new-authz”: “https://acme-v01.api.letsencrypt.org/acme/new-authz”,\n “new-cert”: “https://acme-v01.api.letsencrypt.org/acme/new-cert”,\n “new-reg”: “https://acme-v01.api.letsencrypt.org/acme/new-reg”,\n “revoke-cert”: “https://acme-v01.api.letsencrypt.org/acme/revoke-cert”\n}‘
2017-06-03 09:51:59,203:DEBUG:acme.client:Received response <Response [200]> (headers: {‘Content-Length’: ‘352’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Boulder-Request-Id’: ‘NfhlN_5HNeTSc_bC4dz0PFDSvCSqltCk5t0wjA6ybpU’, ‘Strict-Transport-Security’: ‘max-age=604800’, ‘Server’: ‘nginx’, ‘Connection’: ‘keep-alive’, ‘Pragma’: ‘no-cache’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘X-Frame-Options’: ‘DENY’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘oLhlffrN-p975lg4N_4Kr9l64A2rdRBYGvHLabi892s’}): ‘{\n “key-change”: “https://acme-v01.api.letsencrypt.org/acme/key-change”,\n “new-authz”: “https://acme-v01.api.letsencrypt.org/acme/new-authz”,\n “new-cert”: “https://acme-v01.api.letsencrypt.org/acme/new-cert”,\n “new-reg”: “https://acme-v01.api.letsencrypt.org/acme/new-reg”,\n “revoke-cert”: “https://acme-v01.api.letsencrypt.org/acme/revoke-cert”\n}‘
2017-06-03 09:51:59,227:DEBUG:root:Requesting fresh nonce
2017-06-03 09:51:59,227:DEBUG:root:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz. args: (), kwargs: {}
2017-06-03 09:51:59,411:DEBUG:requests.packages.urllib3.connectionpool:“HEAD /acme/new-authz HTTP/1.1” 405 0
2017-06-03 09:51:59,412:DEBUG:root:Received <Response [405]>. Headers: {‘Content-Length’: ‘91’, ‘Pragma’: ‘no-cache’, ‘Boulder-Request-Id’: ‘qigpJE0ZOxNzHkwhFL1uYn8TxbxcY-1-O6Y1VeZoW8E’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Server’: ‘nginx’, ‘Connection’: ‘keep-alive’, ‘Allow’: ‘POST’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Content-Type’: ‘application/problem+json’, ‘Replay-Nonce’: ‘CfyCl5lD6KqFpjRfIq7pXAOT5_1erraqXc3KLTaSBd8’}. Content: ‘‘
2017-06-03 09:51:59,412:DEBUG:acme.client:Storing nonce: ‘\t\xfc\x82\x97\x99C\xe8\xaa\x85\xa64_"\xae\xe9\\x03\x93\xe7\xfd^\xae\xb6\xaa]\xcd\xca-6\x92\x05\xdf’
2017-06-03 09:51:59,413:DEBUG:acme.jose.json_util:Omitted empty fields: expires=None, combinations=None, status=None, challenges=None
2017-06-03 09:51:59,414:DEBUG:acme.client:Serialized JSON: {“identifier”: {“type”: “dns”, “value”: “demo.cust.thingdust-dev.io”}, “resource”: “new-authz”}
2017-06-03 09:51:59,415:DEBUG:acme.jose.json_util:Omitted empty fields: x5c=(), jku=None, x5t=None, x5tS256=None, alg=None, typ=None, jwk=None, crit=(), x5u=None, kid=None, cty=None
2017-06-03 09:51:59,417:DEBUG:acme.jose.json_util:Omitted empty fields: x5c=(), jku=None, nonce=None, x5tS256=None, crit=(), x5t=None, typ=None, x5u=None, kid=None, cty=None
2017-06-03 09:51:59,417:DEBUG:root:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz. args: (), kwargs: {‘data’: ‘{“header”: {“alg”: “RS256”, “jwk”: {“e”: “AQAB”, “kty”: “RSA”, “n”: “—redacted—”}}, “protected”: “—redacted—”, “payload”: “eyJpZGVudGlmaWVyIjogeyJ0eXBlIjogImRucyIsICJ2YWx1ZSI6ICJkZW1vLmN1c3QudGhpbmdkdXN0LWRldi5pbyJ9LCAicmVzb3VyY2UiOiAibmV3LWF1dGh6In0”, “signature”: “1Kz4HlUv6j5phNpG3iyNnD1BmTU1uHrJZuF9opTXnW5-GIjJQyJQVp_c6EDDZ_HWGPFv1Oa0W9v9Gm2emS_d52tjI3LGGaCyVMjfDKAi2xLDdZVMJYb8pPL2z9WWuo83Z4yKbjOms_iJcpiJRryl9qGarf2OcgfK_QjdVhuDMtq_ZCr3-axXnr5cC_1JkPMA044XSuJkGwPYSIvt-lZzsOjmgnAEVSAMdn4vBsTEJvYIIVpOoPYCAXw5zCAs5x5iwtornn4CX1TiKUs0DHKMIofab8XF3S4K7OCWbDcUmGk-TdSTNHiRlSWNU-4-t1Qs6kioAMHALMQJe-N40FaFQg”}’}
2017-06-03 09:51:59,616:DEBUG:requests.packages.urllib3.connectionpool:“POST /acme/new-authz HTTP/1.1” 201 1014
2017-06-03 09:51:59,617:DEBUG:root:Received <Response [201]>. Headers: {‘Content-Length’: ‘1014’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Boulder-Request-Id’: ‘Kdh6Z42q7GMV_2mT-o3O9a7fliFFlpEtVn6mLX5Wrn8’, ‘Strict-Transport-Security’: ‘max-age=604800’, ‘Server’: ‘nginx’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Connection’: ‘keep-alive’, ‘Link’: ‘https://acme-v01.api.letsencrypt.org/acme/new-cert;rel=“next”’, ‘Location’: ‘https://acme-v01.api.letsencrypt.org/acme/authz/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8’, ‘Pragma’: ‘no-cache’, ‘Boulder-Requester’: ‘11511344’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘X-Frame-Options’: ‘DENY’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘OIsXv6rUkcR9yqEqWh2NsrYzfeQ-RQIl1x09HbKD8qU’}. Content: ‘{\n “identifier”: {\n “type”: “dns”,\n “value”: “demo.cust.thingdust-dev.io”\n },\n “status”: “pending”,\n “expires”: “2017-06-10T09:52:00.556365182Z”,\n “challenges”: [\n {\n “type”: “http-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239”,\n “token”: “—redacted—”\n },\n {\n “type”: “tls-sni-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433240”,\n “token”: “ijLhV1BTPkrFqBQEzwx34zKeOLuIXgPykP68t0xDbO8”\n },\n {\n “type”: “dns-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241”,\n “token”: “—redacted—”\n }\n ],\n “combinations”: [\n [\n 2\n ],\n [\n 0\n ],\n [\n 1\n ]\n ]\n}‘
2017-06-03 09:51:59,617:DEBUG:acme.client:Storing nonce: ‘8\x8b\x17\xbf\xaa\xd4\x91\xc4}\xca\xa1*Z\x1d\x8d\xb2\xb63}\xe4>E\x02%\xd7\x1d=\x1d\xb2\x83\xf2\xa5’
2017-06-03 09:51:59,618:DEBUG:acme.client:Received response <Response [201]> (headers: {‘Content-Length’: ‘1014’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Boulder-Request-Id’: ‘Kdh6Z42q7GMV_2mT-o3O9a7fliFFlpEtVn6mLX5Wrn8’, ‘Strict-Transport-Security’: ‘max-age=604800’, ‘Server’: ‘nginx’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Connection’: ‘keep-alive’, ‘Link’: ‘https://acme-v01.api.letsencrypt.org/acme/new-cert;rel=“next”’, ‘Location’: ‘https://acme-v01.api.letsencrypt.org/acme/authz/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8’, ‘Pragma’: ‘no-cache’, ‘Boulder-Requester’: ‘11511344’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘X-Frame-Options’: ‘DENY’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘OIsXv6rUkcR9yqEqWh2NsrYzfeQ-RQIl1x09HbKD8qU’}): ‘{\n “identifier”: {\n “type”: “dns”,\n “value”: “demo.cust.thingdust-dev.io”\n },\n “status”: “pending”,\n “expires”: “2017-06-10T09:52:00.556365182Z”,\n “challenges”: [\n {\n “type”: “http-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239”,\n “token”: “—redacted—”\n },\n {\n “type”: “tls-sni-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433240”,\n “token”: “ijLhV1BTPkrFqBQEzwx34zKeOLuIXgPykP68t0xDbO8”\n },\n {\n “type”: “dns-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241”,\n “token”: “—redacted—”\n }\n ],\n “combinations”: [\n [\n 2\n ],\n [\n 0\n ],\n [\n 1\n ]\n ]\n}‘
2017-06-03 09:51:59,619:DEBUG:acme.challenges:dns-01 was not recognized, full message: {u’status’: u’pending’, u’token’: u’—redacted—’, u’type’: u’dns-01’, u’uri’: u’https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241’}
2017-06-03 09:51:59,619:INFO:certbot.auth_handler:Performing the following challenges:
2017-06-03 09:51:59,619:INFO:certbot.auth_handler:http-01 challenge for demo.cust.thingdust-dev.io
2017-06-03 09:51:59,625:INFO:certbot.auth_handler:Waiting for verification…
2017-06-03 09:51:59,626:DEBUG:acme.client:Serialized JSON: {“keyAuthorization”: “—redacted—”, “type”: “http-01”, “resource”: “challenge”}
2017-06-03 09:51:59,627:DEBUG:acme.jose.json_util:Omitted empty fields: x5c=(), jku=None, x5t=None, x5tS256=None, alg=None, typ=None, jwk=None, crit=(), x5u=None, kid=None, cty=None
2017-06-03 09:51:59,629:DEBUG:acme.jose.json_util:Omitted empty fields: x5c=(), jku=None, nonce=None, x5tS256=None, crit=(), x5t=None, typ=None, x5u=None, kid=None, cty=None
2017-06-03 09:51:59,629:DEBUG:root:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239. args: (), kwargs: {‘data’: ‘{“header”: {“alg”: “RS256”, “jwk”: {“e”: “AQAB”, “kty”: “RSA”, “n”: “—redacted—”}}, “protected”: “eyJub25jZSI6ICJPSXNYdjZyVWtjUjl5cUVxV2gyTnNyWXpmZVEtUlFJbDF4MDlIYktEOHFVIn0”, “payload”: “eyJrZXlBdXRob3JpemF0aW9uIjogIkJYOUNkZE4ybGVHdktiMW90dzNINk9sMTdhVzcxYVV5NUtLVWlVQ2hSNjAuOU5EQ205ZWNLWEhuazYzZmtGTXFYQk43ZFhFeG1Rd3dwTVFQeEJPaHU1RSIsICJ0eXBlIjogImh0dHAtMDEiLCAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIn0”, “signature”: “FCu7Nf7nsb8DBx8lvnYH0d43YVNa8prZe2TUjOclrUaw2j4epq3ClpXsg6mokb1kJlF_CTXVTePNfH_6pqhsjATMzKFe9IZAsCBQugVPy-t5P0oRxpOsWzMHfd_xRI0IDcG7De5Ku3pnOQr2s8txTiBqdpy21uOXYotsAt6aA5BX-r4pdyRoEP8WgQcmzUEKWWdcmdkDGFBpUoPRRXNrKvdCxRGc91wnPnYEXDFg2EwVRxPyoFTAOMSQjWxDdiu8l5FIH2AjGfoTtrHBCML_vZSleLtoh0tUwiVAA-M_hsR-sTCeHNFP-rmOypSGuP2RPziFOQYn1cR3eXAQBQH-Hg”}’}
2017-06-03 09:51:59,817:DEBUG:requests.packages.urllib3.connectionpool:“POST /acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239 HTTP/1.1” 202 336
2017-06-03 09:51:59,817:DEBUG:root:Received <Response [202]>. Headers: {‘Content-Length’: ‘336’, ‘Boulder-Request-Id’: ‘x6meo5xR8RUXUlzV4MXn2rjzN0x2-nuE33eUpyPWw1U’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Server’: ‘nginx’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Connection’: ‘keep-alive’, ‘Link’: ‘https://acme-v01.api.letsencrypt.org/acme/authz/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8;rel=“up”’, ‘Location’: ‘https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239’, ‘Pragma’: ‘no-cache’, ‘Boulder-Requester’: ‘11511344’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘6gH9z54vTVSYioR6FUrosEz9q0SXA_ToP_ZgfiSgNCE’}. Content: ‘{\n “type”: “http-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239”,\n “token”: “—redacted—”,\n “keyAuthorization”: “—redacted—”\n}‘
2017-06-03 09:51:59,818:DEBUG:acme.client:Storing nonce: ‘\xea\x01\xfd\xcf\x9e/MT\x98\x8a\x84z\x15J\xe8\xb0L\xfd\xabD\x97\x03\xf4\xe8?\xf6`~$\xa04!‘
2017-06-03 09:51:59,818:DEBUG:acme.client:Received response <Response [202]> (headers: {‘Content-Length’: ‘336’, ‘Boulder-Request-Id’: ‘x6meo5xR8RUXUlzV4MXn2rjzN0x2-nuE33eUpyPWw1U’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Server’: ‘nginx’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Connection’: ‘keep-alive’, ‘Link’: ‘https://acme-v01.api.letsencrypt.org/acme/authz/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8;rel=“up”’, ‘Location’: ‘https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239’, ‘Pragma’: ‘no-cache’, ‘Boulder-Requester’: ‘11511344’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:00 GMT’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘6gH9z54vTVSYioR6FUrosEz9q0SXA_ToP_ZgfiSgNCE’}): ‘{\n “type”: “http-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239”,\n “token”: “—redacted—”,\n “keyAuthorization”: “—redacted—”\n}‘
2017-06-03 09:52:02,822:DEBUG:root:Sending GET request to https://acme-v01.api.letsencrypt.org/acme/authz/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8. args: (), kwargs: {}
2017-06-03 09:52:03,013:DEBUG:requests.packages.urllib3.connectionpool:“GET /acme/authz/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8 HTTP/1.1” 200 1121
2017-06-03 09:52:03,013:DEBUG:root:Received <Response [200]>. Headers: {‘Content-Length’: ‘1121’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:04 GMT’, ‘Boulder-Request-Id’: ‘MNRMqLxxgaMdwVmnR77xd5H9kfPqANVux2oMYXmkerY’, ‘Strict-Transport-Security’: ‘max-age=604800’, ‘Server’: ‘nginx’, ‘Connection’: ‘keep-alive’, ‘Link’: ‘https://acme-v01.api.letsencrypt.org/acme/new-cert;rel=“next”’, ‘Pragma’: ‘no-cache’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:04 GMT’, ‘X-Frame-Options’: ‘DENY’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘-1TqyjdwwYjRGD7rVhloRGa5cEYG_jAAf9LfGtyGLG8’}. Content: ‘{\n “identifier”: {\n “type”: “dns”,\n “value”: “demo.cust.thingdust-dev.io”\n },\n “status”: “pending”,\n “expires”: “2017-06-10T09:52:00Z”,\n “challenges”: [\n {\n “type”: “http-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239”,\n “token”: “—redacted—”,\n “keyAuthorization”: “—redacted—”\n },\n {\n “type”: “tls-sni-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433240”,\n “token”: “ijLhV1BTPkrFqBQEzwx34zKeOLuIXgPykP68t0xDbO8”\n },\n {\n “type”: “dns-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241”,\n “token”: “—redacted—”\n }\n ],\n “combinations”: [\n [\n 2\n ],\n [\n 0\n ],\n [\n 1\n ]\n ]\n}‘
2017-06-03 09:52:03,014:DEBUG:acme.client:Received response <Response [200]> (headers: {‘Content-Length’: ‘1121’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:04 GMT’, ‘Boulder-Request-Id’: ‘MNRMqLxxgaMdwVmnR77xd5H9kfPqANVux2oMYXmkerY’, ‘Strict-Transport-Security’: ‘max-age=604800’, ‘Server’: ‘nginx’, ‘Connection’: ‘keep-alive’, ‘Link’: ‘https://acme-v01.api.letsencrypt.org/acme/new-cert;rel=“next”’, ‘Pragma’: ‘no-cache’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:04 GMT’, ‘X-Frame-Options’: ‘DENY’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘-1TqyjdwwYjRGD7rVhloRGa5cEYG_jAAf9LfGtyGLG8’}): ‘{\n “identifier”: {\n “type”: “dns”,\n “value”: “demo.cust.thingdust-dev.io”\n },\n “status”: “pending”,\n “expires”: “2017-06-10T09:52:00Z”,\n “challenges”: [\n {\n “type”: “http-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239”,\n “token”: “—redacted—”,\n “keyAuthorization”: “—redacted—”\n },\n {\n “type”: “tls-sni-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433240”,\n “token”: “ijLhV1BTPkrFqBQEzwx34zKeOLuIXgPykP68t0xDbO8”\n },\n {\n “type”: “dns-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241”,\n “token”: “—redacted—”\n }\n ],\n “combinations”: [\n [\n 2\n ],\n [\n 0\n ],\n [\n 1\n ]\n ]\n}‘
2017-06-03 09:52:03,014:DEBUG:acme.challenges:dns-01 was not recognized, full message: {u’status’: u’pending’, u’token’: u’—redacted—’, u’type’: u’dns-01’, u’uri’: u’https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241’}
2017-06-03 09:52:06,016:DEBUG:root:Sending GET request to https://acme-v01.api.letsencrypt.org/acme/authz/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8. args: (), kwargs: {}
2017-06-03 09:52:06,199:DEBUG:requests.packages.urllib3.connectionpool:“GET /acme/authz/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8 HTTP/1.1” 200 1710
2017-06-03 09:52:06,200:DEBUG:root:Received <Response [200]>. Headers: {‘Content-Length’: ‘1710’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:07 GMT’, ‘Boulder-Request-Id’: ‘g5DpfVybkV_6EUwBfTBbpPxZbW6SRgU4NeTRTccpm4Y’, ‘Strict-Transport-Security’: ‘max-age=604800’, ‘Server’: ‘nginx’, ‘Connection’: ‘keep-alive’, ‘Link’: ‘https://acme-v01.api.letsencrypt.org/acme/new-cert;rel=“next”’, ‘Pragma’: ‘no-cache’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:07 GMT’, ‘X-Frame-Options’: ‘DENY’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘U_bnZ7fclLzQ4k_i5NEsXM_RfEadF4oA9opp8u0JsDs’}. Content: ‘{\n “identifier”: {\n “type”: “dns”,\n “value”: “demo.cust.thingdust-dev.io”\n },\n “status”: “invalid”,\n “expires”: “2017-06-10T09:52:00Z”,\n “challenges”: [\n {\n “type”: “http-01”,\n “status”: “invalid”,\n “error”: {\n “type”: “urn:acme:error:connection”,\n “detail”: “Could not connect to demo.cust.thingdust-dev.io”,\n “status”: 400\n },\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239”,\n “token”: “—redacted—”,\n “keyAuthorization”: “—redacted—”,\n “validationRecord”: [\n {\n “url”: “http://demo.cust.thingdust-dev.io/.well-known/acme-challenge/---redacted---”,\n “hostname”: “demo.cust.thingdust-dev.io”,\n “port”: “80”,\n “addressesResolved”: [\n “2001:1620:a68:100:101::1”\n ],\n “addressUsed”: “2001:1620:a68:100:101::1”,\n “addressesTried”: []\n }\n ]\n },\n {\n “type”: “tls-sni-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433240”,\n “token”: “ijLhV1BTPkrFqBQEzwx34zKeOLuIXgPykP68t0xDbO8”\n },\n {\n “type”: “dns-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241”,\n “token”: “—redacted—”\n }\n ],\n “combinations”: [\n [\n 2\n ],\n [\n 0\n ],\n [\n 1\n ]\n ]\n}‘
2017-06-03 09:52:06,201:DEBUG:acme.client:Received response <Response [200]> (headers: {‘Content-Length’: ‘1710’, ‘Expires’: ‘Sat, 03 Jun 2017 09:52:07 GMT’, ‘Boulder-Request-Id’: ‘g5DpfVybkV_6EUwBfTBbpPxZbW6SRgU4NeTRTccpm4Y’, ‘Strict-Transport-Security’: ‘max-age=604800’, ‘Server’: ‘nginx’, ‘Connection’: ‘keep-alive’, ‘Link’: ‘https://acme-v01.api.letsencrypt.org/acme/new-cert;rel=“next”’, ‘Pragma’: ‘no-cache’, ‘Cache-Control’: ‘max-age=0, no-cache, no-store’, ‘Date’: ‘Sat, 03 Jun 2017 09:52:07 GMT’, ‘X-Frame-Options’: ‘DENY’, ‘Content-Type’: ‘application/json’, ‘Replay-Nonce’: ‘U_bnZ7fclLzQ4k_i5NEsXM_RfEadF4oA9opp8u0JsDs’}): ‘{\n “identifier”: {\n “type”: “dns”,\n “value”: “demo.cust.thingdust-dev.io”\n },\n “status”: “invalid”,\n “expires”: “2017-06-10T09:52:00Z”,\n “challenges”: [\n {\n “type”: “http-01”,\n “status”: “invalid”,\n “error”: {\n “type”: “urn:acme:error:connection”,\n “detail”: “Could not connect to demo.cust.thingdust-dev.io”,\n “status”: 400\n },\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433239”,\n “token”: “—redacted—”,\n “keyAuthorization”: “—redacted—”,\n “validationRecord”: [\n {\n “url”: “http://demo.cust.thingdust-dev.io/.well-known/acme-challenge/---redacted---”,\n “hostname”: “demo.cust.thingdust-dev.io”,\n “port”: “80”,\n “addressesResolved”: [\n “2001:1620:a68:100:101::1”\n ],\n “addressUsed”: “2001:1620:a68:100:101::1”,\n “addressesTried”: []\n }\n ]\n },\n {\n “type”: “tls-sni-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433240”,\n “token”: “ijLhV1BTPkrFqBQEzwx34zKeOLuIXgPykP68t0xDbO8”\n },\n {\n “type”: “dns-01”,\n “status”: “pending”,\n “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241”,\n “token”: “—redacted—”\n }\n ],\n “combinations”: [\n [\n 2\n ],\n [\n 0\n ],\n [\n 1\n ]\n ]\n}‘
2017-06-03 09:52:06,201:DEBUG:acme.challenges:dns-01 was not recognized, full message: {u’status’: u’pending’, u’token’: u’—redacted—’, u’type’: u’dns-01’, u’uri’: u’https://acme-v01.api.letsencrypt.org/acme/challenge/5QLLjXv0_aZsun9gCJZDTNJhDYgLigc9kvS_3hoH6I8/1272433241’}
2017-06-03 09:52:06,202:INFO:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: demo.cust.thingdust-dev.io
Type: connection
Detail: Could not connect to demo.cust.thingdust-dev.io

To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you’re using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
2017-06-03 09:52:06,202:INFO:certbot.auth_handler:Cleaning up challenges
2017-06-03 09:52:06,202:DEBUG:certbot.plugins.standalone:Stopping server at 0.0.0.0:80…
2017-06-03 09:52:06,631:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 9, in
load_entry_point(‘certbot==0.7.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 693, in main
return config.func(config, plugins)
File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 512, in obtain_cert
_, action = _auth_from_domains(le_client, config, domains, lineage)
File “/usr/lib/python2.7/site-packages/certbot/main.py”, line 93, in _auth_from_domains
lineage = le_client.obtain_and_enroll_certificate(domains)
File “/usr/lib/python2.7/site-packages/certbot/client.py”, line 276, in obtain_and_enroll_certificate
certr, chain, key, _ = self.obtain_certificate(domains)
File “/usr/lib/python2.7/site-packages/certbot/client.py”, line 247, in obtain_certificate
self.config.allow_subset_of_names)
File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 74, in get_authorizations
self._respond(resp, best_effort)
File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 131, in _respond
self._poll_challenges(chall_update, best_effort)
File “/usr/lib/python2.7/site-packages/certbot/auth_handler.py”, line 195, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. demo.cust.thingdust-dev.io (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Could not connect to demo.cust.thingdust-dev.io

Failed authorization procedure. demo.cust.thingdust-dev.io (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Could not connect to demo.cust.thingdust-dev.io

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: demo.cust.thingdust-dev.io
Type: connection
Detail: Could not connect to demo.cust.thingdust-dev.io

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you’re using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.

My web server is (include version):
I use an apache reverse proxy (docker image httpd:2.4) that forwards to a dockerized certbot (alpine:3.4)

The operating system my web server runs on is (include version):
docker running on Ubuntu 16.04

My hosting provider, if applicable, is:
self-hosted

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

rg305 · June 3, 2017, 9:39pm

(edited)
Please answer some questions to better clarify the problem:
Do you have the latest certbot? When was it last updated?
Was the test environment always IPv6 only?
Have you made any changes around the time it stopped working?

schoen · June 4, 2017, 4:39am

I’m sorry to have to disagree with @rg305 and @EnumC, but Certbot should work properly on an IPv6-only system (despite the error message’s slightly obsolete reference to A records), and @adrianimboden says that this setup already works properly with IPv4 hosts.

However, I suspect that the problem is most likely to be in the reverse proxy configuration. Can you look at logs from the reverse proxy to see if it is receiving inbound connections in IPv6? If not, the problem is probably not with Certbot but rather with the reverse proxy, with the container setup, or with your IPv6 routing or firewalls.

EnumC · June 4, 2017, 7:32am

Thanks for joining in! I wasn’t aware certbot would support such scenario. Thanks for clearing it up!

adrianimboden · June 4, 2017, 9:18am

I have to test it once it does not work again. I tried it an hour later, then it worked without a problem.

I see that the reverse proxy cannot reach the certbot when it was asked from 2600:3000:2710:200::1d. I assume that is one of letsencrypts ip addresses. Letsencrypt received an error 503 in this case.
I was misled by the “Could not connect to demo.cust.thingdust-dev.io” that the connection could not even be made, which is obviously not true.

I saw that the certbot version is very old (0.7.0). I used the one that was in the package repository of alpine linux. Probably, that was not such a good idea. I will update the version of certbot first. It is probably a timing problem/race condition of some kind in this old version, but i did not investigate further.

Thank you very much for your help.

ahaw021 · June 5, 2017, 9:23am

hi @adrianimboden

Can you post the log from the second run (sucessful one)?

I see you are using the standalone plugin and it seems to spin up the server ok

However it only bound to the IPV4 interface on your server (0.0.0.0:80)

I am interested if the second time it still bound to an IPV4 Interface and if your upstream proxy and routers figured out how to get there.

Andrei

adrianimboden · June 5, 2017, 10:15am

Hi @ahaw021

It works as follows (in this case):

Internet -> Router -> DevPC -> Vagrant Machine -> Apache Container -> Certbot Container
  ipv6       ipv6     ipv6          ipv4              ipv4                ipv4

The transition from ipv6 to ipv4 is done via 6tunnel (getting ipv6 to work with docker containers is a major pain at the moment, thats because I chose the way via 6tunnel running on my development computer to have the system available on the ipv6 network. The implementation knows nothing about ipv6.

It has to be some sort of timing problem between the Apache Container and the Certbot Application in the Certbot container.

I previously searched in the wrong direction, because the response from the letsencrypt server says, that the connection failed. I thought that that means that the socket connection failed. But the message lacks a little bit of detail in my opinion.

I think it could be a timing problem with dns resolution of the certbot container on my side. It is only curious, because it only happened with the ipv6 implementation for now.

That would be a successful log:
letsencrypt_successsful_log.txt (20.5 KB)

ahaw021 · June 5, 2017, 1:00pm

hi @adrianimboden

The second run still used an IPV4 Standalone Server.

I think convergence may be an issue.

part of the challenge I see is that your port 80 may not be open all the time (assuming that is why you are using standalone to spin up a server?).

I suggest spinning up a temporary web server (before certbot runs) and hitting it a few time (via curl) just to “wake” everything up. Then running the certbot commands. See if that improves things

Andrei

rg305 · June 6, 2017, 2:54am

May be unrelated…
But I’ve seen issues where DNS requests insist on using TCP.
If TCP is not allowed, then it may not resolve your IP properly.
Not sure if you control your DNS, nor the firewall (logs) in front of them , but it’s worth a look.

system · July 6, 2017, 2:54am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.