Timeout on get new cert

Timeout on get new cert via cert-manager.

Hi @pod2metra

please answer the following questions:


Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

my domain is: middle-earth.io
I ran this command: jx upgrade ingress --namespaces jx-production --urltemplate=’{{.Service}}.{{.Domain}}’ --verbose
It produced this output: Waiting for TLS certificates to be issued…
WARNING: Timeout reached while waiting for TLS certificates to be ready
cert-manager version is v0.9.1

I can’t answer to another questions.

I don't know how that client works. But checking your domain - middle-earth.io - Make your website better - DNS, redirects, mixed content, certificates

Host T IP-Address is auth. ∑ Queries ∑ Timeout
middle-earth.io A yes 1 0
AAAA yes
www.middle-earth.io C a2230112d401311e9bc0d0a7f017ff46-22aa7ff772bb69dd.elb.us-east-1.amazonaws.com yes 1 0
A 18.214.175.24 Ashburn/Virginia/United States (US) - Amazon.com, Inc. Hostname: ec2-18-214-175-24.compute-1.amazonaws.com yes
A 54.85.78.205 Ashburn/Virginia/United States (US) - Amazon.com, Inc. Hostname: ec2-54-85-78-205.compute-1.amazonaws.com yes
A 54.208.117.92 Ashburn/Virginia/United States (US) - Amazon Technologies Inc. Hostname: ec2-54-208-117-92.compute-1.amazonaws.com yes

Your non-www version doesn't have an A-record.

Does that tool use http validation and do you want to create a certificate with middle-earth.io? That can't work.

i want to create cert to api.middle-earth.io for example

Hi @pod2metra,

If you have access to the Kubernetes dashboard (or using kubectl get/kubectl describe), you should be able to navigate to the Custom Resource Definitions and see what the reason is that your certificate isn’t becoming available in cert-manager.

Specifically, if you look at the “Challenge” resource, you should find the challenges for the domain you are requesting and the reason that cert-manager might be getting stuck.

It may also help to know whether your ClusterIssuer is HTTP-based or DNS-based.

Checking that domain there are three active certificates

Issuer not before not after Domain names LE-Duplicate next LE
Let's Encrypt Authority X3 2019-10-22 2020-01-20 api.middle-earth.io
1 entries
Let's Encrypt Authority X3 2019-08-23 2019-11-21 api.middle-earth.io
1 entries
Let's Encrypt Authority X3 2019-08-19 2019-11-17 api.middle-earth.io
1 entries

Last isn't old. Why worked it 2019-10-22?

@JuergenAuer Thank you for your help.
In fact this is the flow of jx upgrade ingress. It will reissue cert every time than you try to update ingress. And it was my mistake that I don’t save old certs.

Hi @_az


here is Challenge CRD for cert-manager.
Also I’m usin EKS and it’s version is 1.14

@JuergenAuer , Can you, please, take a look. May by I don’t pass some limits?

Hi,

Right - that’s the definition of the CRD. What I was looking for is the actual instance of the challenge resources.

e.g. While your jx upgrade is “timing out”,

kubectl describe challenges

and it might show the status which could end in something like:

Spec:
  Authz URL:  https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/19062557
  Dns Name:   76da4dd5.ngrok.io
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt
  Key:      BYpcy_65yCFj_7XcwAT9HnWsl0K4WsAxAdiHKgUT5yA.-3VJzGDfAkR-5IC0CxTKro7J9WMpFmGISpqQy8KyhuU
  Solver:
    http01:
      Ingress:
        Class:  nginx
    Selector:
  Token:     BYpcy_65yCFj_7XcwAT9HnWsl0K4WsAxAdiHKgUT5yA
  Type:      http-01
  URL:       https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/19062557/Mn3DHw
  Wildcard:  false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:
  Type    Reason     Age   From          Message
  ----    ------     ----  ----          -------
  Normal  Started    57s   cert-manager  Challenge scheduled for processing
  Normal  Presented  57s   cert-manager  Presented challenge using http-01 challenge mechanism

If you look closely, you could see that “Reason” explains the reason that the issuance is not proceeding.

If you don’t see any challenges, you can also try kubectl describe orders, kubectl describe certificaterequests or kubectl describe certificates, which would be relevant if you are hitting other kinds of issues, like rate limits.

Chalenges are empty, but I see strange logs

pkg/client/informers/externalversions/factory.go:117: watch of *v1alpha1.Challenge ended with: too old resource version: 100486904 (100495194)
@_az do you know what can it be?

I don’t know about that error. A few GitHub issues suggest that it’s not necessarily an indication of a problem.

Did you check for challenges while jx was upgrading?

Did you check the other describe commands as well?

Maybe with --all-namespaces?

Yes and it's empty. kubectl get challenges.certmanager.k8s.io -w --all-namespaces give me nothing at all.
BUT:
I got new error:

Waiting for TLS certificates to be issued...
WARNING: Following TLS certificates are not ready:
WARNING: jx/tls-chartmuseum
WARNING: jx/tls-nexus
WARNING: jx/tls-jenkins
error: not all TLS certificates are ready

@_az @JuergenAuer
I’m trying to do the same localy and i get
2019-11-05 03:51:08,050:DEBUG:certbot.error_handler:Calling registered functions
2019-11-05 03:51:08,050:INFO:certbot.auth_handler:Cleaning up challenges
2019-11-05 03:51:08,050:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 421, in _make_request
six.raise_from(e, None)
File “”, line 3, in raise_from
File “/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py”, line 416, in _make_request
httplib_response = conn.getresponse()
File “/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py”, line 1344, in getresponse
response.begin()
File “/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py”, line 306, in begin
version, status, reason = self._read_status()
File “/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/http/client.py”, line 267, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), “iso-8859-1”)
File “/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/socket.py”, line 589, in readinto
return self._sock.recv_into(b)
File “/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/contrib/pyopenssl.py”, line 326, in recv_into
raise timeout(“The read operation timed out”)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/util/retry.py", line 400, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 423, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/urllib3/connectionpool.py", line 331, in _raise_timeout
    self, url, "Read timed out. (read timeout=%s)" % timeout_value
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.39.0', 'console_scripts', 'certbot')()
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/main.py", line 1378, in main
    return config.func(config, plugins)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/main.py", line 1265, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/main.py", line 121, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/client.py", line 405, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/client.py", line 348, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/client.py", line 384, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/certbot/auth_handler.py", line 87, in handle_authorizations
    self.acme.answer_challenge(achall.challb, resp)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/acme/client.py", line 164, in answer_challenge
    response = self._post(challb.uri, response)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/acme/client.py", line 95, in _post
    return self.net.post(*args, **kwargs)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/acme/client.py", line 1194, in post
    return self._post_once(*args, **kwargs)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/acme/client.py", line 1207, in _post_once
    response = self._send_request('POST', url, data=data, **kwargs)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/acme/client.py", line 1110, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/Cellar/certbot/0.39.0/libexec/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='acme-v02.api.letsencrypt.org', port=443): Read timed out. (read timeout=45)
2019-11-05 03:51:08,055:ERROR:certbot.log:An unexpected error occurred:

With a network timeout on a POST request, I'd be checking whether there are any MTU issues. Reducing your network interface MTU to 1300 can be a good test for this. It regularaly works for other users who encounter similar problems.

Did you manage to try kubectl describe certificates (and orders and certificaterequests)?

One of the resources is clearly getting stuck, just need to identify which one it is.

Reducing your network interface MTU to 1300 can be a good test for this. It regularaly works for other users who encounter similar problems.

I've already done that, but it's not give me anything.

sudo mtr -c 20 -w -r acme-v02.api.letsencrypt.org
    Start: 2019-11-05T05:32:30+0300
    HOST: MacBook-Pro-Segey.local Loss%   Snt   Last   Avg  Best  Wrst StDev
      1.|-- 192.168.100.1            0.0%    20    1.3   1.6   1.1   4.8   0.8
      2.|-- core.10g.net.belpak.by   0.0%    20    3.6   6.0   3.5  17.8   3.6
      3.|-- 93.84.80.61              0.0%    20    7.9   5.4   3.2   8.8   1.5
      4.|-- core2.net.belpak.by      0.0%    20   10.4   7.3   3.7  11.3   2.2
      5.|-- ie1.net.belpak.by        0.0%    20    6.6   6.8   3.3  11.5   2.3
      6.|-- asbr1.net.belpak.by      0.0%    20    3.6   3.9   3.1   5.8   0.7
      7.|-- spx-ix.as13335.net      25.0%    20   24.1  25.3  23.8  32.1   2.2
      8.|-- 172.65.32.248            0.0%    20   24.6  31.0  24.1 128.8  23.6

May be this trace may help?

kubectl describe certificates (and orders and certificaterequests )

Don’t give anything.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.