Letsencrypt txt record creates and automatically disappears

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: xpresside.com

I ran this command:

cat <<EOF | kubectl apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
  name: che-certificate-issuer
spec:
  acme:
    dns01:
      providers:
      - route53:
          region: us-east-1
          accessKeyID: AKIA4ZR4YBFXPFDK7SNR
          secretAccessKeySecretRef:
            name: aws-cert-manager-access-key
            key: CLIENT_SECRET
        name: route53
    email: mplabxpress@gmail.com
    privateKeySecretRef:
      name: letsencrypt
    server: https://acme-v02.api.letsencrypt.org/directory
EOF



 cat <<EOF | kubectl apply -f -
apiVersion: certmanager.k8s.io/v1alpha1
kind: Certificate
metadata:
 name: che-tls
 namespace: che
spec:
 secretName: che-tls
 issuerRef:
   name: che-certificate-issuer
   kind: ClusterIssuer
 dnsNames:
   - '*.xpresside.com'
 acme:
   config:
     - dns01:
         provider: route53
       domains:
         - '*.xpresside.com'
EOF

It produced this output: 
It created DNS TXT entry in route53 and then it disappeared , i was not able to run the che build.

ACME DNS01 validation record propagated for "_acme-challenge.xpresside.com."
I1018 12:39:15.005333       1 sync.go:307] cert-manager/controller/challenges/acceptChallenge "level"=0 "msg"="accepting challenge with ACME server" "dnsName"="xpresside.com" "resource_kind"="Challenge" "resource_name"="che-tls-4211818486-0" "resource_namespace"="che" "type"="dns-01" 
I1018 12:39:15.005356       1 logger.go:63] Calling AcceptChallenge
I1018 12:39:15.268953       1 sync.go:324] cert-manager/controller/challenges/acceptChallenge "level"=0 "msg"="waiting for authorization for domain" "dnsName"="xpresside.com" "resource_kind"="Challenge" "resource_name"="che-tls-4211818486-0" "resource_namespace"="che" "type"="dns-01" 
I1018 12:39:15.268971       1 logger.go:78] Calling WaitAuthorization
I1018 12:39:16.727511       1 controller.go:198] cert-manager/controller/orders "level"=0 "msg"="syncing resource" "key"="che/che-tls-4211818486" 
I1018 12:39:16.727681       1 sync.go:274] Need to create 0 challenges
I1018 12:39:16.727691       1 logger.go:43] Calling GetOrder
I1018 12:39:16.728711       1 controller.go:219] cert-manager/controller/challenges "level"=0 "msg"="finished processing work item" "key"="che/che-tls-4211818486-0" 
I1018 12:39:16.728743       1 controller.go:213] cert-manager/controller/challenges "level"=0 "msg"="syncing resource" "key"="che/che-tls-4211818486-0" 
I1018 12:39:16.819291       1 controller.go:167] cert-manager/controller/certificates "level"=0 "msg"="syncing resource" "key"="che/che-tls" 
I1018 12:39:16.819800       1 issue.go:169] cert-manager/controller/certificates "level"=0 "msg"="Order is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate." "related_resource_kind"="Order" "related_resource_name"="che-tls-4211818486" "related_resource_namespace"="che" "resource_kind"="Certificate" "resource_name"="che-tls" "resource_namespace"="che" 

My web server is (include version): we are trying to delpy eclipse che using kubernetes on aws

The operating system my web server runs on is (include version): MacOS

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

Hi @xpressbeta,

Welcome to the community forum!

I see you have successfully issued a wildcard certificate https://crt.sh/?id=2011772963 (this one is a precert, the final cert will eventually be logged).

Can you confirm if you have the following k8s resources in your tenants namespaces?

kubectl get Issuer -n ${tenant_namespace}
kubectl get Certificates -n ${tenant_namespace}

i was not able to run the che build.

What do you mean by this statement? Can you provide logs from that application and the k8s manifest used to configure it?

thanks Phil, i am actually following this document to build kops cluster on aws for eclipse che , https://www.eclipse.org/che/docs/che-7/deploying-che-on-kubernetes-on-aws/

I will send you the details of the requested, could u plz let me know how to reattach this certificate into the kops cluster certificate-namespace.
We are rebuilding the cluster , i will share the logs aswell.

Thank you

[ec2-user@ip-172-31-90-16 microchip]$ kubectl get Certificates -n che
NAME      READY   SECRET    AGE
che-tls   False   che-tls   1h
[ec2-user@ip-172-31-90-16 microchip]$ kubectl get Issuer che
Error from server (NotFound): issuers.certmanager.k8s.io "che" not found
[ec2-user@ip-172-31-90-16 microchip]$ kubectl get Issuer che-tls
Error from server (NotFound): issuers.certmanager.k8s.io "che-tls" not found

this is the above i get when i run your commands

I assume you have been following the install directions from https://github.com/jetstack/cert-manager/tree/master/deploy/charts/cert-manager to get the latest cert-manager v0.11.0 instead of using the old version from the Che install doc which lists v0.8.1?

Please run this command and paste the output. Specifically we want to check the status of the ACMEv2 order.

kubectl describe Certificates -n che
kubectl describe ClusterIssuer -n che

Please see my in-line notes on the commands you ran.

[ec2-user@ip-172-31-90-16 microchip]$ kubectl get Certificates -n che   <=== correct invocation
NAME      READY   SECRET    AGE
che-tls   False   che-tls   1h

[ec2-user@ip-172-31-90-16 microchip]$ kubectl get Issuer che            <===== are you using cert-manager in the 'default' namespace? If not a namespace should be specified here.
Error from server (NotFound): issuers.certmanager.k8s.io "che" not found

[ec2-user@ip-172-31-90-16 microchip]$ kubectl get Issuer che-tls        <===== are you using cert-manager in the 'default' namespace? If not a namespace should be specified here.
Error from server (NotFound): issuers.certmanager.k8s.io "che-tls" not found

Thanks Phil,

Thank you for help and support phil and apologies for this lengthy reply with all details , i just wanted you to get all required information to help nad trouble shoot this issue please - thanks

first the request out come :


sultans-MacBook-Pro-2:helm-kops devops$ kubectl describe Certificates -n che
Name: che-tls
Namespace: che
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{“apiVersion”:“certmanager.k8s.io/v1alpha1",“kind”:“Certificate”,“metadata”:{“annotations”:{},“name”:“che-tls”,“namespace”:“che”},"spec”:{…
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-10-23T18:16:56Z
Generation: 1
Resource Version: 5699
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/che/certificates/che-tls
UID: 4c22313b-f5c1-11e9-9119-02ab54321114
Spec:
Acme:
Config:
dns01:
Provider: route53
Domains:
.xpressbeta.com’
Dns Names:
.xpressbeta.com’
Issuer Ref:
Kind: ClusterIssuer
Name: che-certificate-issuer
Secret Name: che-tls
Status:
Conditions:
Last Transition Time: 2019-10-23T18:18:23Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events:
Type Reason Age From Message


Warning IssuerNotReady 11m (x3 over 12m) cert-manager Issuer che-certificate-issuer not ready
Normal Generated 11m cert-manager Generated new private key
Normal GenerateSelfSigned 11m cert-manager Generated temporary self signed certificate
Normal OrderCreated 11m cert-manager Created Order resource “che-tls-167678892”
sultans-MacBook-Pro-2:helm-kops devops$


Second output :-1:


sultans-MacBook-Pro-2:helm-kops devops$ kubectl describe ClusterIssuer -n che
Name: che-certificate-issuer
Namespace:
Labels:
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{“apiVersion”:"certmanager.k8s.io/v1alpha1",“kind”:“ClusterIssuer”,“metadata”:{“annotations”:{},“name”:“che-certificate-issuer”},“spec”:{"
API Version: certmanager.k8s.io/v1alpha1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2019-10-23T18:15:56Z
Generation: 1
Resource Version: 5695
Self Link: /apis/certmanager.k8s.io/v1alpha1/clusterissuers/che-certificate-issuer
UID: 287e6674-f5c1-11e9-9119-02ab54321114
Spec:
Acme:
dns01:
Providers:
Name: route53
route53:
Access Key ID: AKIA4ZR4YBFXJZR3PYVV
Region: eu-west-1
Secret Access Key Secret Ref:
Key: CLIENT_SECRET
Name: aws-cert-manager-access-key
Email: mplabxpress@gmail.com
Private Key Secret Ref:
Name: letsencrypt
Server: https://acme-v02.api.letsencrypt.org/directory
Status:
Acme:
Uri: https://acme-v02.api.letsencrypt.org/acme/acct/70108409
Conditions:
Last Transition Time: 2019-10-23T18:18:23Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events:


I followed the procedure but when i am trying to build the cert-manager helm chart using step 1 by applying manifests/00-crds.yaml - it gives me below error.

sultans-MacBook-Pro-2:~ devops$ kubectl apply \

-f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml

error: error validating “https://raw.githubusercontent.com/jetstack/cert-manager/release-0.11/deploy/manifests/00-crds.yaml”: error validating data: ValidationError(CustomResourceDefinition.spec.validation.openAPIV3Schema.properties.spec.properties.solver.properties.dns01.properties.webhook.properties.config): unknown field “x-kubernetes-preserve-unknown-fields” in io.k8s.apiextensions-apiserver.pkg.apis.apiextensions.v1beta1.JSONSchemaProps; if you choose to ignore these errors, turn validation off with --validate=false
sultans-MacBook-Pro-2:~ devops$

I also tried to run the same with --validation=false - however the response was same .

so then i searched for a lower version of the same 0.8 and ran the command , below is its output

: kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.8/deploy/manifests/00-crds.yaml

customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
—> next step

sultans-MacBook-Pro-2:~ devops$ helm repo add jetstack https://charts.jetstack.io

“jetstack” has been added to your repositories

When i ran : sultans-MacBook-Pro-2:helm-kops devops$ helm install --name xpressstaging --namespace cert-manager jetstack/cert-manager

Error: release xpressstaging failed: namespaces “cert-manager” is forbidden: User “system:serviceaccount:kube-system:default” cannot get namespaces in the namespace “cert-manager”

– I tried following instructions at: https://github.com/helm/helm/blob/240e539cec44e2b746b3541529d41f4ba01e77df/docs/rbac.md#Example-Service-account-with-cluster-admin-role

It still gives me the same error , so then follow the che document and give the kubectl apply -f to create certificate-manager and services YAML file.

After completion of th entire process i get the same error back :-

E1023 18:19:40.382419 1 controller.go:200] cert-manager/controller/orders “msg”=“re-queuing item due to error processing” “error”=“error creating new order: acme: urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for “‘.xpressbeta.com’": DNS name had a malformed wildcard label" “key”=“che/che-tls-167678892”
I1023 18:21:00.382700 1 controller.go:198] cert-manager/controller/orders “level”=0 “msg”=“syncing resource” “key”=“che/che-tls-167678892”
I1023 18:21:00.382875 1 logger.go:38] Calling CreateOrder
E1023 18:21:00.525464 1 controller.go:200] cert-manager/controller/orders “msg”=“re-queuing item due to error processing” “error”="error creating new order: acme: urn:ietf:params:acme:error:rejectedIdentifier: Error creating new order :: Cannot issue for "‘
.xpressbeta.com’”: DNS name had a malformed wildcard label” “key”=“che/che-tls-167678892”

Phile,

We build another cluster following the steps again and it does not work .

we ran into the same issue again :-1:

{
“type”: “urn:ietf:params:acme:error:malformed”,
“detail”: “Method not allowed”,
“status”: 405
}

phil/team

i was able to create certs manually by adding the TXT Records value into route 53 as the records were either not created when * wildcard was used in domains, or , if it did create it would have been disappeared after a few minutes.

i also worked on certbot to see to it that it created txt records manually.

Now the next issue is our application is still failing to see the txt record to build the containers and i am getting all false errors for cert ready parameter.


Problems:-

We are not able to create a certificate using * wild card.
We are able to create the certificate from certificate issuer using domain names.
The cert gets automatically cleared out of DNS after a few minutes.

Please advise further

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.