Letsencrypt DCOS failed renew certificates, please help

[wenbolovesnz]

I have a DC/OS (V1.9.0) running on Azure. Marathon V 1.4.2.

The Letencrypt DC/OS has been renewing the certificates in the past without issue until now it is not renewing and the certificates will be expired in 5 days.

My domain is: estimating.fletcherinnovation.com

I ran this command:
dcos marathon app add letsencrypt-dcos.json

Then I can see that deployment was taken place, and it tries to regenerate and renew the certificates.
But then it fails with the error message below:

It produced this output:

NOTES:
 - If you lose your account credentials, you can recover through
   e-mails sent to digital.developer@fbu.com.
 - The following errors were reported by the server:

   Domain: estimating.fletcherinnovation.com
   Type:   unauthorized
   Detail: Invalid response from
   http://estimating.fletcherinnovation.com/.well-known/acme-challenge/7-uu0vj7wlTcnwkd52SAs9K_-CsDoy6Lle9Whkf33eM
   [52.187.235.74]: 503

   Domain: fbwebprod001.fletcherinnovation.com
   Type:   unauthorized
   Detail: Invalid response from
   http://fbwebprod001.fletcherinnovation.com/.well-known/acme-challenge/UdiK8HMk0avn953Awy6jdi6aY_VKcmwFDK4Xi80gwfc
   [52.187.235.74]: 503

   Domain: api.parse.fletcherinnovation.com
   Type:   unauthorized
   Detail: Invalid response from
   http://api.parse.fletcherinnovation.com/.well-known/acme-challenge/CegdIygwD16fZi5CiJM3bArmEV7UBamiEbVT9NXu8zY
   [52.187.235.74]: 503

   Domain: jenkins002.fletcherinnovation.com
   Type:   unauthorized
   Detail: Invalid response from
   http://jenkins002.fletcherinnovation.com/.well-known/acme-challenge/rJndFIsA9nydl61bInXMxE3X68erwHj6u9LH47aTtF4
   [52.187.235.74]: 503

   Domain: api.roofing.fletcherinnovation.com
   Type:   unauthorized
   Detail: Invalid response from
   http://api.roofing.fletcherinnovation.com/.well-known/acme-challenge/1Obkc-FMLR_uo8dXMPcWYAMLK77AiyaWq6NvgeVg9Rw
   [52.187.235.74]: 503

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is: Azure

I can login to a root shell on my machine (yes or no, or I don’t know):
Yes, I can login to root.

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
I can use DCOS UI, or I can also use shell.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): I am not sure about this, as we used the letsencrypt dcos to do the work.

Please help out. and I might be missed some information here. so please if you think I didn’t add enough information to this.

Cheers

[_az]

Are you able to inspect the running config of your marathon-lb haproxy(s)?

It seems like it has been instrumented successfully to intercept the requests to /.well-known/acme-challenge*, but not sending them to the right backend (which should be letsencrypt-dcos).

Perhaps if you take a look at the haproxy.cfg that is currently loaded, you can identify where the breakdown is occurring.

[wenbolovesnz]

Thank you very much _az for your help. Could you tell me where I can find the haproxy.cfg file? I am new to DC/OS and Letsencrypt.

[_az]

I’m not a DC/OS user so I don’t know the answer.

Perhaps you can use the Configuration endpoint shown here to fetch it - https://docs.mesosphere.com/services/marathon-lb/mlb-reference/

I am not sure this forum is the best place to find expertise about your issue - the Mesos mailing lists or issue trackers may be more useful. I can try to help you anyway but my guidance will be vague.

[wenbolovesnz]

Thanks for you help anyway, I will look into the resources you shared. Cheers.