4-5 minute delay when getting certificate

Help
1

My domain is:
carroll-square.com is one of the domains. This is happening for multiple domains

I ran this command:
certbot certonly --webroot -w /mnt/fsetsprodzcusptl01/websites/carroll-square.com/httpdocs -d carroll-square.com -w /mnt/fsetsprodzcusptl01/websites/carroll-square.com/httpdocs -d www.carroll-square.com

It produced this output:
It got the certificate successfully, but it takes 4 -5 minutes

My web server is (include version):
nginx 1.21.6

The operating system my web server runs on is (include version):
Ubuntu 18.04.6

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.27.0

The portion of the logs where this is slow looks like this
Between the lines "Received response" and "Requesting a certificate" there is about a 4 minute delay.
This happens for all our domains.

2022-05-12 20:05:16,644:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2022-05-12 20:05:16,644:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Thu, 12 May 2022 20:05:16 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"i9TpIUJJ87E": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2022-05-12 20:09:50,428:DEBUG:certbot._internal.display.obj:Notifying user: Requesting a certificate for carroll-square.com and www.carroll-square.com
2022-05-12 20:10:02,132:DEBUG:certbot.crypto_util:Generating RSA key (2048 bits): /mnt/fsetsprodzcusptl01/certificates/keys/1018_key-certbot.pem

2

Hello @mattdyer

Do you still need help? I ask because I do not see A or AAAA records for either of those domain names in the public DNS system. Neither does Let's Debug

nslookup  carrol-square.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53
** server can't find carrol-square.com: NXDOMAIN

nslookup  www.carrol-square.com 8.8.8.8
Server:         8.8.8.8
Address:        8.8.8.8#53
** server can't find www.carrol-square.com: NXDOMAIN
9 Likes
3

Root DNS servers don't know about that domain:

nslookup -q=ns carrol-square.com a.gtld-servers.net
Server:  UnKnown
Address: 192.5.6.30
*** UnKnown can't find carrol-square.com: Non-existent domain
8 Likes
4

carrol-square.com WHOIS domain registration information from Network Solutions

9 Likes
5

I concur :slight_smile:

nslookup -q=ns carrol-square.com a.gtld-servers.net
Server:		a.gtld-servers.net
Address:	192.5.6.30#53

** server can't find carrol-square.com: NXDOMAIN

root:svr ~ >> host carrol-square.com
Host carrol-square.com not found: 3(NXDOMAIN)
root:svr ~ >> nslookup carrol-square.com
Server:		127.0.0.53
Address:	127.0.0.53#53

** server can't find carrol-square.com: NXDOMAIN
7 Likes
6

Sorry, at the top in of the question I typed those answers in manually. The domain is actually carroll-square.com. With 2 L's. You can see the correct domain in the log that I pasted in. I also have many other domains that have had this same issue. I can provide some others if needed. Mainly I am just wondering what it is doing between those 2 lines in the log where there is the 4 minute gap.

Thank you

2 Likes
7

Thank you.


 http://carroll-square.com/
HTTP/1.1 301 Moved Permanently
Content-Length: 150
Content-Type: text/html; charset=UTF-8
Location: https://carroll-square.com/
Server: Microsoft-IIS/10.0
Content-Security-Policy: default-src https: 'unsafe-inline' 'unsafe-eval'; font-src * data:;
Referrer-Policy: strict-origin
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Date: Fri, 13 May 2022 01:13:46 GMT

host carroll-square.com
carroll-square.com has address 13.86.127.241
9 Likes
8

It seems like the output of your certbot request was somehow truncated. (in the post)
Would you please post the complete output (if it is not to late)?

9 Likes
9

I corrected all four five instances of "carrol-" with "carroll-" in your OP.

Now on to the actual problem...

This command:

Doesn't seem to lineup with:

Which version of certbot are you running?
Did you deviate from its' default installation, or operation, in any way?

8 Likes
10

I'm using version 1.27.0 of certbot.
I have a cli.ini file in /etc/letsencrypt. It has one line

config-dir = /mnt/fsetsprodzcusptl01/certificates

This changes where the certificates are stored because they are eventually going to be used by multiple servers.

11

Here is a full log for one of the domains. This is for a different one, but they all seem to have the same delay. This is for bryantparkny.com. It has a delay at the same place from 19:17:37 to 19:22:25.

letsencrypt.txt (44.7 KB)

12

I see IIS/10.
Is there a reason why you are using certbot?

7 Likes
13

Right now the sites are being served from Windows servers with IIS. We are switching over to linux servers, so I am trying to get certificates setup with certbot so they will be ready before we switch over. We have about 1500 sites so if there is a better/faster solution to this other than certbot then I would be happy to try something else.

Thank You.

1 Like
14

On our windows servers we have been using Certify The Web for several years.

15

If the names are still pointing to another system, then you will need to use DNS authentication.
HTTP authentication must reach the system making the request (via HTTP).

8 Likes
16

Agree but the request succeeds. There is just a very long delay right after the initial outbound request to .../directory

It's a mystery to me why it even works as I don't see how the challenge file gets on the right system. But lo and behold it does.

Perhaps the delay is related to this entry in the log? Like maybe it takes a long time to access the mnt point at the start? I was just starting to read certbot code but this is probably best handled by the engineers

2022-05-12 19:22:52,346:INFO:certbot._internal.client:Non-standard path(s), might not work with crontab installed by your operating system package manager
2022-05-12 19:22:52,431:DEBUG:certbot._internal.storage:Creating directory /mnt/fsetsprodzcusptl01/certificates/archive/bryantparkny.com.
9 Likes
17

The sites are being served from a shared directory, so the server that is running certbot is adding the verification file to the shared folder that is then served by the Windows servers. That will change eventually when the Windows servers are removed, but for now it seems to be working. It just has this delay. If I have to deal with the delay it is okay, I was just hoping there would be some simple solution.

2 Likes
18

It is possible that this is caused by slow access to our mnt. If between those 2 log entries with the delay it is doing a lot of file activity it could be the case.

1 Like
19

That sort of delay is not seen normally. I can only guess it is environmental with your shared folders. Perhaps even involving snap. I am not skilled enough with certbot code to say. Maybe @_az will have insight.

Any more info about how your Ubuntu system accesses that shared system with IIS would probably be helpful.

8 Likes
20

The only thing Certbot really does between those two points is:

  1. Enumerate the existing certificates in /etc/letsencrypt/renewal/
    • Could be slow if you have a lot of certificates and very poor i/o performance for small files
  2. Run pre-hooks
    • You don't appear to have any

If you like, you can record an strace:

snap run --strace="-t -s 1024 -o certbot.strace" certbot certonly --webroot \
-w /mnt/fsetsprodzcusptl01/websites/carroll-square.com/httpdocs -d carroll-square.com \
--dry-run --non-interactive

and then email it to me at "az @ letsdebug.net". The trace might contain some private key material, which is why I recommend you use --dry-run to ensure you don't accidentally send me your live ACME account key. You can otherwise try to analyze the trace yourself.

9 Likes