Would it be possible to know how long the rate limit would apply?

Feature Requests
1

As I previously had more than five subdomains for which I was using separate certificates, what was expected to happen finally happened: I was blocked because of the rate limit (possibly six or more certificates were renewed in the same week).

I successfully solved this problem by migrating to a wildcard certificate, going from a dozen certificates to just two: a wildcard one, and the one for the domain itself. It was about time.

However, this morning I learned that I hit the same limit again.

Both the time when I hit the limit the first time, and this morning, I found it unfortunate that the API doesn’t tell anything about the time it would take to be unblocked. I find myself with my primary website completely unavailable (because of the Strict-Transport-Security HTTP header, I can’t switch to HTTP, so my only option to have a working site right now is to switch to a paid CA), and more importantly, I have no clue whether I’ll have to wait for the whole week for my website to become available, or I will be able to renew the certificate before that.

According to the logs of certbot, the response from acme-v02.api.letsencrypt.org is simply:

{
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many certificates already issued for exact set of domains: ...: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429
}

and there is no indication as to how long would it take to be able to renew the expired certificate.

Would it be possible to indicate in the API response how long one would be rate-limited? So to have something like:

{
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many certificates already issued for exact set of domains: ...: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429,
  "can-renew-after-seconds": 603821
}
1 Like
2

Hi @arseni-mourzenko

five subdomains aren't a problem. Creating the same certificate 5 times per week is a problem.

If you have hitted that limit, you have five identical certificates. Why do you want to create the next certificate?

Please use one of these 60 - 85 days, then create the next.

If you hit the rate limit one time, that's not good. If you hit the rate limit again and again, your configuration is buggy.

That's not required. Read your log, there you see when you have created the certificates. If you don't have one, it's time to create one. And create a working backup, so you find your created certificates.

It's wrong adding HSTS without a working certificate renew configuration.

Where are all the valid certificates you have created? Deleted? Find one and use it.

1 Like
3

Thank you for your quick reply. However, a few things remain unclear for me:

Well, I don't want to create the next certificate. I just want to renew one which expired yesterday. In other words, I'm running certbot renew, not certbot certonly ...

It is indeed likely that there is something wrong with the configuration, although it seems pretty standard for me: run certbot renew once per day, then call nginx -s reload. I suppose that there was maybe a change in a recent version of certbot (maybe when it moved from ACME-v01 to ACME-v02?), so I have to double-check it. Thank you for your suggestion.

That, I know: the certificate was renewed the last time on March 30th. Recent updates possibly all failed due to the rate limits, and I ended up with a blocked domain.

The configuration worked well for years. However, I would agree with you if you say that it's wrong not to monitor the expiration of the certificates, and to blindly trust certbot, without even checking the logs, and waiting until the production monitoring tool tells that the website is not reachable any longer because the certificate expired. This was absolutely unprofessional from me.

certbot certificates lists only the last ones (the valid one with the wildcard, and the expired one for the site itself). From the backups, it looks like there are nineteen older certificates, but obviously, if the last one is expired, the previous ones expired as well.

So although you explained that there is probably a problem with my configuration, my original suggestion is still valid, I believe. I still would like to know whether I have to wait for the whole week to be able to renew the certificate (in which case it would possibly be a better solution to move to another CA, as 25% downtime for a month is a bit problematic), or not.

1 Like
4

Hi @arseni-mourzenko,

You could also potentially use the Buypass service, although I'm not sure it allows wildcards.

I agree that it would be useful for the API to provide this information.

The rate limit details are described at

and there is a useful tool which has worked well in the past to calculate your rate limit status based on previous issuance at

2 Likes
5

That's not relevant, both commands create a new certificate.

That would never hit a limit. If the certificate is longer then 30 days valid, that command doesn't renew it. If the certificate is older, it's renewed, nothing else.

That's not possible, that wouldn't hit a rate limit.

If you have really that rate limit, you have new created certificates. Then you should be able to find these and use one of these.

What's the domain name?

Runs there a second client? Is the error message really correct? Not the "failed validation" error message?

Your real error is unknown. Hitting the "created certificates" limit you must have certificates if you didn't delete these. Hitting the failed validation limit isn't a problem because that's only a one-hour-limit.

1 Like
6

Stating some of the above points a little differently…

The ratelimit you hit is for 5 duplicate certificates (see https://letsencrypt.org/docs/rate-limits/)

That means your cerbot instance successfully got 5 certificates for an identical set of domains within the last week. You should not need to renew a certificate for this set of domains nor should you have any downtime, because there should be 5 of them on your server that are valid for around 83 days (or more).

There is either:

  1. some configuration issue causing you to constantly re-request these and/or not deploy them
  2. some server issue where these certificates are being deleted/destroyed

If you have an issue where your server destroyed the certificates, you can get around the duplicate certificate ratelimit by either:

A) split the domains in the rate-limited certificate into 2 or more certificates
B) add one or more new domains to the certificate

3 Likes
7

Sometimes if people tamper with the files in /etc/letsencrypt/live, Certbot can get confused and not successfully save the results of a renewal. It should never be possible to exceed Let's Encrypt limits with certbot renew on an intact /etc/letsencrypt, but unfortunately it could be possible in some circumstances with a damaged or corrupted one.

1 Like
8

Nice! I may try it if I can't figure out how to get the certificate with Let's Encrypt.

Thanks! It confirms what Juergen was saying about the incorrect configuration on my side. lectl reports that there was a certificate created on June 29th, one on June 28th, two on June 27th, one June 26th, one on June 22th, etc. Since I none of those certificates were preserved locally by certbot, there is obviously something wrong going on on my server.

I suppose what happened is that certbot were requesting new certificates, but they were somehow lost after being created. So Let's Encrypt counted them in order to apply the rate limit, while the server is still stuck with the expired certificate.

It's pelicandd.com (the one which has an expired certificate). And there is a not yet expired wildcard certificate *.pelicandd.com.

You mean the error message from certbot certificates? It tells more precisely:

Expiry Date: 2020-06-28 05:25:24+00:00 (INVALID: EXPIRED)

As for the error which is shown when renewing the certificates, it's:

Attempting to renew cert (pelicandd.com) from /etc/letsencrypt/renewal/pelicandd.com.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: pelicandd.com: see https://letsencrypt.org/docs/rate-limits/. Skipping.  
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/pelicandd.com/fullchain.pem (failure)
1 Like
9

Could you post the output of ls -lR /etc/letsencrypt? Did you try to rename any of the files in /etc/letsencrypt in the past?

1 Like
10

I didn't rename any files, however I possibly removed some of the files when moving to the wildcard certificate (since I didn't need the old certificates any longer). Maybe this is the actual problem, and I had to remove the certificates through certbot instead.

I can't add attachments, as I'm a new user, so here is the copy-paste of the output of ls -lR /etc/letsencrypt; sorry, it's a bit long.

/etc/letsencrypt:
total 64
drwx------ 5 root root  4096 May 27 17:31 accounts
drwx------ 4 root root  4096 May 27 18:23 archive
-rw-r--r-- 1 root root   121 May 27 17:32 cli.ini
drwxr-xr-x 2 root root 20480 Jun 30 17:24 csr
drwx------ 2 root root 20480 Jun 30 17:24 keys
drwx------ 4 root root  4096 May 27 18:24 live
drwxr-xr-x 2 root root  4096 May 27 18:22 renewal
drwxr-xr-x 5 root root  4096 May 27 17:33 renewal-hooks

/etc/letsencrypt/accounts:
total 12
drwx------ 3 root root 4096 May 27 17:31 acme-staging.api.letsencrypt.org
drwx------ 3 root root 4096 May 27 17:31 acme-v01.api.letsencrypt.org
drwx------ 2 root root 4096 May 27 17:31 acme-v02.api.letsencrypt.org

/etc/letsencrypt/accounts/acme-staging.api.letsencrypt.org:
total 4
drwx------ 3 root root 4096 May 27 17:31 directory

/etc/letsencrypt/accounts/acme-staging.api.letsencrypt.org/directory:
total 4
drwx------ 2 root root 4096 May 27 17:31 bd48d6d7ffc8505d916cd80cb132226b

/etc/letsencrypt/accounts/acme-staging.api.letsencrypt.org/directory/bd48d6d7ffc8505d916cd80cb132226b:
total 12
-rw-r--r-- 1 root root   84 May 27 17:31 meta.json
-r-------- 1 root root 1632 May 27 17:31 private_key.json
-rw-r--r-- 1 root root  710 May 27 17:31 regr.json

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org:
total 4
drwx------ 3 root root 4096 May 27 17:31 directory

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory:
total 4
drwx------ 2 root root 4096 May 27 18:19 36e8444a852eb552c22d07ca2856e253

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/36e8444a852eb552c22d07ca2856e253:
total 12
-rw-r--r-- 1 root root   84 May 27 17:31 meta.json
-r-------- 1 root root 1632 May 27 17:31 private_key.json
-rw-r--r-- 1 root root  753 May 27 17:31 regr.json

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org:
total 4
lrwxrwxrwx 1 root root 64 May 27 17:31 directory -> /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory

/etc/letsencrypt/archive:
total 8
drwxr-xr-x 2 root root 4096 May 27 17:32 pelicandd.com
drwxr-xr-x 2 root root 4096 May 27 17:33 pelicandd.com-0001

/etc/letsencrypt/archive/pelicandd.com:
total 320
-rw-r--r-- 1 root root 2147 May 27 17:32 cert10.pem
-rw-r--r-- 1 root root 2151 May 27 17:32 cert11.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert12.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert13.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert14.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert15.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert16.pem
-rw-r--r-- 1 root root 1903 May 27 17:32 cert17.pem
-rw-r--r-- 1 root root 1903 May 27 17:32 cert18.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert19.pem
-rw-r--r-- 1 root root 1793 May 27 17:32 cert1.pem
-rw-r--r-- 1 root root 1903 May 27 17:32 cert20.pem
-rw-r--r-- 1 root root 1793 May 27 17:32 cert2.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert3.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert4.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert5.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert6.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert7.pem
-rw-r--r-- 1 root root 2151 May 27 17:32 cert8.pem
-rw-r--r-- 1 root root 2147 May 27 17:32 cert9.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain10.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain11.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain12.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain13.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain14.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain15.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain16.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain17.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain18.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain19.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain1.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain20.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain2.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain3.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain4.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain5.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain6.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain7.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain8.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain9.pem
-rw-r--r-- 1 root root 3794 May 27 17:32 fullchain10.pem
-rw-r--r-- 1 root root 3798 May 27 17:32 fullchain11.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain12.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain13.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain14.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain15.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain16.pem
-rw-r--r-- 1 root root 3550 May 27 17:32 fullchain17.pem
-rw-r--r-- 1 root root 3550 May 27 17:32 fullchain18.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain19.pem
-rw-r--r-- 1 root root 3440 May 27 17:32 fullchain1.pem
-rw-r--r-- 1 root root 3550 May 27 17:32 fullchain20.pem
-rw-r--r-- 1 root root 3440 May 27 17:32 fullchain2.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain3.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain4.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain5.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain6.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain7.pem
-rw-r--r-- 1 root root 3798 May 27 17:32 fullchain8.pem
-rw-r--r-- 1 root root 3794 May 27 17:32 fullchain9.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey10.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey11.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey12.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey13.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey14.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey15.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey16.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey17.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey18.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey19.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey1.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey20.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey2.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey3.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey4.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey5.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey6.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey7.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey8.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey9.pem

/etc/letsencrypt/archive/pelicandd.com-0001:
total 16
-rw-r--r-- 1 root root 1911 May 27 17:33 cert1.pem
-rw-r--r-- 1 root root 1647 May 27 17:33 chain1.pem
-rw-r--r-- 1 root root 3558 May 27 17:33 fullchain1.pem
-rw-r--r-- 1 root root 1708 May 27 17:33 privkey1.pem

/etc/letsencrypt/csr:
total 1344
-rw-r--r-- 1 root root 960 May 27 17:31 0000_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0001_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0002_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0003_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0004_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0005_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0006_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0007_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0008_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0009_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0010_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0011_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0012_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0013_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0014_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0015_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0016_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0017_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0018_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0019_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0020_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0021_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0022_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0023_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0024_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0025_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0026_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0027_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0028_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0029_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0030_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0031_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0032_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0033_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0034_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0035_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0036_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0037_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0038_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0039_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0040_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0041_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0042_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0043_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0044_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0045_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0046_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0047_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0048_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0049_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0050_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0051_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0052_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0053_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0054_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0055_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0056_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0057_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0058_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0059_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0060_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0061_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0062_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0063_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0064_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0065_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0066_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0067_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0068_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0069_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0070_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0071_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0072_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0073_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0074_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0075_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0076_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0077_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0078_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0079_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0080_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0081_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0082_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0083_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0084_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0085_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0086_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0087_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0088_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0089_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0090_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0091_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0092_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0093_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0094_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0095_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0096_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0097_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0098_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0099_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0100_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0101_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0102_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0103_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0104_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0105_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0106_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0107_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0108_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0109_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0110_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0111_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0112_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0113_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0114_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0115_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0116_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0117_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0118_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0119_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0120_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0121_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0122_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0123_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0124_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0125_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0126_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0127_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0128_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0129_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0130_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0131_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0132_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0133_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0134_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0135_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0136_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0137_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0138_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0139_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0140_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0141_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0142_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0143_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0144_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0145_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0146_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0147_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0148_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0149_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0150_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0151_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0152_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0153_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0154_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0155_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0156_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0157_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0158_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0159_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0160_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0161_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0162_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0163_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0164_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0165_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0166_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0167_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0168_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0169_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0170_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0171_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0172_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0173_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0174_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0175_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0176_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0177_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0178_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0179_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0180_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0181_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0182_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0183_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0184_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0185_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0186_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0187_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0188_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0189_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0190_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0191_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0192_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0193_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0194_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0195_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0196_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0197_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0198_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0199_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0200_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0201_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0202_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0203_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0204_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0205_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0206_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0207_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0208_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0209_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0210_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0211_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0212_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0213_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0214_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0215_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0216_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0217_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0218_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0219_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0220_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0221_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0222_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0223_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0224_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0225_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0226_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0227_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0228_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0229_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0230_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0231_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0232_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0233_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0234_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0235_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0236_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0237_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0238_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0239_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0240_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0241_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0242_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0243_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0244_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0245_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0246_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0247_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0248_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0249_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0250_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0251_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0252_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0253_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0254_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0255_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0256_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0257_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0258_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0259_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0260_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0261_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0262_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0263_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0264_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0265_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0266_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0267_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0268_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0269_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0270_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0271_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0272_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0273_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0274_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0275_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0276_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0277_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0278_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0279_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0280_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0281_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0282_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0283_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0284_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0285_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0286_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0287_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0288_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0289_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0290_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0291_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0292_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0293_csr-certbot.pem
-rw-r--r-- 1 root root 948 May 27 17:31 0294_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0295_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0296_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0297_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0298_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0299_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0300_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0301_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0302_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0303_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0304_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0305_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0306_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0307_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0308_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0309_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0310_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0311_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0312_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0313_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0314_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0315_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0316_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0317_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0318_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0319_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0320_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0321_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0322_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0323_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0324_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0325_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0326_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0327_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0328_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0329_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0330_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0331_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0332_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0333_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0334_csr-certbot.pem
-rw-r--r-- 1 root root 924 Jun 30 17:24 0335_csr-certbot.pem
11 
/etc/letsencrypt/keys:
total 1344
-rw------- 1 root root 1704 May 27 17:32 0000_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0001_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0002_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0003_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0004_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0005_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0006_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0007_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0008_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0009_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0010_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0011_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0012_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0013_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0014_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0015_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0016_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0017_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0018_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0019_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0020_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0021_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0022_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0023_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0024_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0025_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0026_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0027_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0028_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0029_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0030_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0031_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0032_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0033_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0034_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0035_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0036_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0037_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0038_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0039_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0040_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0041_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0042_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0043_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0044_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0045_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0046_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0047_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0048_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0049_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0050_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0051_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0052_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0053_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0054_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0055_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0056_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0057_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0058_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0059_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0060_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0061_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0062_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0063_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0064_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0065_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0066_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0067_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0068_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0069_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0070_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0071_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0072_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0073_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0074_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0075_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0076_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0077_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0078_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0079_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0080_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0081_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0082_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0083_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0084_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0085_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0086_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0087_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0088_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0089_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0090_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0091_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0092_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0093_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0094_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0095_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0096_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0097_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0098_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0099_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0100_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0101_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0102_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0103_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0104_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0105_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0106_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0107_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0108_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0109_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0110_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0111_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0112_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0113_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0114_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0115_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0116_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0117_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0118_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0119_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0120_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0121_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0122_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0123_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0124_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0125_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0126_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0127_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0128_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0129_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0130_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0131_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0132_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0133_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0134_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0135_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0136_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0137_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0138_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0139_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0140_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0141_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0142_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0143_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0144_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0145_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0146_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0147_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0148_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0149_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0150_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0151_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0152_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0153_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0154_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0155_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0156_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0157_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0158_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0159_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0160_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0161_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0162_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0163_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0164_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0165_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0166_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0167_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0168_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0169_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0170_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0171_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0172_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0173_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0174_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0175_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0176_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0177_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0178_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0179_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0180_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0181_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0182_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0183_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0184_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0185_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0186_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0187_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0188_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0189_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0190_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0191_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0192_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0193_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0194_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0195_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0196_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0197_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0198_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0199_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0200_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0201_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0202_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0203_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0204_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0205_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0206_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0207_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0208_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0209_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0210_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0211_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0212_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0213_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0214_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0215_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0216_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0217_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0218_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0219_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0220_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0221_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0222_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0223_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0224_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0225_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0226_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0227_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0228_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0229_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0230_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0231_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0232_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0233_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0234_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0235_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0236_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0237_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0238_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0239_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0240_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0241_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0242_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0243_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0244_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0245_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0246_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0247_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0248_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0249_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0250_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0251_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0252_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0253_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0254_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0255_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0256_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0257_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0258_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0259_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0260_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0261_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0262_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0263_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0264_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0265_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0266_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0267_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0268_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0269_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0270_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0271_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0272_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0273_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0274_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0275_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0276_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0277_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0278_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0279_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0280_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0281_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0282_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0283_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0284_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0285_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0286_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0287_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0288_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0289_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0290_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0291_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0292_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0293_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0294_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0295_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0296_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0297_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0298_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0299_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0300_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0301_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0302_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0303_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0304_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0305_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0306_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0307_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0308_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0309_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0310_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0311_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0312_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0313_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0314_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0315_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0316_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0317_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0318_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0319_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0320_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0321_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0322_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0323_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0324_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0325_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0326_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0327_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0328_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0329_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0330_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0331_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0332_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0333_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0334_key-certbot.pem
-rw------- 1 root root 1704 Jun 30 17:24 0335_key-certbot.pem

/etc/letsencrypt/live:
total 12
drwxr-xr-x 2 root root 4096 May 27 17:33 pelicandd.com
drwxr-xr-x 2 root root 4096 May 27 17:33 pelicandd.com-0001
-rw-r--r-- 1 root root  740 May 27 17:33 README

/etc/letsencrypt/live/pelicandd.com:
total 0
lrwxrwxrwx 1 root root 38 May 27 17:33 cert.pem -> ../../archive/pelicandd.com/cert20.pem
lrwxrwxrwx 1 root root 39 May 27 17:33 chain.pem -> ../../archive/pelicandd.com/chain20.pem
lrwxrwxrwx 1 root root 43 May 27 17:33 fullchain.pem -> ../../archive/pelicandd.com/fullchain20.pem
lrwxrwxrwx 1 root root 41 May 27 17:33 privkey.pem -> ../../archive/pelicandd.com/privkey20.pem

/etc/letsencrypt/live/pelicandd.com-0001:
total 4
lrwxrwxrwx 1 root root  42 May 27 17:33 cert.pem -> ../../archive/pelicandd.com-0001/cert1.pem
lrwxrwxrwx 1 root root  43 May 27 17:33 chain.pem -> ../../archive/pelicandd.com-0001/chain1.pem
lrwxrwxrwx 1 root root  47 May 27 17:33 fullchain.pem -> ../../archive/pelicandd.com-0001/fullchain1.pem
lrwxrwxrwx 1 root root  45 May 27 17:33 privkey.pem -> ../../archive/pelicandd.com-0001/privkey1.pem
-rw-r--r-- 1 root root 692 May 27 17:33 README

/etc/letsencrypt/renewal:
total 8
-rw-r--r-- 1 root root 587 May 27 17:31 pelicandd.com-0001.conf
-rw-r--r-- 1 root root 565 May 27 17:31 pelicandd.com.conf

/etc/letsencrypt/renewal-hooks:
total 12
drwxr-xr-x 2 root root 4096 May 27 17:33 deploy
drwxr-xr-x 2 root root 4096 May 27 17:33 post
drwxr-xr-x 2 root root 4096 May 27 17:33 pre

/etc/letsencrypt/renewal-hooks/deploy:
total 0

/etc/letsencrypt/renewal-hooks/post:
total 0

/etc/letsencrypt/renewal-hooks/pre:
total 0
12

Thanks, could we also see the full output of certbot certificates?

It’s odd that the certificates would be expired now if they were created on May 27. Did you copy this tree from a different machine at some point?

1 Like
13

There is really a mess - see https://crt.sh/?q=pelicandd.com

Not only the main domain, same with subdomains, daily created certificates.

Do you use force-renew?

1 Like
14

Here it is:

# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
OCSP check failed for /etc/letsencrypt/live/pelicandd.com/cert.pem (are we offline?)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: pelicandd.com-0001
    Domains: *.pelicandd.com
    Expiry Date: 2020-08-25 16:22:14+00:00 (VALID: 55 days)
    Certificate Path: /etc/letsencrypt/live/pelicandd.com-0001/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/pelicandd.com-0001/privkey.pem
  Certificate Name: pelicandd.com
    Domains: pelicandd.com
    Expiry Date: 2020-06-28 05:25:24+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/pelicandd.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/pelicandd.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

So:

  1. There are two Nginx reverse proxies working in failover.
  2. One of them is in charge of renewing the certificates. It has a cron job, which calls certbot renew, then copies the local directory containing the private certificates to an NFS share, and finally reloads Nginx.
  3. The other one has another cron job, which once per day gets the certificates from the NFS share, and reloads Nginx.
  4. Both machines are occasionally destroyed and rebuilt from scratch. When they are rebuilt, they get the certificates from the NFS share. They were rebuilt four times today, once yesterday, but not the days before, if I remember well.

It is a mess, indeed, and if I understand correctly, the mess started in November 2019, so a long time ago. No, I don't use force-renew. I suppose the problem is indeed that my certbot is misconfigured, and just discards the certificates it created. I'll see tomorrow if I can figure it out.

15

Hmmm! So, could you explain a bit more about the NFS share thing? I’m wondering if the pelicandd.com base domain certificate got renewed in a way where the resulting certificates weren’t preserved. The creation time on the certificate files says May 27 for all of them, but I guess that isn’t necessarily accurate in this NFS setup?

1 Like
16

Really copy? Or move? Move - your certbot doesn't find the new certificate, instead the old -> next certificate is created.

Or the cron job runs on both machines.

There are a lot of errors possible.

1 Like
17

One easy way this can go wrong is if you use rsync to copy the certificates to NFS, which will by default dereference symlinks, instead of copying the symlink. You need to use -L in order not to break Certbot, otherwise you end up with renewals every day.

A /var/log/letsencrypt/ log from a renewal that ran unnecessarily might reveal the reason a renewal ran as well.

Probably you have figured out the date by now, but I made this web-based tool to allow quickly checking when a duplicate certificate rate limit is due to expire.

1 Like
18

Wouldn’t we potentially get an “expected X to be a symlink” error when trying to renew in that case? (I haven’t tested this with NFS so I’m not sure, I just remember that Certbot has some code related to checking for this.)

1 Like
19

You’re right, good point. Gave it a test and it just reports the renewal config as broken. Not sure why I remembered otherwise.

20

You do realise you could put those two hostnames in the same certificate, right?

Even is a way to circumvent the current rate limit, once you've got your certbot fixed :wink:

1 Like