Would it be possible to know how long the rate limit would apply?

[JuergenAuer]

Hi @arseni-mourzenko

@arseni-mourzenko

As I previously had more than five subdomains for which I was using separate certificates, what was expected to happen finally happened: I was blocked because of the rate limit (possibly six or more certificates were renewed in the same week).

five subdomains aren’t a problem. Creating the same certificate 5 times per week is a problem.

@arseni-mourzenko

However, this morning I learned that I hit the same limit again.

If you have hitted that limit, you have five identical certificates. Why do you want to create the next certificate?

Please use one of these 60 - 85 days, then create the next.

If you hit the rate limit one time, that’s not good. If you hit the rate limit again and again, your configuration is buggy.

@arseni-mourzenko

I found it unfortunate that the API doesn’t tell anything about the time it would take to be unblocked.

That’s not required. Read your log, there you see when you have created the certificates. If you don’t have one, it’s time to create one. And create a working backup, so you find your created certificates.

@arseni-mourzenko

I find myself with my primary website completely unavailable (because of the Strict-Transport-Security HTTP header, I can’t switch to HTTP

It’s wrong adding HSTS without a working certificate renew configuration.

@arseni-mourzenko

so my only option to have a working site right now is to switch to a paid CA

Where are all the valid certificates you have created? Deleted? Find one and use it.

[arseni-mourzenko]

Thank you for your quick reply. However, a few things remain unclear for me:

@JuergenAuer

If you have hitted that limit, you have five identical certificates . Why do you want to create the next certificate?

Well, I don’t want to create the next certificate. I just want to renew one which expired yesterday. In other words, I’m running certbot renew, not certbot certonly ...

@JuergenAuer

If you hit the rate limit one time, that’s not good. If you hit the rate limit again and again, your configuration is buggy.

It is indeed likely that there is something wrong with the configuration, although it seems pretty standard for me: run certbot renew once per day, then call nginx -s reload. I suppose that there was maybe a change in a recent version of certbot (maybe when it moved from ACME-v01 to ACME-v02?), so I have to double-check it. Thank you for your suggestion.

@JuergenAuer

That’s not required. Read your log, there you see when you have created the certificates. If you don’t have one, it’s time to create one. And create a working backup, so you find your created certificates.

That, I know: the certificate was renewed the last time on March 30^th. Recent updates possibly all failed due to the rate limits, and I ended up with a blocked domain.

@JuergenAuer

It’s wrong adding HSTS without a working certificate renew configuration.

The configuration worked well for years. However, I would agree with you if you say that it’s wrong not to monitor the expiration of the certificates, and to blindly trust certbot, without even checking the logs, and waiting until the production monitoring tool tells that the website is not reachable any longer because the certificate expired. This was absolutely unprofessional from me.

@JuergenAuer

Where are all the valid certificates you have created? Deleted? Find one and use it.

certbot certificates lists only the last ones (the valid one with the wildcard, and the expired one for the site itself). From the backups, it looks like there are nineteen older certificates, but obviously, if the last one is expired, the previous ones expired as well.

So although you explained that there is probably a problem with my configuration, my original suggestion is still valid, I believe. I still would like to know whether I have to wait for the whole week to be able to renew the certificate (in which case it would possibly be a better solution to move to another CA, as 25% downtime for a month is a bit problematic), or not.

[schoen]

Hi @arseni-mourzenko,

@arseni-mourzenko

so my only option to have a working site right now is to switch to a paid CA

You could also potentially use the Buypass service, although I’m not sure it allows wildcards.

https://www.buypass.com/ssl/products/acme

I agree that it would be useful for the API to provide this information.

The rate limit details are described at

Rate Limits - Let's Encrypt - Free SSL/TLS Certificates

and there is a useful tool which has worked well in the past to calculate your rate limit status based on previous issuance at

sahsanu/lectl

[JuergenAuer]

@arseni-mourzenko

In other words, I’m running certbot renew , not certbot certonly ...

That’s not relevant, both commands create a new certificate.

@arseni-mourzenko

certbot renew once

That would never hit a limit. If the certificate is longer then 30 days valid, that command doesn’t renew it. If the certificate is older, it’s renewed, nothing else.

@arseni-mourzenko

That, I know: the certificate was renewed the last time on March 30th.

That’s not possible, that wouldn’t hit a rate limit.

@arseni-mourzenko

Recent updates possibly all failed due to the rate limits, and I ended up with a blocked domain.

If you have really that rate limit, you have new created certificates. Then you should be able to find these and use one of these.

What’s the domain name?

@arseni-mourzenko

certbot certificates lists only the last ones (the valid one with the wildcard, and the expired one for the site itself).

Runs there a second client? Is the error message really correct? Not the “failed validation” error message?

@arseni-mourzenko

I still would like to know whether I have to wait for the whole week to be able to renew the certificate (in which case it would possibly be a better solution to move to another CA, as 25% downtime for a month is a bit problematic), or not.

Your real error is unknown. Hitting the “created certificates” limit you must have certificates if you didn’t delete these. Hitting the failed validation limit isn’t a problem because that’s only a one-hour-limit.

[jvanasco]

Stating some of the above points a little differently…

The ratelimit you hit is for 5 duplicate certificates (see https://letsencrypt.org/docs/rate-limits/)

That means your cerbot instance successfully got 5 certificates for an identical set of domains within the last week. You should not need to renew a certificate for this set of domains nor should you have any downtime, because there should be 5 of them on your server that are valid for around 83 days (or more).

There is either:

1. some configuration issue causing you to constantly re-request these and/or not deploy them
2. some server issue where these certificates are being deleted/destroyed

If you have an issue where your server destroyed the certificates, you can get around the duplicate certificate ratelimit by either:

A) split the domains in the rate-limited certificate into 2 or more certificates
B) add one or more new domains to the certificate

[schoen]

@JuergenAuer

That would never hit a limit. If the certificate is longer then 30 days valid, that command doesn’t renew it. If the certificate is older, it’s renewed, nothing else.

Sometimes if people tamper with the files in /etc/letsencrypt/live, Certbot can get confused and not successfully save the results of a renewal. It should never be possible to exceed Let’s Encrypt limits with certbot renew on an intact /etc/letsencrypt, but unfortunately it could be possible in some circumstances with a damaged or corrupted one.

[arseni-mourzenko]

@schoen

You could also potentially use the Buypass service, although I’m not sure it allows wildcards.

Nice! I may try it if I can’t figure out how to get the certificate with Let’s Encrypt.

@schoen

and there is a useful tool which has worked well in the past to calculate your rate limit status based on previous issuance at

Thanks! It confirms what Juergen was saying about the incorrect configuration on my side. lectl reports that there was a certificate created on June 29^th, one on June 28^th, two on June 27^th, one June 26^th, one on June 22^th, etc. Since I none of those certificates were preserved locally by certbot, there is obviously something wrong going on on my server.

@JuergenAuer

That’s not possible, that wouldn’t hit a rate limit.

I suppose what happened is that certbot were requesting new certificates, but they were somehow lost after being created. So Let’s Encrypt counted them in order to apply the rate limit, while the server is still stuck with the expired certificate.

@JuergenAuer

What’s the domain name?

It’s pelicandd.com (the one which has an expired certificate). And there is a not yet expired wildcard certificate *.pelicandd.com.

@JuergenAuer

Is the error message really correct? Not the “failed validation” error message?

You mean the error message from certbot certificates? It tells more precisely:

Expiry Date: 2020-06-28 05:25:24+00:00 (INVALID: EXPIRED)

As for the error which is shown when renewing the certificates, it’s:

Attempting to renew cert (pelicandd.com) from /etc/letsencrypt/renewal/pelicandd.com.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many certificates already issued for exact set of domains: pelicandd.com: see https://letsencrypt.org/docs/rate-limits/. Skipping.  
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/pelicandd.com/fullchain.pem (failure)

[schoen]

Could you post the output of ls -lR /etc/letsencrypt? Did you try to rename any of the files in /etc/letsencrypt in the past?

[arseni-mourzenko]

@schoen

Could you post the output of ls -lR /etc/letsencrypt ? Did you try to rename any of the files in /etc/letsencrypt in the past?

I didn’t rename any files, however I possibly removed some of the files when moving to the wildcard certificate (since I didn’t need the old certificates any longer). Maybe this is the actual problem, and I had to remove the certificates through certbot instead.

I can’t add attachments, as I’m a new user, so here is the copy-paste of the output of ls -lR /etc/letsencrypt; sorry, it’s a bit long.

/etc/letsencrypt:
total 64
drwx------ 5 root root  4096 May 27 17:31 accounts
drwx------ 4 root root  4096 May 27 18:23 archive
-rw-r--r-- 1 root root   121 May 27 17:32 cli.ini
drwxr-xr-x 2 root root 20480 Jun 30 17:24 csr
drwx------ 2 root root 20480 Jun 30 17:24 keys
drwx------ 4 root root  4096 May 27 18:24 live
drwxr-xr-x 2 root root  4096 May 27 18:22 renewal
drwxr-xr-x 5 root root  4096 May 27 17:33 renewal-hooks

/etc/letsencrypt/accounts:
total 12
drwx------ 3 root root 4096 May 27 17:31 acme-staging.api.letsencrypt.org
drwx------ 3 root root 4096 May 27 17:31 acme-v01.api.letsencrypt.org
drwx------ 2 root root 4096 May 27 17:31 acme-v02.api.letsencrypt.org

/etc/letsencrypt/accounts/acme-staging.api.letsencrypt.org:
total 4
drwx------ 3 root root 4096 May 27 17:31 directory

/etc/letsencrypt/accounts/acme-staging.api.letsencrypt.org/directory:
total 4
drwx------ 2 root root 4096 May 27 17:31 bd48d6d7ffc8505d916cd80cb132226b

/etc/letsencrypt/accounts/acme-staging.api.letsencrypt.org/directory/bd48d6d7ffc8505d916cd80cb132226b:
total 12
-rw-r--r-- 1 root root   84 May 27 17:31 meta.json
-r-------- 1 root root 1632 May 27 17:31 private_key.json
-rw-r--r-- 1 root root  710 May 27 17:31 regr.json

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org:
total 4
drwx------ 3 root root 4096 May 27 17:31 directory

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory:
total 4
drwx------ 2 root root 4096 May 27 18:19 36e8444a852eb552c22d07ca2856e253

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory/36e8444a852eb552c22d07ca2856e253:
total 12
-rw-r--r-- 1 root root   84 May 27 17:31 meta.json
-r-------- 1 root root 1632 May 27 17:31 private_key.json
-rw-r--r-- 1 root root  753 May 27 17:31 regr.json

/etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org:
total 4
lrwxrwxrwx 1 root root 64 May 27 17:31 directory -> /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory

/etc/letsencrypt/archive:
total 8
drwxr-xr-x 2 root root 4096 May 27 17:32 pelicandd.com
drwxr-xr-x 2 root root 4096 May 27 17:33 pelicandd.com-0001

/etc/letsencrypt/archive/pelicandd.com:
total 320
-rw-r--r-- 1 root root 2147 May 27 17:32 cert10.pem
-rw-r--r-- 1 root root 2151 May 27 17:32 cert11.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert12.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert13.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert14.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert15.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert16.pem
-rw-r--r-- 1 root root 1903 May 27 17:32 cert17.pem
-rw-r--r-- 1 root root 1903 May 27 17:32 cert18.pem
-rw-r--r-- 1 root root 1907 May 27 17:32 cert19.pem
-rw-r--r-- 1 root root 1793 May 27 17:32 cert1.pem
-rw-r--r-- 1 root root 1903 May 27 17:32 cert20.pem
-rw-r--r-- 1 root root 1793 May 27 17:32 cert2.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert3.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert4.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert5.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert6.pem
-rw-r--r-- 1 root root 1789 May 27 17:32 cert7.pem
-rw-r--r-- 1 root root 2151 May 27 17:32 cert8.pem
-rw-r--r-- 1 root root 2147 May 27 17:32 cert9.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain10.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain11.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain12.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain13.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain14.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain15.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain16.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain17.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain18.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain19.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain1.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain20.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain2.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain3.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain4.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain5.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain6.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain7.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain8.pem
-rw-r--r-- 1 root root 1647 May 27 17:32 chain9.pem
-rw-r--r-- 1 root root 3794 May 27 17:32 fullchain10.pem
-rw-r--r-- 1 root root 3798 May 27 17:32 fullchain11.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain12.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain13.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain14.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain15.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain16.pem
-rw-r--r-- 1 root root 3550 May 27 17:32 fullchain17.pem
-rw-r--r-- 1 root root 3550 May 27 17:32 fullchain18.pem
-rw-r--r-- 1 root root 3554 May 27 17:32 fullchain19.pem
-rw-r--r-- 1 root root 3440 May 27 17:32 fullchain1.pem
-rw-r--r-- 1 root root 3550 May 27 17:32 fullchain20.pem
-rw-r--r-- 1 root root 3440 May 27 17:32 fullchain2.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain3.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain4.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain5.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain6.pem
-rw-r--r-- 1 root root 3436 May 27 17:32 fullchain7.pem
-rw-r--r-- 1 root root 3798 May 27 17:32 fullchain8.pem
-rw-r--r-- 1 root root 3794 May 27 17:32 fullchain9.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey10.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey11.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey12.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey13.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey14.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey15.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey16.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey17.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey18.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey19.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey1.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey20.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey2.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey3.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey4.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey5.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey6.pem
-rw-r--r-- 1 root root 1708 May 27 17:32 privkey7.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey8.pem
-rw-r--r-- 1 root root 1704 May 27 17:32 privkey9.pem

/etc/letsencrypt/archive/pelicandd.com-0001:
total 16
-rw-r--r-- 1 root root 1911 May 27 17:33 cert1.pem
-rw-r--r-- 1 root root 1647 May 27 17:33 chain1.pem
-rw-r--r-- 1 root root 3558 May 27 17:33 fullchain1.pem
-rw-r--r-- 1 root root 1708 May 27 17:33 privkey1.pem

/etc/letsencrypt/csr:
total 1344
-rw-r--r-- 1 root root 960 May 27 17:31 0000_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0001_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0002_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0003_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0004_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0005_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0006_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0007_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0008_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0009_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0010_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0011_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0012_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0013_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0014_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0015_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0016_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0017_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0018_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0019_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0020_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0021_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0022_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0023_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0024_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0025_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0026_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0027_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0028_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0029_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0030_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0031_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0032_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0033_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0034_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0035_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0036_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0037_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0038_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0039_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0040_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0041_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0042_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0043_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0044_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0045_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0046_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0047_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0048_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0049_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0050_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0051_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0052_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0053_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0054_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0055_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0056_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0057_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0058_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0059_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0060_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0061_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0062_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0063_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0064_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0065_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0066_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0067_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0068_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0069_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0070_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0071_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0072_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0073_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0074_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0075_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0076_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0077_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0078_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0079_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0080_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0081_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0082_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0083_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0084_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0085_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0086_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0087_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0088_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0089_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0090_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0091_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0092_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0093_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0094_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0095_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0096_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0097_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0098_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0099_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0100_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0101_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0102_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0103_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0104_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0105_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0106_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0107_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0108_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0109_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0110_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0111_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0112_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0113_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0114_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0115_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0116_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0117_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0118_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0119_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0120_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0121_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0122_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0123_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0124_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0125_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0126_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0127_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0128_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0129_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0130_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0131_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0132_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0133_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0134_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0135_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0136_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0137_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0138_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0139_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0140_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0141_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0142_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0143_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0144_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0145_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0146_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0147_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0148_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0149_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0150_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0151_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0152_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0153_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0154_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0155_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0156_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0157_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0158_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0159_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0160_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0161_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0162_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0163_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0164_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0165_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0166_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0167_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0168_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0169_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0170_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0171_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0172_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0173_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0174_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0175_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0176_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0177_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0178_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0179_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0180_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0181_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0182_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0183_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0184_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0185_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0186_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0187_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0188_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0189_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0190_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0191_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0192_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0193_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0194_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0195_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0196_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0197_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0198_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0199_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0200_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0201_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0202_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0203_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0204_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0205_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0206_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0207_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0208_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0209_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0210_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0211_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0212_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0213_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0214_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0215_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0216_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0217_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0218_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0219_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0220_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0221_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0222_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0223_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0224_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0225_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0226_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0227_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0228_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0229_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0230_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0231_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0232_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0233_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0234_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0235_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0236_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0237_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0238_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0239_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0240_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0241_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0242_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0243_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0244_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0245_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0246_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0247_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0248_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0249_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0250_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0251_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0252_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0253_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0254_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0255_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0256_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0257_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0258_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0259_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0260_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0261_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0262_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0263_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0264_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0265_csr-certbot.pem
-rw-r--r-- 1 root root 964 May 27 17:31 0266_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0267_csr-certbot.pem
-rw-r--r-- 1 root root 956 May 27 17:31 0268_csr-certbot.pem
-rw-r--r-- 1 root root 989 May 27 17:31 0269_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0270_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0271_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0272_csr-certbot.pem
-rw-r--r-- 1 root root 968 May 27 17:31 0273_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0274_csr-certbot.pem
-rw-r--r-- 1 root root 993 May 27 17:31 0275_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0276_csr-certbot.pem
-rw-r--r-- 1 root root 980 May 27 17:31 0277_csr-certbot.pem
-rw-r--r-- 1 root root 972 May 27 17:31 0278_csr-certbot.pem
-rw-r--r-- 1 root root 960 May 27 17:31 0279_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0280_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0281_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0282_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0283_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0284_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0285_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0286_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0287_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0288_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0289_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0290_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0291_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0292_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0293_csr-certbot.pem
-rw-r--r-- 1 root root 948 May 27 17:31 0294_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0295_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0296_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0297_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0298_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0299_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0300_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0301_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0302_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0303_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0304_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0305_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0306_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0307_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0308_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0309_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0310_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0311_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0312_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0313_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0314_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0315_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0316_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0317_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0318_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0319_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0320_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0321_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0322_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0323_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0324_csr-certbot.pem
-rw-r--r-- 1 root root 936 May 27 17:31 0325_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0326_csr-certbot.pem
-rw-r--r-- 1 root root 928 May 27 17:31 0327_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0328_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0329_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0330_csr-certbot.pem
-rw-r--r-- 1 root root 932 May 27 17:31 0331_csr-certbot.pem
-rw-r--r-- 1 root root 940 May 27 17:31 0332_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0333_csr-certbot.pem
-rw-r--r-- 1 root root 924 May 27 17:31 0334_csr-certbot.pem
-rw-r--r-- 1 root root 924 Jun 30 17:24 0335_csr-certbot.pem

[arseni-mourzenko]

/etc/letsencrypt/keys:
total 1344
-rw------- 1 root root 1704 May 27 17:32 0000_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0001_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0002_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0003_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0004_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0005_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0006_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0007_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0008_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0009_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0010_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0011_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0012_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0013_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0014_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0015_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0016_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0017_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0018_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0019_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0020_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0021_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0022_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0023_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0024_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0025_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0026_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0027_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0028_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0029_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0030_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0031_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0032_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0033_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0034_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0035_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0036_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0037_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0038_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0039_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0040_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0041_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0042_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0043_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0044_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0045_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0046_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0047_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0048_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0049_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0050_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0051_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0052_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0053_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0054_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0055_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0056_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0057_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0058_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0059_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0060_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0061_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0062_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0063_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0064_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0065_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0066_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0067_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0068_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0069_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0070_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0071_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0072_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0073_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0074_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0075_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0076_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0077_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0078_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0079_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0080_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0081_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0082_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0083_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0084_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0085_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0086_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0087_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0088_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0089_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0090_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0091_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0092_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0093_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0094_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0095_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0096_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0097_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0098_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0099_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0100_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0101_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0102_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0103_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0104_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0105_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0106_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0107_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0108_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0109_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0110_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0111_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0112_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0113_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0114_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0115_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0116_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0117_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0118_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0119_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0120_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0121_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0122_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0123_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0124_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0125_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0126_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0127_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0128_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0129_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0130_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0131_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0132_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0133_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0134_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0135_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0136_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0137_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0138_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0139_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0140_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0141_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0142_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0143_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0144_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0145_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0146_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0147_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0148_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0149_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0150_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0151_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0152_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0153_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0154_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0155_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0156_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0157_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0158_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0159_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0160_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0161_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0162_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0163_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0164_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0165_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0166_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0167_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0168_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0169_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0170_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0171_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0172_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0173_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0174_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0175_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0176_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0177_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0178_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0179_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0180_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0181_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0182_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0183_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0184_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0185_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0186_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0187_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0188_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0189_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0190_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0191_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0192_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0193_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0194_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0195_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0196_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0197_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0198_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0199_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0200_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0201_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0202_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0203_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0204_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0205_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0206_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0207_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0208_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0209_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0210_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0211_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0212_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0213_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0214_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0215_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0216_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0217_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0218_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0219_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0220_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0221_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0222_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0223_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0224_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0225_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0226_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0227_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0228_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0229_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0230_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0231_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0232_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0233_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0234_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0235_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0236_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0237_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0238_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0239_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0240_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0241_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0242_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0243_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0244_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0245_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0246_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0247_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0248_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0249_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0250_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0251_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0252_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0253_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0254_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0255_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0256_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0257_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0258_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0259_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0260_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0261_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0262_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0263_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0264_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0265_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0266_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0267_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0268_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0269_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0270_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0271_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0272_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0273_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0274_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0275_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0276_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0277_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0278_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0279_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0280_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0281_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0282_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0283_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0284_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0285_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0286_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0287_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0288_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0289_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0290_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0291_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0292_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0293_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0294_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0295_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0296_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0297_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0298_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0299_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0300_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0301_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:31 0302_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0303_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:31 0304_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0305_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0306_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0307_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0308_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0309_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0310_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0311_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0312_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0313_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0314_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0315_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0316_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0317_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0318_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0319_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0320_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0321_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0322_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0323_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0324_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0325_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0326_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0327_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0328_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0329_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0330_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0331_key-certbot.pem
-rw------- 1 root root 1704 May 27 17:32 0332_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0333_key-certbot.pem
-rw------- 1 root root 1708 May 27 17:32 0334_key-certbot.pem
-rw------- 1 root root 1704 Jun 30 17:24 0335_key-certbot.pem

/etc/letsencrypt/live:
total 12
drwxr-xr-x 2 root root 4096 May 27 17:33 pelicandd.com
drwxr-xr-x 2 root root 4096 May 27 17:33 pelicandd.com-0001
-rw-r--r-- 1 root root  740 May 27 17:33 README

/etc/letsencrypt/live/pelicandd.com:
total 0
lrwxrwxrwx 1 root root 38 May 27 17:33 cert.pem -> ../../archive/pelicandd.com/cert20.pem
lrwxrwxrwx 1 root root 39 May 27 17:33 chain.pem -> ../../archive/pelicandd.com/chain20.pem
lrwxrwxrwx 1 root root 43 May 27 17:33 fullchain.pem -> ../../archive/pelicandd.com/fullchain20.pem
lrwxrwxrwx 1 root root 41 May 27 17:33 privkey.pem -> ../../archive/pelicandd.com/privkey20.pem

/etc/letsencrypt/live/pelicandd.com-0001:
total 4
lrwxrwxrwx 1 root root  42 May 27 17:33 cert.pem -> ../../archive/pelicandd.com-0001/cert1.pem
lrwxrwxrwx 1 root root  43 May 27 17:33 chain.pem -> ../../archive/pelicandd.com-0001/chain1.pem
lrwxrwxrwx 1 root root  47 May 27 17:33 fullchain.pem -> ../../archive/pelicandd.com-0001/fullchain1.pem
lrwxrwxrwx 1 root root  45 May 27 17:33 privkey.pem -> ../../archive/pelicandd.com-0001/privkey1.pem
-rw-r--r-- 1 root root 692 May 27 17:33 README

/etc/letsencrypt/renewal:
total 8
-rw-r--r-- 1 root root 587 May 27 17:31 pelicandd.com-0001.conf
-rw-r--r-- 1 root root 565 May 27 17:31 pelicandd.com.conf

/etc/letsencrypt/renewal-hooks:
total 12
drwxr-xr-x 2 root root 4096 May 27 17:33 deploy
drwxr-xr-x 2 root root 4096 May 27 17:33 post
drwxr-xr-x 2 root root 4096 May 27 17:33 pre

/etc/letsencrypt/renewal-hooks/deploy:
total 0

/etc/letsencrypt/renewal-hooks/post:
total 0

/etc/letsencrypt/renewal-hooks/pre:
total 0

[schoen]

Thanks, could we also see the full output of certbot certificates?

It’s odd that the certificates would be expired now if they were created on May 27. Did you copy this tree from a different machine at some point?

[JuergenAuer]

@arseni-mourzenko

It’s pelicandd.com (the one which has an expired certificate). And there is a not yet expired wildcard certificate *.pelicandd.com .

There is really a mess - see https://crt.sh/?q=pelicandd.com

Not only the main domain, same with subdomains, daily created certificates.

Do you use force-renew?

[arseni-mourzenko]

@schoen

Thanks, could we also see the full output of certbot certificates ?

Here it is:

# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
OCSP check failed for /etc/letsencrypt/live/pelicandd.com/cert.pem (are we offline?)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: pelicandd.com-0001
    Domains: *.pelicandd.com
    Expiry Date: 2020-08-25 16:22:14+00:00 (VALID: 55 days)
    Certificate Path: /etc/letsencrypt/live/pelicandd.com-0001/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/pelicandd.com-0001/privkey.pem
  Certificate Name: pelicandd.com
    Domains: pelicandd.com
    Expiry Date: 2020-06-28 05:25:24+00:00 (INVALID: EXPIRED)
    Certificate Path: /etc/letsencrypt/live/pelicandd.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/pelicandd.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

@schoen

It’s odd that the certificates would be expired now if they were created on May 27. Did you copy this tree from a different machine at some point?

So:

1. There are two Nginx reverse proxies working in failover.
2. One of them is in charge of renewing the certificates. It has a cron job, which calls certbot renew, then copies the local directory containing the private certificates to an NFS share, and finally reloads Nginx.
3. The other one has another cron job, which once per day gets the certificates from the NFS share, and reloads Nginx.
4. Both machines are occasionally destroyed and rebuilt from scratch. When they are rebuilt, they get the certificates from the NFS share. They were rebuilt four times today, once yesterday, but not the days before, if I remember well.

@JuergenAuer

There is really a mess - see https://crt.sh/?q=pelicandd.com

Not only the main domain, same with subdomains, daily created certificates.

Do you use force-renew ?

It is a mess, indeed, and if I understand correctly, the mess started in November 2019, so a long time ago. No, I don’t use force-renew. I suppose the problem is indeed that my certbot is misconfigured, and just discards the certificates it created. I’ll see tomorrow if I can figure it out.

[schoen]

Hmmm! So, could you explain a bit more about the NFS share thing? I’m wondering if the pelicandd.com base domain certificate got renewed in a way where the resulting certificates weren’t preserved. The creation time on the certificate files says May 27 for all of them, but I guess that isn’t necessarily accurate in this NFS setup?

[JuergenAuer]

@arseni-mourzenko

then copies the local directory containing the private certificates to an NFS share, and finally reloads Nginx.

Really copy? Or move? Move - your certbot doesn’t find the new certificate, instead the old -> next certificate is created.

Or the cron job runs on both machines.

There are a lot of errors possible.

[_az]

One easy way this can go wrong is if you use rsync to copy the certificates to NFS, which will by default dereference symlinks, instead of copying the symlink. You need to use -L in order not to break Certbot, otherwise you end up with renewals every day.

A /var/log/letsencrypt/ log from a renewal that ran unnecessarily might reveal the reason a renewal ran as well.

Probably you have figured out the date by now, but I made this web-based tool to allow quickly checking when a duplicate certificate rate limit is due to expire.

[schoen]

Wouldn’t we potentially get an “expected X to be a symlink” error when trying to renew in that case? (I haven’t tested this with NFS so I’m not sure, I just remember that Certbot has some code related to checking for this.)

[_az]

You’re right, good point. Gave it a test and it just reports the renewal config as broken. Not sure why I remembered otherwise.

[Osiris]

@arseni-mourzenko

(…) to just two: a wildcard one, and the one for the domain itself.

You do realise you could put those two hostnames in the same certificate, right?

Even is a way to circumvent the current rate limit, once you’ve got your certbot fixed

[bruncsak]

Is the date-time correct on the servers?