Can't renew certs, rate limit after errors

Help
21

Ok, our choices now are very limited.
Here is what can work:

  • we can combine the two separate (limited) cert requests into one (new) cert request.

To that end, let's review the current renewal config files.
with:
cat /usr/local/etc/letsencrypt/renewal/*

6 Likes
22

cat /usr/local/etc/letsencrypt/renewal/*

# renew_before_expiry = 30 days
version = 1.22.0
archive_dir = /usr/local/etc/letsencrypt/archive/solar.schamschula.com
cert = /usr/local/etc/letsencrypt/live/solar.schamschula.com/cert.pem
privkey = /usr/local/etc/letsencrypt/live/solar.schamschula.com/privkey.pem
chain = /usr/local/etc/letsencrypt/live/solar.schamschula.com/chain.pem
fullchain = /usr/local/etc/letsencrypt/live/solar.schamschula.com/fullchain.pem
# Options used in the renewal process
[renewalparams]
account = 9f7cd967483dcd89aa13600a77eef307
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory

# renew_before_expiry = 30 days
version = 1.22.0
archive_dir = /usr/local/etc/letsencrypt/archive/wx.schamschula.com
cert = /usr/local/etc/letsencrypt/live/wx.schamschula.com/cert.pem
privkey = /usr/local/etc/letsencrypt/live/wx.schamschula.com/privkey.pem
chain = /usr/local/etc/letsencrypt/live/wx.schamschula.com/chain.pem
fullchain = /usr/local/etc/letsencrypt/live/wx.schamschula.com/fullchain.pem
# Options used in the renewal process
[renewalparams]
account = 9f7cd967483dcd89aa13600a77eef307
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
23

OK, hurdle #1:

Why did you (have to) get a cert using --standalone authentication?
[or did you?]

2 Likes
24

I don't recall ever specifying that. However, I've been using certbot for quite a long time. On my FreeBSD server since I first set it up in 2015, and on my Mac home server before it when Let's Encrypt first came out.

25

apache is answering on port 80, so I think we need to include that in our solution:

curl -Ii solar.schamschula.com
HTTP/1.1 403 Forbidden
Date: Sun, 26 Jun 2022 19:45:56 GMT
Server: Apache/2.4.54 (FreeBSD) OpenSSL/1.1.1o-freebsd PHP/7.4.30
Content-Type: text/html; charset=iso-8859-1

Let's have a look at both HTTP server blocks.

5 Likes
26

That's no surprise that you get a 403 one solar.schamschula.com. It only allows access from a particular C block (work). You are better off trying wx.schamschula.com, but it also has some restrictions. Also I have to turn off ipfw when I renew the certs, as I block a large part of ipv4 space.

<VirtualHost *:443>
ServerAdmin marius@schamschula.com
DocumentRoot "/usr/local/www/apache24/data"
ServerName wx.schamschula.com:443
CustomLog "/var/log/httpd-access.log" combined

#   SSL
SSLEngine on
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile         /usr/local/etc/letsencrypt/live/wx.schamschula.com/cert.pem
SSLCertificateKeyFile      /usr/local/etc/letsencrypt/live/wx.schamschula.com/privkey.pem

<Directory "/usr/local/www/apache24/data">
    #Include /usr/local/etc/apache24/extra/httpd-rewrite.conf
    Options Indexes FollowSymLinks
    AllowOverride All
    Order allow,deny
    RewriteEngine On

    # IP ranges
    Include /usr/local/etc/apache24/extra/ip.conf

    Allow from all
    Require all granted
</Directory>

<VirtualHost *:443>
ServerAdmin marius@schamschula.com
ServerName solar.schamschula.com:443
ErrorLog "/var/log/solar-error_log"
CustomLog "/var/log/solar-access_log" combined

#   SSL
SSLEngine on
    SSLProtocol -all +TLSv1.2 +TLSv1.3
SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
SSLCertificateFile         /usr/local/etc/letsencrypt/live/solar.schamschula.com/cert.pem
SSLCertificateKeyFile      /usr/local/etc/letsencrypt/live/solar.schamschula.com/privkey.pem

ProxyPass / http://10.0.1.33/
ProxyPassReverse / http://10.0.1.33/

<Proxy "*">
    Require ip 198.180.133
</Proxy>
27

Read this while a digest your config:

3 Likes
28

hmm...
Nothing to digest.
I see HTTPS server blocks.
I asked for:

[It even included the starting line numbers]

3 Likes
29

I keep port 80 open on all my servers. However, my home server is unique in that it doesn't automatically forward http to https.

30

I only care about renewing your cert OR getting a new one.

You are free to do whatever you like.

But I need to see the HTTP server blocks in order to continue [down the path of least resistance].

3 Likes
31

I have to step away for a long while - maybe someone else can continue where we left off.

2 Likes
32

Hmm. Line numbers?

I'm not sure what you are looking for. There are some bits of the config I'd rather not make public, i.e. anything about the directory structure, and anything commented out, like old proxy setups.

The port 80 config closely mirrors the port 443 setup:

<VirtualHost *:80>
    ServerAdmin marius@schamschula.com
    DocumentRoot "/usr/local/www/apache24/data"
    ServerName wx.schamschula.com:80
    ErrorLog "/var/log/httpd-error.log"
    CustomLog "/var/log/httpd-access.log"

    <Directory "/usr/local/www/apache24/data">
	#Include /usr/local/etc/apache24/extra/httpd-rewrite.conf
        Options Indexes FollowSymLinks
        AllowOverride All
	RewriteEngine On

	# IP ranges
    	Include /usr/local/etc/apache24/extra/ip.conf

    </Directory>
</VirtualHost>

and

<VirtualHost *:80>
    ServerAdmin marius@schamschula.com
    ServerName solar.schamschula.com:80
    ErrorLog "/var/log/solar-error_log"
    CustomLog "/var/log/solar-access_log" combined

    ProxyPass / http://10.0.1.33/
    ProxyPassReverse / http://10.0.1.33/

    <Proxy "*">
        Require ip 198.180.133
    </Proxy>
</VirtualHost>
33

I guess there is no way of copying the last certs from the log file?

34

No, there is not. Your private key should never be exposed.

I see you have 42 valid certs for your wx subdomain. There are 22 valid certs for your solar domain. Some of these overlap.

You have been creating certs frequently for the past month. Is there any backups for any of those?

I also see you will soon be past your rate limit. You will have a chance to get one cert for these names in roughly a day.

I don't have time to work on this messy problem right now. I wanted to inform you about prior certs in case your backups cover that. And, inform of the status of your current rate limit.

5 Likes
35

I ran a script to check for expired certs on a daily basis (I've since changed the cron job to run once a week). However, there was a configuration error (see this thread), which caused multiple certs to be generated and I didn't notice it until I got an email that my certs are about to expire. Unfortunately, I didn't correctly debug the issue at the time (last week). Due to the configuration error, certs weren't correctly installed on my server, so there are no back-ups. The script didn't make back-ups anyway. I'll add that!

36
  1. You can remove ":80" from the `servername" line [a port is not part of the name].
  2. We now have a webroot we can use for that cert [part 1 of 2].
  3. You will need to add a "documentroot" line to the second vhost [part 2 of 2].

Then we can proceed with getting a combined cert.
Let me know once that has been added (and what the path is).

2 Likes
37

It seems that we will have to add another name to the list (to make the cert request unique).
So, I propose (for the time being), to add:

Name:    173-26-107-252.client.mchsi.com
Address: 173.26.107.252

to the two names:

Name:    wx.schamschula.com
Address: 173.26.107.252
Aliases: solar.schamschula.com

Then, once we get your going with that cert, we can clean up the other certs and unused folders (etc.).

2 Likes
38

OK. I removed the port from the server names and added a temporary DocumentRoot to solar.schamschula.com (/usr/local/www/apache24/data/solar). Remember, it normally proxies my inverter.

39

I'm not sure how to add 173-26-107-252.client.mchsi.com. I have no access to my ISP's DNS. It might be easier to add a temporary host name to schamschula.com.

40

All you have to "add" is an HTTP vhost server using that servername.
OR
serveralias

2 Likes