Certificate renew :-( again fails

My domain is:
gollum.at

The server names of the main server are:
gollum.at www.gollum.at ipv4-www.gollum.at ipv6-www.gollum.at
and a second vhost runs at:
nc.gollum.at

I ran this command:
certbot renew -vvv

It produced this output:
Sorry - can not reproduce because now it get:
There were too many requests of a given type :: Error creating new order :: too many failed authorizations recently

I correctly created the certificate for all the domains. Now I got an expiration email. Then I tried to renew and always get this annoying error "The Certificate Authority failed to verify the temporary Apache configuration". Even with -vvv i was not able to understand what happens. It updated my apache configs but finally failed when it tries to fetch the acme-challenge.

Whats the corect way to debug this certbot stuff? It writes something into the config but on any error it immediately rollbacks the config so I cannot check whats really going wrong.I think it's because my hssp.config is really old (and often migrated) to the certbot just puts in some garbage. What should I do now?

Is there any other way of getting a stable cert without this crazy hacks into my http-conf. I have also almost full acces to manage my DNS - is there another way without rewriting and craching my configs every month? Maybe can it be a problem because ipv4-* has only A-records and ipv6-* only AAAA- records but the renew process fails on www.gollum.at so I think this is not the problem.

My web server is (include version):
apache 2.4.41-4ubuntu3.12

The operating system my web server runs on is (include version):
Ubuntu 20.04.4 LTS

My hosting provider, if applicable, is:
self hosting

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
1.31.0

You can use the staging system to test.

Yes, the apache and nginx plug-ins can sometimes be trickier to debug.
The /var/log/letsencrypt/letsencrypt.log is often helpful. Copy it to a .txt file and upload it. Or, ask us.

Yes. Use the webroot authentication or even DNS (docs here). webroot is often much easier to setup

I don't see an obvious reason for that to fail. Perhaps the log will help.
But, I cannot connect to your ipv4 domain (ipv6 was fine)

curl -I http://ipv4-www.gollum.at/.well-known/acme-challenge/Test123
curl: (7) Failed to connect to ipv4-www.gollum.at port 80 after 113 ms: Connection refused
4 Likes

Both IPv4 and IPv6 were accessible by SSL Labs.
https://www.ssllabs.com/ssltest/analyze.html?d=gollum.at

Is not present in the (SAN) Alternative names gollum.at ipv4-www.gollum.at ipv6-www.gollum.at oc.gollum.at www.gollum.at

Best Practice - Keep Port 80 Open

Testing and debugging are best done using the Staging Environment as the Rate Limits are much higher. Rate Limits are per week (rolling).

And to assist with debugging there is a great place to start is Let's Debug.

And a list of issued certificates can be found here: crt.sh | gollum.at

2 Likes

Checked with https://semonto.com/tools/website-reachability-check.html: All sites are reachable via ipv4 and ipv6
So not sure what SSL Server Test (Powered by Qualys SSL Labs) is trying? It seems it just fails when there is no ipv4 present?

gollum.at www.gollum.at ipv4-www.gollum.at ipv6-www.gollum.at and oc.gollum.at are going to the same vhost
the host on port 80 has a permant redirect to https:
The config is:

        RewriteEngine On
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1
        <Directory /var/lib/letsencrypt/http_challenges>
                Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
                Require all granted
        </Location>

        RewriteCond %{SERVER_NAME} =gollum.at [OR]
        RewriteCond %{SERVER_NAME} =www.gollum.at [OR]
        RewriteCond %{SERVER_NAME} =ipv4-www.gollum.at [OR]
        RewriteCond %{SERVER_NAME} =ipv6-www.gollum.at
        RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]

the hvost on 443 has (except for oc.gollum.at):

                RewriteEngine On
                RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1
                <Directory /var/lib/letsencrypt/http_challenges>
                        Require all granted
                </Directory>
                <Location /.well-known/acme-challenge>
                        Require all granted
                </Location>
                SSLCertificateFile /etc/letsencrypt/live/gollum.at/fullchain.pem
				SSLCertificateKeyFile /etc/letsencrypt/live/gollum.at/privkey.pem

And for the hvost on 443 oc.gollum.at I now added also the Rewrite-rule for the acme-challenge

I created also a test file Test123 so this works: http://www.gollum.at/.well-known/acme-challenge/Test123
but of course, curl returns: Location: https://www.gollum.at/.well-known/acme-challenge/Test123

Can it be that certbot does not work correctly with the HTTP-Redirect?

certbot renew --test-cert -vvv returned:
certbot-test.txt (7.0 KB)

This Check website performance and response: Check host - online website monitoring show Port 80 is not open from various locations around the world.

2 Likes

Okay, now I removed the permanent redirect (which I really do not want...)
Not Check website performance and response: Check host - online website monitoring is almost green.

certbot renew -vvv is still not working :frowning:
certbot3.txt (36.9 KB)

Not always check this permanent link Check website performance and response: Check host - online website monitoring

2 Likes

Okay, when checking from this site I am not aways reachable. Using some other check sites it is always reachable.

What can I do now?

The server connected via an business SDSL line - so the line cannot be the problem (also the other hosts using this line have no connection problems)

Understand why and debug the connectivity issue.

2 Likes

Also observer there are 2 IP address for the Domain Name:

  1. IPv4 address of 80.64.140.85
  2. IPv6 address of 2001:470:6cb8:2::2

Both must work for inbound Port 80 for HTTP-01 challenge

$ nslookup
> gollum.at
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
Name:   gollum.at
Address: 80.64.140.85
> set q=soa
> gollum.at
Server:         127.0.0.1
Address:        127.0.0.1#53

Non-authoritative answer:
gollum.at
        origin = dns1.netplanet.at
        mail addr = support.netplanet.at
        serial = 2022080301
        refresh = 1200
        retry = 3600
        expire = 604800
        minimum = 300

Authoritative answers can be found from:
> server dns1.netplanet.at
Default server: dns1.netplanet.at
Address: 89.185.121.251#53
> gollum.at
Server:         dns1.netplanet.at
Address:        89.185.121.251#53

gollum.at
        origin = dns1.netplanet.at
        mail addr = support.netplanet.at
        serial = 2022080301
        refresh = 1200
        retry = 3600
        expire = 604800
        minimum = 300
> set q=a
> gollum.at
Server:         dns1.netplanet.at
Address:        89.185.121.251#53

Name:   gollum.at
Address: 80.64.140.85
> set q=aaaa
> gollum.at
Server:         dns1.netplanet.at
Address:        89.185.121.251#53

gollum.at       has AAAA address 2001:470:6cb8:2::2
>
2 Likes

I see this with https://letsdebug.net/ HTTP-01
Let's Debug

2 Likes

As Bruce has noted, HTTP (port 80) connections are not working well. Your Apache server may be mis-configured. Can you show result of this

sudo apachectl -t -D DUMP_VHOSTS

(might be apachectl2)

I say that because I see this

www domain fails HTTP w/IPv4 but works HTTP w/IPv6
curl -4 http://www.gollum.at/.well-known/acme-challenge/Test123
curl: (7) Failed to connect to www.gollum.at port 80 after 112 ms: Connection refused
curl -6 http://www.gollum.at/.well-known/acme-challenge/Test123
test

apex domain works for both IPv4/6 with HTTP
curl -4 http://gollum.at/.well-known/acme-challenge/Test123
test
curl -6 http://gollum.at/.well-known/acme-challenge/Test123
test

and this showing ipv4 domain failing HTTP but ipv6 domain working

curl -I http://ipv4-www.gollum.at
curl: (7) Failed to connect to ipv4-www.gollum.at port 80 after 113 ms: Connection refused

curl -I http://ipv6-www.gollum.at
HTTP/1.1 200 OK
Date: Mon, 17 Oct 2022 19:42:11 GMT
Server: Apache
3 Likes

The maybe even some weird DNS or Routing issue going on here.

First attempt:

Second attempt:

1 Like

I think that website tool is faulty. Did the same failed/ok sequence to me. But, DNS checks from my own system are fine. And, Let's Debug and unboundtest get the DNS fine.

It's just the HTTP checks that fail and are easily reproduced

3 Likes

Maybe it's an IPv4/IPv6 fault/no fault???

Name:      gollum.at
Addresses: 2001:470:6cb8:2::2
           80.64.140.85
2 Likes

There is definitely something wrong with IPv4 / 6 HTTP connects. My post #13 shows some of them. Peculiar collection of symptoms. I wanted to see Apache config

dnsviz shows an odd warning so perhaps DNS has problems in addition to HTTP issues

3 Likes

The IPv6 is from tunnelbroker - so I have no control over this

What id did:

  • Removed ipv4-www.gollum.at and ipv6-www.gollum.at from the vhost into a separate vhost
  • Removed AAAA bindings von gollum.at and www.gollum.at :frowning:
  • Waited some days for the DNS-Changes to get active...

Certbot was still not working - and still trying ipv4-www.gollum.at

  • Executed: certbot certonly --cert-name gollum.at -d gollum.at,www.gollum.at,oc.gollum.at -v
  • Selected 1: Apache Web Server plugin (apache)
  • First run (without -v) failed
  • Second run was successfull
  • Manually changed the SSLCertificateFile and SSLCertificateKeyFile in the configs.

Now I have a cert till January - let's see what happens next year.

Even if the problem is still not solved (that sometimes my web server is not reachable) thanks for all the help! Maybe I will ask my internet provider (which also manages the microtik-router) for analyzing the problem.

lg Robert

1 Like
[Thu Oct 20 20:32:06.475553 2022] [so:warn] [pid 632282] AH01574: module dav_module is already loaded, skipping
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/gollum.at-ssl.conf:2
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server gollum.at (/etc/apache2/sites-enabled/gollum.at-ssl.conf:3)
         port 443 namevhost gollum.at (/etc/apache2/sites-enabled/gollum.at-ssl.conf:3)
                 alias www.gollum.at
         port 443 namevhost oc.gollum.at (/etc/apache2/sites-enabled/owncloud-ssl.conf:2)
*:80                   is a NameVirtualHost
         default server gollum.at (/etc/apache2/sites-enabled/gollum.at.conf:14)
         port 80 namevhost gollum.at (/etc/apache2/sites-enabled/gollum.at.conf:14)
                 alias www.gollum.at
         port 80 namevhost ipv4-www.gollum.at (/etc/apache2/sites-enabled/ipv-gollum.at.conf:12)
                 alias ipv6-www.gollum.at
         port 80 namevhost oc.gollum.at (/etc/apache2/sites-enabled/owncloud.conf:1)
  • The ipv4/ipv6 is now just using the same data directory like gollum.at but not using the same vhost.
  • ipv6 ist the only one with an AAAA record. The AAAA records from gollum.at and www.gollum.at have been removed.
  • The automatic-redirection from http to https in gollum.at / www.gollum.at is now commented out.
  • The automatic-redirection from http to https in oc.gollum.at is still active.
  • gollum.at / www.gollum.at and oc.gollum.at are using the same certificate.

Have you disabled port 80 for IPv4 now? Because I can reach your apex, www, and oc domains using IPv4 and HTTPS but not with HTTP. They all fail with "connection refused". See letsdebug.net

Yet, your ipv6-www.gollum.at connects both HTTP and HTTPS over IPv6 (it only has an AAAA record). Although, it returns an invalid cert but this is expected since you did not include this name in your cert request.

And, as expected, HTTPS to ipv4-www.gollum.at fails because the cert you got did not include this name either.

UPDATE: I just realized you did not setup a VHost for your ipv4 / ipv6 domains for port 443 so the cert is just from the default server used by Apache. So, that's fine. Except that ipv4-www domain fails to connect with HTTP with "connection refused" like your others.

3 Likes