Suddenly cert problem

Khymick · December 3, 2019, 9:42am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: https://khymon.homelinux.net/

My web server is (include version):
Server version: Apache/2.4.25 (Debian)
Server built: 2019-08-19T19:25:31

The operating system my web server runs on is (include version): Debian GNU/Linux 9

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.28.0

Problem description:
Today, all of a sudden, I cannot access my server anymore due to cert issues. My cert is still valid. What I get in Firefox is: MOZILLA_PKIX_ERROR_SELF_SIGNED_CERT.
Strange enough, because I am not aware of any changes on the web server.

Can somebody please help me in fixing this?

_az · December 3, 2019, 9:50am

Works fine for me in Firefox.

Is it possible that you are connecting on the domain’s IPv6 address, and that Apache hasn’t been setup to serve that certificate on IPv6?

Could you check whether these produce different results?

curl -X GET -I -vv -4 https://khymon.homelinux.net

curl -X GET -I -vv -6 https://khymon.homelinux.net

Khymick · December 3, 2019, 9:54am

Thanks for your reply.
It seems that the problem is only coming from inside my home network. I can reproduce the problem on 2 machines with 2 different OS.

I never had something like this. Can you please tell me where I can start looking for the cause? I am not at all sure where to start…

For the commands:
$ curl -X GET -I -vv -4 https://khymon.homelinux.net

Rebuilt URL to: https://khymon.homelinux.net/
Trying 178.27.9.15…
TCP_NODELAY set
Connected to khymon.homelinux.net (178.27.9.15) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (IN), TLS handshake, Server key exchange (12):
TLSv1.2 (IN), TLS handshake, Server finished (14):
TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.2 (OUT), TLS handshake, Finished (20):
TLSv1.2 (IN), TLS handshake, Finished (20):
SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
ALPN, server accepted to use http/1.1
Server certificate:
subject: CN=khymon.homelinux.net
start date: Nov 7 21:43:15 2019 GMT
expire date: Feb 5 21:43:15 2020 GMT
subjectAltName: host “khymon.homelinux.net” matched cert’s “khymon.homelinux.net”
issuer: C=US; O=Let’s Encrypt; CN=Let’s Encrypt Authority X3
SSL certificate verify ok.

GET / HTTP/1.1
Host: khymon.homelinux.net
User-Agent: curl/7.61.1
Accept: /

< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Tue, 03 Dec 2019 09:54:51 GMT
Date: Tue, 03 Dec 2019 09:54:51 GMT
< Server: Apache/2.4.25 (Debian)
Server: Apache/2.4.25 (Debian)
< Strict-Transport-Security: max-age=15552000; includeSubDomains
Strict-Transport-Security: max-age=15552000; includeSubDomains
< Vary: Accept-Encoding
Vary: Accept-Encoding
< Content-Length: 1361
Content-Length: 1361
< Content-Type: text/html;charset=UTF-8
Content-Type: text/html;charset=UTF-8

<

Excess found in a non pipelined read: excess = 1361 url = / (zero-length body)
Connection #0 to host khymon.homelinux.net left intact

$ curl -X GET -I -vv -6 https://khymon.homelinux.net

Rebuilt URL to: https://khymon.homelinux.net/
Trying 2a02:810d:0:8b:75d8:c055:252f:b333…
TCP_NODELAY set
Connected to khymon.homelinux.net (2a02:810d:0:8b:75d8:c055:252f:b333) port 443 (#0)
ALPN, offering h2
ALPN, offering http/1.1
successfully set certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.2 (IN), TLS handshake, Certificate (11):
TLSv1.2 (OUT), TLS alert, unknown CA (560):
SSL certificate problem: self signed certificate
Closing connection 0
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

JuergenAuer · December 3, 2019, 9:54am

Hi @Khymick

there are some problems, but not your problem - https://check-your-website.server-daten.de/?q=khymon.homelinux.net

http has a http status 500 Server Error, so creating the next certificate may not work.

You have ipv4 and ipv6, ipv6 has a timeout -> Letsencrypt prefers ipv6, that's critical.

But your certificate

CN=khymon.homelinux.net
	07.11.2019
	05.02.2020
expires in 64 days	khymon.homelinux.net - 1 entry

is correct.

PS: Perhaps you use internal ipv6 with a wrong certificate. But external -> there is a timeout, so it's not possible to check that.

_az · December 3, 2019, 9:57am

Great, that confirms that your IPv6 is the problem.

When you visit http://[2a02:810d:0:8b:75d8:c055:252f:b333] in your browser , what do you see?

(Can't seem to make a clickable link, but you can copy it)

Khymick · December 3, 2019, 9:59am

With this IP, I see the login of my router.

_az · December 3, 2019, 10:02am

Aha, that’s what I suspected was happening.

I think this is a port forwarding problem. Your port 80 and port 443 forwarding is applying for IPv4, but not IPv6. (That, or NAT loopback only works for IPv4).

I’m not sure on the exact steps to fix it though, it’s going to depend on your router’s administration interface. Is there a way to forward the ports for IPv6 as well?

If you aren’t able to work it out, you could also consider just removing the AAAA/IPv6 address from your domain name, which will cause your browser to always connect via IPv4, and show the right website.

Khymick · December 3, 2019, 10:13am

What concerns the port forwarding: In my router, port 443 is both active for IPv4 and IPv6. So this should work correctly.

What is really strange is that yesterday, all worked correctly - and this morning, I got the problem. I did not make any changes on the router nor the 2 systems here from which I try to connect in my LAN. Can somebody think of any possible cause? I really wonder…

From outside (=internet), it seems that I do not have the problem.

_az · December 3, 2019, 10:19am

From the internet, your IPv6 port does not respond at all, so the wrong certificate never has a chance to be sent. v4 fallback saves the day.

All signs point to v6 port forwarding not working. No idea why that would randomly break, though.

Khymick · December 3, 2019, 10:32am

Hmmm… I restarted the router and now it seems to work.
Can you please counter-check if you get a reply from v6?

What concerns the possible cause: My provider seemed to made an update of the router last night. But if that would have been the cause, this would mean that it shouldn´t work after the reboot as well. I am really kind of insecure in terms of what happens here.

_az · December 3, 2019, 10:49am

I still can’t connect on IPv6, same as when you originally posted the thread. Your router rejects the connection with ICMP Destination Unreachable (Host administratively prohibited).

In my country, all the customer premises equipment has awful support for IPv6 that breaks all the time, I’m not too surprised if the issues seem “random” .

Does the curl -6 at least work for you now?

Khymick · December 3, 2019, 11:10am

No, it´s even getting worse:

$ curl -X GET -I -vv -6 https://khymon.homelinux.net

Rebuilt URL to: https://khymon.homelinux.net/
Could not resolve host: khymon.homelinux.net
Closing connection 0
curl: (6) Could not resolve host: khymon.homelinux.net

I have to admin that I am not that deep in IPv6 configuration.
Let me ask a question to better understand: May it be that the server behind the router is not configured for IPv6? In other words: Perhaps the port forwarding itself is working, but the server behind does not reply to IPv6 requests? Is there a way to check easily if the forwarding itself works (eg temporarily start any service on the server to check this?)

_az · December 3, 2019, 11:20am

Well, you can test that theory by trying a curl request from the web server itself:

curl -X GET -I -6 --connect-to khymon.homelinux.net:443:localhost:443 https://khymon.homelinux.net

I don't think that's the case though, because we sort of disproved it earlier when you connected to the IP and it showed you the router login screen.

This may have just been a temporary side effect of restarting your router. It should be fine now (when your DNS updates anyway).

Khymick · December 3, 2019, 11:34am

gives as output:

# curl -X GET -I -6 --connect-to khymon.homelinux.net:443:localhost:443 https://khymon.homelinux.net
HTTP/1.1 200 OK
Date: Tue, 03 Dec 2019 11:25:29 GMT
Server: Apache/2.4.25 (Debian)
Strict-Transport-Security: max-age=15552000; includeSubDomains
Vary: Accept-Encoding
Content-Length: 1361
Content-Type: text/html;charset=UTF-8

Generally, the router says:

Internet, IPv6||verbunden seit 03.12.2019, 11:26 Uhr,
IPv6-Adresse: 2a02:810d::8b:75d8:c055:252f:b333, Gültigkeit: 4370/1670s,
IPv6-Präfix: 2a02:810d:45c0:4d7c::/62, Gültigkeit: 4370/1670s|

Concerning my address, this shouldn´t be any problem as well, as the router says:

[DynDNS](http://<router-dyndns-url>) aktiviert, khymon.homelinux.net, IPv4-Status: erfolgreich angemeldet, IPv6-Status: erfolgreich angemeldet

"erfolgreich angemeldet" means "succesfully connected".
The port forwarding for IPv6 is (as I mentioned earlier) active.

So, if I got that correctly, the router is still somehow intercepting the IPv6 connection?!? I am already searching in google for any known problems here, but I didn´t find anything helpful up to now.

_az · December 3, 2019, 8:11pm

Yes.

Do you know if it is somehow possible to change the IPv6 ports that the router administration interface listens on. For example, to change it from 80 and 443 to 8080 and 8443?

This might "free up" those ports to be available for port forwarding.

Just blind guesses at this point - depends on your router.

Which shows that your web server can handle IPv6 just fine - it's just never getting the connection from the router.

Khymick · December 4, 2019, 7:09am

Slowly, it´s getting very interesting. Today, I booted my PC and tried to access once again. This time, Firefox gave me the following error:

SEC_ERROR_INADEQUATE_KEY_USAGE

Yesterday evening, all worked fine. In the logs of the router, nothing changed. Also, although I ran an update of my linux (Fedora via dnf update), I guess on the client side, nothing changed as well.
I slowly getting the impression that something really strange is happening here…

EDIT: I rebooted my router once again and now, it´s working again. What I can try to do: Switch off the “MyFritz” service on the router, which gives accesibility to certain services of the router via internet.

EDIT2: I noticed something: Before the reboot of the router, I made a ping (from inside my LAN) which showed the following:

$ ping khymon.homelinux.net
PING khymon.homelinux.net(2a02:810d:0:8b:75d8:c055:252f:b333 (2a02:810d:0:8b:75d8:c055:252f:b333)) 56 data bytes
64 bytes from 2a02:810d:0:8b:75d8:c055:252f:b333 (2a02:810d:0:8b:75d8:c055:252f:b333): icmp_seq=1 ttl=64 time=0.394 ms

After the reboot of the router:

$ ping khymon.homelinux.net
PING khymon.homelinux.net (178.27.10.70) 56(84) bytes of data.
64 bytes from ipb21b0a46.dynamic.kabel-deutschland.de (178.27.10.70): icmp_seq=1 ttl=63 time=0.370 ms

On the router, I configured the dyndns to be active. May it be possible that the router is first forwarding via IPv4 which works fine - and after some time, it switched to IPv6 which doesn´t get forwarded?

Strange though… Any thoughts on this?

_az · December 4, 2019, 7:32am

Well, it's your desktop computer that chooses whether to use IPv4 or IPv6. Your router doesn't really get a say. In theory, software is supposed to prefer IPv6, if it's available at all.

Right now, your domain does not have an IPv6/AAAA record at all. Perhaps you changed something in your DynDNS?

The issue should be absent at the moment, since your domain only resolves via IPv4 (for now).

Khymick · December 4, 2019, 1:06pm

No, nothing. If I look into the settings right now, IPv6 address (optional) is 2a02:810d::8b:75d8:c055:252f:b333. This seems to be correct, if I´m right.

Now, I just tried - and the ping switched back to IPv6 which means that it is not working at the moment.

JuergenAuer · December 4, 2019, 1:50pm

If this is the ipv6 of your router, that may be wrong. Normally, every ipv6 device has it's own address, so no port forwarding is required.

Khymick · December 4, 2019, 1:59pm

Hm, okay, just let me get into this. When I login into my DynDNS accounts and go on the config site of my host (khymon…), there are two fields: one for “IP Address” (which is the IPv4) and one “IPv6 Address (optional)” - and there is the word “optional” in it. Though, there was always written something in it.

I deleted the entry in the IPv6 field - and now, the ping went back to the IPv4, as far as I can judge.
So I have to admin that I am a very noob in IPv6, because I don´t get it: If there is no IPv6 address in my DynDNS configuration, then how can my domain “khymon.homelinux.net” be translated in any IPv6? I guess there is something missing in my big picture about this.

Generally speaking: Now, I am back on IPv4 and it works. But I remember that it was so kind of discussion with my internet provider to give me an IPv4 address. If this reoccurs: How the heck should I then go on?

If anybody has a short explanation and/or a link with an easy start of IPv6, I would really appreciate this. All things about IPv6 I found so far were quite technical - and I guess I need some basic introduction.