I am running my first k3s single node cluster on a ubuntu VPS which works when I curl the http endpoints of an angular application. Via Helm I install cert-manager and followed this tutorial to implement letsEncrypt Tutorial . However, when I run the clusterissuer, cert etc. the challenge fails with the following:
I have added the domain to the VPS's /etc/hosts/ with the external IP of treafik ingress
When you browse to the domain you get the default DNS providers page and when you curl to http/https on the vps you get webpage on http and cert error on https, as expected I think because of the challenge that fails.
I am a bit lost now and do not know what or how to debug this? Is more info needed? Just ask and thanks in advance?
Hi @Chriskick, and welcome to the LE community forum
That sounds like a default landing page.
OR
It is using HTML to redirect or shows your content within a frame, etc.
None of which will work with HTTP-01 authentication.
Also, be sure your site works via IPv6: 2a03:b0c0:2:d0::14bf:4001
Yes, a lot.
Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
My domain is:
I ran this command:
It produced this output:
My web server is (include version):
The operating system my web server runs on is (include version):
My hosting provider, if applicable, is:
I can login to a root shell on my machine (yes or no, or I don't know):
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Thanks for the welcoming first post! Appreciated! So first try for all the remaining information, hope I do not break any rules by naming my providers by name.
@Bruce5051 lol was typing when I got your message. We have builders here at home, so my time is limited, sorry for that!
As i understand it correctly, it is the default landing page that I can delete in their local storage, but I would assume that there is some functionality that redirects to this html. Curling to https://nutrizone.nl responds with curl: (60) SSL certificate problem: self-signed certificate, so that is why assume this.
My VPS had IPv6 turned off and I have turned it on now, only this, because does my loadbalancer also have its external ip switched to this? Now it is on its IPv4 adress (same as VPS ip)
Nutrizone.nl
So I ran through the tuturial and ended up with a clusterIssuer, certificate and ingress (just ask if needed)
Installed cert-manager --version v1.10.1 via helm
It is a local domain registy company with DirectAdmin as their DNS management system
VPS is digital ocean droplet
Ubuntu 22.10
I SSH onto the server, but on my pc I can access k3s single node cluster.
Hope this give you guys a start, all help appreciated, I will go look into the IPv6 and try the same curls I tried with IPv4
edit after trying IPv6: @Bruce5051 I have tried the curls you mentioned and I get a curl: (7) Couldn't connect to server. Looking further into this...
MultipleIPAddressDiscrepancy
Warning
nutrizone.nl has multiple IP addresses in its DNS records. While they appear to be accessible on the network, we have detected that they produce differing results when sent an ACME HTTP validation request. This may indicate that some of the IP addresses may unintentionally point to different servers, which would cause validation to fail.
[Address=2a03:b0c0:2:d0::14bf:4001,Address Type=IPv6,Server=nginx,HTTP Status=301,Number of Redirects=1,Final HTTP Status=404] vs [Address=206.189.101.199,Address Type=IPv4,Server=nginx/1.17.1,HTTP Status=404]
From here I see there is both an IPv4 and an IPv6 Address, the both have to give the same answers. Best Practice - Keep Port 80 Open for both IPv4 & IPv6 Addresses.
Again, from an IPv4 only network. This is what I get for the certificate being served.
CN = TRAEFIK DEFAULT CERT
verify error:num=18:self-signed certificate
$ openssl s_client -showcerts -servername nutrizone.nl -connect nutrizone.nl:443 < /dev/null
CONNECTED(00000003)
depth=0 CN = TRAEFIK DEFAULT CERT
verify error:num=18:self-signed certificate
verify return:1
depth=0 CN = TRAEFIK DEFAULT CERT
verify return:1
---
Certificate chain
0 s:CN = TRAEFIK DEFAULT CERT
i:CN = TRAEFIK DEFAULT CERT
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Jan 19 16:21:21 2023 GMT; NotAfter: Jan 19 16:21:21 2024 GMT
-----BEGIN CERTIFICATE-----
MIIDXTCCAkWgAwIBAgIQbgw8fcai6Okb5bjY/j43HTANBgkqhkiG9w0BAQsFADAf
MR0wGwYDVQQDExRUUkFFRklLIERFRkFVTFQgQ0VSVDAeFw0yMzAxMTkxNjIxMjFa
Fw0yNDAxMTkxNjIxMjFaMB8xHTAbBgNVBAMTFFRSQUVGSUsgREVGQVVMVCBDRVJU
MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAktg4Uu9h6SviJ+5/S8b5
d/Msk6PSUXJPXrnl+jNO8TaBUi+BUcQI+Q9Zfc3Zt4jLGVzXXXLkMdCcDopbeRxV
h47+qJ6qzXGa0cTJDOAmurP9TLqJ/vsji4VO+ndJeDOBg1yZ17Y3rokg7e0CfjQn
6jmVius7FoPRhWvDSSYhKVIDe7cqzP3RNtmPrJ116IxT7sBbl80FsI6t4OSTSDEi
HX7UPCAoTGeXMMmIgn73p6AInX400Ij3rP3FbXRbA/TYVyNWpP6sh9VE6qJYvor9
TArdEn/vfbtMS21N6Ed8nHiO+i3PcxrSa1lDV5bezB1qdqvQVIXVMsIwdwGaH8OX
UwIDAQABo4GUMIGRMA4GA1UdDwEB/wQEAwIDuDATBgNVHSUEDDAKBggrBgEFBQcD
ATAMBgNVHRMBAf8EAjAAMFwGA1UdEQRVMFOCUTZkMWFiOTU3N2IxYzkxOWFhNWRi
NjJmMmEzMDM0M2M4LmRmZTM5ZmM3NDcwNTNmYzdjMDg1NDZiYTkzMWE2OTVhLnRy
YWVmaWsuZGVmYXVsdDANBgkqhkiG9w0BAQsFAAOCAQEAY0GsDyw0r2AlAUqF/iIE
2mvZmJ155yiimhfPfqsk2cki0N9cwRn3aeRFKAkamJD7c4vYwDRMsBagb1KCee5n
yzpdheWRYGGRdy11cnWBYOwdaqgmEZJ/qjDd9NPILSzjeD4S5waYJUcfTYe06xJ+
dTZc9he+o9Cn9jureM8/NkeScmeZoLAKwuq/EoRTzPNv/HUAf3OZqv69w2DEOORZ
bMq9wAQP6/FzPBarLNkK1/GkwhAvV50ZW7lduIh++hrEWFDIiHZ8vZ4gk3O8zjBF
ykevH6juWrcE5q5hPZ62BQ3LO/SxdAs2jasSGCKdXobLckESFBl/xOoxcmc1na6e
fQ==
-----END CERTIFICATE-----
---
Server certificate
subject=CN = TRAEFIK DEFAULT CERT
issuer=CN = TRAEFIK DEFAULT CERT
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 1405 bytes and written 378 bytes
Verification error: self-signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_128_GCM_SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self-signed certificate)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
Protocol : TLSv1.3
Cipher : TLS_AES_128_GCM_SHA256
Session-ID: 965A12BB7BDB6A72CCC6B58D6DBE7343C773A60755224FE7FA4DC49CC2D7586B
Session-ID-ctx:
Resumption PSK: 3C6CC88E587C38B31D43F1F14FA05C3352E67071DA2322EC442C8631CD12EE16
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 604800 (seconds)
TLS session ticket:
0000 - 8b 37 3c d2 58 0f 60 db-e9 03 fc 37 f5 1f b7 87 .7<.X.`....7....
0010 - b4 70 45 af 3f d0 bc c7-30 33 02 7e 39 ad f5 14 .pE.?...03.~9...
0020 - ab 97 62 0a 36 c7 46 c3-20 63 ed e8 69 45 17 3b ..b.6.F. c..iE.;
0030 - 79 21 3e 03 5e 8d 7c be-e5 61 af aa be 2d c9 13 y!>.^.|..a...-..
0040 - f4 79 f0 fc ea e3 7d 1d-2d 95 51 de c0 b8 80 39 .y....}.-.Q....9
0050 - 14 fa ff 17 cd 64 73 35-5e 81 3b 90 e4 3b 9f a7 .....ds5^.;..;..
0060 - 87 68 2c de 96 45 75 b2-97 f7 ef 0e 95 01 12 e2 .h,..Eu.........
0070 - d2 .
Start Time: 1674145962
Timeout : 7200 (sec)
Verify return code: 18 (self-signed certificate)
Extended master secret: no
Max Early Data: 0
---
read R BLOCK
DONE
Currently I am blocklisted from my DNS management system, but with little knowledge of IPv6 I think maybe there is a whole IPv6 list not pointing to the IPv6 of my VPS. I will fix this as soon I have access.
So far I have on my action list
Switch VPS to IPv6 to be able to use the cert (done)
DNS management point change default IPv6 records to VPS IPv6 (TODO)
The loadbalancers IP has port 80 open so I guess this is no problem?
Yes I used your curl -6 -k -Ii https://\[2a03:b0c0:2:d0::d7d:d001\]/ command, which is the IPv6 and go the mentioned couldnt connect
Lots of info already, thanks, those websites to check certain things are already helpful, although I feel I do not automatically all the errors, so if more is needed to the above action list, feel free to add
No preference really, since IPv6 is needed for LetsEncrypt AND I hear everyone (my work environment) is switching to also use IPv6 I would choose IPv6 as the most important. I cannot find reasons to keep IPv4, but all post on stackoverflow says keep both haha.
The -6 ones are not working, because my dns management is still pointing to the default ip. There is no way it could work now right? I read the above error as an error that DirectAdmin gives on it default ip.
-4 == 404 not found because there is no cert working setup via the IPv6, so when I have access to my DNS management and change IPv6 I can check again right? I think this is blocking at the moment.
To make sure we understand eachother
The DirectAdmin IPv6 IP that is default and I still need to change in my DNS management
2a03:b0c0:2:d0::d7d:d001
Below the new IPv6 I opened on my VPS
2a03:b0c0:2:d0::d7d:d001
So until I update my DNS management, the results from curl -6 are not correctly directing to my VPS
If you have both IPv4 & IPv6 (my personal opinion is that is good) but the Let's Encrypt Challenges are for Domain Validation, which means that the Fully Qualified Domain Name is being looked up from DNS for an A, AAAA, or CNAME record. So to pass the Challenge what ever path is being taken from Let's Encrypt must give the correct response. Thus both all IP Addresses must give the same response.
Thus I have an impediment from not able to access my DNS management, right? I hope tomorrow I have access again. I will update again when I have changed it and waited the DNS to change.
Thanks so far! Not one step further but so much learned already! I will update soon.
The problem when IPv4 and IPv6 addresses return different results is that this is a symptom of misconfiguration. You want "regular" clients like browsers to see the same result regardless of whether they used IPv4 or IPv6.
The LE HTTP Challenge is different. The Let's Encrypt Servers will try IPv6 if an AAAA record is present. If it works it's done and IPv4 is not involved. Only if there are specific comms failures will IPv4 be tried when an AAAA record is present.
When no AAAA record is present, the IPv4 A address is used.
LE does not need IPv6. IPv4-only is perfectly acceptable. But as I noted above if you have AAAA it will be favored by LE for HTTP Challenge
I can't follow what's happening in this thread so not sure what suggestions to make. Just hoping this clarity will help.
This clarity certainly helps, but one thing to be clear in order of importance;
AAAA IPv6 Record
AAAA IPv4 record
A IPv4 Record
Where is IPv6 A Record in this? In between 1 and 2 or between 2 and 3?
In my DNS management so far is an A record IPv6 with the default ip (
2a03:b0c0:2:d0::14bf:4001
)
And an IPv4 A record pointing to my VPS
206.189.101.199
On the DNS Management side I have to change the IPv6 A record to my VPS's IPv6
2a03:b0c0:2:d0::d7d:d001
That is what I have learned so far.
That being said, the opening post still is valid. I get that error while solving the challenge and this afternoon I also looked in the cert-manager controller logs and found the following.
