Renewing Certificate Failing on QNAP NAS

[pg1]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: brak.myqnapcloud.com

I ran this command: clicked on "Certificate Renewal" button in SSL Certificate & Private Key tab of Security section of Control Panel.

It produced this output:

A domain validation challenge was not received from the ACME Server. Ensure that your router and QNAP device both accept inbound traffic on ports 80 and 443 which is a requirement from Let's Encrypt.

My web server is (include version): Apache (can't get version because ServerTokens is set to Prod and I can't figure out how to change it)

The operating system my web server runs on is (include version):
Linux U16BuildServer56 4.4.0-178-generic #208-Ubuntu SMP Sun Apr 5 23:45:10 UTC 2020 x86_64 GNU/Linux

My hosting provider, if applicable, is: local NAS

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): QNAP QTS 5.1.4.2596

Ports 80 and 443 are open. My DDNS connectivity test shows green on both ports.

Other posts I've read mentioned the logs, so I found these:

acme_error_log_http:

02/06/24 23:15:43 - args: Namespace(account_key='/mnt/ext/opt/QcloudSSLCertificate/cert/account/key', acme_dir='/mnt/ext/opt/QcloudSSLCertificate/cert/.well-known/acme-challenge', ca='https://acme-v02.api.letsencrypt.org', cert_file='/mnt/ext/opt/QcloudSSLCertificate/cert/cert_tmp', chain_file='/mnt/ext/opt/QcloudSSLCertificate/cert/chain_tmp', contact=['mailto:myemail@gmail.com'], csr='/mnt/ext/opt/QcloudSSLCertificate/cert/csr', directory_url='https://acme-v02.api.letsencrypt.org/directory', disable_check=False, qpkg_dir='/mnt/ext/opt/QcloudSSLCertificate', quiet=40, verify_type='http', web_document_root='/Web', well_known_dir='/mnt/ext/opt/QcloudSSLCertificate/cert/.well-known')

Traceback (most recent call last):
File "/mnt/ext/opt/QcloudSSLCertificate/bin/acme-tiny/acme_tiny.py", line 385, in main
qpkg_path=args.qpkg_dir, challenge_type=challenge_type, ca_certs=ca_certs, web_document_root=web_document_root_list)
File "/mnt/ext/opt/QcloudSSLCertificate/bin/acme-tiny/acme_tiny.py", line 271, in get_crt
wellknown_path, tmp_wellknown_url), ERROR_CODE_CHALLENGE_NOT_FOUND)
CustomError: Wrote file to /mnt/ext/opt/QcloudSSLCertificate/cert/.well-known/acme-challenge/scFvB5xfx3ZQSbpuMpm9O8X6FtRBZNReCc_i3npJcRA, but couldn't download http://localhost/.well-known/acme-challenge/scFvB5xfx3ZQSbpuMpm9O8X6FtRBZNReCc_i3npJcRA

acme_error_log_https:

02/06/24 23:15:48 - args: Namespace(account_key='/mnt/ext/opt/QcloudSSLCertificate/cert/account/key', acme_dir='/mnt/ext/opt/QcloudSSLCertificate/cert/.well-known/acme-challenge', ca='https://acme-v02.api.letsencrypt.org', cert_file='/mnt/ext/opt/QcloudSSLCertificate/cert/cert_tmp', chain_file='/mnt/ext/opt/QcloudSSLCertificate/cert/chain_tmp', contact=['mailto:myemail@gmail.com'], csr='/mnt/ext/opt/QcloudSSLCertificate/cert/csr', directory_url='https://acme-v02.api.letsencrypt.org/directory', disable_check=False, qpkg_dir='/mnt/ext/opt/QcloudSSLCertificate', quiet=40, verify_type='https', web_document_root='/Web', well_known_dir='/mnt/ext/opt/QcloudSSLCertificate/cert/.well-known')

Traceback (most recent call last):
File "/mnt/ext/opt/QcloudSSLCertificate/bin/acme-tiny/acme_tiny.py", line 385, in main
qpkg_path=args.qpkg_dir, challenge_type=challenge_type, ca_certs=ca_certs, web_document_root=web_document_root_list)
File "/mnt/ext/opt/QcloudSSLCertificate/bin/acme-tiny/acme_tiny.py", line 282, in get_crt
raise ex
ValueError: Challenge did not pass for brak.myqnapcloud.com: {u'status': u'invalid', u'challenges': [{u'status': u'invalid', u'validationRecord': [{u'hostname': u'brak.myqnapcloud.com', u'addressUsed': u'69.218.230.195', u'port': u'443', u'addressesResolved': [u'69.218.230.195']}], u'url': u'https://acme-v02.api.letsencrypt.org/acme/chall-v3/312484283987/j-3grw', u'token': u'scFvB5xfx3ZQSbpuMpm9O8X6FtRBZNReCc_i3npJcRA', u'error': {u'status': 403, u'type': u'urn:ietf:params:acme:error:unauthorized', u'detail': u'Incorrect validation certificate for tls-alpn-01 challenge. Requested brak.myqnapcloud.com from 69.218.230.195:443. Received certificate which is not self-signed.'}, u'validated': u'2024-02-07T04:15:48Z', u'type': u'tls-alpn-01'}], u'identifier': {u'type': u'dns', u'value': u'brak.myqnapcloud.com'}, u'expires': u'2024-02-14T04:15:43Z'}

I set up my LE certificate several years ago. It's been auto-renewing fine for years. I haven't changed anything on the NAS or on my router that I can think of that would cause this to break.

Thanks for any help!

[Osiris]

@pg1

Ensure that your router and QNAP device both accept inbound traffic on ports 80 and 443 which is a requirement from Let's Encrypt.

Have you doublechecked this? Because I can't reach your domain (IP address 69.218.230.195) from my location. Please doublecheck using online services or your mobile phone without using WiFi. Because often something works perfectly fine through the local network, but not using the internet.

[pg1]

Thank you for the speedy response!

Yes, I have checked quite a few times actually. But I turned off the port forwarding rules on my router last night, for security. But they're back on right now.

Here's a screenshot of the connectivity test on the NAS. If I turn off the port forwarding rules on my router, they all fail, as expected. If I turn them back on, they pass.

Also, looking at the logs that I posted, it appears that the uploading or writing of the challenge files is working, but it's unable to get the files, or it doesn't like what it's getting. Maybe. I'm definitely not an expert on these logs or this whole process.

[MikeMcQ]

I'm not sure what this means exactly but your ACME TLS-ALPN Challenge fails with:

"type": "tls-alpn-01",
    "status": "invalid",
    "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Incorrect validation certificate for tls-alpn-01 challenge. 
Requested brak.myqnapcloud.com from 69.218.230.195:443. 
Received certificate which is not self-signed.",
        "status": 403
    },

We don't see problems with TLS-ALPN challenge that often. So, I am not sure what is causing the error about "not self-signed" cert.

The problem is easily reproduced using Let's Debug and selecting TLS-ALPN challenge (link here)

I see that using HTTPS (port 443) to your domain there is a problem with the certificate chain. So, maybe that is related. I have no idea how to adjust that with a QNAP system though.

Have you tried posting on a QNAP forum?

See your faulty chain with a tool like this

SSL Checker

[pg1]

MikeMcQ - Thank you for that insight and the links to those tests. Very interesting!

I don't know where the last certificate in the chain came from. The first two are from Let's Encrypt. I don't know what Digital Signature Trust Co. is. I found a post here that references that issuer, but I'm quickly getting beyond my depth of understanding. But there are no custom root certificates installed on my NAS.

Any ideas how I fix this issue with the chain?

Fwiw, I did just post to the QNAP forums here.

[pg1]

Since there seemed to be a problem with the LE certificate chain, I released the LE certificate and went back to the default QNAP certificate.

The Let's Debug test shows green - All OK:

Test result for brak.myqnapcloud.com: OK

So I tried to get a new certificate from Let's Encrypt. It took a long time and then said "The server is busy or the network is disconnected! Make sure the NAS can be reached and then reload the page."

I reloaded the page and - low and behold - I have a new LE certificate! (It expires in May.)

But the SSL checker test still shows a problem with the chain:

SSL Checker

So I guess I'm going to have to go through this process of "release and get a new certificate" in May, and maybe going forward, until someone figures out this chain issue.

[Osiris]

SSL Checker complains about the expired DST Root CA X3, but that's not something to complain about. Your chain is perfectly fine (for most circumstances). In a few days this won't be an issue any longer if you/QNAP use/uses the default provided chain. See Shortening the Let's Encrypt Chain of Trust for more info.

[MikeMcQ]

@Osiris

SSL Checker complains about the expired DST Root CA X3

It is complaining that a self-signed cert shows up in intermediate chain. Normally that checker properly validates the DST Intermediate. In this case it's the DST Root in that chain. It won't go away when the default chain changes soon as that cert is being inserted by something else.

Their failing chain:

From letsencrypt.org with expected DST intermediate

[Osiris]

Huh, I guess I read too fast. I thought it was the signed by DST root ISRG Root X1 intermediate.. Weird? Why would QNAP do that?

[MikeMcQ]

The only good news with that wrong DST root is that clients should ignore it. A well-behaved client (modern browser) will see the ISRG Root X1 intermediate and be able to chain to a trusted root on its own system. It won't look any farther in the chain.

A less well-behaved client might, for some reason, object to that being there. I can't name one other than SSL Testing type sites.

It points to a problem in the QNAP or possibly a device you have in front of that. Something on your system is inserting this extra cert. It does not come from Let's Encrypt.

The bigger problem is if that faulty component is creating the whole intermediate chain from a hard-coded file of its own. That will be problematic in the future as Let's Encrypt changes its chains (and it will).

[Bruce5051]

Supplemental information QNAP requires DDNS enabled for doing Let's Encrypt domain validation.
And you have stated that that is happening correctly; I would suggest double checking that myQNAPcloud really believes it from the QNAP side of the world.

[MikeMcQ]

Which is even more puzzling. The ACME Challenge that failed was TLS-ALPN (not a DNS Challenge). TLS-ALPN is similar to HTTP Challenge and shouldn't care who the DNS provider is.

[Bruce5051]

@MikeMcQ

and shouldn't care who the DNS provider is

Correct. But never the less that is what my QNAP NAS shows as a necessity; I think it is so they have some monitoring or control and to "encourage" use of the QNAP Certificate Authority.

[MikeMcQ]

I get that but how is a TLS-ALPN Challenge even done on QNAP then?

[Bruce5051]

This link has QNAP's description https://support.myqnapcloud.com/faq/_faq_q_would-qts-ssl-certificate-qpkg-renew-my-expiring-certificate-from-lets-encrypt-automatically?lang=en

and this one too https://support.myqnapcloud.com/faq/_faq_q_why-cant-i-apply-ssl-certificate-from-lets-encrypt?lang=en