SSL is now failing after switching from certbot to deyhdrated with the namecheap handler


#1

I would like some guidance about what certs to put where, as you can guess this is not exactly my area of expertise :wink:

I was using certbot quite happily until the tls challenge was axed and since port 80 is blocked by the ISP I’ve switched to the dns-01 challenge using dehydrated and the dehydrated_namecheap_dns_api_hook. This does result in new certificates but they don’t seem to be accepted by Apache.

With certbot I would get cert.pem, chain.pem, fullchain.pem and privkey.pem but now with dehydrated I get domain.net-chain.crt, domain.net.crt, domain.net.key and letsencrypt-intermediate-certificates.pem. For Apache I was using cert.pem for the Server Certificate and privkey.pem for the Server Private Key and all was wonderful. Now using the domain.net.crt for the Server Certificate and domain.net.key for the Server Private Key Apache fails to start. Looking in the ssl_error_log I see the following:

[Sat Jan 05 16:45:41 2019] [error] Init: Private key not found
[Sat Jan 05 16:45:41 2019] [error] SSL Library Error: 218529960 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Sat Jan 05 16:45:41 2019] [error] SSL Library Error: 218640442 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error
[Sat Jan 05 16:45:41 2019] [error] SSL Library Error: 218529960 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Sat Jan 05 16:45:41 2019] [error] SSL Library Error: 218595386 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error
[Sat Jan 05 16:45:41 2019] [error] SSL Library Error: 67710980 error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib
[Sat Jan 05 16:45:41 2019] [error] SSL Library Error: 218529960 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag
[Sat Jan 05 16:45:41 2019] [error] SSL Library Error: 218595386 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error

Any suggestions about what certs to use for Apache? This is running on a Centos 6 machine

Thanks


#2

Check for TYPO in path or file name.


#3

Heh, that was my first thought too. I have verified them and even tried copy-paste of the names with the same result. How can I tell what is in the files other than by the file name? Is the choice of files correct? How do I tell? This is all smoke from burning snake oil to me …


#4

Hi @Maturity,

You can use openssl commands to see what’s in the files.

openssl x509 -in somefile -text -noout

to view a file that you think contains a cerificate, and

openssl x509 -in somefile -pubout | openssl -pubin -text -noout

to view the public key associated with a file that you think contains an RSA private key. (I didn’t suggest the commands that view the private key directly because I think this risks having people share the private key accidentally or having other people be in a position to see it, and there’s normally no reason that you have to look at the actual secrets in order to confirm that something is a private key.)

You could try using these on the various files in question to see what they contain. Another option for PEM-formatted files in general is

grep -- ----- somefile

(the exact number of hyphens is important here!) in order to see what kinds of PEM objects are present in somefile, without looking at their contents.

The one important limitation of the commands above is that openssl is only willing to look at the first object in a file that contains multiple objects. So for example if you used the command I suggested on the Certbot-generated fullchain.pem, it would only show the end-entity (leaf) certificate, not the intermediate (issuer) certificate, even though both are present one after another in the same file.


#5

You can also view the contents of PEM files as text with cat, but they might not make very much sense because they’re all likely to consist mainly of encoded random data something like

y5rT2CiGWT5JM/FILuI/sQTx2kiGImY4qijxVeMDWn1kK017bAN9d3dWPF+QgbKzH0qUfaejUXt5
iCsqrZVN6ZcLVMtk1cXjeqgKaAMeHcZWEn4HYP/neiW8mvWxsU5bFhYF2gTDfexg4TEWMedxXczW
ibrUn51HfqdoFf/TavEiXuAPqBVpvhy4HP6ov0ojv7SMYUvAKoqyGmkWq/U4Au+DAdpIFWfoUhkW
UWtXN5rv9DHux0QOEqFg/ebaH8TmsiKdW6rWmA39ZlOgBOrHn79q9ZADA4Es8NysmJBW/Li/Iv1Z
G9sAjc61nOrS1Vg9JHbgsCJA+e75K/xUxbGqfGb5JkjpnOd9ycHrfJ4kvUjVBtzlLl6ny/2DDxqj

regardless of what kind of object is represented. :slight_smile:


#6

Hi @schoen, that was extremely helpful. It appears that the .key file that is generated is bad for some reason. The content begins with
-----BEGIN RSA PRIVATE KEY-----
and ends with
-----END RSA PRIVATE KEY-----
with a lot of encoded content similar to what you show

But openssl says:

140207933949768:error:0906D06C:PEM routines:PEM_read_bio:no start line:pem_lib.c:703:Expecting: TRUSTED CERTIFICATE

So I am guessing that something inside has not been generated correctly for some reason?

— hmm, as I was writing this I thought I would try reading the previously working privkey.pem that certbot generated and it seems to show the same message so now I am back to not understanding what is going on. The new .key file is about twice the size of the old privkey.pem file with more encoded content but same header/footer line.

Just for reference the .crt file is read fine by openssl and it’s BEGIN/END lines indicate just CERTIFICATE

So the files all respond similarly to the working ones and the content looks similar so I don’t see why Apache won’t happily consume them.


#7

If you run openssl x509 on a private key file, you’ll get that error. openssl x509 only understands certificates, not keys.

Could you show us the relevant part of your Apache configuration?


#8

Looking at the two key files, the working privkey.pem from certbot and the kahli.net.key from the namecheap handler for dehydrated I see that the working one starts with
-----BEGIN PRIVATE KEY-----
and the failing one starts with
-----BEGIN RSA PRIVATE KEY-----
Could it be that this RSA text is indicating that it is encoded differently?

The change in the Apache config is minor, I just changed the files pointed to by the config values
SSLCertificateFile
and
SSLCertificateKeyFile
in the /etc/httpd/conf.d/ssl.conf file. I keep the old values just commented out so I can switch back and forth easily. I have never set the certificate chain or certificate authority values and it has worked for Apache and the IMAP server just fine (well, until this update). Currently I’ve set it back to the certbot generated values as they are still fine for the next week or so.


#9

I found this page where it gives some guidance about checking a cert-key pair:
https://support.comodo.com/index.php?/Knowledgebase/Article/View/684/17/how-do-i-verify-that-a-private-key-matches-a-certificate-openssl
And following the simple instructions there the key is fine and matches the certificate just fine. So I guess it is down to figuring out what is different about this new key from letsencrypt via dehydrated that is different from the previous one from certbot


#10

You might check whether all the files are in PEM format (that is, human-readable text with the hyphens and BEGIN and END and so on), whether they can all be read by Apache according to their file permissions, and whether they have any other extraneous information before the PEM data.

Also, of course, make sure that your Apache configuration isn’t pointing to the private key as if it were the cert and the cert as if it were the private key!


#11

Both the certificate and the private key produced by dehydrate appear to be formatted correctly and all the checks that I have been able to find indicate that they are valid and matching. I am simply not seeing what Apache is not happy with. The only difference that I can see is that the new private key contains about twice the encoded content as the old private key that Apache is happy with. I have found a number of instances of this on the internet but either the cause is obvious or no solution was reported - very frustrating.

And yes, the permissions are fine (more open than the Apache docs indicate even) and the cert and private key are configured correctly in the ssl config.


#12

Well I have to say that it is resolved but I don’t 100% understand what the problem was.

That is to say I removed and then re-applied all the changes to move to the new cert/key set set the permission to 777 then back to 744 because I didn’t want it being too open and then restarted httpd and it came up fine. So I can only imagine that it was a typo somewhere?

Thanks for looking at this issue, your help and comments were greatly appreciated
.


closed #13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.