Certificate name mismatch error on new certs recently made after working correctly for months

Although we have routinely been able to successfully make a couple thousand (2,477) certificates with our script to create and/or renew certs (some details below), yesterday several new certs did not get created correctly. Nothing changed on the script or servers. We had a similar problem a couple months ago (2022-07-01) and I don't think the changes we made back then actually solved the problem. It just started working again.

The cert created has an entirely different name inside the cert. Usually this is one of the recent domains from the past 48 hours with the same IP address. This generates an error that is variously described as:

NET::ERR_CERT_COMMON_NAME_INVALID in Chrome

Domain Matching: Your SSL Certificate does not match your domain name in WhyNoPadlock.com

Certificate name mismatch in SSLlabs.com

name does not match in check-your-website.server-daten.de

Chain - too much certificates, don't send root certificates in the same tool for one of the affected domains.

I do not know why the cert does not use the domain name invoked in the certbot command. Perhaps I have an error there. But it has been working for many months, even years, without only these two problems. Of course the clients and customer service teams want the new sites to go live as quickly as possible so I am motivated to find a real and lasting solution.

Through the same LE account we generate smaller numbers of certs on about a dozen servers. Some of these use hitch but others use nginx for termination. So far they do not seem to be affected but I don't have a lot of new certs (mainly since yesterday) to try out.

I was able to renew 37 certs on this server with the issue yesterday with no problems. All seem to be working unless there is a hidden problem.

In some of the recent cert creations they work most of the time but about 1 in 20 requests will show as insecure, often with a different domain name that is supposedly encoded in the cert. An example of a yesterday cert that mostly works is https://unitedfundingco.homes/


Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: There are several recent certs which are causing problems for me. One is http://pacwestfunding.com/

I ran this command: Inside a script I have a call to

certbot certonly --standalone -n --preferred-challenges http --http-01-port 888 --expand -d "$domain" -d "$wwwdomain" --cert-name "$domain" --post-hook="/usr/local/bin/hitch-post-hook $domain"

where $domain contains pacwesfunding.com and $wwwdomain contains www.pacwestfunding.com. The script checks to see that the DNS for each domain resolves to the expected load balancer IP address. If only one resolves, another script is used to make a single domain (not combined www and non-www in one) cert.

It produced this output: The usual Congratulations message.

My web server is (include version):

We terminate SSL on this server with an older version of hitch (1.5.2, latest is 1.7.3).

This server also uses Varnish and four node servers running Apache/2.4.6 (CentOS)

The operating system my web server runs on is (include version): The server is based on CentOS 7. The uname -a signature is:

Linux mortgage-varnish 3.10.0-1160.49.1.el7.x86_64 #1 SMP Tue Nov 30 15:51:32 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

My hosting provider, if applicable, is: Rackspace

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): This is an older version because we had problems after an upgrade: certbot 1.11.0

Your hitch is sending a totally incorrect certificate for the hostname www.pacwestfunding.com. (It's a cert for unitedfundingco.homes and www.unitedfundingco.homes.

You should double and triple-check your Apache configuration.

4 Likes

Yeah, it sounds like your problem isn't with the certbot command (as you're running certonly, it's just getting the new certificate) but with whatever part of the process (that you haven't expounded on) that configures your servers to use those certificates.

3 Likes

You got certificates yesterday crt.sh | pacwestfunding.com

1 Like

On the affected domains, we were finding that the new ones are getting diferent domains in the certs from hitch (SSL termination made to work with Varnish).

Looking back at the last time this happened, we found that memory consumption of the then 15 GB server was very high. Part of the solution then was to increase the virtual server's memory to 30 GB.

Today the consumption was again very high, particularly for hitch. We have resized this server to 60 GB and things are working better again.

It appears that when the server is running short of RAM, hitch starts to malfunction.

In terms of looking at the Apache configurations, I would note that the other older certs domains were working fine even with the problem. it was only the new ones.

For now I think we have a solution though some tuning may be needed for both Varnish and hitch on this server to ensure they use the appropriate amount of memory, child processes, etc.

Perhaps this post will help others who may have similar situations.

1 Like

Yes, they were new client domains. After making the certs, when we were looking at various tools, including crt.sh, sslshopper.com, that the certs themselves seemed OK. But when the browser and certain tools saw them, there was the name mismatch.

This appears to have been a RAM resource issue where hitch was running out of resources. At this time I am 95% sure this is the cause but am looking at things to be 100% sure.

1 Like

Hitch is a fairly simple and small program, right? Is it proxying millions of sites that it has such a high memory consumption? Or does it have a memory leak perhaps?

3 Likes