Let's Debug says to let you know; "All OK!" but certbot didn't work

Help
1

MANY decades experienced system administrator now with 54 domains served by Apache's httpd v 2.4.42 on Fedora Core 28 with recent updates. I decided to get them all on https (instead of just one based on IP) and needed to do them one at a time since they're for different orgs. All went well until one of them wouldn't validate and from then on NO MORE validated, totaling 15 domains that certbot couldn't validate. (Before I started I went through the entire set of 54 domains' DNS to confirm none were being served by a backup server as sometimes happens - all are served by the same web server with nearly identical zone layout.)

I suspect there's a root cause that would apply to all of the ones that didn't validate, but I don't know what to look for. ... For now, I'll just list one - the very first one - that wouldn't validate. Note that I performed the exact same steps for all the domains, including all the ones that worked and all the ones that didn't, the only exception being the very first one I had redirect http to https, but the script went wrong and while I fixed it, I figured I'd do that part by hand later.

It may be worth noting that I went back and ran the same command again (same format as below) on a domain that did validate, and told it not to create a new cert but just attempt to reinstall and that worked. And I tried it telling it to create a new cert, and that worked too, but neither seemed to revalidate...

...I'm assuming it's a simple httpd configuration file error or difference from what certbot expects that's tripping things up since once it failed the first time, ALL subsequent attempts also failed. So, I came here, found the pointer to Let's Debug, ran it and when it said things were "All OK!", I knew it was time to post here!

My domain is: clep-energy.org

I ran this command: # certbot run -d clep-energy.org

It produced this output: .. Yall are intelligent, here's an excerpt of the pertinent bits:

Challenge failed for domain clep-energy.org
http-01 challenge for clep-energy.org

Followed later by:

My web server is (include version): Apache httpd 2.4.42

The operating system my web server runs on is (include version): From uname -a

Linux server1 4.16.3-301.fc28.x86_64 #1 SMP Mon Apr 23 21:59:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

My hosting provider, if applicable, is: None (I'm the hosting provider!)

I can login to a root shell on my machine yes

I'm using a control panel to manage my site no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.0.0 (installed last night)

Thanks. ... And if anyone can tell me if certbot notices the server aliases and includes those in the certs it generates, that'd be great! ... Or what to do about it, if not! :slight_smile:

OH! I just noticed the "verbose" data from Let's Debug - it might be helpful:

Under HTTPCheck:

Request to: clep-energy.org/67.101.113.4, Result: [Address=67.101.113.4,Address Type=IPv4,Server=Apache/2.4.41 (Fedora) OpenSSL/1.1.1d mod_perl/2.0.10 Perl/v5.28.2,HTTP Status=400], Issue:
Trace:
@0ms: Making a request to http://clep-energy.org/.well-known/acme-challenge/letsdebug-test (using initial IP 67.101.113.4)
@0ms: Dialing 67.101.113.4
@191ms: Server response: HTTP 400 Bad Request

HTTPRecords

clep-energy.org. 0 IN A 67.101.113.4
InternalProblem

Failed to query certwatch database to check rate limits: pq: unnamed prepared statement does not exist

LetsEncryptStaging

Challenge update failures for clep-energy.org in order https://acme-staging-v02.api.letsencrypt.org/acme/order/5751349/77115307

acme: error code 403 "urn:ietf:params:acme:error:unauthorized": Invalid response from http://clep-energy.org/.well-known/acme-challenge/153AoBgfPFrS4afVbEivt2T0E_WH0CA1hPcwF3_YOEQ [67.101.113.4]: "\n\n400 Bad Request\n\n

Bad Request</h1"

StatusIO was "Operational"

1 Like
2

The URL currently says:

Bad Request

Your browser sent a request that this server could not understand.
Reason: You're speaking plain HTTP to an SSL-enabled server port.
Instead use the HTTPS scheme to access this URL, please.

Somehow HTTPS got enabled on port 80, the HTTP port.

If I access https://clep-energy.org:80/, it responds with the certificate and website content for clepenergy.org. Try looking in that virtual host first for issues.

Can you also post the virtual hosts for clepenergy.org here? I'm not a Certbot developer, but if Certbot did that, it sounds like a bug.

(By the way, your server also seems to take 5 seconds to respond. Is it on satellite or something?)

When you run Certbot with the -d option, it includes only the names listed on the command line. So you'd have to do "sudo certbot -d example.org -d www.example.org" or so forth.

If you run Certbot without any -d options, it parses the Apache (or Nginx) configuration and gives you a list of alll ServerNames and ServerAliases to choose from.

FYI, the status page says issuance is down now. :grimacing: That has nothing to do with your issue, though.

2 Likes
3

We're working diligently on it :slight_smile:

2 Likes
4

I think mnordhoff i sright - your server configuration seems to be expecting HTTPS requests on port 80. This might be because you have SSLEngine on inside your port 80 virtualhost in your Apache configuration. (Or it might be that Certbot got confused with your Apache config and did the wrong thing).

I just wanted to chime in to add that I've updated Let's Debug to detect and report this problem: Let's Debug

and that the below problem should be fixed now (I think crt.sh added pgbouncer or something and it's being weird with prepared statements):

2 Likes
5

I appreciate how some of you have taken the time to check that domain and how it was behaving, but I think that I was changing the config as some of you, like _az, came looking to try and help. And in any event, I’m not sure where things went wrong, but as I continued to try and listen to the advice here, things seemed to get worse - or maybe they already were worse, I’m not sure. … I think that MAYBE I made things worse when in interrupted my collection of certs for domains to react to the “B” rating based on only having TLS 1.0 and 1.1 as production versions and 1.3 as “experimental”. I definitely made a change to the SSL directives, trying to make things better, and at first I thought that at least they didn’t make anything worse.

However, when I went to figure out the port 80 vs 443 issue I found that the certificate being presented on 80 wasn’t related to the virtual host at all!
Whatever the cause, things had gotten so bad that the only certificate being presented was a generic one that I think was created during the OS install, no matter what site I tried or what the virtual host settings were, and many (maybe all? - didn’t check) of the non https versions if sites didn’t load correctly either, but instead went back to the default document path. …So I did what any sensible administrator would do: I restored the configuration using the backup files I’d created just before beginning - at least, then, everything worked as before.

Since I already had the certificates for a huge fraction of the domains, I decided to try to get the rest of them, and plodded along with certbot run -d commands, and it just worked. This time I made NO ATTEMPT to un-do what I saw as the wrong moves (changes not compatible with the previous configuration strategy) certbot was doing to the config files - wait until the end, I figured. I got all the certs successfully, including subdomains as per mnordhoff’s advice, and then went back and added the appropriate subdomains to their corresponding certs - the few that wouldn’t validate probably are missing the www entries in their zone files. So far, so good!

I then went through all the domains, one at a time and reorganized the config files using the virtual host and ssl directives that certbot had created - rewrite where that’s needed, and then specifying the cert files, of course, testing each domain along the way, http and https, and it all worked out perfectly - or, at least so far as I think I have a few left to deal with - and a few domains that apparently are laying fallow.

I want to thank everyone who has tried to help me! I’ve learned a lot and appreciate the LetsEncrypt / certbot effort immensely!

One small suggestion (I think I have to post in a different location) would be to add a tip in the directions for someone with lots of domains to do, suggesting maybe they validate everything, get the certs, and then, domain at a time, reconfig to their environment’s taste, without addressing other sundry issues like which versions of TLS are being used. … If I had done this, I’d have probably saved my own and several other people’s time.

Oh, one more thing to suggest in docs: Use a browser that doesn’t keep a cache! I found it a LOT more reliable to use lynx than to use Firefox, for example.

Thanks everyone. I think I have a few lingering questions, but maybe I can find them in docs or otherwise, but I know where to come if I get stumped! (I’m now keen on learning about this “…site works only in browsers with SNI support…” and the “experimental” TLS 1.3 stuff. Oh, and I need to learn more about how to securely use “deep linking” to other sites since one site I host uses this technique {appropriately}. )

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.