Certbot Returning "unauthorized" and 302 Error on Two Domains

Help
#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domains are:


csnet.live

I ran these two commands:

sudo certbot certonly --dry-run --webroot --webroot-path /Users/MyAccountNameHere/Sites/FirstDomain

sudo certbot certonly --dry-run --webroot --webroot-path /Users/MyAccountNameHere/Sites/SecondDomain

It produced this output:

For billkochman.com and www.billkochman.com:

Waiting for verification...
Challenge failed for domain billkochman.com
Challenge failed for domain www.billkochman.com
http-01 challenge for billkochman.com
http-01 challenge for www.billkochman.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

For csnet.live and www.csnet.live:

Waiting for verification...
Challenge failed for domain csnet.live
Challenge failed for domain www.csnet.live
http-01 challenge for csnet.live
http-01 challenge for www.csnet.live
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

My web server is (include version):

Apache 2.4.46 (part of MAMP PRO 6.0.1 for macOS)

The operating system my web server runs on is (include version):

macOS Big Sur 11.0.1 (previously macOS Catalina 10.15.7)

My hosting provider, if applicable, is:

Not applicable

I can login to a root shell on my machine (yes or no, or I don't know):

Not necessary. I have direct access to my iMac and use the Terminal app with certbot

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

1.9.0

For a number of years now, I have had no problem manually renewing my certificates for billkochman.com and www.billkochman.com using certbot in the macOS Terminal app, and then copying the resulting two certificates and key file over to my MAMP PRO installation.

However, recently, three major changes occurred:

  1. I upgraded from macOS 10.15.7 to macOS 11.0.1
  2. I upgraded MAMP PRO from 5.7.0 to 6.0.1
  3. I registered a second domain name to include as a new host in MAMP PRO

While Apple is constantly tightening machine security, at this point, I am not convinced that the upgrade to macOS 11.0.1 is the source of my problem. In fact, while my memory is a bit fuzzy, I think the certificate issues I am having may have started under macOS 10.15.7.

My suspicion is that the problem may lie with the MAMP PRO upgrade, because some major changes were made between MAMP PRO versions 5.7.0 and 6.0.1.

Chief among these changes are two things which may be causing my certificate difficulties:

  1. MAMP PRO now auto-creates its own self-signed certificates. Well, actually, only two of the three: the certificate itself, and the key file. It does not create a chain file, which I definitely need, being as I am running my sites on Apache.

  2. After many years, Appsolute -- the developers of MAMP PRO -- forced us to move our document roots from the "htdocs" folder -- which resides inside of the main MAMP folder -- to /Users/AccountName/Sites/. While I could be wrong, I suspect that this may be the root cause of the problems I am having now. Worse yet, it is no longer even possible now to choose the "htdocs" folder, or the document root folders it formerly contained.

The way I discovered this current problem regarding certificate creation and renewals is when I used the Terminal app and certbot to create the first certificates for my new domain, csnet.live. That is when I first encountered the "unauthorized" and 302 errors.

After getting shut out due to your rate limits, I became curious and decided to see if I could renew my certificates for billkochman and www.billkochman.com, which I last renewed on October 6th. When I encountered the very same "unauthorized" and 302 errors, that is when I became alarmed.

Following is a list, in order, of everything I have done in order to try to resolve this issue:

  1. Installed XCode Command Line Tools 12.2
  2. Reinstalled Homebrew
  3. Upgraded Certbot from 0.32.0 to 1.9.0
  4. Uninstalled all LetsEncrypt files and folders and started from scratch
  5. Removed all HSTS Preload List redirect code from MAMP PRO’s httpd.conf file
  6. Totally removed htaccess files from both billkochman.com and csnet.live document roots
  7. Totally shut down my LAN and rebooted my two iMacs, router and MAMP PRO

There is NO rewrite or redirection code anywhere that I am aware of, so I don't understand why my server is saying "unauthorized" and throwing the 302 error.

Despite taking all of the above steps, when I use certbot in the Terminal app to try to obtain LetsEncrypt certificates for my two domains, I am still getting the aforementioned feedback in the Terminal app.

I don’t know what else to do or try to fix this. I have done all that I can. Does anyone here have a clear solution that does not involve starting totally from scratch? And actually, that shouldn't be necessary anyway, because my main website is working fine, my blog is working fine, and my social network is working fine.

As I said, I last renewed my certificates for my main website -- billkochman.com -- in October, so they are still good for now. However, with my social network site -- csnet.live -- I do have the SSL warning because it is using a self-signed certificate that was created by MAMP PRO, which is useless. As a result, with csnet.live, for the time being, you have to allow a security exception in your web browser, if you want to access the site. Here are the three URLs:



https://www.csnet.live
2 Likes
Doesn't work with MAMP?
#2

I suspect the Apache configuration may be faulty.
Please show the output of:
apachectl -S

2 Likes
#3

I believe that rg305 has exactly the right idea. I just wanted to clarify a couple of things. The message on the redirected page is misleading. A 302 Found redirect is not actually an error (though it is sometimes used when an error occurs). Most often a 302 Found redirect is used after processing a form on a webpage to direct the visitor to a confirmation or results page. This prevents the visitor from being asked if he/she wants to "resubmit" the form if he/she uses the back button.

Screenshot_20201119-115043_Samsung Internet
Screenshot_20201119-115043_Samsung Internet1231×449 159 KB

Screenshot_20201119-115232_Samsung Internet
Screenshot_20201119-115232_Samsung Internet1316×482 177 KB

2 Likes
#4

Hello rg305. Thank you for your response. While it is helpful, it wasn't exactly the solution that was required. That is because I am NOT using macOS's default Apache installation. In case you are not familiar with it, MAMP PRO is a stack which includes its OWN apache, mySQL, PHP and Nginx binaries -- and a few other things -- and associated files. They are totally separate from the components which are installed by macOS. As a result, the apachectl -S terminal command has no bearing on my particular MAMP PRO installation of the apache server.

But the good news is this: Appsolute -- who is the developer of MAMP PRO -- suggested that I use the "Reset to Factory Settings" command in MAMP PRO, which I did. So, they, and you, were actually thinking along the very same lines. That is, that somehow, something got messed up in the configuration.

Well, upon doing that, I ran certbot again in the macOS Terminal app. First for billkochman.com and www.billkochman.com, and then for csnet.live and www.csnet.live. In both cases, certbot came through this time, and I am happy to report that I now have a set of up-to-date certificates and keys for BOTH domains. You can't imagine how happy I am at this point. :slight_smile:

I have one other related issue, which you may be able to help me with. But I am going to post it as a reply to griffin, who commented after you did, so that I can acknowledge his response as well. Thank you again for your input.

3 Likes
#5

Hello griffin. Thanks for your reply. Yes, rg305 was indeed headed in the right direction; and if you read my reply to him/her, you will see that the certificate issue is now resolved, and I am so relieved!

I have one other related issue which I am hoping either you or rg305 -- or maybe even someone else -- can help me to resolve.

My first domain has been on the HSTS Preload List for some time now. I want to add my new domain -- csnet.live -- to the HSTS Preload List as well. I am not very experienced at this, so what I did is duplicate the redirect directives I am already using in my httpd.conf file, and simply changed the domains from billkochman.com to csnet.live, as you can see here:

For HSTS preload list, we must redirect to https://billkochman.com first,

and then to https://www.billkochman.com after that

ServerAdmin wordweaver777@gmail.com
ServerName billkochman.com
ServerAlias www.billkochman.com
RedirectMatch 301 (.) https://billkochman.com$1
RedirectMatch 301 (.) https://www.billkochman.com$1

For HSTS preload list, we must redirect to https://csnet.live first,

and then to https://www.csnet.live after that

ServerAdmin wordweaver777@gmail.com
ServerName csnet.live
ServerAlias www.csnet.live
RedirectMatch 301 (.) https://csnet.live$1
RedirectMatch 301 (.) https://www.csnet.live$1

While my first domain continues to be preloaded according to https://hstspreload.org, when I check for the new domain -- csnet.live -- it is telling me the following:

Error: HTTP does not redirect to HTTPShttp://csnet.live (HTTP) redirects to https://www.billkochman.com/errors/302.html. The first redirect from http://csnet.live should be to a secure page on the same host (https://csnet.live).

So how do I fix my above code in the httpd.conf file, so that all http requests for billkochman.com first go to https://billkochman.com, and then to https://www.billkochman.com, and all http requests for csnet.live first go to https://csnet.live, and then to https://www.csnet.live, as the HSTS Preload List requires?

I am sure it just requires a small code change to the above, but I am not sure how to combine those two sets of directives into one, so that all http requests go to the right https domain.

Thanks in advance, griffin, rg305, and anyone else who may be able to help me to straighten this out. I appreciate your time and effort. I am an old fellow, and some of this stuff is just a bit over my head.

3 Likes
#6

I'll get back to you after lunch. :blush:

2 Likes
#7

Great! I appreciate it. I'll be going out for an early morning doctor's appointment anyway, and I won't be back until close to noon Friday, our local time, which is GMT+10. I don't know where you live, but when I get home, U.S. East Coast time will be around 10:00 PM Thursday night.

3 Likes
#8

Your redirects should all be 301 Permanent as follows:

http://domain.com -> https://domain.com -> https://www.domain.com

http://sub.domain.com -> https://sub.domain.com

I recommend using a handy-dandy redirect-checker to verify things. :blush:

Right now I'm struggling to interact with your sites. I'm seeing port 443 (https) as closed for both sites.

I'm under the belief that it's where your redirects are occurring in your apache configuration that is causing most (or all) of the headaches. Rudy (@rg305) rightfully suspected this immediately. Hence why he wanted to trace through your apache configuration, which brings us full circle. Many strange things can be afoot in webserver configuration files.

So...

These sites will be our closest friends going forward:

Certificate Histories
2 Likes
#9

Hello Griffin and rg305. I apologize for this late response. I hope that neither of you felt that I was being unappreciative of your help. The truth is that the past week has been rather rough insofar as my computer, web server and websites are concerned. In fact, the situation became so bad, that I was almost ready to give up and throw in the towel. But, I didn't. Instead, I switched to different web server software, and it is a whole new ballgame. To my delight and surprise, the new software works better and faster than MAMP Pro, which I have now ditched after at least nine years of using it, and paying for it every year. But what is also really cool is the fact that the new web server software has automatic certificate generation and renewal built right into it. As a result, I already have certificates for both of my websites. I am still working on the HSTS angle with the new software, and I hope to have that working as well very soon. If either of you have experience with setting up redirects properly in the Abyss Web Server software, please let me know. Thanks!

3 Likes
#10

Glad to hear this.

If you can't figure out how to do the redirects within the web server code, you can always do them in HTML.
If you can separate where the HTTP site is served from where the HTTPS site is served from, you can code the HTTP site with something like this:
[and also send all the HTTP 404 a similar line]

<html>
 <head>
  <meta http-equiv="REFRESH" content="0;url=https://my.domain/">
 </head>
<body>
...
2 Likes
#11

Thanks rg305. I already came across that code yesterday or the day before in an online article which explained three different ways to do a redirect. From what I recall reading, using an HTML header is the last of the ways that should be used. So, I am hoping that the company from whom I purchased my new web server software will provide me with a clear solution within the next day or two. If anyone understands their software, and how to get things done correctly it in it, it is them. So we shall see. Thanks for your contribution though. I will use it as a last resort.

3 Likes
#12

You can use it as an interim solution; which will, al least, get some of the visitors to their correct destination.
It can even stay after they provide the "fix", as the fix will/should override that file altogether.

2 Likes
#13

Well, this seems like a really crazy setup to me, but here is what I've got set up:

Because I have two domains, and because either my web server software -- or perhaps Firefox -- does not seem to recognize having a TLD and its www subdomain in the same certificate, I had to create FOUR document roots.

https://billkochman.com and https://www.billkochman.com point to one document root.

http://billkochman.com and http://www.billkochman.com point to one document root.

https://csnet.live and https://www.csnet.live point to one document root.

http://csnet.live and http://www.csnet.live point to one document root.

So in all, I created EIGHT different hosts which point to FOUR different document roots.

In the two http document roots, there is only an index.html file which contains the edited HTML redirect in the header section, as per your instructions.

While the HSTS website is recognizing the HSTS headers in the https versions of my site, it is NOT recognizing the HTML redirect code in the head sections of the two http index.html files.

I don't know why it doesn't, because if I try to go to the http version of either of my two sites, I am in fact automatically redirected to the https version of the same.

Any ideas?

2 Likes
#14

I can't really replicate the problem you describe.

I can say that the HTTP 404 page should be changed to also redirect to HTTPS.

Did you say that you cleared your browser cache?

2 Likes
#15

Yes, I did clear the cache, and probably more than once.

Are you suggesting that my visitors shouldn't even see a 404 page, and that I should put the same redirect code in its head section as well?

What difference would it make? I mean, if the HSTS site is not recognizing the redirect code in the index.html file, why would it recognize it in the 404 error file?

2 Likes
#16

Anyone visiting an HTTP page should be automatically redirected to the same URL (but HTTPS).
Until you can do that, you should redirect them to HTTPS://base.domain/ instead of HTTP://404.html

Which would you rather see:

  • your HTTPS homepage
  • an HTTP 404 Error page

They are two different things:
The 404 error page is a specifically assigned page shown whenever a file is not found.
The index.html page is the page shown when someone reaches a path that "exists".

Again, all this is just until you get the redirection working correctly.

2 Likes
#17

HSTS should not be deployed until the reliability of the site is confirmed. It could come back to bite you. Other headers as well.

3 Likes
#18

rg305, my friend, I have been a webmaster for over 20 years now. As such, I fully understand what purpose an index.html page and a 404 page each serve. I was simply asking you to clarify if you truly meant that I should place the same redirection code in the 404 error document.

In other words, I don't even understand why the 404 is even a part of this conversation, because as far as I know -- aside from my blog where I am still fixing some links due to the changeover to a new web server software -- no one is getting a 404 on my site. All of the links on my site's many pages are all good and verified more than once with a link-checking program that I use.

And, as I said before, whenever I type any http URL for a page on my site, my web browser automatically takes me to the htttps version instead. My point is, redirection is occurring, so why does the HSTS Preload List site insist that it isn't?

Also, for the record, until just recently, my original site -- billkochman.com -- was on the HSTS Preload List for several years. It was only with these recent problems with MAMP Pro that it was dropped. On the other hand, csnet.live, being a new site, has never been on the HSTS list.

When I was still using MAMP Pro, I was using the following redirection code in my httpd.conf file, and the HSTS Preload List site had no problem with it:

ServerAdmin myaddress@gmail.com
ServerName billkochman.com
ServerAlias www.billkochman.com
RedirectMatch 301 (.) https://billkochman.com$1
RedirectMatch 301 (.) https://www.billkochman.com$1

Again, I don't understand what my 404 error page has to do with any of this? Are you saying that you have visited one or both of my sites, and that you are getting 404s for some pages? Again, you will with my blog, but not with my main BBB site, or with my CSN site, unless there is a link I missed fixing with this recent changeover of web server software.

To prove my point, just visit my CSN site, and you will see that folks are participating on it today. They are not being sent to a 404 page.

2 Likes
#19

I'm not getting a 404 page now. Your message is clear.

2 Likes
#20

The only reason I brought that up was (maybe I read too much into this):

Which I presumed that the HTTP and HTTPS document roots differed.
So that although HTTPS://site/folder/file may work, the same link in HTTP would produce a 404 error.

Again, typing is far from speaking, so I may not get all that you say and visa versa.

2 Likes