Certbot Returning "unauthorized" and 302 Error on Two Domains

Help
#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domains are:


csnet.live

I ran these two commands:

sudo certbot certonly --dry-run --webroot --webroot-path /Users/MyAccountNameHere/Sites/FirstDomain

sudo certbot certonly --dry-run --webroot --webroot-path /Users/MyAccountNameHere/Sites/SecondDomain

It produced this output:

For billkochman.com and www.billkochman.com:

Waiting for verification...
Challenge failed for domain billkochman.com
Challenge failed for domain www.billkochman.com
http-01 challenge for billkochman.com
http-01 challenge for www.billkochman.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

For csnet.live and www.csnet.live:

Waiting for verification...
Challenge failed for domain csnet.live
Challenge failed for domain www.csnet.live
http-01 challenge for csnet.live
http-01 challenge for www.csnet.live
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:

My web server is (include version):

Apache 2.4.46 (part of MAMP PRO 6.0.1 for macOS)

The operating system my web server runs on is (include version):

macOS Big Sur 11.0.1 (previously macOS Catalina 10.15.7)

My hosting provider, if applicable, is:

Not applicable

I can login to a root shell on my machine (yes or no, or I don't know):

Not necessary. I have direct access to my iMac and use the Terminal app with certbot

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

1.9.0

For a number of years now, I have had no problem manually renewing my certificates for billkochman.com and www.billkochman.com using certbot in the macOS Terminal app, and then copying the resulting two certificates and key file over to my MAMP PRO installation.

However, recently, three major changes occurred:

  1. I upgraded from macOS 10.15.7 to macOS 11.0.1
  2. I upgraded MAMP PRO from 5.7.0 to 6.0.1
  3. I registered a second domain name to include as a new host in MAMP PRO

While Apple is constantly tightening machine security, at this point, I am not convinced that the upgrade to macOS 11.0.1 is the source of my problem. In fact, while my memory is a bit fuzzy, I think the certificate issues I am having may have started under macOS 10.15.7.

My suspicion is that the problem may lie with the MAMP PRO upgrade, because some major changes were made between MAMP PRO versions 5.7.0 and 6.0.1.

Chief among these changes are two things which may be causing my certificate difficulties:

  1. MAMP PRO now auto-creates its own self-signed certificates. Well, actually, only two of the three: the certificate itself, and the key file. It does not create a chain file, which I definitely need, being as I am running my sites on Apache.

  2. After many years, Appsolute -- the developers of MAMP PRO -- forced us to move our document roots from the "htdocs" folder -- which resides inside of the main MAMP folder -- to /Users/AccountName/Sites/. While I could be wrong, I suspect that this may be the root cause of the problems I am having now. Worse yet, it is no longer even possible now to choose the "htdocs" folder, or the document root folders it formerly contained.

The way I discovered this current problem regarding certificate creation and renewals is when I used the Terminal app and certbot to create the first certificates for my new domain, csnet.live. That is when I first encountered the "unauthorized" and 302 errors.

After getting shut out due to your rate limits, I became curious and decided to see if I could renew my certificates for billkochman and www.billkochman.com, which I last renewed on October 6th. When I encountered the very same "unauthorized" and 302 errors, that is when I became alarmed.

Following is a list, in order, of everything I have done in order to try to resolve this issue:

  1. Installed XCode Command Line Tools 12.2
  2. Reinstalled Homebrew
  3. Upgraded Certbot from 0.32.0 to 1.9.0
  4. Uninstalled all LetsEncrypt files and folders and started from scratch
  5. Removed all HSTS Preload List redirect code from MAMP PRO’s httpd.conf file
  6. Totally removed htaccess files from both billkochman.com and csnet.live document roots
  7. Totally shut down my LAN and rebooted my two iMacs, router and MAMP PRO

There is NO rewrite or redirection code anywhere that I am aware of, so I don't understand why my server is saying "unauthorized" and throwing the 302 error.

Despite taking all of the above steps, when I use certbot in the Terminal app to try to obtain LetsEncrypt certificates for my two domains, I am still getting the aforementioned feedback in the Terminal app.

I don’t know what else to do or try to fix this. I have done all that I can. Does anyone here have a clear solution that does not involve starting totally from scratch? And actually, that shouldn't be necessary anyway, because my main website is working fine, my blog is working fine, and my social network is working fine.

As I said, I last renewed my certificates for my main website -- billkochman.com -- in October, so they are still good for now. However, with my social network site -- csnet.live -- I do have the SSL warning because it is using a self-signed certificate that was created by MAMP PRO, which is useless. As a result, with csnet.live, for the time being, you have to allow a security exception in your web browser, if you want to access the site. Here are the three URLs:



https://www.csnet.live
2 Likes
#2

I suspect the Apache configuration may be faulty.
Please show the output of:
apachectl -S

2 Likes
#3

I believe that rg305 has exactly the right idea. I just wanted to clarify a couple of things. The message on the redirected page is misleading. A 302 Found redirect is not actually an error (though it is sometimes used when an error occurs). Most often a 302 Found redirect is used after processing a form on a webpage to direct the visitor to a confirmation or results page. This prevents the visitor from being asked if he/she wants to "resubmit" the form if he/she uses the back button.

Screenshot_20201119-115043_Samsung Internet
Screenshot_20201119-115043_Samsung Internet1231×449 159 KB

Screenshot_20201119-115232_Samsung Internet
Screenshot_20201119-115232_Samsung Internet1316×482 177 KB

2 Likes
#4

Hello rg305. Thank you for your response. While it is helpful, it wasn't exactly the solution that was required. That is because I am NOT using macOS's default Apache installation. In case you are not familiar with it, MAMP PRO is a stack which includes its OWN apache, mySQL, PHP and Nginx binaries -- and a few other things -- and associated files. They are totally separate from the components which are installed by macOS. As a result, the apachectl -S terminal command has no bearing on my particular MAMP PRO installation of the apache server.

But the good news is this: Appsolute -- who is the developer of MAMP PRO -- suggested that I use the "Reset to Factory Settings" command in MAMP PRO, which I did. So, they, and you, were actually thinking along the very same lines. That is, that somehow, something got messed up in the configuration.

Well, upon doing that, I ran certbot again in the macOS Terminal app. First for billkochman.com and www.billkochman.com, and then for csnet.live and www.csnet.live. In both cases, certbot came through this time, and I am happy to report that I now have a set of up-to-date certificates and keys for BOTH domains. You can't imagine how happy I am at this point. :slight_smile:

I have one other related issue, which you may be able to help me with. But I am going to post it as a reply to griffin, who commented after you did, so that I can acknowledge his response as well. Thank you again for your input.

3 Likes
#5

Hello griffin. Thanks for your reply. Yes, rg305 was indeed headed in the right direction; and if you read my reply to him/her, you will see that the certificate issue is now resolved, and I am so relieved!

I have one other related issue which I am hoping either you or rg305 -- or maybe even someone else -- can help me to resolve.

My first domain has been on the HSTS Preload List for some time now. I want to add my new domain -- csnet.live -- to the HSTS Preload List as well. I am not very experienced at this, so what I did is duplicate the redirect directives I am already using in my httpd.conf file, and simply changed the domains from billkochman.com to csnet.live, as you can see here:

For HSTS preload list, we must redirect to https://billkochman.com first,

and then to https://www.billkochman.com after that

ServerAdmin wordweaver777@gmail.com
ServerName billkochman.com
ServerAlias www.billkochman.com
RedirectMatch 301 (.) https://billkochman.com$1
RedirectMatch 301 (.) https://www.billkochman.com$1

For HSTS preload list, we must redirect to https://csnet.live first,

and then to https://www.csnet.live after that

ServerAdmin wordweaver777@gmail.com
ServerName csnet.live
ServerAlias www.csnet.live
RedirectMatch 301 (.) https://csnet.live$1
RedirectMatch 301 (.) https://www.csnet.live$1

While my first domain continues to be preloaded according to https://hstspreload.org, when I check for the new domain -- csnet.live -- it is telling me the following:

Error: HTTP does not redirect to HTTPShttp://csnet.live (HTTP) redirects to https://www.billkochman.com/errors/302.html. The first redirect from http://csnet.live should be to a secure page on the same host (https://csnet.live).

So how do I fix my above code in the httpd.conf file, so that all http requests for billkochman.com first go to https://billkochman.com, and then to https://www.billkochman.com, and all http requests for csnet.live first go to https://csnet.live, and then to https://www.csnet.live, as the HSTS Preload List requires?

I am sure it just requires a small code change to the above, but I am not sure how to combine those two sets of directives into one, so that all http requests go to the right https domain.

Thanks in advance, griffin, rg305, and anyone else who may be able to help me to straighten this out. I appreciate your time and effort. I am an old fellow, and some of this stuff is just a bit over my head.

3 Likes
#6

I'll get back to you after lunch. :blush:

2 Likes
#7

Great! I appreciate it. I'll be going out for an early morning doctor's appointment anyway, and I won't be back until close to noon Friday, our local time, which is GMT+10. I don't know where you live, but when I get home, U.S. East Coast time will be around 10:00 PM Thursday night.

3 Likes
#8

Your redirects should all be 301 Permanent as follows:

http://domain.com -> https://domain.com -> https://www.domain.com

http://sub.domain.com -> https://sub.domain.com

I recommend using a handy-dandy redirect-checker to verify things. :blush:

Right now I'm struggling to interact with your sites. I'm seeing port 443 (https) as closed for both sites.

I'm under the belief that it's where your redirects are occurring in your apache configuration that is causing most (or all) of the headaches. Rudy (@rg305) rightfully suspected this immediately. Hence why he wanted to trace through your apache configuration, which brings us full circle. Many strange things can be afoot in webserver configuration files.

So...

These sites will be our closest friends going forward:

Certificate Histories
2 Likes