What is supposed to return for <site>/hello-world/?

I see this:

3 Likes

So did they :slight_smile:

5 Likes

Rex Swain's HTTP Viewer

Code last updated 21 March 2020

Request:

GET http://riseandthrive2022.com/ HTTP/1.1 Host: riseandthrive2022.com User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0 Referer: Rex Swain's HTTP Viewer Connection: Close

Response Header:

HTTP/1.1 301 Moved Permanently Connection: close Date: Wed, 17 Aug 2022 20:34:43 GMT Location: https://riseandthrive2022.com/ Server: nginx Content-Length: 178 Content-Type: text/html

Content (Length = 178):

(CR)(LF) 301·Moved·Permanently(CR)(LF) (CR)(LF)

301·Moved·Permanently

(CR)(LF)
nginx(CR)(LF) (CR)(LF) (CR)(LF)

Done

Elapsed time so far: 1 seconds


The Location: line in the header above would redirect your browser to a new URL:

Location 2

Request:

GET https://riseandthrive2022.com/ HTTP/1.1 Host: riseandthrive2022.com User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:103.0) Gecko/20100101 Firefox/103.0 Referer: Rex Swain's HTTP Viewer Connection: Close

Response Header:

HTTP/1.1 200 OK Connection: close Date: Wed, 17 Aug 2022 20:34:46 GMT Server: nginx Content-Type: text/html; charset=UTF-8 Link: https://riseandthrive2022.com/wp-json/; rel="https://api.w.org/" Link: https://riseandthrive2022.com/; rel=shortlink P3P: CP="ALL DSP NID CURa ADMa DEVa HISa OTPa OUR NOR NAV DEM" Set-Cookie: PHPSESSID=92ob6apgm7r1nctq6m3p7gga9k; path=/ Set-Cookie: pmpro_visit=1; path=/ Strict-Transport-Security: max-age=15768000 X-Content-Type-Options: nosniff X-Frame-Options: SAMEORIGIN

Content (Length = 80863):

2 Likes

Here's the server block for port 80 for this domain:

# -- HTTP
server {
   listen                       INTERNAL.IP.ADDRESS.HIDDEN:80;
   server_name                  riseandthrive2022.com;

   location / {
      return 301 https://$host$request_uri;
   }
}

This hasn't changed any time recently, and we run basically the same block on not only this server but dozens of others.

However if you can confirm that the request should actually be going to /.well-known/, I may be able to check with the client and see if they're doing anything in their app to further redirect.

3 Likes

The redirection looks to be working correctly.

2 Likes

Yes, absolutely. That's how it starts but the Let's Encrypt servers will follow redirects. The error you see is just the last URL it was redirected to that failed (not the first).

Could the internal IP address have changed? My first guess was that the request was now falling into your default nginx server and not the one with the matching server_name. Is that IP address even needed on the listen statement? Couldn't you just listen 80;

5 Likes

Are there any issues with TLS 1.0/1.1 deprecation and SHA-1 deprecation not trying TLS 1.2 first?

3 Likes

And I do not see any issues using https://letsdebug.net/

3 Likes

I found the redirect but I don't quite understand why it is triggered. With the nginx plug-in which shows in your first post, it will insert a response in the port 80 server block.

But, let's pretend that somehow went awry. Your server block would redirect to https and that response looks like below. Note the location directing to hello-world. The other response headers point to wordpress and a possible network device being involved (the p3p header is a clue).

What is puzzling is why the nginx plug-in should even go let it go there though. I still wonder about that internal IP address I mentioned previously.

curl -i https://riseandthrive2022.com/.well-known/acme-challenge/Forum123

HTTP/2 301
server: nginx
date: Wed, 17 Aug 2022 20:52:29 GMT
content-type: text/html; charset=UTF-8
set-cookie: PHPSESSID=99ant1kgcqaic0c3994i2sd97n; path=/
expires: Wed, 11 Jan 1984 05:00:00 GMT
cache-control: no-transform, no-cache, no-store, must-revalidate
p3p: CP="ALL DSP NID CURa ADMa DEVa HISa OTPa OUR NOR NAV DEM"
set-cookie: pmpro_visit=1; path=/
link: <https://riseandthrive2022.com/wp-json/>; rel="https://api.w.org/"
x-redirect-by: WordPress
location: https://riseandthrive2022.com/hello-world/
strict-transport-security: max-age=15768000
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
5 Likes

Hardenize show an error:

3 Likes

I only see a Hardenize error for the www domain name but they have never gotten a cert with that name. The www name is not shown in the first post error message or the nginx server block for port 80. It's not required to use www names.

5 Likes

In many cases you can just use listen 80;, and we often do. It probably could even be done here, but it would require changing it on all the sites on the box simultaneously and it's not needed. (And yes, I've verified that the IP is the correct IP. There's only one on the box so the fact that you can see the site at all is further confirmation that it's correct.)

Thanks for the confirmation that it's hitting the /hello-world/ because of a redirect in the code, not in Nginx. This particular client has a customized variant of WordPress, and that's most likely where the redirect is coming from. I'll be talking with our client a bit more as it's sounding like a code issue, though yes I'm also wondering how it got past the nginx plug-in. Is there a way I can look at that plug-in to see what exactly it's inserting?

2 Likes

Yes. Look at /var/log/letsencrypt/letsencrypt.log

They rotate so if you've done something since just search the older ones with number suffix.

You will see the entire nginx config twice. The original and the updated. I think you'll understand but ask if any questions.

EDIT: Certbot and nginx plug-in source is on github but likely see enough in log

5 Likes

What shows?:
nginx -T | grep -i hello | grep -i world

4 Likes

Just a follow up, since this was causing us to pull hair out, and I finally found the issue:

There was a 'ghost' Nginx process running, separate from the instance being managed by SystemD. So the commands to reload the Nginx config didn't do anything, as it applied to an nginx process that somehow was disconnected from the actual nginx that was listening on the relevant ports.

I had to do a full kill/stop of all nginx processes, and then a reload - but after that it works correctly.

5 Likes

Glad you found it. Several confounding factors to work through - good job.

Just fyi, the certbot nginx plug-in can cause that in an unusual set of circumstances. It happens because, in rare situations, the plug-in does an nginx start without using systemd. If you want more details about this for future reference let me know.

5 Likes

Definitely interested in more details. We run Certbot+Nginx on various Unix-like platforms (mostly Ubuntu, CentOS, and FreeBSD) for at least a dozen different clients.

2 Likes

Sure, no problem. The nginx plug-in briefly does:

  1. Modify the nginx conf to respond to the authentication challenge.
  2. Reload nginx: nginx -c /etc/nginx/nginx.conf -s reload
  3. Request/get cert which has LE servers sending challenge
  4. Remove the nginx conf updates from step 1
  5. Reload nginx again to remove updates and pickup new cert

A problem can occur if the reload in step 2 fails (or I suppose step 5 too but if it worked in step 2 it should work in step 5).

When the reload fails certbot tries to start nginx but does not use systemd. Instead, it does nginx -c /etc/nginx/nginx.conf. There is now an nginx that systemd is not aware of so you cannot control it as you'd expect.

Now, why might the reload fail? There may be various reasons but the two most common are:

  1. nginx was not running when you started certbot (can't reload if not running)
  2. enabling perl in nginx can sometimes cause a segv fault during reload

The thread below has comments from a certbot dev with more details. It is long and involved. That was my first experience with this, um, quirk :slight_smile: Since then I've personally dealt with a handful of cases.

@_az In that thread you said you'd try to see about using systemd. Have you evaluated that yet?

4 Likes

One of the reasons I prefer to use --webroot instead of nginx or apache plugin.

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.