Googlebot gives Bad Gateway 502 but website works in Firefox and Chrome

Help
1

Migrating my website from http to https using Certbot gave me some issues with the Google search engine. Somehow it is not showing the site other than 502 error. I used Certbot auto install/configure to configure my NginX. All other browsers open my website perfectly but somehow Google will not index my website anymore because of the error

My Nginx configuration:

server {

  server_name larsvanamsterdam.nl;

  location / {
    proxy_pass http://localhost:3000; # Change the port if needed
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_cache_bypass $http_upgrade;
  }
                                                                                                                                                                                                                                                listen 443 ssl; # managed by Certbot                                                                                                                                                                                                         ssl_certificate /etc/letsencrypt/live/larsvanamsterdam.nl/fullchain.pem; # managed by Certbot                                                                                                                                                ssl_certificate_key /etc/letsencrypt/live/larsvanamsterdam.nl/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}


server {

    if ($host = www.larsvanamsterdam.nl) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = larsvanamsterdam.nl) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


  listen 80;

  server_name larsvanamsterdam.nl;
    return 404; # managed by Certbot
}

My domain is: larsvanamsterdam.nl

I ran this command: Problem with migrating from http to https using Certbot and NginX on a server with ExpressJS Platform

It produced this output: Running Chrome with Googlebot as useragent gives Bad Gateway 502 error. All other browsers run the website with no problems

My web server is (include version): Linux

The operating system my web server runs on is (include version): Linux Ubuntu 16.04

My hosting provider, if applicable, is: Myself

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0

2

Do you have a log line from the nginx log with the Googlebot request? Does your nginx log_format directive include TLS parameters? If not, see https://docs.nginx.com/nginx/admin-guide/monitoring/logging/#tls_sample on how to add this. Could help.

3

Hi @Larsmanson

there

is your answer.

If you have really that problem, you have a not working (and bad) useragent sniffing in your system.

Find that and change that.

4

I agree with @JuergenAuer here. Simple test:

  • curl -Li http://larsvanamsterdam.nl

Returns a single HTTP 301 redirect to the HTTPS site after which the HTTPS request answers with HTTP 200 immediately and returns your site contents.

However:

  • curl -LiA "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" http://larsvanamsterdam.nl

returns a HTTP 301 redirect to HTTPS (so far so good), but after that your backend keeps returning HTTP 301 redirects too! And it keeps doing so in a loop.

This is also the case for even curl -LiA "Googlebot" http://larsvanamsterdam.nl by the way.

So Express is malfunctioning.

5

Thanks, this will help me finding the issue here

6

Probably some kind of plugin installed I think.

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.