Certificate Renewal Failed: Invalid Response

No, I wanted you to add the 3 lines for the location for well-known/acme-challenge like below.

Also show the command you used to restart nginx after changing the conf and any messages from that.

       listen       80;
       server_name  192.168.100.60;

       #charset koi8-r;
       #access_log  /var/log/nginx/host.access.log  main;

       location /.well-known/acme-challenge/ {
         root  /srv/www/htdocs;
       }

       location / {
           proxy_pass http://backend;
           proxy_set_header Host $host;
           proxy_set_header X-Real-IP $remote_addr;
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
           proxy_set_header X-Forwarded-Proto $scheme;
       } 
2 Likes

Ok, the file was like this

#user  nginx;
worker_processes  1;

# load_module lib64/nginx/modules/ngx_http_fancyindex_module.so;
# load_module lib64/nginx/modules/ngx_http_geoip_module.so;
# load_module lib64/nginx/modules/ngx_http_headers_more_filter_module.so;
# load_module lib64/nginx/modules/ngx_http_image_filter_module.so;
# load_module lib64/nginx/modules/ngx_http_perl_module.so;
# load_module lib64/nginx/modules/ngx_http_xslt_filter_module.so;
# load_module lib64/nginx/modules/ngx_mail_module.so;
# load_module lib64/nginx/modules/ngx_rtmp_module.so;
# load_module lib64/nginx/modules/ngx_stream_geoip_module.so;
# load_module lib64/nginx/modules/ngx_stream_module.so;
load_module /usr/lib64/nginx/modules/ngx_http_sticky_module.so;

#error_log  /var/log/nginx/error.log;
#error_log  /var/log/nginx/error.log  notice;
#error_log  /var/log/nginx/error.log  info;

#pid        /run/nginx.pid;


events {
    worker_connections  1024;
    use epoll;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    include conf.d/*.conf;

    server {
        listen       80;
        server_name  192.168.100.60;

        #charset koi8-r;
        #access_log  /var/log/nginx/host.access.log  main;

	 location /.well-known/acme-challenge/ {
         root  /srv/www/htdocs;
       }
        location / {
            proxy_pass http://backend;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;
        }


        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /srv/www/htdocs/;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           /srv/www/htdocs/;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   /srv/www/htdocs/;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;
    
    #    Allow TLS version 1.2 only, which is a recommended default these days
    #    by international information security standards.
    #    ssl_protocols        TLSv1.2;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   /srv/www/htdocs/;
    #        index  index.html index.htm;
    #    }
    #}

    include vhosts.d/*.conf;

}

use this command

service nginx restart
1 Like

Before you had that server defined in your load-balancer.conf file

Did you remove it from there when you put it in the nginx.conf file?

And, why did you delete your server for localhost?
It is harder to debug when you make many changes at one time.

Please check your nginx config running this and show results
Some server name problems do not show up with restart command.

nginx -t

I am starting to think your nginx is faulty. I still do not see the root-test.txt file

Please show results of these. Place the 3 backticks before and after each command please.

find / -name nginx.pid
ps -ef | grep nginx | grep -v grep
sudo netstat -pant | grep -Ei '80|443|nginx|apache'
2 Likes
telsur-lb:~ # nginx -t
nginx: [warn] conflicting server name "192.168.100.60" on 0.0.0.0:80, ignored
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

I'm not sure about the questions, I have only done what we have done here but before someone else had it. I add the results of the commands

telsur-lb:~ # find / -name nginx.pid
/run/nginx.pid
telsur-lb:~ # ps -ef | grep nginx | grep -v grep
nginx     1238  1170  0 Dec04 ?        00:00:00 php-fpm: pool www
nginx     1239  1170  0 Dec04 ?        00:00:00 php-fpm: pool www
root      6273     1  0 08:47 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon off;
nginx     6274  6273  3 08:47 ?        00:02:33 nginx: worker process
telsur-lb:~ # sudo netstat -pant | grep -Ei '80|443|nginx|apache'
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      6273/nginx: master
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      6273/nginx: master
tcp        0      0 0.0.0.0:446             0.0.0.0:*               LISTEN      6273/nginx: master
1 Like

I can understand your frustration. But, your system is behaving odd. We should easily be able to see that root-test.txt file and we cannot. That is usually a simple matter to get working.

Something started going wrong after Aug25. Your system was renewing your certs every week for a long time until then. No certs for this domain have been created since Aug25. Was anything important changed between Aug25 and Sept2?

Right now your nginx config needs change. You have two server definitions using the same name which should not be. Can you put it back the way it was with the two server sections with "listen 80;". See your post #9. You had a server defined for "localhost" in your nginx.conf and you had a server defined for "192.168.100.60" defined in here:

/etc/nginx/conf.d/load-balancer.conf

So,

  1. Put back the "localhost" server definition unless you know you do not want it anymore.
  2. Decide where you want the server definition for the "198.168..." server and delete the one you do not want. Make sure the one that remains has the 3 lines for "location" of the well-known URI that I provided.

After your changes, do

nginx -t
service nginx restart
nginx -T

And again put 3 backticks before and after each command results

Also try:

systemctl status nginx -l

And, let's make sure the root test file still exists:

ls -l /srv/www/htdocs/root-test*

I know this is a lot but something odd is happening. We need to have a consistent and cleanly working nginx to proceed to more detailed checks.

2 Likes

I did not have the server at that time, this is what I see in the history

 2021-08-26 13:06:12 ls
 2021-08-26 13:06:27 crontab -e
 2021-08-26 13:08:45 cd /etc/certbot
 2021-08-26 13:08:47 ls
 2021-08-26 13:08:51 vim certbotrenew9.sh
 2021-08-26 13:09:27 rm certbotrenew9.sh
 2021-08-26 13:10:57 exit
 2021-09-07 13:36:43 vim /etc/ssh/sshd_config
 2021-09-07 13:37:08 service sshd reload

Ok, I already returned it and it remains like this

nginx.conf


#user  nginx;
worker_processes  1;

# load_module lib64/nginx/modules/ngx_http_fancyindex_module.so;
# load_module lib64/nginx/modules/ngx_http_geoip_module.so;
# load_module lib64/nginx/modules/ngx_http_headers_more_filter_module.so;
# load_module lib64/nginx/modules/ngx_http_image_filter_module.so;
# load_module lib64/nginx/modules/ngx_http_perl_module.so;
# load_module lib64/nginx/modules/ngx_http_xslt_filter_module.so;
# load_module lib64/nginx/modules/ngx_mail_module.so;
# load_module lib64/nginx/modules/ngx_rtmp_module.so;
# load_module lib64/nginx/modules/ngx_stream_geoip_module.so;
# load_module lib64/nginx/modules/ngx_stream_module.so;
load_module /usr/lib64/nginx/modules/ngx_http_sticky_module.so;

#error_log  /var/log/nginx/error.log;
#error_log  /var/log/nginx/error.log  notice;
#error_log  /var/log/nginx/error.log  info;

#pid        /run/nginx.pid;


events {
    worker_connections  1024;
    use epoll;
}


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    include conf.d/*.conf;

    server {
        listen       80;
        server_name  localhost;

        #charset koi8-r;

        #access_log  /var/log/nginx/host.access.log  main;

        location / {
            root   /srv/www/htdocs/;
            index  index.html index.htm;
        }


        #error_page  404              /404.html;

        # redirect server error pages to the static page /50x.html
        #
        error_page   500 502 503 504  /50x.html;
        location = /50x.html {
            root   /srv/www/htdocs/;
        }

        # proxy the PHP scripts to Apache listening on 127.0.0.1:80
        #
        #location ~ \.php$ {
        #    proxy_pass   http://127.0.0.1;
        #}

        # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
        #
        #location ~ \.php$ {
        #    root           /srv/www/htdocs/;
        #    fastcgi_pass   127.0.0.1:9000;
        #    fastcgi_index  index.php;
        #    fastcgi_param  SCRIPT_FILENAME  /scripts$fastcgi_script_name;
        #    include        fastcgi_params;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #    deny  all;
        #}
    }


    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   /srv/www/htdocs/;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;
    
    #    Allow TLS version 1.2 only, which is a recommended default these days
    #    by international information security standards.
    #    ssl_protocols        TLSv1.2;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   /srv/www/htdocs/;
    #        index  index.html index.htm;
    #    }
    #}

    include vhosts.d/*.conf;

}

and load-balancer.conf

upstream backend {
	server web1.telsurcallcenter.com;
        server web2.telsurcallcenter.com;
        server web3.telsurcallcenter.com;
}

upstream backendssl {
        server web1.telsurcallcenter.com:443;
        server web2.telsurcallcenter.com:443;
        server web3.telsurcallcenter.com:443;
}    


server {

	listen 443 ssl;
    	server_name viciremote.telsurcallcenter.com;

    	ssl on;
    	ssl_certificate         /etc/certbot/live/viciremote.telsurcallcenter.com/cert.pem;
    	ssl_certificate_key     /etc/certbot/live/viciremote.telsurcallcenter.com/privkey.pem;
    	ssl_trusted_certificate /etc/certbot/live/viciremote.telsurcallcenter.com/fullchain.pem;

        location / {

            	proxy_pass https://backendssl;

		proxy_set_header Host $host;
        	proxy_set_header X-Real-IP $remote_addr;
        	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        	proxy_set_header X-Forwarded-Proto $scheme;
        }

}


server {

        listen 446 ssl;
        server_name viciremote.telsurcallcenter.com;

        ssl on;
        ssl_certificate         /etc/certbot/live/viciremote.telsurcallcenter.com/cert.pem;
        ssl_certificate_key     /etc/certbot/live/viciremote.telsurcallcenter.com/privkey.pem;
        ssl_trusted_certificate /etc/certbot/live/viciremote.telsurcallcenter.com/fullchain.pem;

    	root /srv/www/vhosts/dynportal/;
    	index index.php index.html index.htm index.nginx-debian.html;


    	location / {
        	try_files $uri $uri/ =404;
    	}

    	location ~ \.php$ {
        	try_files $uri =404;
        	include fastcgi_params;
        	fastcgi_pass unix:/var/run/php7-fpm.sock;
        	fastcgi_index index.php;
        	fastcgi_intercept_errors on;
        	fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
    	}

    	location ~ /\.ht {
        	deny all;
    	}
}   
 
server {

    	listen 80;
    	server_name 192.168.100.60;

        location / {
            	proxy_pass http://backend;

        	proxy_set_header Host $host;
        	proxy_set_header X-Real-IP $remote_addr;
        	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        	proxy_set_header X-Forwarded-Proto $scheme;

    	}
}

Is this correct? stop being like this, do the tests that you told me

1 Like

No, your server for the '192...' name in your load-balancer.conf should look like this:

server {

    	listen 80;
    	server_name 192.168.100.60;

        location /.well-known/acme-challenge/ {
           root  /srv/www/htdocs;
        }

        location / {
            	proxy_pass http://backend;

        	proxy_set_header Host $host;
        	proxy_set_header X-Real-IP $remote_addr;
        	proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        	proxy_set_header X-Forwarded-Proto $scheme;

    	}
}

After that changed proceed with these commands I described earlier:
And again put 3 backticks before and after group of commands

nginx -t
service nginx restart
nginx -T

Also try:

systemctl status nginx -l

And, let's make sure the root test file still exists:

ls -l /srv/www/htdocs/root-test*
2 Likes

Do you have a backup of that file from before Aug26? Maybe there is some special processing it did that would explain why this all stopped working on that date.

2 Likes

Ok, ready the changes, these were the results

telsur-lb:~ # nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
You have new mail in /var/mail/root
telsur-lb:~ # service nginx restart
You have new mail in /var/mail/root
telsur-lb:~ # nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
telsur-lb:~ # systemctl status nginx -l
● nginx.service - The nginx HTTP and reverse proxy server
   Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; vendor preset:>
   Active: active (running) since Thu 2021-12-09 12:13:28 PST; 1min 6s ago
  Process: 11877 ExecStartPre=/usr/sbin/nginx -t (code=exited, status=0/SUCCESS)
 Main PID: 11878 (nginx)
    Tasks: 2
   CGroup: /system.slice/nginx.service
           ├─11878 nginx: master process /usr/sbin/nginx -g daemon off;
           └─11879 nginx: worker process

Dec 09 12:13:28 telsur-lb systemd[1]: Starting The nginx HTTP and reverse proxy s>
Dec 09 12:13:28 telsur-lb nginx[11877]: nginx: the configuration file /etc/nginx/>
Dec 09 12:13:28 telsur-lb nginx[11877]: nginx: configuration file /etc/nginx/ngin>
Dec 09 12:13:28 telsur-lb systemd[1]: Started The nginx HTTP and reverse proxy se>

You have new mail in /var/mail/root
telsur-lb:~ # ls -l /srv/www/htdocs/root-test*
-rw-r--r-- 1 root root 52 Dec  8 21:11 /srv/www/htdocs/root-test.txt
1 Like

Unfortunately I have not, they removed the file with all the intention

1 Like

I still cannot see the root-test.txt file from the public internet. I get an http error 404.

From your network, what does this show:

curl -i http://192.168.100.60/root-test.txt
2 Likes

Not Found :confused:

telsur-lb:~ # curl -i http://192.168.100.60/root-test.txt
HTTP/1.1 404 Not Found
Server: nginx/1.14.0
Date: Thu, 09 Dec 2021 20:24:56 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: keep-alive
Vary: accept-language,accept-charset,Accept-Encoding
Accept-Ranges: bytes
Content-Language: en
Expires: Thu, 09 Dec 2021 20:24:56 GMT

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Object not found!</title>
<link rev="made" href="mailto:admin@company.com" />
<style type="text/css"><!--/*--><![CDATA[/*><!--*/
    body { color: #000000; background-color: #FFFFFF; }
    a:link { color: #0000CC; }
    p, address {margin-left: 3em;}
    span {font-size: smaller;}
/*]]>*/--></style>
</head>

<body>
<h1>Object not found!</h1>
<p>


    The requested URL was not found on this server.



    If you entered the URL manually please check your
    spelling and try again.



</p>
<p>
If you think this is a server error, please contact
the <a href="mailto:admin@company.com">webmaster</a>.

</p>

<h2>Error 404</h2>
<address>
  <a href="/">192.168.100.60</a><br />
  <span>Apache</span>
</address>
</body>
</html>


1 Like

Let's see what the logs say. Enable error and access logs for nginx doing this:
Find this line in nginx.conf and uncomment it

#error_log  /var/log/nginx/error.log  notice;

And, change the access log in nginx.conf too. Look for this:


http {
    include       mime.types;
    default_type  application/octet-stream;

    #log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
    #                  '$status $body_bytes_sent "$http_referer" '
    #                  '"$http_user_agent" "$http_x_forwarded_for"';

    #access_log  /var/log/nginx/access.log  main;

Change to these below. Make sure to add the $server_name on the end of the log format

http {
    include       mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for" $server_name';

    access_log  /var/log/nginx/access.log  main;

Then, restart nginx (again) and we will try reaching the root-test.txt file again and check the logs.

2 Likes

The first "t" is lowercase.
The last "T" is uppercase.

Your output shown was lowercase and lowercase (again):

2 Likes

Yeah, I know. I think the logs will tell us more soon. Will probably need another -T after that anyway. Thanks

2 Likes

Ok, ready the changes, the logs of the 2021/12/09 13:06:48 they are from the last changes

2021/12/09 10:07:21 [warn] 8431#8431: conflicting server name "192.168.100.60" on 0.0.0.0:80, ignored
2021/12/09 10:50:55 [crit] 6274#6274: *1541375 SSL_do_handshake() failed (SSL: error:1417D18C:SSL routines:tls_process_client_hello:version too low) while SSL handshaking, client: 192.241.213.115, server: 0.0.0.0:443
2021/12/09 11:56:01 [crit] 6274#6274: *2682200 SSL_do_handshake() failed (SSL: error:1417D18C:SSL routines:tls_process_client_hello:version too low) while SSL handshaking, client: 185.142.236.43, server: 0.0.0.0:443
2021/12/09 13:06:48 [notice] 13502#13502: using the "epoll" event method
2021/12/09 13:06:48 [notice] 13502#13502: nginx/1.14.0
2021/12/09 13:06:48 [notice] 13502#13502: built by gcc 7.4.0 (SUSE Linux) 
2021/12/09 13:06:48 [notice] 13502#13502: OS: Linux 4.12.14-lp151.28.87-default
2021/12/09 13:06:48 [notice] 13502#13502: getrlimit(RLIMIT_NOFILE): 1024:4096
2021/12/09 13:06:48 [notice] 13502#13502: start worker processes
2021/12/09 13:06:48 [notice] 13502#13502: start worker process 13503
2021/12/09 13:16:05 [crit] 13503#13503: *151760 SSL_do_handshake() failed (SSL: error:1417D18C:SSL routines:tls_process_client_hello:version too low) while SSL handshaking, client: 192.241.213.162, server: 0.0.0.0:443

1 Like

Ok, I just made some requests for root-test.txt and some others.

What do both the access log and error log have in them? Put backticks around each separately

1 Like

Well, I just made a number of requests to your domain name with http and https and none of them show in the access log (update: that log was later removed from prior post). Examples:

curl -ik https://viciremote.telsurcallcenter.com/Test-To-https
curl -i https://viciremote.telsurcallcenter.com/MikeMcQ-Forum-Test
curl -i http://viciremote.telsurcallcenter.com/root-test.txt

I do not know how this can happen. It seems like this nginx you are showing is not the one receiving requests for that domain name. It looks like it should but it is not.

I need a break to think about what I suggest next.

Oh, you do not need to delete those logs. It will be better to leave them for reference until problem resolved.

=================

I also am not sure why the 404 error message is formatted like it is. It is not the default error page format and you do not have any custom error pages defined for 4xx error codes. Can you try to find the source for that error?

Maybe try commands like these:

grep -ri 'Object not found!' /srv
grep -ri 'Object not found!' /etc

even try just / instead of /srv or /etc but it will probably take a very long time so just try other folders in your system that might have sources if that is too much.

1 Like

Unrelated to solution:
How long have you had this IP?

1 Like

The first certificate was on 2020-03-16 I believe that approximately from that date

2 Likes