Problem with staging environment


#1

Hello everyone,

After days of research, I couldn’t find a clear answer to my question, so I’m seeking your help.

We are using 2 environments for our websites.

  • 1 server for production / 1VPS for staging.

I duplicate the /etc/letsencrypt directory and recreate links from my production environment (where the cert working just fine) to the staging one.

The configuration seems to be good, the ls -l /etc/letsencrypt/renewal /etc/letsencrypt/live/* command shows :

/etc/letsencrypt/live/domain.com:
total 4
lrwxrwxrwx 1 www-data www-data 46 Jun 1 16:48 cert.pem -> /etc/letsencrypt/archive/domain.com/cert3.pem
lrwxrwxrwx 1 www-data www-data 47 Jun 1 16:48 chain.pem -> /etc/letsencrypt/archive/domain.com/chain3.pem
lrwxrwxrwx 1 www-data www-data 51 Jun 1 16:48 fullchain.pem -> /etc/letsencrypt/archive/domain.com/fullchain3.pem
lrwxrwxrwx 1 www-data www-data 49 Jun 1 16:48 privkey.pem -> /etc/letsencrypt/archive/domain.com/privkey3.pem
-rw-r–r-- 1 www-data www-data 543 Apr 5 15:27 README

(same result on production environment).

My problem is that my certificat not completely working on my staging environment.

The browser reads the cert on the staging env. , but after several page refresh, it drops and shows the old one with this error : net::ERR_CERT_DATE_INVALID

Does anyone has a clue to fix this ?

Thank you very much !!


#2

You should ensure that dns is working fine. Maybe there are multiple distinct address records for the hostname of your staging environment? On subsequent reloads, you may hit different machines.


#3

Thanks for your answer @bytecamp !

The problem is that there is no DNS on our staging environment.

We are toggling between prod and staging with the hosts file :confused:


#4

What do you mean “the old one”? The previous Let’s Encrypt certificate for your site, but from before the most recent certificate renewal?


#5

hi @schoen, yes exactly

Yes exactly, I have one certificate, renewed two times (february, may and july ).

The cert on the production environment expires 27 july.

The cert on staging environment shows the one of july, but after refresh, it shows the may one


#6
  1. It’s a browser problem -> clear the browser cache, use another browser.
  2. It’s a problem that the server sends the wrong certificate.

What happens, if you rename / move / delete the may - certificate?


#7

hi @JuergenAuer,

I already cleared cache / change browser, same result.

In the archive directory, I removed all files and kept only the last cert/chain/fullchain/privkey (cert3/chain3/fullchain3/privkey3)
no change

rename the file cert3/chain3/fullchain3/privkey3 without 3 and recreate links
no change

Dont know what else to try


#8

Does certbot certificates show the right expiration date?

Are you sure you restarted your web server after syncing the certiticates from production?


#9

hi @Patches,

yup , certbot certificates shows me the right expiration date :

-------------------------------------------------------------------------------
Found the following certs:
  Certificate Name: domain.com
    Domains: domain.com www.domain.com
    Expiry Date: 2018-07-27 09:28:29+00:00 (VALID: 51 days)
    Certificate Path: /etc/letsencrypt/live/domain.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/domain.com/privkey.pem
------------------------------------------------------------------------------- 

and yes I restarted the web server :confused:


#10

Maybe revoke the cert on staging, clear all files and directories and try to install again the one from prod would help ?


#11

You probably don’t want to revoke it if it is a copy of your production certificate!

It’s only necessary to revoke a certificate if someone has compromised the private key. Otherwise just deleting it is fine.

If you’re going to restart and try again, I would suggest using tar to copy your /etc/letsencrypt directory so the permissions and symlinks are preserved instead of manually recreating them. It looks like you did that right because certbot certificates worked, but this is often a source of issues.

prod $ sudo tar -cJf certs.tar.xz /etc/letsencrypt
stage $ sudo tar -x certs.tar.xz -C /

#12

just did it, restarted the server and nothing changed :exploding_head:


#13

What certificate file is the web server configured to use?

Are you connecting directly to that server, or a TLS terminating load balancer or CDN or something?

Are you sure the DNS records are right?


#14

hi @mnordhoff,

sorry for answering late.

You mean in the /etc/apache2/sites-available/ssl.conf ?
If yes, it is configured to use the fullchain and privkey in live directory

I’m connecting to the stage in SSH and HTTPS which encapsulates the TLS protocol and there is no ip load balancer.
I switch between stage and prod env. by editing the /etc/hosts file and commenting the ip of the website

the stage env. has no DNS, you can’t access it unless by editing the etc/hosts


#15

Changes in /etc/hosts might not be instant. Look into /etc/nsswitch.conf, section hosts:. If there is a keyword nscd before the keyword file, then you have to reset the name service caching daemon (nscd) after touching /etc/hosts.

On the other hand, the browser might also cache dns data. After switching the environment, you should close all instances of it before connecting to your environment again.


#16

Hi @bytecamp,

my /etc/nsswitch.conf looks like this :

# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         compat
group:          compat
shadow:         compat
gshadow:        files

hosts:          files dns
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis 

I already tried to clear browser cache, close all instance and open again the website, always the same result.
At first , the cert works fine, I have the https and all, but after few refresh, the css fails to load, and the valid cert is not recognized and shows me the old one


#17

Can you see error-details via FireFox - Web-Console or Chrome - Console?


#18

I would check with e.g. the firefox developer tools which host you actually land on.


#19

@JuergenAuer, @bytecamp,

After refreshing 10K times XD to find all errors I could have, It seems to be “better”
Most of the time, the cert is correct, sometimes it fails, but after refresh get back to normal

tests In Chrome:

  1. cert good

  2. refresh -->
    GET https://www.domain.com/wp-includes/css/admin-bar.min.css?ver=4.9.6 0 ()
    (index):47 GET https://www.domain.com/wp-content/plugins/pilipop-decision-tree/decisiontree_local_style.css?ver=4.9.6 0 ()
    (index):49 GET https://www.domain.com/wp-content/plugins/popup-maker/assets/css/site.min.css?ver=1.6.6 0 ()
    (index):56 GET https://www.domain.com/wp-content/themes/pili-pop/style.css?ver=3.6.0 0 ()
    (index):59 GET https://www.domain.com/wp-includes/js/jquery/jquery-migrate.min.js?ver=1.4.1 0 ()
    Uncaught ReferenceError: jQuery is not defined
    at front.js?ver=1.2.39:101
    etc…

    cert still valid

  3. refresh -->
    untrusted website page
    Certificate - missing
    This site is missing a valid, trusted certificate (net::ERR_CERT_DATE_INVALID).

  4. continue anyway -->
    4.1 sometimes all is ok, i have https + browser reads the correct cert, with no errors

    4.2 sometimes cert is ok, i have https + browser reads the correct cert, with errors
    GET https://www.domain.com/wp-includes/css/admin-bar.min.css?ver=4.9.6 0 ()
    (index):45 GET https://www.domain.com/wp-content/plugins/cookie-notice/css/front.css?ver=4.9.6 0 ()
    (index):49 GET https://www.domain.com/wp-content/plugins/popup-maker/assets/css/site.min.css?ver=1.6.6 0 ()
    jquery-migrate.min.js?ver=1.4.1:2 JQMIGRATE: Migrate is installed, version 1.4.1

    4.3 sometimes (rare) no https with cert ok
    Resources - active content with certificate errors
    You have recently allowed content loaded with certificate errors (such as scripts or iframes) to run on this site.
    Resources - content with certificate errors
    This page includes resources that were loaded with certificate errors.
    Certificate - valid and trusted
    The connection to this site is using a valid, trusted server certificate issued by Let’s Encrypt Authority X3.

  5. refresh --> back to step 3, etc etc

In Firefox:

  1. all is fine https / cert good

  2. refresh -->
    non secured connexion but with valid cert

  3. refresh -->
    connexion is not secured page
    error cord : SEC_ERROR_EXPIRED_CERTIFICATE


#20

Wouldn’t it be smarter to just give staging and production different host names instead of juggling the same hostname to different IPs in the /etc/hosts of the client and confusing the browsers that way?