Cannot load certificate

I am trying to deploy an application using these instructions (How To Secure a Containerized Node Application with Let's Encrypt | DigitalOcean) and am seeing the same error as described here (Nginx: [emerg] BIO_new_file("/etc/letsencrypt/live/domain.com/fullchain.pem") failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/domain.com/fullchain.pem','r')).

My certificates are in the expected location and pass the following dry-run:

sudo certbot certonly --nginx --dry-run -d domain.com -d www.domain.com

Here is the answer!

1 Like

Welcome to the Let's Encrypt Community, Diana :slightly_smiling_face:

Make sure that the correct certificate name shown by:

sudo certbot certificates

is used in place of domain.com here in your nginx configuration:

/etc/letsencrypt/live/domain.com/fullchain.pem

1 Like

Hi @griffin,

Thank you very much for your reply. I do have the correct domain. I edited it for this forum since I am new to security culture and not certain what you should and should not share. In retrospect, I suppose the domain is not bad, so I will post it in case you can spot another problem.

server {
        listen 80;
        listen [::]:80;
        server_name hippocampusanalytics.com www.hippocampusanalytics.com;

        location ~ /.well-known/acme-challenge {
          allow all;
          root /var/www/dsp-ui;
        }

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ /index.html;
        }
}

server {
        listen 443 ssl http2;
        listen [::]:443 ssl http2;
        server_name hippocampusanalytics.com www.hippocampusanalytics.com;

        server_tokens off;

        ssl_certificate /etc/letsencrypt/live/www.hippocampusanalytics.com/fullchain.pem;
        ssl_certificate_key /etc/letsencrypt/live/www.hippocampusanalytics.com/privkey.pem;

        ssl_buffer_size 8k;

        ssl_dhparam /etc/ssl/certs/dhparam-2048.pem;

        ssl_protocols TLSv1.2 TLSv1.1 TLSv1;
        ssl_prefer_server_ciphers on;

        ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DH+3DES:!ADH:!AECDH:!MD5;

        ssl_ecdh_curve secp384r1;
        ssl_session_tickets off;

        ssl_stapling on;
        ssl_stapling_verify on;
        resolver 8.8.8.8;

        location / {
                try_files $uri $uri/ /index.html;
        }

        location @nodejs {
                proxy_pass http://nodejs:8080;
                add_header X-Frame-Options "SAMEORIGIN" always;
                add_header X-XSS-Protection "1; mode=block" always;
                add_header X-Content-Type-Options "nosniff" always;
                add_header Referrer-Policy "no-referrer-when-downgrade" always;
                add_header Content-Security-Policy "default-src * data: 'unsafe-eval' 'unsafe-inline'" always;
                # add_header Strict-Transport-Security "max-age=31536000; includeSubDomains; preload" always;
                # enable strict transport security only if you understand the implications
        }

        root /var/www/dsp-ui;
        index index.html;
}

I also found this discussion that suggested pointing to the archives rather than the symlinks (docker - Reverse proxy cannot load ssl certificates - Stack Overflow). That produced the same error. Each time I think I have fixed it, I am wrong.

Is there a way of checking if I have requested a certificate too many times. I am using the --staging setting for my docker-compose file, but I have tried many different potential solutions.

Also, here is my docker-compose in case you can spot the problem there.

version: '3'

services:
  nodejs:
    build:
      context: .
      dockerfile: Dockerfile
    image: nodejs
    container_name: dsp
    restart: unless-stopped
    networks:
      - app-network

  webserver:
    image: nginx:mainline-alpine
    container_name: webserver
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - web-root:/var/www/html
      - ./nginx-conf:/etc/nginx/conf.d
      - certbot-etc:/etc/letsencrypt
      - certbot-var:/var/lib/letsencrypt
      - dhparam:/etc/ssl/certs
    depends_on:
      - nodejs
    networks:
      - app-network

  certbot:
    image: certbot/certbot
    container_name: certbot
    volumes:
      - certbot-etc:/etc/letsencrypt
      - certbot-var:/var/lib/letsencrypt
      - web-root:/var/www/html
    depends_on:
      - webserver
    command: certonly --webroot --webroot-path=/var/www/html --email diana.ma@hippocampusanalytics.com --agree-tos --no-eff-email --staging -d hippocampusanalytics.com  -d www.hippocampusanalytics.com

volumes:
  certbot-etc:
  certbot-var:
  web-root:
    driver: local
    driver_opts:
      type: none
      device: /home/weilidma/dsp-ui/views/
      o: bind
  dhparam:
    driver: local
    driver_opts:
      type: none
      device: /home/weilidma/dsp-ui/dhparam/
      o: bind

networks:
  app-network:
    driver: bridge
1 Like

I'm at lunch now. I'll check back later. :slightly_smiling_face:

For now though, here's a link to your certificate history:

https://crt.sh/?Identity=hippocampusanalytics.com&deduplicate=Y

and here's a link to the rate limits:

Having helped thousands of people with similar problems, I can say for my part that what you've posted is very common and shouldn't contain anything particularly useful to an attacker. It might identify weak protocols and/or such, but by the time we're done it should serve as more of a deterrent than an aid to an attacker. :slightly_smiling_face:

That said...

You generated two certificates today: one covering hippocampusanalytics.com and one covering www.hippocampusanalytics.com. These will have different certificate names in certbot. What you really want is one certificate covering both hippocampusanalytics.com and www.hippocampusanalytics.com. That certificate should be named "hippocampusanalytics.com".

If anything fails at any point in the following process, please stop there and let us know.


Let's set the correct certificate name.

Change this:

ssl_certificate /etc/letsencrypt/live/www.hippocampusanalytics.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/www.hippocampusanalytics.com/privkey.pem;

to this (by removing the two instances of "www."):

ssl_certificate /etc/letsencrypt/live/hippocampusanalytics.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/hippocampusanalytics.com/privkey.pem;

Let's reload the nginx configuration to reflect the changes.

docker kill -s HUP webserver


Let's get the correct certificate acquired and renewal configuration file created.

sudo certbot certonly --cert-name hippocampusanalytics.com --nginx -d "hippocampusanalytics.com,www.hippocampusanalytics.com" --deploy-hook "docker kill -s HUP webserver"


Let's test the renewal.

sudo certbot renew --cert-name hippocampusanalytics.com --dry-run


Let's run a real renewal to test the deployment hook that restarts nginx:

sudo certbot renew --cert-name hippocampusanalytics.com --force-renewal


If you've made it this far, things should be working as expected. You can now remove any unneeded certificates by...

finding the name of any unneeded certificate:

sudo certbot certificates

deleting the certificate using its name:

sudo certbot delete --cert-name name

Keep in mind that a certificate's name is just a label and thus deleting a certificate that happens to have a particular name (e.g. www.hippocampusanalytics.com) does not remove coverage by another certificate of any domain names. Multiple certificates can independently cover the same domain name (though this is usually a sign of misunderstanding of what names should be covered by a particular certificate).

For your security, you should remove these:

TLSv1.1 TLSv1

from:

ssl_protocols

because those protocols are considered weak.

You can consider adding:

TLSv1.3

in front of:

TLSv1.2

Be sure to reload nginx once you do any of those:

docker kill -s HUP webserver

1 Like

@griffin Thank you for your detailed instructions and encouragement :slight_smile:

I made it successfully up to the dry-run test and revoked the certificate for www.hippocampusanalytics.com so you can reuse it (if that is even a concern).

The dry-run test suggests that the domain is invalid

Challenge failed for domain hippocampusanalytics.com
http-01 challenge for hippocampusanalytics.com
Cleaning up challenges
Failed to renew certificate hippocampusanalytics.com with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/hippocampusanalytics.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: hippocampusanalytics.com
   Type:   unauthorized
   Detail: Invalid response from
   http://hippocampusanalytics.com/.well-known/acme-challenge/S5eBcRWKVfaL5dk8xVDBXX9_DJNY-cYKsYIqWbTUAYE
   [159.65.162.220]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body>\r\n<center><h1>404 Not
   Found</h1></center>\r\n<hr><center>nginx/1.19.10</c"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

docker-compose shows that the server is up, but a curl test fails.

The A record on my ISP's cPanel contains the inet address and the AAAA record contains the inet6 address.

in the docker-container logs it appears that certbot wants me to make a choice, but I do not know how I would answer the question:

Plugins selected: Authenticator webroot, Installer None
certbot      | Cert not yet due for renewal
certbot      |
certbot      | You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
certbot      | (ref: /etc/letsencrypt/renewal/hippocampusanalytics.com.conf)
certbot      |
certbot      | What would you like to do?
certbot      | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
certbot      | 1: Keep the existing certificate for now
certbot      | 2: Renew & replace the certificate (may be subject to CA rate limits)
certbot      | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
certbot      | An unexpected error occurred:
certbot      | EOFError
certbot      | Please see the logfiles in /var/log/letsencrypt for more details.
certbot      | Select the appropriate number [1-2] then [enter] (press 'c' to cancel): Saving debug log to /var/log/letsencrypt/letsencrypt.log
certbot      | Plugins selected: Authenticator webroot, Installer None
certbot      | Cert not yet due for renewal
certbot      |
certbot      | You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
certbot      | (ref: /etc/letsencrypt/renewal/hippocampusanalytics.com.conf)
certbot      |
certbot      | What would you like to do?
certbot      | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
certbot      | 1: Keep the existing certificate for now
certbot      | 2: Renew & replace the certificate (may be subject to CA rate limits)
certbot      | - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
certbot      | An unexpected error occurred:
certbot      | EOFError
certbot      | Please see the logfiles in /var/log/letsencrypt for more details.
certbot      | Select the appropriate number [1-2] then [enter] (press 'c' to cancel):

Finally, there were errors in the webserver but I updated the ssl protocols and updated nginx per your last instructions and they went away!

1 Like

I think you might have a saved certbot configuration inside your docker instance. I noticed from your certificate history that you did acquire a correct certificate (covering hippocampusanalytics.com and www.hippocampusanalytics.com), but then you acquired another incorrect certificate afterwards (only covering hippocampusanalytics.com). You should first delete all of the unnecessary certificates from your certbot image (using certbot delete as I instructed above) then update the certificate named hippocampusanalytics.com to cover both using the nginx authenticator instead of the webroot authenticator. When using docker instances, you need to be sure that you are saving the correct certificate in your persistent image and not acquiring a new certificate with each run because you will almost definitely hit the rate limits in short order! The persistent certificate should then be renewed as needed using certbot renew.

I believe the command here is involved with the issue:

It should be:

command: certonly --cert-name hippocampusanalytics.com --nginx -d "hippocampusanalytics.com,www.hippocampusanalytics.com" --keep

The --keep tells certbot not to acquire a new certificate if a correct one exists that is not near expiration. It prevents you from hitting the rate limits. Its (very dangerous) opposite is --force-renewal, which tells certbot to acquire a new certificate even if a correct one exists that is not near expiration.

The problem with both the original command and the one that I just gave you is that unlike the run command, the certonly command will not reload nginx when a certificate is acquired/renewed. In non-docker situations, you could just add --deploy-hook "nginx -s reload" to have certbot send a reload signal to the nginx process. In docker though, my understanding (note: I'm not a docker expert) is that you don't have command line access to the nginx process, so you need to use docker through certbot to signal the nginx container to reload via --deploy-hook "docker kill -s HUP webserver".

Hopefully all of this makes things easier. :crossed_fingers:

:upside_down_face:

Hi @griffin

YEA!!! I think I am almost there. :blush:

When I bring the server up, the certificates are served correctly and clients will get a response, but someone (not certain if it is nginx or letsencrypt) runs a test calling for a static response. This is failing. I have included a snippet of the error below. The file .well-known and its contents are not present in the root directory (/var/www/html). I am not certain when that was established but I have been seeing this problem all along. This time, I think the error is real (the server can't find the file) Do I need to reinitialize something to fill that directory?

snippet of the error

webserver    | 18.224.20.83 - - [29/Apr/2021:17:12:27 +0000] "GET /.well-known/acme-challenge/0yo076FL3htjOzMbViolHzfscebpaKW_-HUYDAP8w0c HTTP/1.1" 404 154 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
1 Like

By the way, this is kind of a misconception (revoking certificates has no benefit in terms of the ability to use or issue other future certificates).

1 Like

Going along with what @schoen has wisely said, I created an animated clarification of revocation a while back:

That is most definitely a Let's Encrypt server attempting to validate an http-01 challenge. Keep in mind that the certbot nginx authenticator adds an exception to your nginx configuration to serve the challenge file. Since you already have a similar exception, there might be a conflict. Additionally, your webroot directory (root) for port 80 is only defined inside of your current exception, which might cause a failure to serve the challenge file if certbot's exception interferes. To avoid this issue entirely, we can switch back to the webroot authenticator, but include the correct webroot this time in the certbot command.

1 Like

@griffin Thank you! Also thank you to you and @schoen It makes much more sense to me now :slight_smile: On to the next problem!

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.