Reinstalling Certbot After Running Into Nginx Renewal Issues

[dezshaw]

Hello,

I am working on a project as a contractor that was previously worked on by someone else. I'm using Amazon Lightsail with an Ubuntu 16.04.03 instance which I can log onto as root. They actually set up the server using Python Daphne. When they set up the SSL certification, they were using Certbot 0.19.0. This is where it got a bit confusing. In the set of instructions that he gave me for certificate renewal (which is manual), he said to basically run this command

'sudo certbot --authenticator standalone --installer nginx -d prod.thefelixapps.com --pre-hook "service nginx stop" --post-hook "service nginx start"'

and then copy over the new certificates over to the Daphne server/django folder. However, nginx isn't the server being used here, it's Daphne. When I asked him about this, the previous contractor emailed me and said,

I was using nginx just because that was one of the few webservers that certbot worked with at the time. So I ran it and allowed it to copy the certificate files to where nginx would expect them and then just copied the files to where the Python code needed them. Nginx is not used at all and as long as it's directory structure is there this process should be fine. Certbot is more capable now so using the nginx structure might not even be needed anymore.

The last sentence is definitely true. I see from the Certbot installation instructions that they provide an option for unspecified webservers. Since I was running into weird nginx errors that I can't seem to get rid of testing renewal with certbot renew --dry-run that look like this

Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator standalone, Installer nginx
Running pre-hook command: service nginx stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for prod.thefelixapps.com
Waiting for verification...
Cleaning up challenges
nginx: [error] invalid PID number "" in "/run/nginx.pid"
Attempting to renew cert (prod.thefelixapps.com) from /etc/letsencrypt/renewal/prod.thefel
ixapps.com.conf produced an unexpected error: nginx restart failed:

. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/prod.thefelixapps.com/fullchain.pem (failure)

---

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/prod.thefelixapps.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

Running post-hook command: service nginx start
Hook command "service nginx start" returned error code 1
Error output from service:
Job for nginx.service failed because the control process exited with error code. See "syst
emctl status nginx.service" and "journalctl -xe" for details.

1 renew failure(s), 0 parse failure(s)

I figured that it was time to just delete the current SSL certificates, uninstall certbot, and try again. However, I've noticed that there exists an nginx folder of which holds a sites-available and sites-enabled folder with code in it for the domain.

server {
# listen 80;
server_name prod.thefelixapps.com;

location = /favicon.ico { access_log off; log_not_found off; }

location /static/ {
    alias /home/felix/felixserver/staticserver/;
    autoindex off;
}

location / {
    include proxy_params;
    proxy_pass http://unix:/home/felix/felix.sock;
}

listen 443 ssl; # managed by Certbot

ssl_certificate /etc/letsencrypt/live/prod.thefelixapps.com/fullchain.pem; # managed by Ce
rtbot
ssl_certificate_key /etc/letsencrypt/live/prod.thefelixapps.com/privkey.pem; # managed by
Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

if ($scheme != "https") {
    return 301 https://$host$request_uri;
} # managed by Certbot

}

My question can pretty much sum up to, how can I safely start over without getting mixed up in the previous configuration?

Excuse the weird type setting. I don't know why that's happening.

[JuergenAuer]

Hi @dezshaw

@dezshaw

‘sudo certbot --authenticator standalone --installer nginx -d prod.thefelixapps.com --pre-hook “service nginx stop” --post-hook “service nginx start”’

your setup is "a little bit untypical".

You have a running nginx, but you stop it and use the standalone-version.

It should be easier if you don't stop that nginx, instead use it direct with webroot.

So check if your certbot supports the webroot-option. If not, perhaps update - 0.19 is very old.

If there is no need to stop your nginx, this problem

@dezshaw

nginx: [error] invalid PID number “” in “/run/nginx.pid”

doesn't exist.

Check something like

sudo certbot run -a webroot -i nginx -w yourWebRoot -d prod.thefelixapps.com

But that requires, that your running nginx answers ( https://check-your-website.server-daten.de/?q=prod.thefelixapps.com ):

Now, that looks like a firewall:

Domainname	Http-Status	redirect	Sec.	G
• http://prod.thefelixapps.com/
18.220.15.251	-2		1.343	V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 18.220.15.251:80
• https://prod.thefelixapps.com/
18.220.15.251	200		1.370	B
• http://prod.thefelixapps.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
18.220.15.251	-2		1.340	V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 18.220.15.251:80
Visible Content:

The last row should have a http status 404 - Not Found.

[dezshaw]

Hi @JuergenAuer,

nginx isn’t running as a server though. Python Daphne is the running server. The previous contractor only used nginx in conjunction with certbot to obtain certificates which was passed to the Daphne server folder. It isn’t being used at all.

For webroot, the documentation states that the server must be configured to serve files from hidden directories. That currently isn’t the case so I rather scrap that option.

Going back to my original question, since this set up is atypical, I rather start from scratch. Do you know how I could safely do that? Does that only require me deleting the certificates, nginx folder, letsencrypt folder, and uninstalling certbot?

[schoen]

@JuergenAuer is right that it’s quite unlikely that starting from scratch in this sense is more helpful than not doing so.

You don’t need to uninstall and reinstall Certbot; you can start from scratch by deleting all of the contents of /etc/letsencrypt. Note that if you have any web server application that currently refers to files within /etc/letsencrypt/live, it will fail to start once those files are deleted.

[dezshaw]

Ahh ok I see. I did some more reading about webroot and so I think I must've misread what @JuergenAuer said and forgot to mention a couple of things.

The domain name, prod.thefelixapps.com, is actually a database site which can only be accessed with admin privileges. I can reach the admin site just fine and it actually shouldn't respond to the regular domain name. When I run the command

ps aux | grep nginx

I get,

ubuntu 20002 0.0 0.0 12944 1088 pts/2 S+ 03:41 0:00 grep --color=auto nginx

however, I do not see nginx when I run the command ps aux on it's own. I'm not sure what to make of this but I believe that nginx is running.

Now, a webroot is the folder in which my files are being served to the web. In my case this would be my Python Daphne folder which is where the actual server is and also where I was copying my certificates on over to. When I run the command @JuergenAuer wrote, yourWebRoot should be replaced with the path to the Python Daphne directory.

However, the server directory should be made into a hidden directory which I now know how to do. Would you say that this line of thinking is all correct?

[schoen]

@dezshaw

however, I do not see nginx when I run the command ps aux on it’s own. I’m not sure what to make of this but I believe that nginx is running.

You're seeing your own grep command. Compare ps aux | grep foo, ps aux | grep bar.

You can more readily find out what web servers are running with a command like sudo ss -plat.

@dezshaw

However, the server directory should be made into a hidden directory which I now know how to do. Would you say that this line of thinking is all correct?

The only part about "hidden directories" is that the path that Certbot itself will create (as required by the ACME standard) starts with /.well-known; it's "hidden" only in the sense that, since it starts with a . character, ls won't display that directory unless you run it with ls -a.

You don't have to do anything to hide any directories; you just have to ensure that your web server is willing to serve files and directories whose names begin with . (which most are, but some aren't, because of the existence of sensitive files like .htaccess and .htpasswd that webmasters might accidentally allow to be served to users otherwise).

Right now, there is no web server listening on port 80 of prod.thefelixapps.com, so webroot validation is guaranteed to fail. (The Let's Encrypt CA connects via HTTP on port 80, not by HTTPS on port 443, for this kind of validation.)

[dezshaw]

@schoen

Right now, there is no web server listening on port 80 of prod.thefelixapps.com, so webroot validation is guaranteed to fail. (The Let's Encrypt CA connects via HTTP on port 80, not by HTTPS on port 443, for this kind of validation.)

This is in regards to nginx not actually running, correct? My Python Daphne server is running on port 443. What exactly do I do in this situation?

[JuergenAuer]

@dezshaw

My Python Daphne server is running on port 443.

Then your port 80 isn't used. So let nginx run on port 80 and use certonly, so the new certificate isn't installed, no port 443 is created, no port conflict.

[schoen]

Or, you can use certbot --standalone, where Certbot itself will listen on port 80 (temporarily during the validation process).

[dezshaw]

@JuergenAuer I tried running nginx which according to it’s default file in the sites-available directory is set to run on port 80 but it failed to run. The error seemed to center around nginx not being able to bind to port 443. I saw that there was another file in there made for my server folder which had a line that said listen 443 ssl. I figured out that nginx was trying to run my domain from that port so I moved the same file in the sites-enabled folder into a temporary folder outside of the nginx directory. I typed in the command service nginx start and it is now running on port 80!

I’m now going to try the webroot authenticator.

[dezshaw]

I now feel like I'm getting very close. The dry run was successful in testing the command

sudo certbot certonly -a webroot -i nginx -w webroot -d mydomain --dry-run

however, when I run the actual command to replace the certificate, I get in error that looks like this.

Renewing an existing certificate
Performing the following challenges:
http-01 challenge for prod.thefelixapps.com
Using the webroot path /home/felix/felixserver for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. prod.thefelixapps.com (http-01): urn:acme:error:unauthoriz
ed :: The client lacks sufficient authorization :: Invalid response from http://prod.thefe
lixapps.com/.well-known/acme-challenge/1eXFT-_ketooCn-5uWKILbSR4DGegwnOmwgXNQnbBdI [18.220.15.251]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">
\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: prod.thefelixapps.com
   Type:   unauthorized
   Detail: Invalid response from
   http://prod.thefelixapps.com/.well-known/acme-challenge/1eXFT-_ketooCn-5uWKILbSR4DGegwn
OmwgXNQnbBdI
   [18.220.15.251]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I think this has to do with what @JuergenAuer was saying about there being a firewall up. However, on AWS Lightsail server instance page, I have made sure that in the firewall section, under the networking tab, that port 80 is open. This is what is puzzling me at the moment.

[schoen]

@dezshaw

however, when I run the actual command to replace the certificate, I get in error that looks like this.

Is the command completely identical to the --dry-run command, except for omitting the --dry-run option?

[dezshaw]

Yes, it is identical

[dezshaw]

Hey! I finally got my certificates. What I did at the end was stop the nginx service to free up port 80 and then used this command.

sudo certbot certonly --standalone --preferred-challenges http -d mydomain.com

Thank you so much @JuergenAuer and @schoen for your help!