Renew issue with nginx reverse proxy and raspberry pi

Hallo community,
I have the b.m. issue only randomly meaning I found a workaround for it when it appears but it requires always manual intervention, what is really annoying. My configuration:

  • nginx server with reverse proxy on a virtual machine: radulov.net www.radulov.net - receives all 80 and 443 requests from the router
  • apache server on another virtual machine: cloud.radulov.net
  • two raspberry pi station with nginx: pi1.radulov.net , pi2.radulov.net
    All were configured with letsencrypt and at the beginning everything worked as expected. First I noticed that the one pi had expired certificated, what obviously was not automatically renewed. I tried manual renewal what didn't work as well (see error below). The only workaround I found was to delete the certificates and to comment out the changes from the certbot in the nginx config files (pi and proxy), and them made "fresh" certificate installation.
    The weird thing is that after fresh install --dry-run and --force-renew work without the b.m. error but after some (undefined) time, it just stops and gives the below error and then only the workaround works.
    This issue is only with the two raspberries, with the cloud server for example was never the case.
    First I thought I did some configuration change without noticing but it happened already few times and with both raspberries, so I am a bit annoyed doing the manual delete-reinstall workaround.
    Currently the pi1 station is Ok (after the workaround), the pi2 station gives the b.m. error. Both PIs have the same configuration.

My domain is: pi2.radulov.net

I ran this command: sudo certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/pi2.radulov.net.conf


Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for pi2.radulov.net
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (pi2.radulov.net) from /etc/letsencrypt/renewal/pi2.radulov.net.conf produced an unexpected error: Failed authorization procedure. pi2.radulov.net (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://pi2.radulov.net/.well-known/acme-challenge/YjgIqEVdGzB_RIKOg-ONdV_g0_O7zhC8MN9dUNSftd4 [80.108.240.176]: "\n404 Not Found\n

Not Found

\n

The requested URL was". Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/pi2.radulov.net/fullchain.pem (failure)


** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/pi2.radulov.net/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)


1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My web server is (include version): nginx

The operating system my web server runs on is (include version): debian (proxy) and raspbian (pi)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0

nginx proxy config:

server {

     server_name pi2.radulov.net;

    location / {
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $remote_addr;
      proxy_set_header Host $host;
      proxy_pass https://192.168.0.116:443;
    }

listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/pi2.radulov.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/pi2.radulov.net/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
if ($host = pi2.radulov.net) {
return 301 https://$host$request_uri;
} # managed by Certbot

    listen 80;
    server_name pi2.radulov.net;
return 404; # managed by Certbot

}

raspberry nginx config:

server {
server_name pi2.radulov.net;

location / {
    include uwsgi_params;
    uwsgi_pass unix:/var/www/pi2.radulov.net/rasp2.sock;
}

listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/pi2.radulov.net/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/pi2.radulov.net/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
if ($host = pi2.radulov.net) {
return 301 https://$host$request_uri;
} # managed by Certbot

listen 80;
server_name pi2.radulov.net;
return 404; # managed by Certbot
}

1 Like

Welcome to the Let's Encrypt Community :slightly_smiling_face:

I'm a bit confused where (i.e. on which device(s)) you're running certbot. This has a huge impact on the outcome. You would probably benefit from terminating SSL/TLS (i.e. handshaking, encrypting, decrypting) at the reverse proxy if your other devices are behind it in a private network. It would save you loads of headache and configuration issues by taking advantage of SNI in nginx.


You should avoid using this parameter at all costs, especially when dealing with configuration issues. It never results in anything useful except under the most bizarre circumstances. It almost always ends with rate-limiting. You never need to use that parameter with --dry-run.


It looks like you successfully acquired a certificate today for pi1.radulov.net.

Complete Certificate History
1 Like

Thanks for the answer. I am running certbot on all devices, at least I always thought that it should be the correct way.
But what I understood (also in another post today), it is not necessary to have the certificates on both machines, i.e. proxy and device behind it, at least when they are in local network? And I am not sure how exactly to configure them with the certificates on the proxy only but I'll have a look on your link.
But here I may have another issue: I am accessing the devices from the LAN as well at their public names (dnsmasq). I am not sure if it will work if I don't have the certificates on the devices as well.

2 Likes

It's really a matter of trust and access point. The reverse proxy (usually) acts as the gatekeeper for the devices for untrusted (e.g. web) requests. If I'm understanding you correctly, you're also trying to access (the service-end of) the devices around the reverse proxy? Why use https for that? If you terminate SSL/TLS at the reverse proxy (and the devices do not serve outside requests directly), you can just use http between the [reverse proxy and your internal means] and the devices. I hope that makes sense.

Basically, it comes down to who is answering to whom for either a domain name or an IP address. :grin:

1 Like

Actually because of my sync clients - I am synchronizing mobile devices with the cloud server from the local LAN (when at home) and from the Web (when outside), thus I use only one client configuration (https). Some time ago I had similar configuration https form the web and http from the LAN but the sync clients were giving permanently sync error, therefore I decided for the current configuration.

1 Like

Ah... that's a horse of a different color. This should really just come down to IP address and port within your network. Check out the links I gave you and plan out your devices and ports. This should yield the setup you want. I don't believe that anything should stop you from accessing the public interface of your reverse proxy from within your own network (to simplify your world). This would eliminate any duplication/confusion of configuration and functionality.

1 Like

Thanks griffin for your support, I got your point. I'll play around with my configuration and try to simplify it, as I realize now, it is a bit complicated for my needs :slight_smile:

1 Like

If you proxy HTTP, you can get a cert directly from any device in your internal network.
But when those devices apply HTTP redirection to HTTPS, then you start to have a problem.
The reverse proxy must also have the cert that is now only on the inside system.
How do you request the same cert from both systems...?
Short answer: You don't (even try to).
Why? To simplify things. [there are many ways, but some are difficult to configure and maintain]
You can choose from these "simplest" configurations:

  • TLS/HTTPS termination at the proxy (via HTTP auth) & only use HTTP internally.
  • TLS/HTTPS termination at the proxy (via DNS auth for wildcard) & also use HTTPS internally (via HTTP auth).

If your DNS service provider allows for updates via API, there is likely an ACME client that can work for you. If not, then you may opt for using only HTTP internally and have all the certs only on the proxy.

1 Like

Thank you all for your support!
Now I have the solution - TLS/HTTPS termination at the proxy and HTTP internally.
For this to work I had to do only minor changes in my configurations.
Proxy server:
proxy_pass https://192.168.0.116:443;
to
proxy_pass http://192.168.0.116:80;
And then set up the end devices (raspberries) to serve http only and deleted the certificates, even purged the certbot there :slight_smile: .
With the cloud server is a bit more tricky but I have an idea what to do there.
My confusion was that I would need ssl on the end devices for the local network clients but didn't realise that the reverse proxy serves all, i.e. it is doing the termination for all external and internal requests.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.