Unable to renew cert (docker + nginx + letsencrypt)

Help
1

My domain is: efirma.rs (under dev)

I ran this command: sudo docker run -it --rm -p 4343:443 -p 8080:80 --name certbot -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" quay.io/letsencrypt/letsencrypt:latest renew --dry-run --preferred-challenges http (ALSO I TRIED WITHOUT --preferred-challenges http)

It produced this output: SEE BELOW

My operating system is (include version): UBUNTU 16.04

My web server is (include version): NGINX latest release from official docker image

My hosting provider, if applicable, is: digital ocean

I can login to a root shell on my machine (yes or no, or I donā€™t know): yes

Iā€™m using a control panel to manage my site (no, or provide the name and version of the control panel): -

When trying to renew and perform dry-run without stating --preferred-challenges http the ERROR is:

Processing /etc/letsencrypt/renewal/efirma.rs.conf

Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for efirma.rs
tls-sni-01 challenge for www.efirma.rs
Waiting for verificationā€¦
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/efirma.rs.conf produced an unexpected error: Failed authorization procedure. www.efirma.rs (tls-sni-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect validation certificate for TLS-SNI-01 challenge. Requested 83f910c6e9e9d0d64224ec530d582571.1f6e1497c1e6c2af7fe7679e2cf04f18.acme.invalid from 207.154.204.11:443. Received certificate containing ā€˜efirma.rs, www.efirma.rsā€™, efirma.rs (tls-sni-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect validation certificate for TLS-SNI-01 challenge. Requested b0bb872b7e2869a1d003183bc687548b.6cf754297deb33512778a35300ba0c02.acme.invalid from 207.154.204.11:443. Received certificate containing ā€˜efirma.rs, www.efirma.rsā€™. Skipping.
** DRY RUN: simulating ā€˜certbot renewā€™ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/efirma.rs/fullchain.pem (failure)
** DRY RUN: simulating ā€˜certbot renewā€™ close to cert expiry
** (The test certificates above have not been saved.)
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: www.efirma.rs
    Type: unauthorized
    Detail: Incorrect validation certificate for TLS-SNI-01 challenge.
    Requested
    83f910c6e9e9d0d64224ec530d582571.1f6e1497c1e6c2af7fe7679e2cf04f18.acme.invalid
    from 207.154.204.11:443. Received certificate containing
    ā€™efirma.rs, www.efirma.rsā€™

    Domain: efirma.rs
    Type: unauthorized
    Detail: Incorrect validation certificate for TLS-SNI-01 challenge.
    Requested
    b0bb872b7e2869a1d003183bc687548b.6cf754297deb33512778a35300ba0c02.acme.invalid
    from 207.154.204.11:443. Received certificate containing
    ā€™efirma.rs, www.efirma.rsā€™

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A record(s) for that domain
    contain(s) the right IP address.

When --preferred-challenges http is added the ERROR is:

Processing /etc/letsencrypt/renewal/efirma.rs.conf

Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for efirma.rs
http-01 challenge for www.efirma.rs
Waiting for verificationā€¦
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/efirma.rs.conf produced an unexpected error: Failed authorization procedure. efirma.rs (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://efirma.rs/.well-known/acme-challenge/aDqGkwKQDWwu0eYjwUyxQRxy5fxnf0Zl4U1UqNn6nmU: "

502 Bad Gateway

502 Bad Gateway

<cen", www.efirma.rs (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.efirma.rs/.well-known/acme-challenge/6b9YvePOONDm0mTzlWVDrsor-yf7O_v1tG4Pe82ApbM: " 502 Bad Gateway

502 Bad Gateway

<cen". Skipping. ** DRY RUN: simulating 'certbot renew' close to cert expiry ** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/efirma.rs/fullchain.pem (failure)
** DRY RUN: simulating ā€˜certbot renewā€™ close to cert expiry
** (The test certificates above have not been saved.)
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My nginx configuration is:

server {
listen 80;
server_name efirma.rs, www.efirma.rs;

if ($scheme = http) {
    return 301 https://$host$request_uri;
}

location /.well-known {
    proxy_pass http://127.0.0.1:8080/;
}

location / {
  return 301 https://$server_name$request_uri;
}

}

server {
listen 443 ssl;
server_name efirma.rs, www.efirma.rs;

ssl_certificate      /etc/letsencrypt/live/efirma.rs/fullchain.pem;
ssl_certificate_key  /etc/letsencrypt/live/efirma.rs/privkey.pem;

ssl_session_cache    shared:SSL:1m;
ssl_session_timeout  5m;

ssl_ciphers  HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers  on;

location / {
  root /usr/share/nginx/html/;
  index  index.html index.htm;
}

location /.well-known {
    proxy_pass http://127.0.0.1:4343/;
}

}

Any ideas?

2

Iā€™m not familiar enough with nginx configuration to be 100% sure, but I think whatā€™s happening is that that first redirect block is taking precedence over the proxy_pass and redirecting to HTTPS rather than allowing certbot to respond on port 80/8080. The HTTPS config then passes /.well-known to port 4343/443 but certbot isnā€™t listening there for http-01.

Iā€™d try putting that first redirect (in the http config) below the proxy_pass, or changing the https proxy_pass to port 8080.

3

Commenting out:

if ($scheme = http) {

return 301 https://$host$request_uri;

}

doesnā€™t seem to solve the problemā€¦

4

@MladenKobiljski I believe thereā€™s a possible issue here that could relate to the docker networking mode in use.

When nginx is proxying the letsencrypt validation request upstream to 127.0.0.1:8080, itā€™s generating the 502 error because nginx, running inside a container, is not able to connect to 127.0.0.1:8080.

Certbot, running inside a separate container, is the one that is listening on port 8080 on the docker host.

With dockerā€™s default networking mode of ā€œbridgeā€ the address of the 127.0.0.1:8080 relative to the nginx container would be looking for a service on the same container rather than on the docker host. However, with networking mode of host all containers share the hostā€™s networking stack and thatā€™s the scenario where this is expected to work.

Warning, Iā€™m fairly new to both docker and letā€™s encrypt but I think thatā€™s the issue: the nginx upstream mapping 127.0.0.1:8080 is looking on itā€™s own container rather than on the docker host where certbot is listening which causes the 502 gateway error.

1 Like
5

8080 port is only used to proxy requests to internal port 80 that certbot is using for renew authenticationā€¦ Nginx is used only to forward incoming traffic with .well-known location to certbot. I donā€™t know what is cerbot serving on port 80 that is mapped as 8080 so the nginx listening to all traffic on standard http port 80 can forward to it. Do you know any other way I could renew certificate from docker container proxied by nginx?

6

@MladenKobiljski do you know what docker network mode the nginx container is using?

If itā€™s using the default docker networking mode (bridge), one option might be to try using an upstream of 172.17.0.1:8080 instead of 127.0.0.1:8080

Another option, depending on your environment setup and needs, might be to try starting the nginx container using the host networking mode instead which should give it access to 127.0.0.1:8080 without changing nginx.conf.

Iā€™m going to try these out myself in a bit (as Iā€™m still fighting the same issue as you) and will report back here

7

Ok, Iā€™ll also check and try using upstreamā€¦ Whoever solves this first should post correct solution, because I believe anybody with similar setup (docker + nginx + LE) will have the same problem.

8

The first thing I tried was changing network mode to host. Unfortunately, I hit some unrelated issues with host mode networking on AWS ECS (where I'm hosting) which I don't want to sort through at the moment. Next I tried changing my nginx.conf to use 172.17.0.1:8080 (leaving my networking mode at bridge) and that worked.

However, I'm not a huge fan of this hard-coded IP (which happens to be the default IP used by the latest version of docker for the default docker0 network) because it could change (as it has with past versions of docker). In any case, I now feel confident that the 502 errors are not related to certbot or nginx but specifically with whether the networking setup allows for the nginx container to reach 127.0.0.1:8080 (or where ever certbot is launched).

1 Like
9

Thanks, it worked. If I find a better solution than hardcoding IP Iā€™ll post it here.

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.