Certbot reports likely firewall problem with nginx

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: emtpush.emetrotel.net

I ran this command: certbot run --nginx -d emtpush.emetrotel.net

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Requesting a certificate for emtpush.emetrotel.net
Performing the following challenges:
http-01 challenge for emtpush.emetrotel.net
Waiting for verification...
Challenge failed for domain emtpush.emetrotel.net
http-01 challenge for emtpush.emetrotel.net
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: emtpush.emetrotel.net
   Type:   connection
   Detail: 3.227.199.105: Fetching
   http://emtpush.emetrotel.net/.well-known/acme-challenge/GKuL2tkXHFI3iTqayGX67hHqEhWCAbM4jU5khnjGX2U:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version): nginx version: nginx/1.20.1 (I hope that's the web server.)

The operating system my web server runs on is (include version): Centos 7

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): not that I know of

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.11.0

Background: We are trying to move our push server from one place to another. I'm trying to handle this, from the point where a now retired coworker cloned the server into AWS infrastructure. The certificate was copied over but since it has a new domain it doesn't work so I'm trying to generate a new cert for the new domain. I keep getting told that a firewall is blocking something.

I'm not familiar with nginx or anything else in this, I was thrown at the problem, so as much as practical please assume I don't understand what you are talking about and give detailed instructions. ;-}

Port 80 is working enough that if I browse to http://emtpush.emetrotel.net:80 it redirects me as expected (based on the original server's operation) to https://emtpush.emetrotel.net/sessions/new.

Here's the content of my nginx.conf file, as it now exists. I've played with it, to not much benefit. The commented section at the bottom was previously not commented and I though might be part of the problem. Getting rid of it, and making the 80 section up higher have the correct name didn't help at all.

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  emtpush.emetrotel.net;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        location /.well-known/acme-challenge {
          default_type text/plain;
          root /etc/letsencrypt/webroot;
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }


    server {
        server_name emtpush.emetrotel.net; # managed by Certbot
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        # prevents 502 bad gateway error
        proxy_buffers 8 32k;
        proxy_buffer_size 64k;

        # redirect all HTTP traffic to localhost:8088;
        proxy_pass http://localhost:4900;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $http_host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        #proxy_set_header X-NginX-Proxy true;

        # enables WS support
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        proxy_read_timeout 999999999;
        }

        location /.well_known {
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }


        listen [::]:443 ssl ipv6only=on; # managed by Certbot
        listen 443 ssl; # managed by Certbot
        ssl_certificate /etc/letsencrypt/live/push-server.emetrotel.org/fullchain.pem; # managed by Certbot
        ssl_certificate_key /etc/letsencrypt/live/push-server.emetrotel.org/privkey.pem; # managed by Certbot
        include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

#    server {
#    if ($host = emtpush.emetrotel.net) {
#        return 301 https://$host$request_uri;
#    } # managed by Certbot
#
#
#        listen       80 ;
#        listen       [::]:80 ;
#    server_name emtpush.emetrotel.net;
#    return 404; # managed by Certbot
#
#
#}
}

I inserted the section under port 80 for the .well-known directory, but that doesn't seem to have helped.

I ran a test yesterday, and another just now, on https://check-your-website.server-daten.de/. It mostly seems to tell me that trying to get a certificate from you will fail. Not much help. Most things look pretty good.

I have no experience with any of this. Is there something obvious that needs fixing?

...so it seems like that's the place to look for the answer. Does your AWS security policy allow access to this instance on port 80 (and 443)?

3 Likes

I agree with danb35 that your EC2 Security Group is likely not allowing inbound requests from port 80. It is allowing port 443.

But, the nginx you show doesn't have any redirects in the http server block. So, I'm not sure how you would see one.

Also, you have the below location block which is not needed when using the --nginx plug-in of certbot. That plug-in makes needed temp changes directly in your nginx config and reverses that when done. Your location won't interfere it just isn't needed.

4 Likes

Buried somewhere in all that text it says that if I browse to http port 80 it redirects me to https://emtpush.emetrotel.org/sessions/new. I think that means that AWS must be allowing me to port 80. Or is it possible that it is AWS that is doing the redirection, and effectively blocking the certbot attempt?

I can probably figure out who manages the AWS instances now that the other guy retired. What specific question should I ask him?

I got that little section from someone else's post about firewall problems. I can remove it.

If the security group is blocking access is there a recommended solution? Do I have to get the security to poke a hole and allow the certbot through, or open up port 80 completely? Or do I have to change my certificate acquisition process more significantly?

I don't have access to DNS changes, but I'm sure someone must if I have to switch to the DNS query.

Yes, open port 80 :slight_smile: See this Best Practices Recommendation

Your DNS is in Route53 which is supported as a plug-in by Certbot for DNS Challenge. That can be a little tricky to setup but is your best option if you cannot get port 80 opened. It almost certainly is blocked in the EC2 Security Group (although could be blocked elsewhere like VPC Network ACL Rule).

5 Likes

What would a redirect block look like? Anything at all like the commented out section at the bottom of my nginx.conf that I provided?

This would do a 301 redirect

4 Likes

After consulting with my AWS configuration expert he pointed out that we have a domain certificate and recommended that we install that instead of using LetsEncrypt. He is already managing renewals for that and has an automated process to push updates to a group of servers. So I now have that cert and bundle installed and will reset my nginx configuration to be in conf.d as it should be instead of manipulating the base file.

I think the information here may be useful to other people. Thanks for the help.

2 Likes

I see you indeed have a wildcard certificate from a different CA (Sectigo). Note that Let's Encrypt also issues wildcard certificates, so no reason to pay for such a certificate. Let's Encrypt requires the dns-01 challenge, but as your domains DNS is hosted on AWS, this shouldn't be a problem.

4 Likes

I'll pass this point on to the man in charge. Saving money is always appealing in principle. Maybe the next renewal.

2 Likes

Maybe prior to that.
[to ensure all goes when before your paid cert expires]

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.