Certbot certificate renewal failure

My domain is: abs.skvare.com, civi-current-dev.skvare.com, d8.learning.skvare.com, dev.uschess.org, etc.

I ran this command: sudo certbot renew --dry-run

It produced this output:

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: abs.skvare.com
  Type:   connection
  Detail: 23.239.28.150: Fetching https://abs.skvare.com/.well-known/acme-challenge/rNM_AKNkHJHHwe5jQbj7PpGxdVUnnzc0gOI50otvH4U: Timeout during connect (likely firewall problem)

My web server is (include version): nginx/1.18.0

The operating system my web server runs on is (include version): Ubuntu 22.04

My hosting provider, if applicable, is: Linode

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

Certbot version: 2.7.4

#########################

Hello. We have an issue with certbot SSL renewal. We have several sites under /etc/nginx/sites-available/.

We have this location block for .well-known directory

location ~ /\.well-known {
    auth_basic off;
    root /etc/letsencrypt;
    allow all;
}

and we have renewal configurations for each site under /etc/letsencrypt/renewal directory.
This is an example for one of them.

#renew_before_expiry = 30 days
version = 2.7.4
archive_dir = /etc/letsencrypt/archive/redacted.com
cert = /etc/letsencrypt/live/redacted.com/cert.pem
privkey = /etc/letsencrypt/live/redacted.com/privkey.pem
chain = /etc/letsencrypt/live/redacted.com/chain.pem
fullchain = /etc/letsencrypt/live/redacted.com/fullchain.pem

#Options used in the renewal process
[renewalparams]
allow_subset_of_names = True
account = 670273d7a9a89f2d3494cf6e38739b1c
rsa_key_size = 4096
post_hook = /bin/systemctl reload nginx
authenticator = webroot
webroot_path = /etc/letsencrypt,
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa
[[webroot_map]]
redacted.com = /etc/letsencrypt

Our certbot version is 2.7.4 we upgraded from 1.32.0 and didnt work either.

We know our issue is related with ip6tables but we already have ACCEPT rules for 443 and 80 ports.

When we try to access acme-challenge files during certbot renew --dry-run we can successfully access the files. So 80 and 443 port shouldn’t be the problem.

When we change the ip6tables INPUT filter from DROP to ACCEPT all sites can renew but when we use INPUT DROP filter most of the sites fails to renew with this error.

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: redacted.com
Type: connection
Detail: xxx.xxx.xxx.xxx: Fetching https://redacted.com/.well-known/acme-challenge/EIJFF3UFqtZJCZtG_Kv9Ca7BGA5LiuBdb9JIWxXIhVg: Timeout during connect (likely firewall problem)

We already tried with minimum set of rules and tried to add 80 and 443 ports ACCEPT rules on top of the input chain but didnt work either. So we stuck in here.

Example nginx configuration can be found in following lines.

server {
    listen [::]:80;
    server_name .redacted.com;
    return 301 https://redacted.com$request_uri;
}

server {
    listen [::]:443 ssl http2;
    server_name redacted.com;
    access_log /var/www/log/access/redacted.access.log main buffer=32k;
    error_log /var/www/log/error/redacted.com.error.log notice;
    limit_conn gulag 200;
    root /var/www/web/redacted.com/web;
    index index.php;

    ssl_certificate /etc/letsencrypt/live/redacted.com/fullchain.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/redacted.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/redacted.com/privkey.pem;
    include ssl_params;

    ## Standard site protection
    include		snippets/standard.conf;

    location ~ /\.well-known {
        auth_basic off;
        root /etc/letsencrypt;
        allow all;
    }

    ## Deny illegal Host headers
    if ($host !~* ^(redacted.com|redacted.com)$ ) {
        return 444;
        break;
    }

    ## Drupal configuration
    include snippets/drupal7-php7.4.conf;

    ## php handling
    include snippets/php7.4.conf;
}

By the way we can see HTTP 200 codes in nginx logs for acme files during the certbot renewal.

We use multi-perspective validation. This means that if your site is going to use the HTTP-01 validation method ( on either IPv4 or IPv6 ), then port 80 should be completely open, as we will perform a distributed set of validations and our validation attempts could come from anywhere on the internet.

Does that help at all?

It does seem that the issue you describe stems from firewall troubles, so if it doesn't help to know that 80 needs to be completely open, then let's continue identifying your firewall setup.

2 Likes

Hi Ezekiel. Thank you for your reply.

Port 80 is completely open for both ipv4 and ipv6. We have these rules at the top of the INPUT chain for both ipv4 and ipv6

-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT

But we cannot renew some of the sites with certbot renew even if our firewall completely open for port 80 and 443

1 Like

Not sure if it's very helpful, but from my point of view port 80 and 443 are indeed open using IPv4, but using IPv6 it's completely down.

The log at the top mentions the IPv4 address, but it also mentions https://, meaning there has been an HTTP to HTTPS redirect. I'm not sure which IP address is shown, the first (successful) attempt or the last, failing attempt?

2 Likes

We are pretty sure it's related with ipv6 but we already have the ACCEPT rules for 80 and 443.

When we change the INPUT filter from DROP to ACCEPT in ip6tables all sites can renew the certificates.

Then your (specific) ACCEPT rules aren't working properly :slight_smile:

2 Likes

Right, this line seems like the most critical point. If you're employing a filter that DROPS some ipv6 traffic, and when that filter is removed, then everything works, then it stands that the filter is causing the necessary ports to NOT be completely open.

How would you feel about posting the complete set of rules/chains?

1 Like
# Generated by ip6tables-save v1.8.7 on Tue Nov 21 13:20:14 2023

*mangle

:PREROUTING ACCEPT [37611:16600551]

:INPUT ACCEPT [37611:16600551]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [37291:16578055]

:POSTROUTING ACCEPT [37291:16578055]

COMMIT

# Completed on Tue Nov 21 13:20:14 2023

# Generated by ip6tables-save v1.8.7 on Tue Nov 21 13:20:14 2023

*security

:INPUT ACCEPT [35545:16446047]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [37291:16578055]

COMMIT

# Completed on Tue Nov 21 13:20:14 2023

# Generated by ip6tables-save v1.8.7 on Tue Nov 21 13:20:14 2023

*raw

:PREROUTING ACCEPT [37611:16600551]

:OUTPUT ACCEPT [37291:16578055]

COMMIT

# Completed on Tue Nov 21 13:20:14 2023

# Generated by ip6tables-save v1.8.7 on Tue Nov 21 13:20:14 2023

*filter

:INPUT DROP [2066:154504]

:FORWARD DROP [0:0]

:OUTPUT DROP [0:0]

#:bad_packets - [0:0]

#:bad_tcp_packets - [0:0]

#:ban - [0:0]

:icmpv6_packets - [0:0]

:tcp_inbound - [0:0]

:tcp_outbound - [0:0]

:udp_inbound - [0:0]

:udp_outbound - [0:0]

:whitefilter - [0:0]

-A INPUT -p tcp --dport 80 -j ACCEPT

-A INPUT -p tcp --dport 443 -j ACCEPT

-A INPUT -i lo -j ACCEPT

#-A INPUT -j bad_packets

-A INPUT -m set --match-set whitelist_v6 src -j whitefilter

-A INPUT -i eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -s ::1/128 -j ACCEPT

-A INPUT -p tcp -m tcp --dport 27271 -j ACCEPT

#-A INPUT -j ban

-A INPUT -j whitefilter

-A INPUT -m limit --limit 5/min -j LOG --log-prefix "IP6Tables-Input-Dropped: "

-A OUTPUT -o eth0 -j ACCEPT

-A OUTPUT -o lo -j ACCEPT

-A OUTPUT -p tcp -j ACCEPT

-A OUTPUT -p udp -j ACCEPT

#-A OUTPUT -p ipv6-icmp -m state --state INVALID -j DROP

-A OUTPUT -s ::1/128 -j ACCEPT

-A OUTPUT -m limit --limit 5/min -j LOG --log-prefix "IP6Tables-Output-Dropped: "

#-A bad_packets -m state --state INVALID -j DROP

#-A bad_packets -p tcp -j bad_tcp_packets

#-A bad_packets -m limit --limit 5/min -j LOG --log-prefix "Bad Packet: "

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j LOG --log-prefix "fp=bad_tcp_packets:2 a=DROP "

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j LOG --log-prefix "fp=bad_tcp_packets:3 a=DROP "

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -j LOG --log-prefix "fp=bad_tcp_packets:4 a=DROP "

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,ACK,URG -j LOG --log-prefix "fp=bad_tcp_packets:5 a=DROP "

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j LOG --log-prefix "fp=bad_tcp_packets:6 a=DROP "

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j LOG --log-prefix "fp=bad_tcp_packets:7 a=DROP "

#-A bad_tcp_packets -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -j DROP

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH,URG -j DROP

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,ACK,URG -j DROP

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -j DROP

#-A bad_tcp_packets -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -j DROP

#-A bad_tcp_packets -m limit --limit 5/min -j LOG --log-prefix "Bad TCP Packet: "

#-A ban -m set --match-set spamhausv6_drop src -j DROP

#-A ban -m set --match-set stopforumspam180dv6_drop src -j DROP

#-A ban -j RETURN

#-A ban -m limit --limit 5/min -j LOG --log-prefix "Banned: "

#-A icmpv6_packets -p ipv6-icmp -m icmp6 --icmpv6-type 128 -j DROP

#-A icmpv6_packets -p ipv6-icmp -m icmp6 --icmpv6-type 11 -j ACCEPT

-A tcp_inbound -p tcp -m tcp --dport 27271 -j ACCEPT

-A tcp_inbound -p tcp -m tcp --dport 2812 -j ACCEPT

-A tcp_outbound -p tcp -j ACCEPT

#-A udp_inbound -p udp -m udp --dport 137 -j DROP

#-A udp_inbound -p udp -m udp --dport 138 -j DROP

-A udp_inbound -p udp -m udp --dport 123 -j ACCEPT

-A udp_outbound -p udp -j ACCEPT

-A whitefilter -p tcp -m tcp --dport 27271 -j ACCEPT

COMMIT

# Completed on Tue Nov 21 13:20:14 2023

# Generated by ip6tables-save v1.8.7 on Tue Nov 21 13:20:14 2023

*nat

:PREROUTING ACCEPT [774:61888]

:INPUT ACCEPT [69:5504]

:OUTPUT ACCEPT [2036:162880]

:POSTROUTING ACCEPT [2036:162880]

COMMIT

These are the current complete set of rules for ipv6

When the DROP filter is in place, and renewal attempts fail, are you seeing messages containing:

IP6Tables-Input-Dropped: 
IP6Tables-Output-Dropped: 

in your firewall log? That seems like that could help a lot.

2 Likes

Yeah. I already tried it but there wasn't any dropped packets related with certbot renewal. There was only dropped packets for port 19999 which is belongs to netdata.

And we have this snippet as standard on all sites for protection.

## Lock-down prometheus metrics endpoint
location /metrics {
    auth_basic           "Administrator’s Area";
    auth_basic_user_file /etc/nginx/auth.passwd;
}

## Password protect directory
location ~* /sites/(.*)/files/protect {
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/auth.passwd;
}

## protect private file download directory
location ^~ /sites/(.*)/files/private {
    internal;
    error_page 404 /index.php;
}

## Outlook verification request
location /autodiscover/autodiscover.xml {
    return 444;
    break;
}

## Only allow request method if set false in not_allowed_method map
if ($not_allowed_method) {
    return 405;
    break;
}

## Block put
set $block_http_put_str "";
if ($request_method = 'PUT') {
    set $block_http_put_str "${request_method}:${host_blocks_put}";
}
if ($block_http_put_str = 'PUT:1') {
    return 405;
    break;
}

## Block options
set $block_http_options_str "";
if ($request_method = 'OPTIONS') {
    set $block_http_options_str "${request_method}:${host_blocks_options}";
}
if ($block_http_options_str = 'OPTIONS:1') {
    return 405;
    break;
}

## Block delete
set $block_http_delete_str "";
if ($request_method = 'DELETE') {
    set $block_http_delete_str "${request_method}:${host_blocks_delete}";
}
if ($block_http_put_str = 'DELETE:1') {
    return 405;
    break;
}

## Deny certain User-Agents
## The ~* makes it case insensitive as opposed to just a ~
set $block_ua_str "";
if ($http_user_agent ~* (bot|domainreanimator|httrack|htmlparser|libwww|JikeSpider|proximic|Sosospider|Jullo|BBBike|WWWOFFLE|Widow|SuperHTTP|BlackWidow|HTTrack|Pixray|CPython|Spinn3r|Abonti|MSIECrawler|Baiduspider|Yandex|Siteimprove|Aboundex|80legs|360Spider|^bot|^Java|^asterias|^attach|^BackWeb|Bandit|^BatchFTP|^Bigfoot|^Black.Hole|^BlackWidow|^BlowFish|^BotALot|Buddy|^BuiltBotTough|^Bullseye|^BunnySlippers|^Cegbfeieh|^CherryPicker|^ChinaClaw|Collector|Copier|^CopyRightCheck|^cosmos|^Crescent|^Custo) ) {
    set $block_ua_str "UA:${host_blocks_ua}";
}
if ($block_ua_str = 'UA:1') {
    return 444;
    break;
}

error_page 500 501 502 504 505 /5xx.html;
location ~* /5xx.html {
    root /var/www/utils;
}

# Common directory for LetsEncrypt.org verifications
location ~ /\.well-known {
    auth_basic off;
    root /etc/letsencrypt;
    allow all;
}

# deny access to dotfiles
location ~ /\. {
    return 444;
}

# Deny access to web.config
location ~* web\.config$ {
    deny all;
    return 444;
}

## Limit certain crawlers
limit_req zone=limitbots;

Oh - we used IPv4 for this particular authorization - is your filter similar for IPv4?

2 Likes

Actually all sites that fails during renewal process return our servers ipv4 address. But the issue rely on ipv6 actually. This is confusing for me too tbh. I thought its related with ipv4 first but we pretty sure its related to ip6tables.

And yes we have similar rules for ipv4.

The IP addr shown in this error response is the only attempt we made, and it failed.

2 Likes

So if it's attempting to ipv4 Is there a chance that ipv4 and ipv6 are conflicting ? It's like ipv6 interfering the connection.

That's not possible, as the error message shows https://. And the http-01 challenge always starts with HTTP. So there had to be 2 requests: one successful HTTP with a redirect and the unsuccessful HTTPS from the error?

Or perhaps I didn't understand you correctly (probably).

4 Likes

Of course you're right - I see that over a span of about 20 seconds we tried IPv6 port 80 AND IPv4 port 80, then IPv6 port 443. I'm not totally confident on these individual requests, but I think the IPv6:80 connection failed, IPv4:80 successfully redirected to HTTPs, then IPv6:443 failed/timed-out.

Good call.

3 Likes

Yes, LE will retry an IPv6 timeout with IPv4 but only for original HTTP request. It will not do the IPv4 fallback retry for the redirect to HTTPS

So, if IPv6 times out, and IPv4 works but redirects to HTTPS the challenge will fail.

3 Likes

So then, that narrows us back down again to the ip6tables filter.

3 Likes

Community tools outside of the CA environment also help verify that IPv6 is not working:
Let's Debug - for abs.skvare.com
Let's Debug - for dev.uschess.org

3 Likes