Certbot (auto)renew failed with apache server on IPV6

Brakelmann · January 14, 2019, 9:34pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
office.feste-ip.net
ce-stan.feste-ip.net

I ran this command:
sudo certbot renew --dry-run

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/ce-stan.feste-ip.net-0002.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ce-stan.feste-ip.net
http-01 challenge for office.feste-ip.net
Waiting for verification…
Cleaning up challenges
Attempting to renew cert (ce-stan.feste-ip.net-0002) from /etc/letsencrypt/renewal/ce-stan.feste-ip.net-0002.conf produced an unexpected error: Failed authorization procedure. ce-stan.feste-ip.net (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://ce-stan.feste-ip.net/.well-known/acme-challenge/15PUG9FeKp97aEj_apqDPg0g0rktiayCGwXB7WOdUBE: Error getting validation data, office.feste-ip.net (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://office.feste-ip.net/.well-known/acme-challenge/iS81TF1DUxScZlAGBXUj3diSWkKHuHsIYV62Q_v61nM: Error getting validation data. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/ce-stan.feste-ip.net-0002/fullchain.pem (failure)

** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/ce-stan.feste-ip.net-0002/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: ce-stan.feste-ip.net
Type: connection
Detail: Fetching
http://ce-stan.feste-ip.net/.well-known/acme-challenge/15PUG9FeKp97aEj_apqDPg0g0rktiayCGwXB7WOdUBE:
Error getting validation data

Domain: office.feste-ip.net
Type: connection
Detail: Fetching
http://office.feste-ip.net/.well-known/acme-challenge/iS81TF1DUxScZlAGBXUj3diSWkKHuHsIYV62Q_v61nM:
Error getting validation data

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you’re using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.

My web server is (include version):
Server version: Apache/2.4.18 (Ubuntu)
Server built: 2018-06-07T19:43:03
Server’s Module Magic Number: 20120211:52
Server loaded: APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture: 64-bit

The operating system my web server runs on is (include version):
Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-141-generic x86_64)

My hosting provider, if applicable, is:
Unitymedia with DSL-Lite, Apache only reachable via IP V6

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The automatic renewal worked like a charm for the last year.
The last renewal was on 5th of November last year.
I didn’t change much on my system. Latest change was updating PHP from version 7.0 to 7.2.

Any help is much appreciated.

Thanks, Stefan.

stevenzhu · January 14, 2019, 9:52pm

Hi,

Could you please dig into the logs and share us that challenge URL?

Thank you

schoen · January 14, 2019, 9:53pm

The defaults in Certbot have changed because the old method that it used to validate your control of a domain on port 443 is being permanently retired in February. Certbot is now trying to use port 80, but it seems to be blocked by a firewall or something in your environment.

Brakelmann · January 14, 2019, 10:05pm

I created a file to check port 80.
http://office.feste-ip.net/.well-known/acme-challenge/1234
http://ce-stan.feste-ip.net/.well-known/acme-challenge/1234

Both links are working for me an show the text I wrote into the file 1234 "It works".

So it seems that port 80 is open. I use HTTP to HTTPS redirection.
Maybe you can click on the links above an try it out from your site.

Thanks for you quick response.

rg305 · January 14, 2019, 10:08pm

Did you take those files down?
I can't seem to reach them.

Both sites, and paths, return:
failed: Permission denied.

schoen · January 14, 2019, 10:13pm

Yes, I see the same result as @rg305 which makes me think that this really is the fault of a firewall that doesn’t allow inbound connections on port 80 from outside of your network (or ISP or country or something).

Brakelmann · January 14, 2019, 10:14pm

I am not able to upload the letsencrypt log as new user
The Challenge URLs are:
2019-01-14 14:31:55,614:INFO:certbot.auth_handler:Performing the following challenges:
2019-01-14 14:31:55,614:INFO:certbot.auth_handler:http-01 challenge for ce-stan.feste-ip.net
2019-01-14 14:31:55,615:INFO:certbot.auth_handler:http-01 challenge for office.feste-ip.net
2019-01-14 14:31:55,706:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: ce-stan.feste-ip.net in: /etc/apache2/sites-enabled/nextcloud.conf
2019-01-14 14:31:55,706:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: office.feste-ip.net in: /etc/apache2/sites-enabled/collabora.conf
2019-01-14 14:31:55,707:DEBUG:certbot_apache.http_01:writing a pre config file with text:
RewriteEngine on
RewriteRule ^/.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

2019-01-14 14:31:55,707:DEBUG:certbot_apache.http_01:writing a post config file with text:
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted

<Location /.well-known/acme-challenge>
Require all granted

2019-01-14 14:31:55,747:DEBUG:certbot.reverter:Creating backup of /etc/apache2/sites-enabled/collabora.conf
2019-01-14 14:31:55,747:DEBUG:certbot.reverter:Creating backup of /etc/apache2/sites-enabled/nextcloud.conf

But I do not see any files for challenging in the the folder .well-known/acme.challenges/

Thanks for your fast response.

rg305 · January 14, 2019, 10:15pm

Those changes are "temporary".
What changes did you make to allow for the challenge requests?
Do you have any GEO-fencing/blocking configured?

Brakelmann · January 14, 2019, 10:16pm

That is strange, I will double check with a mobile hotspot and get back.

Brakelmann · January 14, 2019, 10:19pm

None, I guess but maybe I don't exactly understand your question.

[/quote]
Do you have any GEO-fencing/blocking configured?
[/quote]
No, I don't have any GEO blocking activated.

rg305 · January 14, 2019, 10:21pm

How are those "1234" files reached?
[they are to simulate a challenge request]
Did you add the path to your document root?
Did you add rewrite logic to send them elsewhere?
Please explain.

Here I am assuming you created those files (and a way to reach them) after the last log showing:

[or why else would certbot have needed to add the rewrite?]

rg305 · January 14, 2019, 10:28pm

Maybe you could just show these two files:

They should "explain" what is going on

Brakelmann · January 14, 2019, 10:37pm

<VirtualHost *:80>
 ServerName office.feste-ip.net
 DocumentRoot "/var/www/nextcloud"

 ErrorLog ${APACHE_LOG_DIR}/error_collabora.log
 CustomLog ${APACHE_LOG_DIR}/access_collabora.log combined

<Directory /var/www/html/>
 Options +FollowSymlinks
 AllowOverride All

 <IfModule mod_dav.c>
 Dav off
 </IfModule>

 SetEnv HOME /var/www/html
 SetEnv HTTP_HOME /var/www/html
 Satisfy Any

</Directory>

</VirtualHost>



<VirtualHost *:443>
ServerName office.feste-ip.net
# SSL configuration, you may want to take the easy route instead and use Lets Encrypt!
SSLEngine on
#Include    /etc/letsencrypt/options-ssl-apache.conf

# Encoded slashes need to be allowed
AllowEncodedSlashes NoDecode

# Container uses a unique non-signed certificate
SSLProxyEngine On
SSLProxyVerify None
SSLProxyCheckPeerCN     Off
SSLProxyCheckPeerName Off

# keep the host
ProxyPreserveHost On


# static html, js, images, etc. served from loolwsd
# loleaflet is the client part of LibreOffice Online
ProxyPass /loleaflet https://ce.fritz.box:9980/loleaflet retry=0
ProxyPassReverse /loleaflet https://ce.fritz.box:9980/loleaflet

# WOPI discovery URL
ProxyPass /hosting/discovery https://ce.fritz.box:9980/hosting/discovery retry=0
ProxyPassReverse /hosting/discovery https://ce.fritz.box:9980/hosting/discovery

# Main websocket
ProxyPassMatch           "/lool/(.*)/ws$" wss://ce.fritz.box:9980/lool/$1/ws nocanon

# Admin Console websocket
ProxyPass    /lool/adminws wss://ce.fritz.box:9980/lool/adminws

# Download as, Fullscreen presentation and Image upload operations
ProxyPass           /lool https://ce.fritz.box:9980/lool
ProxyPassReverse    /lool https://ce.fritz.box:9980/lool


#<Location />
#    Require all granted
#</Location>

SSLCertificateFile /etc/letsencrypt/live/ce-stan.feste-ip.net-0002/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/ce-stan.feste-ip.net-0002/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>


nextcloud.conf:
<VirtualHost *:80>
 ServerName ce-stan.feste-ip.net
 DocumentRoot "/var/www/nextcloud"

 ErrorLog ${APACHE_LOG_DIR}/error_nextcloud.log
 CustomLog ${APACHE_LOG_DIR}/access_nextcloud.log combined

<Directory /var/www/nextcloud/>
 Options +FollowSymlinks
 AllowOverride All

 <IfModule mod_dav.c>
 Dav off
 </IfModule>

 SetEnv HOME /var/www/nextcloud
 SetEnv HTTP_HOME /var/www/nextcloud
 Satisfy Any

</Directory>

</VirtualHost>

### Nextcloud
<VirtualHost *:443>
    ServerName ce-stan.feste-ip.net
    ServerAlias ccc-ce-stan.feste-ip.net
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/ce-stan.feste-ip.net-0002/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/ce-stan.feste-ip.net-0002/privkey.pem

    <IfModule mod_headers.c>
      Header always set Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    </IfModule>

DocumentRoot            "/var/www/nextcloud"

# BandWidthModule On
# ForceBandWidthModule On
# BandWidth 192.168.1.0/24 0
# BandWidth all 550000


ErrorLog          ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

<Directory /var/www/nextcloud/>
 Options +FollowSymlinks
 AllowOverride All

 <IfModule mod_dav.c>
 Dav off
 </IfModule>

 SetEnv HOME /var/www/nextcloud
 SetEnv HTTP_HOME /var/www/nextcloud
 Satisfy Any
</Directory>

Oh sh... what happened to the format, isn't it possible to use plain text without formatting...

rg305 · January 14, 2019, 10:38pm

For legibility, please modify your last post and include this at before the start and after the end. (three backtics)
```

rg305 · January 14, 2019, 10:43pm

OK.
So did you create the folder:
/var/www/nextcloud/.well-known/acme-challenge
[to place the “1234” into?]

They both share the same
DocumentRoot "/var/www/nextcloud"

schoen · January 14, 2019, 10:44pm

I don’t think this line of investigation is very relevant at this stage. If @Brakelmann can see the existing test files over HTTP within the local network yet we can’t see them from outside, this isn’t likely to be a problem of any kind within the Apache configuration, but rather with a firewall rule.

rg305 · January 14, 2019, 10:46pm

Yes, but maybe because something abnormal is going on…
Like:
SetEnv HOME /var/www/html
SetEnv HTTP_HOME /var/www/html

I do agree that externally inbound port 80 access is a must but think there is more than meets the eye here.

Brakelmann · January 14, 2019, 10:47pm

Yes, I created the file manually in the folder /var/www/nextcloud/.well-known/acme-challenge
I tried to reach the file 1234 using my mobile device using a hotspot which also worked fine.
So it is reachable from outside my network. Did you consider having an IP V6 environment?

rg305 · January 14, 2019, 10:48pm

Yes on the IPv6:
–2019-01-14 22:08:56-- (try: 2) http://office.feste-ip.net/.well-known/acme-challenge/1234
Connecting to office.feste-ip.net (office.feste-ip.net)|2a02:908:8a3:1960:921b:eff:fe9f:a15f|:80… failed: Permission denied.
Retrying.

–2019-01-14 22:08:59-- (try: 3) http://office.feste-ip.net/.well-known/acme-challenge/1234
Connecting to office.feste-ip.net (office.feste-ip.net)|2a02:908:8a3:1960:921b:eff:fe9f:a15f|:80… failed: Permission denied.
Retrying.

rg305 · January 14, 2019, 10:50pm

Which points to GEO-fencing/blocking.
Your IP(v6) doesn’t like my IP(v6) - LOL