Certbot (auto)renew failed with apache server on IPV6


#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
office.feste-ip.net
ce-stan.feste-ip.net

I ran this command:
sudo certbot renew --dry-run

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/ce-stan.feste-ip.net-0002.conf


Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ce-stan.feste-ip.net
http-01 challenge for office.feste-ip.net
Waiting for verification…
Cleaning up challenges
Attempting to renew cert (ce-stan.feste-ip.net-0002) from /etc/letsencrypt/renewal/ce-stan.feste-ip.net-0002.conf produced an unexpected error: Failed authorization procedure. ce-stan.feste-ip.net (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://ce-stan.feste-ip.net/.well-known/acme-challenge/15PUG9FeKp97aEj_apqDPg0g0rktiayCGwXB7WOdUBE: Error getting validation data, office.feste-ip.net (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://office.feste-ip.net/.well-known/acme-challenge/iS81TF1DUxScZlAGBXUj3diSWkKHuHsIYV62Q_v61nM: Error getting validation data. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/ce-stan.feste-ip.net-0002/fullchain.pem (failure)


** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/ce-stan.feste-ip.net-0002/fullchain.pem (failure)
** DRY RUN: simulating ‘certbot renew’ close to cert expiry
** (The test certificates above have not been saved.)


1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My web server is (include version):
Server version: Apache/2.4.18 (Ubuntu)
Server built: 2018-06-07T19:43:03
Server’s Module Magic Number: 20120211:52
Server loaded: APR 1.5.2, APR-UTIL 1.5.4
Compiled using: APR 1.5.2, APR-UTIL 1.5.4
Architecture: 64-bit

The operating system my web server runs on is (include version):
Ubuntu 16.04.5 LTS (GNU/Linux 4.4.0-141-generic x86_64)

My hosting provider, if applicable, is:
Unitymedia with DSL-Lite, Apache only reachable via IP V6

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The automatic renewal worked like a charm for the last year.
The last renewal was on 5th of November last year.
I didn’t change much on my system. Latest change was updating PHP from version 7.0 to 7.2.

Any help is much appreciated.

Thanks, Stefan.


#2

Hi,

Could you please dig into the logs and share us that challenge URL?

Thank you


#3

The defaults in Certbot have changed because the old method that it used to validate your control of a domain on port 443 is being permanently retired in February. Certbot is now trying to use port 80, but it seems to be blocked by a firewall or something in your environment.


#4

I created a file to check port 80.
http://office.feste-ip.net/.well-known/acme-challenge/1234
http://ce-stan.feste-ip.net/.well-known/acme-challenge/1234

Both links are working for me an show the text I wrote into the file 1234 “It works”.

So it seems that port 80 is open. I use HTTP to HTTPS redirection.
Maybe you can click on the links above an try it out from your site.

Thanks for you quick response.


#5

Did you take those files down?
I can’t seem to reach them.

Both sites, and paths, return:
failed: Permission denied.


#6

Yes, I see the same result as @rg305 which makes me think that this really is the fault of a firewall that doesn’t allow inbound connections on port 80 from outside of your network (or ISP or country or something).


#7

I am not able to upload the letsencrypt log as new user :frowning:
The Challenge URLs are:
2019-01-14 14:31:55,614:INFO:certbot.auth_handler:Performing the following challenges:
2019-01-14 14:31:55,614:INFO:certbot.auth_handler:http-01 challenge for ce-stan.feste-ip.net
2019-01-14 14:31:55,615:INFO:certbot.auth_handler:http-01 challenge for office.feste-ip.net
2019-01-14 14:31:55,706:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: ce-stan.feste-ip.net in: /etc/apache2/sites-enabled/nextcloud.conf
2019-01-14 14:31:55,706:DEBUG:certbot_apache.http_01:Adding a temporary challenge validation Include for name: office.feste-ip.net in: /etc/apache2/sites-enabled/collabora.conf
2019-01-14 14:31:55,707:DEBUG:certbot_apache.http_01:writing a pre config file with text:
RewriteEngine on
RewriteRule ^/.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

2019-01-14 14:31:55,707:DEBUG:certbot_apache.http_01:writing a post config file with text:
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted

<Location /.well-known/acme-challenge>
Require all granted

2019-01-14 14:31:55,747:DEBUG:certbot.reverter:Creating backup of /etc/apache2/sites-enabled/collabora.conf
2019-01-14 14:31:55,747:DEBUG:certbot.reverter:Creating backup of /etc/apache2/sites-enabled/nextcloud.conf

But I do not see any files for challenging in the the folder .well-known/acme.challenges/

Thanks for your fast response.


#8

Those changes are “temporary”.
What changes did you make to allow for the challenge requests?
Do you have any GEO-fencing/blocking configured?


#9

That is strange, I will double check with a mobile hotspot and get back.


#10

None, I guess but maybe I don’t exactly understand your question.

[/quote]
Do you have any GEO-fencing/blocking configured?
[/quote]
No, I don’t have any GEO blocking activated.


#11

How are those “1234” files reached?
[they are to simulate a challenge request]
Did you add the path to your document root?
Did you add rewrite logic to send them elsewhere?
Please explain.

Here I am assuming you created those files (and a way to reach them) after the last log showing:

[or why else would certbot have needed to add the rewrite?]


#12

Maybe you could just show these two files:

They should “explain” what is going on :wink:


#13
<VirtualHost *:80>
 ServerName office.feste-ip.net
 DocumentRoot "/var/www/nextcloud"

 ErrorLog ${APACHE_LOG_DIR}/error_collabora.log
 CustomLog ${APACHE_LOG_DIR}/access_collabora.log combined

<Directory /var/www/html/>
 Options +FollowSymlinks
 AllowOverride All

 <IfModule mod_dav.c>
 Dav off
 </IfModule>

 SetEnv HOME /var/www/html
 SetEnv HTTP_HOME /var/www/html
 Satisfy Any

</Directory>

</VirtualHost>



<VirtualHost *:443>
ServerName office.feste-ip.net
# SSL configuration, you may want to take the easy route instead and use Lets Encrypt!
SSLEngine on
#Include    /etc/letsencrypt/options-ssl-apache.conf

# Encoded slashes need to be allowed
AllowEncodedSlashes NoDecode

# Container uses a unique non-signed certificate
SSLProxyEngine On
SSLProxyVerify None
SSLProxyCheckPeerCN     Off
SSLProxyCheckPeerName Off

# keep the host
ProxyPreserveHost On


# static html, js, images, etc. served from loolwsd
# loleaflet is the client part of LibreOffice Online
ProxyPass /loleaflet https://ce.fritz.box:9980/loleaflet retry=0
ProxyPassReverse /loleaflet https://ce.fritz.box:9980/loleaflet

# WOPI discovery URL
ProxyPass /hosting/discovery https://ce.fritz.box:9980/hosting/discovery retry=0
ProxyPassReverse /hosting/discovery https://ce.fritz.box:9980/hosting/discovery

# Main websocket
ProxyPassMatch           "/lool/(.*)/ws$" wss://ce.fritz.box:9980/lool/$1/ws nocanon

# Admin Console websocket
ProxyPass    /lool/adminws wss://ce.fritz.box:9980/lool/adminws

# Download as, Fullscreen presentation and Image upload operations
ProxyPass           /lool https://ce.fritz.box:9980/lool
ProxyPassReverse    /lool https://ce.fritz.box:9980/lool


#<Location />
#    Require all granted
#</Location>

SSLCertificateFile /etc/letsencrypt/live/ce-stan.feste-ip.net-0002/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/ce-stan.feste-ip.net-0002/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>


nextcloud.conf:
<VirtualHost *:80>
 ServerName ce-stan.feste-ip.net
 DocumentRoot "/var/www/nextcloud"

 ErrorLog ${APACHE_LOG_DIR}/error_nextcloud.log
 CustomLog ${APACHE_LOG_DIR}/access_nextcloud.log combined

<Directory /var/www/nextcloud/>
 Options +FollowSymlinks
 AllowOverride All

 <IfModule mod_dav.c>
 Dav off
 </IfModule>

 SetEnv HOME /var/www/nextcloud
 SetEnv HTTP_HOME /var/www/nextcloud
 Satisfy Any

</Directory>

</VirtualHost>

### Nextcloud
<VirtualHost *:443>
    ServerName ce-stan.feste-ip.net
    ServerAlias ccc-ce-stan.feste-ip.net
    Include /etc/letsencrypt/options-ssl-apache.conf
    SSLCertificateFile /etc/letsencrypt/live/ce-stan.feste-ip.net-0002/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/ce-stan.feste-ip.net-0002/privkey.pem

    <IfModule mod_headers.c>
      Header always set Strict-Transport-Security "max-age=15768000; includeSubDomains; preload"
    </IfModule>

DocumentRoot            "/var/www/nextcloud"

# BandWidthModule On
# ForceBandWidthModule On
# BandWidth 192.168.1.0/24 0
# BandWidth all 550000


ErrorLog          ${APACHE_LOG_DIR}/error.log
CustomLog ${APACHE_LOG_DIR}/access.log combined

<Directory /var/www/nextcloud/>
 Options +FollowSymlinks
 AllowOverride All

 <IfModule mod_dav.c>
 Dav off
 </IfModule>

 SetEnv HOME /var/www/nextcloud
 SetEnv HTTP_HOME /var/www/nextcloud
 Satisfy Any
</Directory>

Oh sh… what happened to the format, isn’t it possible to use plain text without formatting…


#14

For legibility, please modify your last post and include this at before the start and after the end. (three backtics)
```


#15

OK.
So did you create the folder:
/var/www/nextcloud/.well-known/acme-challenge
[to place the “1234” into?]

They both share the same
DocumentRoot "/var/www/nextcloud"


#16

I don’t think this line of investigation is very relevant at this stage. If @Brakelmann can see the existing test files over HTTP within the local network yet we can’t see them from outside, this isn’t likely to be a problem of any kind within the Apache configuration, but rather with a firewall rule.


#17

Yes, but maybe because something abnormal is going on…
Like:
SetEnv HOME /var/www/html
SetEnv HTTP_HOME /var/www/html

I do agree that externally inbound port 80 access is a must but think there is more than meets the eye here.


#18

Yes, I created the file manually in the folder /var/www/nextcloud/.well-known/acme-challenge
I tried to reach the file 1234 using my mobile device using a hotspot which also worked fine.
So it is reachable from outside my network. Did you consider having an IP V6 environment?


#19

Yes on the IPv6:
–2019-01-14 22:08:56-- (try: 2) http://office.feste-ip.net/.well-known/acme-challenge/1234
Connecting to office.feste-ip.net (office.feste-ip.net)|2a02:908:8a3:1960:921b:eff:fe9f:a15f|:80… failed: Permission denied.
Retrying.

–2019-01-14 22:08:59-- (try: 3) http://office.feste-ip.net/.well-known/acme-challenge/1234
Connecting to office.feste-ip.net (office.feste-ip.net)|2a02:908:8a3:1960:921b:eff:fe9f:a15f|:80… failed: Permission denied.
Retrying.


#20

Which points to GEO-fencing/blocking.
Your IP(v6) doesn’t like my IP(v6) - LOL