Problem renewing certificates

h3ctic · July 17, 2021, 1:31pm

My domain is: many domains

I ran this command: certbot renew

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/example.com-0001.conf

Renewing an existing certificate for mail.ends.no and 55 more domains

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: cloud.austadmaskin.no
Type: unauthorized
Detail: Invalid response from http://example.com/.well-known/acme-challenge/3BegyUKoDpMiNTNironlRK68EWMtRv4VN0D2AFSYPtY [x.x.x.x]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Domain: example.com
Type: unauthorized
Detail: Invalid response from http://example.com/.well-known/acme-challenge/Zmq5D5Gwiqmgi36qDtB11Avxdd5Wv3ORGLEqvT5q72E [x.x.x.x]: "\n\n403 Forbidden\n\n

Forbidden

\n<p"

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Failed to renew certificate example.com-0001 with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/example.com-0001/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):

Various.

The operating system my web server runs on is (include version):

Various.

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I can access my certbot by root; yes.

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

Various.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 1.17.0

I have rebooted my certbot but it did not help.

Latest renewal was a few months ago and it went fine; I do not think anything has changed on the certbot-server since then so Im confused, any help greatly appreciated!

griffin · July 17, 2021, 9:32pm

Welcome to the Let's Encrypt Community

It looks like your port 80 (http) is returning 403 Forbidden. Upon closer inspection, it appears that the culprit likely lies in either your Apache configuration for port 80 or an .htaccess file.

What is the output of:

sudo apachectl -S

_az · July 18, 2021, 7:17am

It looks like this certificate was setup to renew using --standalone, but you are already running an Apache server. One would usually use --apache in a case like this, because the standalone authenticator is generally only applicable when you are not already running a webserver.

If you try this, does it renew?

sudo certbot renew --cert-name example.com \
--pre-hook "service apache2 stop" \
--post-hook "service apache2 restart" \
--dry-run

h3ctic · July 18, 2021, 9:39am

Thank you very much for your replies @griffin and @_az - it is really appreciated.

I am a bit confused, because the certbot says:

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

So...is it a temporary webserver existing on my certbot-server, or is it the one residing on my Nextcloud server (in this case)?


(this is run on Nextcloud since it will fail with "sudo: apachectl: command not found" if run on Certbot)
root@nc:/var/www# sudo apachectl -S

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.16.0.20. Set the 'ServerName' directive globally to suppress this message VirtualHost configuration: *:80 is a NameVirtualHost default server 192.168.0.20 (/etc/apache2/sites-enabled/000-default.conf:1) port 80 namevhost 192.168.0.20 (/etc/apache2/sites-enabled/000-default.conf:1) port 80 namevhost cloud.redacted.net (/etc/apache2/sites-enabled/nextcloud-redacted.conf:1) alias sky.redacted.net port 80 namevhost cloud.ed2.no (/etc/apache2/sites-enabled/nextcloud-ed2.conf:1) alias sky.ed2.no alias nc.ed2.no port 80 namevhost cloud.ed3.no (/etc/apache2/sites-enabled/nextcloud-ed3.conf:1) alias sky.ed3.no port 80 namevhost sky.ed4.no (/etc/apache2/sites-enabled/nextcloud-ed4.conf:1) port 80 namevhost cloud.ed5.no (/etc/apache2/sites-enabled/nextcloud-ed5.conf:1) alias sky.ed5.no ServerRoot: "/etc/apache2" Main DocumentRoot: "/var/www/html" Main ErrorLog: "/var/log/apache2/error.log" Mutex mpm-accept: using_defaults Mutex watchdog-callback: using_defaults Mutex rewrite-map: using_defaults Mutex default: dir="/var/run/apache2/" mechanism=default PidFile: "/var/run/apache2/apache2.pid" Define: DUMP_VHOSTS Define: DUMP_RUN_CFG User: name="www-data" id=33 Group: name="www-data" id=33 You have new mail in /var/mail/root

and


root@certbot:/etc/letsencrypt# sudo certbot renew --cert-name mail.redacted.net-0001 \

--pre-hook "service apache2 stop" 

--post-hook "service apache2 restart" 

--dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/mail.redacted.net-0001.conf

Hook 'pre-hook' reported error code 5

Hook 'pre-hook' ran with error output:

Failed to stop apache2.service: Unit apache2.service not loaded.

Simulating renewal of an existing certificate for mail.redacted.net and 55 more domains
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:

Domain: cloud.redacted.net

Type:   unauthorized

Detail: Invalid response from http://cloud.redacted.net/.well-known/acme-challenge/1UrFvxAaDcZFjNW_Rs-lpgS1VE6-NU9l3BAnHF2T_5w [1.2.3.4]: "\n\n403 Forbidden\n\n
Forbidden
\n<p"
Domain: sky.redacted.net

Type:   unauthorized

Detail: Invalid response from http://sky.redacted.net/.well-known/acme-challenge/YMi03GgKEFVuAydcZs4Ajui3spoXJGT1lAoMhQkNFL4 [1.2.3.4]: "\n\n403 Forbidden\n\n
Forbidden
\n<p"
Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.
Failed to renew certificate mail.redacted.net-0001 with error: Some challenges have failed.

All simulated renewals failed. The following certificates could not be renewed:

/etc/letsencrypt/live/mail.redacted.net-0001/fullchain.pem (failure)

Hook 'post-hook' reported error code 5 Hook 'post-hook' ran with error output: Failed to restart apache2.service: Unit apache2.service not found. 1 renew failure(s), 0 parse failure(s) Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details. root@certbot:/etc/letsencrypt#

Osiris · July 18, 2021, 9:53am

The temporary server is activated on the server where certbot is running. If Nextcloud is on a different server, then this error is to be expected: the Let's Encrypt validation server is connecting to the hostname sky.austadmaskin.no, but if that's running on a different server than certbot, it can't connect to the temporary server spinned up by certbot obviously.

Please read more about how Let's Encrypt works here:

h3ctic · July 18, 2021, 12:44pm

Thanks @Osiris - probably worth mentioning that I've "inherited" this setup and I have not set it up so I'm in the phase of learning everything from the ground up.I am a total newbie when it comes to both certbot, letsencrypt, nextcloud.....much to learn

h3ctic · July 18, 2021, 4:13pm

Ok, so I investigated a bit more. Theres "Path-specific routing" enabled on the firewall that handles all requests to the two (above) mentioned urls (and many more). If someone is trying to access "/.well-known/acme-challenge/" they will be sent to the cerbot-server, all other requests "/" will be sent to the nextcloud-server, on-prem exchange etc. This rule however is "Listening port: 443"

The communication during renewal are coming to the certbot on port 80 or is it 443?

Osiris · July 18, 2021, 4:23pm

That depends on the challenge used. Please see the Let's Encrypt documentation:

(It's categorized under " Advanced Subscriber Information" in the documentation page, but in my opinion its part of the basics of understanding how Let's Encrypt works.)

griffin · July 18, 2021, 5:23pm

An http-01 challenge starts from a domain name on port 80 (http) then follows up to 10 redirects to domain names on either port 80 (http) or port 443 (https). IPv6 addresses (DNS AAAA records) are given priority over IPv4 addresses (DNS A records) for challenge requests.

h3ctic · July 18, 2021, 7:10pm

Thank you @griffin and @Osiris

I found a "checkbox" on the firewall-rule for the domains that failed earlier named "Redirect HTTP". It was not checked in the firewall-rule. When checked::

root@certbot:/etc/letsencrypt# sudo certbot renew --cert-name mail.redacted.net-0001 --dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mail.redacted.net-0001.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Simulating renewal of an existing certificate for mail.redacted.net and 55 more domains

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Congratulations, all simulated renewals succeeded:
  /etc/letsencrypt/live/mail.redacted.net-0001/fullchain.pem (success)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

There are rules for all domains and none of them has the "Redirect HTTP" checked, my guess is that the domains failing above is the first one that is tried, and after this it does not matter whether its 80 or 443(?) as long as the first one goes through...so to speak?

griffin · July 18, 2021, 7:29pm

I don't exactly understand your question, but the same requirements independently apply for all domain names being certified via http-01 challenges.

h3ctic · July 19, 2021, 12:27am

Thank you everyone for your help in this. All my certificates are now renewed.

griffin · July 19, 2021, 12:41am

You are quite welcome.

h3ctic · July 19, 2021, 6:52am

@_az - thanks again for suggestion; Could you please redact our DNS from your post now that the problem is solved? Replace the dns with "example.com" or similar?

system · August 18, 2021, 6:52am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.