Auto-renew stopped working!

Help
1

I have had this certificate in place for a long time and it always auto-renewed previously, but now it is failing to auto-renew. Thanks!

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

neuroxy.langneurosci.org

I ran this command:

sudo certbot renew --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/neuroxy.langneurosci.org.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for neuroxy.langneurosci.org
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (neuroxy.langneurosci.org) from /etc/letsencrypt/renewal/neuroxy.langneurosci.org.conf produced an unexpected error: Failed authorization procedure. neuroxy.langneurosci.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: 160.129.198.244: Invalid response from http://neuroxy.langneurosci.org/.well-known/acme-challenge/EHU1nWjpHzFs1O87h_oM2XHiz0fPDqlPSdmlSpXX8c8: 503. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/neuroxy.langneurosci.org/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/neuroxy.langneurosci.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My web server is (include version):

apache2 2.4.29-1ubuntu4.22

The operating system my web server runs on is (include version):

Ubuntu 18.04.6 LTS

My hosting provider, if applicable, is:

medical center

I can login to a root shell on my machine (yes or no, or I don't know):

yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 0.31.0

1 Like
2

image
image2832×1006 239 KB

See also: Palo Alto firewall users with failing HTTP-01 challenges: enable "acme-protocol"

10 Likes
3

A post was split to a new topic: Problems renewing via IPv6

6

This is very helpful. I am now asking my employer (VUMC) to adjust their firewall so that Let's Encrypt can do what it needs to do with the server. But I don't understand what that is exactly. Could you please help me understand what kind of access is needed, that is being blocked?

1 Like
7

HTTP access to the /.well-known/acme-challenge/ path is required to fulfill the challenge request.

7 Likes
8

I have now learned that my employer uses a Palo Alto firewall, and I suspect we are running into this issue:

However they are saying the acme-protocol IS enabled on the firewall.

They are asking is there any chance the request is coming from overseas, since they block a lot of non-US countries.

And also, is it possible to find out the address from which the request is coming, so they can check their logs.

Thanks a lot for your help.

1 Like
9

Yes, LE uses multiple vantage points from around the globe. Currently, I think all AWS though, but from different regions.

5 Likes
10

Then they are mistaken - they should test things:

curl -Ii http://neuroxy.langneurosci.org/.well-known/acme-challenge/EHU1nWjpHzFs1O87h_oM2XHiz0fPDqlPSdmlSpXX8c8 -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
HTTP/1.1 503 Service Unavailable
Content-Type: text/html; charset=UTF-8
Content-Length: 2042
Connection: close
P3P: CP="CAO PSA OUR"
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

curl -Ii http://neuroxy.langneurosci.org/.well-known/acme-challenge/EHU1nWjpHzFs1O87h_oM2XHiz0fPDqlPSdmlSpXX8c8 -A "Mozilla/5.0 everyone else"
HTTP/1.1 301 Moved Permanently
Date: Mon, 06 Jun 2022 17:19:43 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https://neuroxy.langneurosci.org/.well-known/acme-challenge/EHU1nWjpHzFs1O87h_oM2XHiz0fPDqlPSdmlSpXX8c8
Content-Type: text/html; charset=iso-8859-1

We see two very different handlings.

7 Likes
11

Furrthremore:

curl http://neuroxy.langneurosci.org/.well-known/acme-challenge/EHU1nWjpHzFs1O87h_oM2XHiz0fPDqlPSdmlSpXX8c8 -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
<html>
<head>
<title>Application Blocked</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<META HTTP-EQUIV="PRAGMA" CONTENT="NO-CACHE">
<meta name="viewport" content="initial-scale=1.0">
<style>
#content {
        border: 9px solid #91D6E3;
        background-color: #fff;
    margin: 1.5em;
    padding: 1.5em;
        font-family: Tahoma, Helvetica, Arial, sans-serif;
        font-size: 1em;
}
h1 {
        font-size: 1.3em;
        font-weight: bold;
        color: #000505;
        text-align: center;
}
p {
    font-size: 0.9em;
}
b {
        font-weight:bold;
        color: #000505;
}
.center {
    display: block;
    margin-left: auto;
    margin-right: auto;
        width: 50%;
}
</style>
</head>
<body bgcolor="#D9DDDD">
<div id="content">
<img src="https://wjm.s3.amazonaws.com/todayshospital/uploads/Vanderbilt.png" alt="Vanderbilt University Medical Center" class="center" style="width:300px;" />
<h1>VUMC Unauthorized Application Use</h1>
<p>To protect its staff and faculty, Vanderbilt University Medical Center has implemented an institution-wide program to detect applications inappropriate for VUMC networks on Internet websites.</p>
<p>The connection you are attempting to make has been flagged as utilizing an unauthorized / high risk application.</p>
<p>To protect your privacy, the privacy of others, and the integrity of our computer networks, this application cannot be accessed at this time.</p>
<p>If you have reason to believe this application should not be blocked, please submit a request for review at <a target="_blank" href="https://pegasus.vumc.org/request/discover/info/?id=4862">
VEC Security Operations and Services Blocked Website and Application Review</a> or call the Help Desk at 615-343-4357.</p>
<br /><br />
<p><b>User:</b> [MY.IP] </p>
<p><b>Application:</b> <span style="color: #FF0000;">acme-protocol</span> </p>
<p>VUMC Acceptable Use Policy - <a target="_blank" href="https://www.vumc.org/enterprisecybersecurity/security-policy-and-compliance">VUMC AUP</a></p>
</div>
</body>
7 Likes
12
6 Likes
13

Hm, VUMC, Vanderbilt UMC, there was another thread with Vanderbilt in a screenshot from the error recently.

7 Likes
14

Thank you all for your help. We have solved the problem, it was caused by the Palo Alto firewall change.

3 Likes