Cannot renew certificate (unauthorized)


#1

I ran this command:
/opt/letsencrypt/letsencrypt-auto renew --verbose

It produced this output:
(output is at the bottom)

My web server is (include version):
Apache/2.4.7 (Ubuntu)

The operating system my web server runs on is (include version):
Ubuntu 14.04.5 LTS

My hosting provider, if applicable, is:
AWS

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

Problem:

There are two hosts on the machine. One auto-renews fine. The second auto-renewed fine for months but recently has been failing with urn:ietf:params:acme:error:unauthorized. It is an internal CMS I am not authorised to put here. Both have somewhat different configurations.

I have confirmed:

  • from Apache logs that the LetsEncrypt validation requests are reaching the machine.
  • that (docroot)/.well-known/acme-challenge/ is accessible to the web server and remote clients.
  • that symlinks from /var/lib/letsencrypt/http_challenges/ are served by apache at /.well-known/acme-challenge (this is without running the renew script)
  • that the site has a valid DNS A record and points to the correct IP address

I have not (cannot?) confirmed:

  • that the challenge files are actually being written to httpd_challenge.

Below is the verbose output of the renewal attempt when triggered manually. I have mangled the urls in the output (http > ht_p) because of the link limit in the forum.

Root logging level set at 10
Saving debug log to /var/log/letsencrypt/letsencrypt.log


Processing /etc/letsencrypt/renewal/REDACTED_URL.conf


Requested authenticator <certbot.cli._Default object at 0x7f9927320390> and installer <certbot.cli._Default object at 0x7f9927320390>
Should renew, less than 30 days before certificate expiry 2018-12-29 12:01:31 UTC.
Cert is due for renewal, auto-renewing…
Requested authenticator apache and installer apache
Apache version is 2.4.7
Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7f9927324e10>
Prep: True
Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7f9927324e10>
Prep: True
Selected authenticator <certbot_apache.override_debian.DebianConfigurator object at 0x7f9927324e10> and installer <certbot_apache.override_debian.DebianConfigurator object at 0x7f9927324e10>
Plugins selected: Authenticator apache, Installer apache
Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=u’ht_ps:://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf’, only_return_existing=None, contact=(u’mailto:andrew.adamson@scriptfoundry.com’,), key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7f99272af410>)>), external_account_binding=None), uri=u’ht_ps:://acme-v01.api.letsencrypt.org/acme/reg/702972’, new_authzr_uri=u’ht_ps:://acme-v01.api.letsencrypt.org/acme/new-authz’, terms_of_service=u’ht_ps:://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf’), b109ebc02659cfd7e903e5de51d0829b, Meta(creation_host=u’ip-172-31-40-6.us-west-2.compute.internal’, creation_dt=datetime.datetime(2016, 3, 4, 7, 16, 6, tzinfo=)))>
Sending GET request to ht_ps:://acme-v02.api.letsencrypt.org/directory.
Starting new ht_ps: connection (1): acme-v02.api.letsencrypt.org
ht_ps:://acme-v02.api.letsencrypt.org:443 “GET /directory HTTP/1.1” 200 658
Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 658
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 23 Dec 2018 14:02:43 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 23 Dec 2018 14:02:43 GMT
Connection: keep-alive

{
“keyChange”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/key-change”,
“meta”: {
“caaIdentities”: [
letsencrypt.org
],
“termsOfService”: “ht_ps:://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf”,
“website”: “ht_ps:://letsencrypt.org”
},
“newAccount”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/new-acct”,
“newNonce”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/new-nonce”,
“newOrder”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/new-order”,
“pjTtwy0bsv4”: “ht_ps:://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417”,
“revokeCert”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/revoke-cert”
}
Renewing an existing certificate
Generating key (2048 bits): /etc/letsencrypt/keys/0051_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0051_csr-certbot.pem
Requesting fresh nonce
Sending HEAD request to ht_ps:://acme-v02.api.letsencrypt.org/acme/new-nonce.
ht_ps:://acme-v02.api.letsencrypt.org:443 “HEAD /acme/new-nonce HTTP/1.1” 204 0
Received response:
HTTP 204
Server: nginx
Replay-Nonce: 5r4QknutYITEkE8aSJkQ1QRg7pjDVVp9-LtfAlOKJw4
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 23 Dec 2018 14:02:44 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 23 Dec 2018 14:02:44 GMT
Connection: keep-alive

Storing nonce: 5r4QknutYITEkE8aSJkQ1QRg7pjDVVp9-LtfAlOKJw4
JWS payload:
{
“identifiers”: [
{
“type”: “dns”,
“value”: “REDACTED_URL”
}
]
}
Sending POST request to ht_ps:://acme-v02.api.letsencrypt.org/acme/new-order:
{
“protected”: “eyJub25jZSI6ICI1cjRRa251dFlJVEVrRThhU0prUTFRUmc3cGpEVlZwOS1MdGZBbE9LSnc0IiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDEuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL3JlZy83MDI5NzIiLCAiYWxnIjogIlJTMjU2In0”,
“payload”: “ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwgCiAgICAgICJ2YWx1ZSI6ICJycGFkbWluLnJlbmV3bWVkaWEuY2EiCiAgICB9CiAgXQp9”,
“signature”: “n5thhTCpx36LESL4mEoNQsnYyBxwi0si02X6_8lrk9eHcKz0Gqz8tCaj94vCMhM66q7zqunW75wEyKfjFIkdiabKhloPGJbjUCi5rX7Rxwh8aC1CkfNSQZROOtn242Vn5-r-J1HCcsc31YU4bhAfn1w-ILCHTR-B5HQvGo0mZzdnWWNxcLIYdhZfCe_AKB4mdqvBrirKO7qlVchIQlSWCAoWwPvehuxVhjrxjXP9YUs0AWtHSRw8eG6BHSIyTiPqAQnmTR8aTi2J7oIvZnpzFGmQ562CWJh9OQ5-x9g5VLRb6-sDLeNrsbdHMSccuTuTSKtjjjMVFcfD6jkFN6jYEQ”
}
ht_ps:://acme-v02.api.letsencrypt.org:443 “POST /acme/new-order HTTP/1.1” 201 378
Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 378
Boulder-Requester: 702972
Location: ht_ps:://acme-v02.api.letsencrypt.org/acme/order/702972/235582654
Replay-Nonce: KK4x5gxmJgREnYa3ezfQrL-aE3ZNrY7MlzZxDaJTjDI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 23 Dec 2018 14:02:44 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 23 Dec 2018 14:02:44 GMT
Connection: keep-alive

{
“status”: “pending”,
“expires”: “2018-12-30T14:02:44.381689691Z”,
“identifiers”: [
{
“type”: “dns”,
“value”: “REDACTED_URL”
}
],
“authorizations”: [
“ht_ps:://acme-v02.api.letsencrypt.org/acme/authz/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk”
],
“finalize”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/finalize/702972/235582654”
}
Storing nonce: KK4x5gxmJgREnYa3ezfQrL-aE3ZNrY7MlzZxDaJTjDI
JWS payload:

Sending POST request to ht_ps:://acme-v02.api.letsencrypt.org/acme/authz/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk:
{
“protected”: “eyJub25jZSI6ICJLSzR4NWd4bUpnUkVuWWEzZXpmUXJMLWFFM1pOclk3TWx6WnhEYUpUakRJIiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei91VWVvcXFuZ3poUUpFdjRCR0hkeTN5TFNjTUpmQ0Z0T3Y4UW9abmhKVVlrIiwgImtpZCI6ICJodHRwczovL2FjbWUtdjAxLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9yZWcvNzAyOTcyIiwgImFsZyI6ICJSUzI1NiJ9”,
“payload”: “”,
“signature”: “Ce08HJJO25tB_C7BRFShuRVYWEAhnWd5lDj5Mdz8IdgcbRC0VF29-bvn5mTcvSH5TuzsHTTQfznv5aNfnmMOaY12gUJz9EaevSD7wwXpo1lMr_8WK32x59SoPB0Y-Y9vAXg2RZuYecSmXE_0MuKR5KXeys9HFlFYGlET1_ug3Rv3tIjiNmo9Qc-krH-rxyD1JEg-NDo4Q3wlB6bPFanpA-cJ9nBIbW6dWYdHXmihb40AX5B_CqltUGBnRMP4rNBzJdwyvJuIFiB2JcX2wEJhrTwfCPkL1s9gwy-JJYJOBO-1-vzuwlYGhK5Vs0n9SoRTtdGcVoIMd7NEXf_CF9mf7g”
}
ht_ps:://acme-v02.api.letsencrypt.org:443 “POST /acme/authz/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk HTTP/1.1” 200 1169
Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 1169
Boulder-Requester: 702972
Replay-Nonce: jcdyQe7cdZ9-Dns5qdCe9gC8fqse_E8QHI2Xg-1wLM4
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 23 Dec 2018 14:02:44 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 23 Dec 2018 14:02:44 GMT
Connection: keep-alive

{
“identifier”: {
“type”: “dns”,
“value”: “REDACTED_URL”
},
“status”: “pending”,
“expires”: “2018-12-30T14:02:44Z”,
“challenges”: [
{
“type”: “http-01”,
“status”: “pending”,
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490171”,
“token”: “MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8”
},
{
“type”: “dns-01”,
“status”: “pending”,
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490172”,
“token”: “9w9Eh4R1GV1EiazjOYcJU_JPkPZy6nvdeVIqzHUm_xY”
},
{
“type”: “tls-sni-01”,
“status”: “pending”,
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490173”,
“token”: “2qJM0ZO_5AGc4kX3E_Y4ib28xGwv8Qs7s3Ou1vl7tDo”
},
{
“type”: “tls-alpn-01”,
“status”: “pending”,
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490174”,
“token”: “GnR0sZee7dtZR7n4xlKQPfHlYGVsgp2N9vXEhB0qfjA”
}
]
}
Storing nonce: jcdyQe7cdZ9-Dns5qdCe9gC8fqse_E8QHI2Xg-1wLM4
Performing the following challenges:
http-01 challenge for REDACTED_URL
Adding a temporary challenge validation Include for name: REDACTED_URL in: /etc/apache2/sites-enabled/REDACTED_CONFIG.conf
writing a pre config file with text:
RewriteEngine on
RewriteRule ^/.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

writing a post config file with text:
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted

<Location /.well-known/acme-challenge>
Require all granted

Creating backup of /etc/apache2/sites-enabled/REDACTED_CONFIG.conf
Waiting for verification…
JWS payload:
{
“keyAuthorization”: “MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8.6swKuPYAihdBMOm9jJNC3PeWnd0UUR7ILYXSgLqtcVE”,
“type”: “http-01”,
“resource”: “challenge”
}
Sending POST request to ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490171:
{
“protected”: “eyJub25jZSI6ICJqY2R5UWU3Y2RaOS1EbnM1cWRDZTlnQzhmcXNlX0U4UUhJMlhnLTF3TE00IiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9jaGFsbGVuZ2UvdVVlb3Fxbmd6aFFKRXY0QkdIZHkzeUxTY01KZkNGdE92OFFvWm5oSlVZay8xMDYzODQ5MDE3MSIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvcmVnLzcwMjk3MiIsICJhbGciOiAiUlMyNTYifQ”,
“payload”: “ewogICJrZXlBdXRob3JpemF0aW9uIjogIk1zQVBZMlgxX192UmQwQnpZLU0ybjBmSXVZY2tjb2g0M0lFWXpVaEhQUDguNnN3S3VQWUFpaGRCTU9tOWpKTkMzUGVXbmQwVVVSN0lMWVhTZ0xxdGNWRSIsIAogICJ0eXBlIjogImh0dHAtMDEiLCAKICAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIgp9”,
“signature”: “aRYzU224YIwW8iAU_roNKFBdpo1b4uRSrA5X-y4XyKNf4lfR7INC2SlalTWa7xvv0XOEd-ENmtMBuyMIZPC-czYHFMvpmqLY3J6d2SVUFjxYHFnKz9kVlOhr2nXZj1lMuPfHxhSdgGldG88cuYLHQZFenAEimz2x2bcLSqx43if5Cs1Rgilp0rU3FpxBIeEFnv_H1JJ4gUfIw0p5KbSKTHVcflkLuNZQBph8NDj_1vNFzpA1YJiNTH-xNbH8_usyS7_JWcvfrRKjSuWGStTPz0aZm3LlQdZ26nhpSyokZwMGZkrRuGLmfzLPqGzfagZNqWdgiRF_9g4EYWaE-6cRqw”
}
ht_ps:://acme-v02.api.letsencrypt.org:443 “POST /acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490171 HTTP/1.1” 200 224
Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 224
Boulder-Requester: 702972
Link: <ht_ps:://acme-v02.api.letsencrypt.org/acme/authz/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk>;rel=“up”
Location: ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490171
Replay-Nonce: RzpN7KhYiu_T1YpqL6OguJ8qoaYZhv24-FytI1264CU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 23 Dec 2018 14:02:48 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 23 Dec 2018 14:02:48 GMT
Connection: keep-alive

{
“type”: “http-01”,
“status”: “pending”,
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490171”,
“token”: “MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8”
}
Storing nonce: RzpN7KhYiu_T1YpqL6OguJ8qoaYZhv24-FytI1264CU
JWS payload:

Sending POST request to ht_ps:://acme-v02.api.letsencrypt.org/acme/authz/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk:
{
“protected”: “eyJub25jZSI6ICJSenBON0toWWl1X1QxWXBxTDZPZ3VKOHFvYVlaaHYyNC1GeXRJMTI2NENVIiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei91VWVvcXFuZ3poUUpFdjRCR0hkeTN5TFNjTUpmQ0Z0T3Y4UW9abmhKVVlrIiwgImtpZCI6ICJodHRwczovL2FjbWUtdjAxLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9yZWcvNzAyOTcyIiwgImFsZyI6ICJSUzI1NiJ9”,
“payload”: “”,
“signature”: “bGdQKYbvc9qBPIcRs05Voj9XNaNPThWm-eVVeRsF7tIzjJntvt0SIv5fdSt4Zje8WLsOUm1IIaoMLMEPOWJOSvCqcD9WauFMu1ySKT8F8KEUcHd1VNKQg5DHaA6wYn-7BIFuWGfjaDuaDC0mt33eJ8NBESONNwu9pwDksXwetZH6X_dOgZ3fawQj5nsfMz2aw7vIkK8OLwJRb-tTfjYV-TTXDSRaVobu7TUAJw1DlTh5gqmLq6tdS82QFaio38XgJoY4guXOGpok-WZvyiqpg7aOtUbYXRX5gfPqjEZcbj_WwwmCwYMQhUUljkjke-ZGjGA8njHdjV63Qjcr6iVPgQ”
}
ht_ps:://acme-v02.api.letsencrypt.org:443 “POST /acme/authz/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk HTTP/1.1” 200 1980
Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 1980
Boulder-Requester: 702972
Replay-Nonce: _AezPQ1tWPeF1f0vcVYlSnSTvrQi-W0eiSIGDD35_OQ
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Sun, 23 Dec 2018 14:02:51 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Sun, 23 Dec 2018 14:02:51 GMT
Connection: keep-alive

{
“identifier”: {
“type”: “dns”,
“value”: “REDACTED_URL”
},
“status”: “invalid”,
“expires”: “2018-12-30T14:02:44Z”,
“challenges”: [
{
“type”: “http-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Invalid response from ht_p://REDACTED_URL/.well-known/acme-challenge/MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8: “\u003chtml\u003e\n\u003chead\u003e\n\t\u003ctitle\u003e404\u003c/title\u003e\n\t\u003clink href=‘ht_p://fonts.googleapis.com/css?family=VT323’ rel=‘stylesheet’ type=‘text/css’\u003e\n\t””,
“status”: 403
},
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490171”,
“token”: “MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8”,
“validationRecord”: [
{
“url”: “ht_p://REDACTED_URL/.well-known/acme-challenge/MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8”,
“hostname”: “REDACTED_URL”,
“port”: “80”,
“addressesResolved”: [
“REDACTED_IP”
],
“addressUsed”: “REDACTED_IP”
}
]
},
{
“type”: “dns-01”,
“status”: “invalid”,
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490172”,
“token”: “9w9Eh4R1GV1EiazjOYcJU_JPkPZy6nvdeVIqzHUm_xY”
},
{
“type”: “tls-sni-01”,
“status”: “invalid”,
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490173”,
“token”: “2qJM0ZO_5AGc4kX3E_Y4ib28xGwv8Qs7s3Ou1vl7tDo”
},
{
“type”: “tls-alpn-01”,
“status”: “invalid”,
“url”: “ht_ps:://acme-v02.api.letsencrypt.org/acme/challenge/uUeoqqngzhQJEv4BGHdy3yLScMJfCFtOv8QoZnhJUYk/10638490174”,
“token”: “GnR0sZee7dtZR7n4xlKQPfHlYGVsgp2N9vXEhB0qfjA”
}
]
}
Storing nonce: _AezPQ1tWPeF1f0vcVYlSnSTvrQi-W0eiSIGDD35_OQ
Reporting to user: The following errors were reported by the server:

Domain: REDACTED_URL
Type: unauthorized
Detail: Invalid response from ht_p://REDACTED_URL/.well-known/acme-challenge/MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8: “\n\n\t404\n\t\n\t”

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
Encountered exception:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 161, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 232, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. REDACTED_URL (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from ht_p://REDACTED_URL/.well-known/acme-challenge/MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8: “\n\n\t404\n\t\n\t”

Calling registered functions
Cleaning up challenges
Attempting to renew cert (REDACTED_URL) from /etc/letsencrypt/renewal/REDACTED_URL.conf produced an unexpected error: Failed authorization procedure. REDACTED_URL (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from ht_p://REDACTED_URL/.well-known/acme-challenge/MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8: “\n\n\t404\n\t\n\t”. Skipping.
Traceback was:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/renewal.py”, line 432, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 1170, in renew_cert
renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 118, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/renewal.py”, line 307, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py”, line 353, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/client.py”, line 389, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 82, in handle_authorizations
self._respond(aauthzrs, resp, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 161, in _respond
self._poll_challenges(aauthzrs, chall_update, best_effort)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 232, in _poll_challenges
raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. REDACTED_URL (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from ht_p://REDACTED_URL/.well-known/acme-challenge/MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8: “\n\n\t404\n\t\n\t”


Processing /etc/letsencrypt/renewal/3345.ca.conf


Cert not yet due for renewal
Requested authenticator apache and installer apache
Selecting plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7f9923076250>
Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/REDACTED_URL/fullchain.pem (failure)


The following certs are not due for renewal yet:
/etc/letsencrypt/live/3345.ca/fullchain.pem expires on 2019-03-01 (skipped)
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/REDACTED_URL/fullchain.pem (failure)


Exiting abnormally:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/bin/letsencrypt”, line 11, in
sys.exit(main())
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 1352, in main
return config.func(config, plugins)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/main.py”, line 1259, in renew
renewal.handle_renewal_request(config)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/renewal.py”, line 457, in handle_renewal_request
len(renew_failures), len(parse_failures)))
Error: 1 renew failure(s), 0 parse failure(s)
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: REDACTED_URL
    Type: unauthorized
    Detail: Invalid response from
    ht_p://REDACTED_URL/.well-known/acme-challenge/MsAPY2X1__vRd0BzY-M2n0fIuYckcoh43IEYzUhHPP8:
    “\n\n\t404\n\t\n\t”

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address.


#2

Do you have any port 80 VirtualHosts with duplicate ServerNames??

apachectl -t -D DUMP_VHOSTS

#3

How are you handling the challenge requests in the site that renews properly?


#5

I went through the apache sites-enabled directory and removed / merged everything that I could to simplify the config. There weren’t any duplicate domains, but the SSL and non-SSL domain configs were previously in separate files, so that were two files I merged. After this, the renewal did work. Thank you for the suggestion.