Recently encountered errors renewing on an HAProxy server using Standalone method with Certbot

Help
1

My domain is:
helpdesk.suu.edu
I ran this command:
certbot -v certonly --standalone --cert-name helpdesk.suu.edu -d helpdesk.suu.edu --non-interactive --agree-tos --email noreply@suu.edu --http-01-address 134.250.252.101 > /tmp/certbot-helpdesk.log

It produced this output:
Snippet:

2022-05-03 18:00:23,793:DEBUG:urllib3.connectionpool:http://localhost:None "GET http://snapd/v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2022-05-03 18:00:24,449:DEBUG:certbot._internal.main:certbot version: 1.26.0
2022-05-03 18:00:24,450:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/1952/bin/certbot
2022-05-03 18:00:24,450:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-05-03 18:00:24,464:DEBUG:certbot._internal.log:Root logging level set at 20
2022-05-03 18:00:24,465:DEBUG:certbot._internal.plugins.selection:Requested authenticator standalone and installer None
2022-05-03 18:00:24,469:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: Authenticator, Plugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator
Initialized: <certbot._internal.plugins.standalone.Authenticator object at 0x7f7cbdf89ac0>
Prep: True
2022-05-03 18:00:24,469:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.standalone.Authenticator object at 0x7f7cbdf89ac0> and installer None
2022-05-03 18:00:24,470:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator standalone, Installer None
2022-05-03 18:00:24,478:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v02.api.letsencrypt.org/acme/acct/47429246', new_authzr_uri=None, terms_of_service=None), 558912c4ab5f571a06a39937466e0f24, Meta(creation_dt=datetime.datetime(2018, 12, 10, 16, 42, 12, tzinfo=), creation_host='lb4.suu.edu', register_to_eff=None))>
2022-05-03 18:00:24,479:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2022-05-03 18:00:24,481:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2022-05-03 18:00:24,723:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2022-05-03 18:00:24,724:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Wed, 04 May 2022 00:00:25 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"oLgQNQ6xTes": "Adding random entries to the directory",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2022-05-03 18:00:24,745:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): proxy.suu.edu:8888
2022-05-03 18:00:24,786:DEBUG:urllib3.connectionpool:http://proxy.suu.edu:8888 "POST http://r3.o.lencr.org/ HTTP/1.1" 200 503
2022-05-03 18:00:24,788:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/helpdesk.suu.edu/cert19.pem is signed by the certificate's issuer.
2022-05-03 18:00:24,788:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/helpdesk.suu.edu/cert19.pem is: OCSPCertStatus.GOOD
2022-05-03 18:00:24,793:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2022-05-22 09:03:18 UTC.
2022-05-03 18:00:24,793:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2022-05-03 18:00:24,793:DEBUG:certbot._internal.display.obj:Notifying user: Renewing an existing certificate for helpdesk.suu.edu
2022-05-03 18:00:25,183:DEBUG:certbot.crypto_util:Generating RSA key (2048 bits): /etc/letsencrypt/keys/0552_key-certbot.pem
2022-05-03 18:00:25,194:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0552_csr-certbot.pem
2022-05-03 18:00:25,195:DEBUG:acme.client:Requesting fresh nonce
2022-05-03 18:00:25,195:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2022-05-03 18:00:25,288:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2022-05-03 18:00:25,288:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Wed, 04 May 2022 00:00:25 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: https://acme-v02.api.letsencrypt.org/directory;rel="index"
Replay-Nonce: 0002Ec9hmqhM47SGI69_K5ZUnDXx32FqwlbB7MQFdOQDgMk
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

2022-05-03 18:00:25,288:DEBUG:acme.client:Storing nonce: 0002Ec9hmqhM47SGI69_K5ZUnDXx32FqwlbB7MQFdOQDgMk
2022-05-03 18:00:25,289:DEBUG:acme.client:JWS payload:
b'{\n "identifiers": [\n {\n "type": "dns",\n "value": "helpdesk.suu.edu"\n }\n ]\n}'
2022-05-03 18:00:25,291:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{

2022-05-03 18:00:25,507:INFO:certbot._internal.auth_handler:Performing the following challenges:
2022-05-03 18:00:25,507:INFO:certbot._internal.auth_handler:http-01 challenge for helpdesk.suu.edu
2022-05-03 18:00:25,507:DEBUG:acme.standalone:Failed to bind to 134.250.252.101:80 using IPv6
2022-05-03 18:00:25,509:DEBUG:acme.standalone:Successfully bound to 134.250.252.101:80 using IPv4
2022-05-03 18:00:25,511:DEBUG:acme.client:JWS payload:
b'{}'
2022-05-03 18:00:25,513:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/104840453096/jC6DtQ:gqO9D_helailwuDp-Z7ykFY",
"hostname": "helpdesk.suu.edu",
"port": "80",
"addressesResolved": [
"134.250.252.101"
],
"addressUsed": "134.250.252.101"
}
],
"validated": "2022-05-04T00:00:26Z"
}
]
}
2022-05-03 18:00:26,694:DEBUG:acme.client:Storing nonce: 00028mhBxyzRO15_klBdOl57LASmIwxgJM6K4qTE4BVSry0
2022-05-03 18:00:26,695:INFO:certbot._internal.auth_handler:Challenge failed for domain helpdesk.suu.edu
2022-05-03 18:00:26,695:INFO:certbot._internal.auth_handler:http-01 challenge for helpdesk.suu.edu
2022-05-03 18:00:26,695:DEBUG:certbot._internal.display.obj:Notifying user:
Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: helpdesk.suu.edu
Type: unauthorized
Detail: 134.250.252.101: Invalid response from http://helpdesk.suu.edu/.well-known/acme-challenge/OVEDIk0LYZMImnu74Jx-gqO9D_helailwuDp-Z7ykFY: 503

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on 134.250.252.101:80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

2022-05-03 18:00:26,696:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 206, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2022-05-03 18:00:26,696:DEBUG:certbot._internal.error_handler:Calling registered functions
2022-05-03 18:00:26,696:INFO:certbot._internal.auth_handler:Cleaning up challenges
2022-05-03 18:00:26,697:DEBUG:certbot._internal.plugins.standalone:Stopping server at 134.250.252.101:80...
2022-05-03 18:00:27,018:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/snap/certbot/1952/bin/certbot", line 8, in
sys.exit(main())
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/main.py", line 1723, in main
return config.func(config, plugins)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/main.py", line 1582, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/main.py", line 129, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 344, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/client.py", line 441, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/client.py", line 493, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 106, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/snap/certbot/1952/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 206, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2022-05-03 18:00:27,021:ERROR:certbot._internal.log:Some challenges have failed.

My web server is (include version):
HAProxy load balancer (2.2.22)

The operating system my web server runs on is (include version):
Ubuntu 20.0.4.4
My hosting provider, if applicable, is:
Self
I can login to a root shell on my machine (yes or no, or I don't know):
Yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
1.26.0

This has been working fine until just recently. Searches on the error haven't yielded much.

Thanks for all you do - any leads or help greatly appreciated!

2 Likes
2

Hi @foremaster, and welcome to the LE community forum :slight_smile:

Just to clarify, please show which certs are already being handled by certbot on that system, with:
certbot certificates

3 Likes
3

Thank you! Never had to bother you folks until now! Hoping it isn't some dumb error I (probably) made.

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Found the following certs:
Certificate Name: helpdesk.suu.edu
Serial Number: 34f19b01672e41bab3afac58ecde1a12c47
Key Type: RSA
Domains: helpdesk.suu.edu
Expiry Date: 2022-05-22 09:03:18+00:00 (VALID: 18 days)
Certificate Path: /etc/letsencrypt/live/helpdesk.suu.edu/fullchain.pem
Private Key Path: /etc/letsencrypt/live/helpdesk.suu.edu/privkey.pem
Certificate Name: suu.edu
Serial Number: 4f733c6744cd5d431ea5d1b0ba3ce80b280
Key Type: RSA
Domains: suu.edu
Expiry Date: 2022-05-14 09:03:27+00:00 (VALID: 10 days)
Certificate Path: /etc/letsencrypt/live/suu.edu/fullchain.pem
Private Key Path: /etc/letsencrypt/live/suu.edu/privkey.pem

3 Likes
4

I see that

And yet I see this (now):

curl -Ii helpdesk.suu.edu
HTTP/1.1 302 Found
content-length: 0
location: https://helpdesk.suu.edu/
cache-control: no-cache

Which implies that even when certbot isn't running a webserver (in --standalone mode), there is still a webserver that responds to that name/IP.

3 Likes
8

This looks like a Palo Alto firewall problem:

curl -I helpdesk.suu.edu/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 503 Service Unavailable
Content-Type: text/html; charset=UTF-8
Content-Length: 930
Connection: close
P3P: CP="CAO PSA OUR"
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache

Introduced recently with one of their updates.

3 Likes
9

See: Renewal failure: "Connection reset by peer" - Help - Let's Encrypt Community Support (letsencrypt.org)

3 Likes
10

Thank you! Yes, it is a service that sits behind a floating IP at 134.250.252.101 on the haproxy server. We have a script that kills haproxy, then runs the renew script and fires up the standalone service on port 80. It has been auto-renewing for at least a couple of years now using a cron job that calls certbot with that original command. I can do some more digging tomorrow - thanks so much for the prompt reply. We recently re-signed our ZSK, KSK, et. al. for DNSSEC and upgraded from RSA256 which they list as deprecated to the newer ECDSA algorithm. We wondered if that might be it, but one of our other servers which was upgraded is working fine. I will check with firewall guys too- just saw your reply.

3 Likes
11

Start with the FW guys.
PA is now interfering with inbound HTTP connections of type "acme-protocol".
[unless explicitly allowed]

4 Likes
12

Thanks! In process; I will update here the second I have resolution. Thanks for all the work you guys do. I'm sure it is often a thankless job.

3 Likes
13

That is the job description LOL

5 Likes
14

I agree. Looks more like this one though. @foremaster Your responses are identical with 301 and 503 and the P3P response header. Look a couple posts in the thread after this for a Palo Alto article that might also be the cause.

5 Likes
15

Thank you folks! It was firewall. I've marked rg305's comment as the solution. Thanks so much!

2 Likes
16

That's great. Can you provide any more details? Like was it Palo Alto brand and where was the setting you changed?

I ask because we've already had a couple very much like yours (503 and P3P) and it helps when advising others. Thanks

4 Likes
17

Yes! Thanks again @rg305 and @MikeMcQ I will post details from the security team this afternoon.

3 Likes
18

Additional info from FW guys:
We are running a Palo Alto firewall. Adding the acme-protocol application to the security policy fixed the issue.

3 Likes
19

Thanks. And thanks to your FW guys too.

3 Likes
20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.