Certificate validation failing on renewal "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180,

Help
1

Hello

After running year without any problem my mail server, I am not able to update certificates with certbot, although they have been successfully updated before. I decided to do it manually and executed this command;

[root@mail ~]# certbot certonly --standalone --debug -d mail.hristov.org
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.hristov.org
Waiting for verification...
Challenge failed for domain mail.hristov.org
http-01 challenge for mail.hristov.org
Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 9, in <module>
    load_entry_point('certbot==1.9.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1362, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1243, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 117, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/renewal.py", line 330, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 351, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 398, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

Here is what have changes since the last : I did "yum upgrade" and i remember certbot was one of the packages that had to be upgraded. Now i have

certbot --version
certbot 1.9.0

I would like to note that I do NOT have a quad A record in my DSN provided. i..e. AAAA i do not have ipv6 address! so none the problems related to ipv6 should i.e. ([https://letsencrypt.org/de/docs/ipv6-support/](https://IPv6 support))

After reading this forum i saw that many times the reason is the IPV6 and i did checked it myself i.e.
https://check-your-website.server-daten.de/
this says Name Error for the AAAA record but there is NONE! I validated this with the command ip addr on the machines itself.

and the output of the dig output

dig ALL mail.hristov.org @8.8.8.8
;ALL.                           IN      A

;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Di Dez 01 15:47:53 CET 2020
;; MSG SIZE  rcvd: 32

;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 20424
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;mail.hristov.org.              IN      A

;; ANSWER SECTION:
mail.hristov.org.       21593   IN      A       134.255.239.166

the following ports are open and reachable from Internet : 80 and 443
i tested this by looking and the traffic i.e.

tcpdump -i eth0 not port 22

and then i did

"nc 134.255.239.166 80" 
"nc 134.255.239.166 443"

from and outside host on the internet and i saw the packets arrive and the mail server. So there should not be any problem. The server have default route
In order to be sure i disabled all the IDS and IPS. The firewall is however still active. But like i mentioned the ports 80, 443, 25, 455 and 587 are reachable from internet, tested this from different locations.

In letsencrypt.log i see a status 400. Why i am getting this error if the port is reachable from outside?

2020-12-01 15:32:48,388:DEBUG:urllib3.connectionpool:"POST /acme/authz-v3/8999502957 HTTP/1.1" 200 987
2020-12-01 15:32:48,389:DEBUG:acme.client:Received response:
HTTP 200
content-length: 987
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 75563645
date: Tue, 01 Dec 2020 14:32:48 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0004T9YK4PfIdkiob5SxsII2scF_xcPMu-_oXS99qZXfW8A

{
  "identifier": {
    "type": "dns",
    "value": "mail.hristov.org"
  },
  "status": "invalid",
  "expires": "2020-12-08T14:32:46Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "Fetching http://mail.hristov.org/.well-known/acme-challenge/REMOVED: Error getting validation data",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/8999502957/R9GzyQ",
      "token": "REMOVED",
      "validationRecord": [
        {
          "url": "http://mail.hristov.org/.well-known/acme-challenge/REMOVED",
          "hostname": "mail.hristov.org",
          "port": "80",
          "addressesResolved": [
            "134.255.239.166"
          ],
          "addressUsed": "134.255.239.166"
        }
      ]
    }
  ]
}
2020-12-01 15:32:48,390:DEBUG:acme.client:Storing nonce: 0004T9YK4PfIdkiob5SxsII2scF_xcPMu-_oXS99qZXfW8A
2020-12-01 15:32:48,391:WARNING:certbot._internal.auth_handler:Challenge failed for domain mail.hristov.org
2020-12-01 15:32:48,391:INFO:certbot._internal.auth_handler:http-01 challenge for mail.hristov.org
2020-12-01 15:32:48,392:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server:

Domain: mail.hristov.org
Type:   connection
Detail: Fetching http://mail.hristov.org/.well-known/acme-challenge/REMOVED: Error getting validation data

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
2020-12-01 15:32:48,393:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

2020-12-01 15:32:48,393:DEBUG:certbot._internal.error_handler:Calling registered functions
2020-12-01 15:32:48,393:INFO:certbot._internal.auth_handler:Cleaning up challenges
2020-12-01 15:32:48,394:DEBUG:certbot._internal.plugins.standalone:Stopping server at :::80...
2020-12-01 15:32:48,523:ERROR:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/bin/certbot", line 9, in <module>
    load_entry_point('certbot==1.9.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1362, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1243, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 117, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/renewal.py", line 330, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 351, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 398, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

I tried reinstalling the certbot but that did not help. When i run the certbot-2 renew i see the same error. I was curious to see if certbot is using port 80, so i did
ss -lt
and saw for a very short period of time the server listening on http port i.e.
LISTEN 0 5 [::]:http [::]:*

any idea why i am getting this exception and how to update my certificate are highly appreciated since i have only 4 more days to upgrade that certificate.

2

Hi @penguin

Name Error = No AAAA record, that's not a problem. An AAAA isn't required if there is an A-record.

--standalone is hard to debug, because there is no running webserver. So it's unclear why you have a connection problem.

Isn't it possible that you run a webserver? Easier to debug.

3

How did you test port 80?
I get a very curious response:

curl -Iki mail.hristov.org
curl: (7) Failed to connect to mail.hristov.org port 80: No route to host
4

well currently there is nothing listening on that port. Like i said i use the standalone instalation. I find this the most attractive way since the http port is used only when the certbot tryes to update. Like i have mentioned before when i initiate the command certbot-2 renew i see for a short period of time the http server being started on the port and then it closes i.e.
ss -lt
shows for a very short period of time the following
LISTEN 0 5 [::]:http [::]:*

5

OK i figured it out. As always it is the last place you expect. The IPS systems was in a mode that did not completely pass the packes as i expected i.e. it was still dopping frames. Then i saw that it is dropping from the IP Addresses from letsencrypt. After moving those to the whitelist all work well. I will write to with the vendor in order to figure out how to prevent this in the future. A question on the side . Does anyone know if there is loop of public networks IPv4 and IPv6 that i can whitelist for letsencrypt i.e. in case letsencrypt is using some static pool?

1 Like
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.