Certificate renewals fail on all mail and web servers

georgep · October 7, 2020, 5:49pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mail.anadigi.net

I ran this command: certbot renew

It produced this output:
Challenge failed for domain mail.anadigi.net
Challenge failed for domain mail2.anadigi.net
http-01 challenge for mail.anadigi.net
http-01 challenge for mail2.anadigi.net
Cleaning up challenges
Attempting to renew cert (mail.anadigi.net) from /etc/letsencrypt/renewal/mail.anadigi.net.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mail.anadigi.net/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mail.anadigi.net/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: mail.anadigi.net
Type: connection
Detail: During secondary validation: Fetching
http://mail.anadigi.net/.well-known/acme-challenge/LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs:
Timeout during connect (likely firewall problem)

Domain: mail2.anadigi.net
Type: connection
Detail: During secondary validation: Fetching
http://mail2.anadigi.net/.well-known/acme-challenge/RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g:
Timeout during connect (likely firewall problem)

My web server is (include version): Server version: Apache/2.4.6 (CentOS)

The operating system my web server runs on is (include version):
NAME="CentOS Linux"
VERSION="7 (Core)"
3.10.0-957.27.2.el7.x86_64

My hosting provider, if applicable, is: N/A

I can login to a root shell on my machine (yes or no, or I don't know): Y

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): N

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.7.0

I have several separate servers:

1 x mail server running Nethserver under Centos 7
2 x web servers (multiple virtual stes) running Apache2 under Fedora 32
1 x web server (multiple virual sites) running Apache under Centos 6

All have used Letsencrypt certificates without problem - including automatic backups (run as cron jobs) - for several years.

The servers are fully patched up-to-date. No configuration or other changes (eg; file permissions / ownership) have been made EXCEPT the servers' INTERNAL (ie; private) IP addresses have been changed (from the 192.168.100.xxx range to 192.168.200.xxs) but PUBLIC IP addresses remain constant and unchanged..

The servers are all correctly DNS resolvable (no changes made for years). No changes have been made to firewall settings (other than NAT to reflect internal IP changes) or other protective measures (fail2ban was temporarily disabled on email and 1 web server while attempts to renew were made with no change in outcome). Ports 80 and 443 are open and serving web traffic as normal.

ALL servers (regardless of use or configuration) have failed to renew certificates due to expire during the past month. All fail (most usually) during the http-01 challenge leading to an output such as above. Very infrequently I have seen a dns-01 error - though the site in question remains fully resolvable via any well-known DNS server.

I have placed a test file in the .well-known/acme-challenge directory and confirmed that it can be retrieved via http web browser or command line wget. I can also retrieve all the challenge files in that directory.

As far as I have been able to get anywhere toward resolving this it seems that the mechanism that populates the .well-known/acme-challenge directory has stopped working - in that the http-01 challenge requests files that do not exist.

I have also completely removed certbot and certbot-apache from one web server on which all certificates had expired (including removal of /etc/letsencrypt and /var.log/letsencrypt directories) and manually verified that the webserver was only serving plain HTTP on port 80 (though 443 remained open). On reinstalling certbot and certbot-apache and running certbot --apache EXACTLY the same failures occur with the attempt to obtain new certificates - ie; failure of http-01 challenge requesting files that exist nowhere on the machine.

So, not only do renewals fail but attempts to install NEW certificates on a server cleaned down to basics also fail with the same errors.

The full (lengthy) log output of the latest attempt to run certbot renew on the email server - whose certificate has now expired so leaving all users without email is in the following post.

I would be very grateful if anyone can tell me what in heaven's name is going on and point me toward a solution.

georgep · October 7, 2020, 5:51pm

=========== LOG =============
2020-10-07 17:31:06,581:DEBUG:certbot._internal.main:certbot version: 1.7.0
2020-10-07 17:31:06,582:DEBUG:certbot._internal.main:Arguments:
2020-10-07 17:31:06,582:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntry$
2020-10-07 17:31:06,629:DEBUG:certbot._internal.log:Root logging level set at 20
2020-10-07 17:31:06,629:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-10-07 17:31:06,643:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot.internal.cli.cli_utils. 2020-10-07 17:31:06,686:DEBUG:certbot.ocsp:Querying OCSP for /etc/letsencrypt/archive/mail.anadigi.net/cert21.pem 2020-10-07 17:31:06,686:DEBUG:certbot.ocsp:openssl ocsp -no_nonce -issuer /etc/letsencrypt/archive/mail.anadigi.net/chain21.
2020-10-07 17:31:06,738:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2020-11-01$
2020-10-07 17:31:06,738:INFO:certbot._internal.renewal:Cert is due for renewal, auto-renewing...
2020-10-07 17:31:06,739:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2020-10-07 17:31:06,743:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot._internal.plugins.webroot:Authenticator
Initialized: <certbot._internal.plugins.webroot.Authenticator object at 0x7ff3ba68f6d0>
Prep: True
2020-10-07 17:31:06,743:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.webroot. 2020-10-07 17:31:06,743:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator webroot, Installer None 2020-10-07 17:31:06,747:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(status=
2020-10-07 17:31:06,755:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2020-10-07 17:31:06,764:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsen$
2020-10-07 17:31:07,308:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 658
2020-10-07 17:31:07,308:DEBUG:acme.client:Received response:
HTTP 200
content-length: 658
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
cache-control: public, max-age=0, no-cache
date: Wed, 07 Oct 2020 15:31:07 GMT
x-frame-options: DENY
content-type: application/json

{
  "VEUvnSPLB_M": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2020-10-07 17:31:07,309:INFO:certbot._internal.main:Renewing an existing certificate
2020-10-07 17:31:07,484:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0037_key-certbot.pem
2020-10-07 17:31:07,487:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0037_csr-certbot.pem
2020-10-07 17:31:07,487:DEBUG:acme.client:Requesting fresh nonce
2020-10-07 17:31:07,487:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2020-10-07 17:31:07,621:DEBUG:requests.packages.urllib3.connectionpool:"HEAD /acme/new-nonce HTTP/1.1" 200 0
2020-10-07 17:31:07,622:DEBUG:acme.client:Received response:
HTTP 200
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
cache-control: public, max-age=0, no-cache
date: Wed, 07 Oct 2020 15:31:07 GMT
x-frame-options: DENY
replay-nonce: 0102NUfz87ofNBaTbYgzNCpPKjqxR3SBcecjMDSvn-S_mXQ


2020-10-07 17:31:07,622:DEBUG:acme.client:Storing nonce: 0102NUfz87ofNBaTbYgzNCpPKjqxR3SBcecjMDSvn-S_mXQ
2020-10-07 17:31:07,623:DEBUG:acme.client:JWS payload:
{
  "identifiers": [
    {
      "type": "dns",
      "value": "mail.anadigi.net"
    },
    {
      "type": "dns",
      "value": "mail2.anadigi.net"
    }
  ]
}
2020-10-07 17:31:07,625:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJub25jZSI6ICIwMTAyTlVmejg3b2ZOQmFUYllnek5DcFBLanF4UjNTQmNlY2pNRFN2bi1TX21YUSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21l$
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwgCiAgICAgICJ2YWx1ZSI6ICJtYWlsLmFuYWRpZ2kubmV0Ig$
  "signature": "VIOoZX64W3CSescwMYC5eVBxRHZ5YRoUYHbiPPyQgdhA498KVLl5_sNBMmIzmrecVwpzseqk7Y9q_iTZu72XLbi4vsLbCi2oan6Ktnme0J7o$
}
2020-10-07 17:31:07,940:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/new-order HTTP/1.1" 201 484
2020-10-07 17:31:07,940:DEBUG:acme.client:Received response:
HTTP 201
content-length: 484
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
location: https://acme-v02.api.letsencrypt.org/acme/order/17431926/5569456631
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:07 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0101u7b2864_ke89vbec5A56nrbFLjeAx3SZ9ON9a0SQWQU

{
  "status": "pending",
  "expires": "2020-10-14T15:31:07.826346772Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "mail.anadigi.net"
    },
    {
      "type": "dns",
      "value": "mail2.anadigi.net"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898080",
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898081"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/17431926/5569456631"
}
2020-10-07 17:31:07,941:DEBUG:acme.client:Storing nonce: 0101u7b2864_ke89vbec5A56nrbFLjeAx3SZ9ON9a0SQWQU
2020-10-07 17:31:07,941:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:07,944:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898$
{
  "protected": "eyJub25jZSI6ICIwMTAxdTdiMjg2NF9rZTg5dmJlYzVBNTZucmJGTGplQXgzU1o5T045YTBTUVdRVSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21l$
  "payload": "",
  "signature": "fUUL2EsI7VH8QqVuY729TWDyZTNKVCCqhCUgEiYY78bjQXf_DZCw-Jg6g9hjdV2HnG00CHgM18ZER6JmP440POWmQjKHzmO5TccJ4DaTgphV$
}
2020-10-07 17:31:08,105:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898080 HTTP/1.1" 200 794
2020-10-07 17:31:08,105:DEBUG:acme.client:Received response:
HTTP 200
content-length: 794
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:08 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0101W3MVed5YicIGmonjF07pFJT2L2-6jpFoON2kNfZjhvY

{
  "identifier": {
    "type": "dns",
    "value": "mail.anadigi.net"
  },
  "status": "pending",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/6Z0fhg",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/ac4Mng",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/l8PVyQ",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    }
  ]
}
2020-10-07 17:31:08,106:DEBUG:acme.client:Storing nonce: 0101W3MVed5YicIGmonjF07pFJT2L2-6jpFoON2kNfZjhvY
2020-10-07 17:31:08,106:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:08,108:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898$
{
  "protected": "eyJub25jZSI6ICIwMTAxVzNNVmVkNVlpY0lHbW9uakYwN3BGSlQyTDItNmpwRm9PTjJrTmZaamh2WSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21l$
  "payload": "",
  "signature": "NgH3BdYaySbGJbjTYVq6AwZ01Oh2rn35ZUWFsq8r-Psx5vzLklM_JQTbn2VuRg1RsJPyLLlZcfNVGlW0pDtf4HVglse9TE_WWBmrHQr65zxg$
}
2020-10-07 17:31:08,298:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898081 HTTP/1.1" 200 795
2020-10-07 17:31:08,299:DEBUG:acme.client:Received response:
HTTP 200
content-length: 795
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:08 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0102lT0An7Qmaqx3UF78QH2fl895tOYR4p0j5Pm-3AgmBW0

{
  "identifier": {
    "type": "dns",
    "value": "mail2.anadigi.net"
  },
  "status": "pending",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/3DZ_Cw",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/kIctug",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/U8shgg",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    }
  ]
}
2020-10-07 17:31:08,299:DEBUG:acme.client:Storing nonce: 0102lT0An7Qmaqx3UF78QH2fl895tOYR4p0j5Pm-3AgmBW0
2020-10-07 17:31:08,300:INFO:certbot._internal.auth_handler:Performing the following challenges:
2020-10-07 17:31:08,300:INFO:certbot._internal.auth_handler:http-01 challenge for mail.anadigi.net
2020-10-07 17:31:08,300:INFO:certbot._internal.auth_handler:http-01 challenge for mail2.anadigi.net
2020-10-07 17:31:08,300:INFO:certbot._internal.plugins.webroot:Using the webroot path /var/www/html for all unmatched domain$
2020-10-07 17:31:08,301:DEBUG:certbot._internal.plugins.webroot:Creating root challenges validation dir at /var/www/html/.we$
2020-10-07 17:31:08,301:DEBUG:certbot._internal.plugins.webroot:Creating root challenges validation dir at /var/www/html/.well-known/acme-challenge
2020-10-07 17:31:08,305:DEBUG:certbot._internal.plugins.webroot:Attempting to save validation to /var/www/html/.well-known/acme-challenge/LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs
2020-10-07 17:31:08,308:DEBUG:certbot._internal.plugins.webroot:Attempting to save validation to /var/www/html/.well-known/acme-challenge/RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g
2020-10-07 17:31:08,308:INFO:certbot._internal.auth_handler:Waiting for verification...
2020-10-07 17:31:08,309:DEBUG:acme.client:JWS payload:
{}
2020-10-07 17:31:08,310:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/6Z0fhg:
{
  "protected": "eyJub25jZSI6ICIwMTAybFQwQW43UW1hcXgzVUY3OFFIMmZsODk1dE9ZUjRwMGo1UG0tM0FnbUJXMCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGwtdjMvNzczMTg5ODA4MC82WjBmaGciLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDEuYXBpLmx$
  "payload": "e30",
  "signature": "aMIbJyKERsCnMS4WyYC6FcfUZ3CwsOogRu7YMC2RfqF86cTA5Jhpuj2laCLDOGmzem9_yNqTCs39_6-UUTnvQpZb3OPSBXK3iH569wP8O9LVAyJq0K2uc6aq3H50PN45QiBnMTnIoSUiCm7EDfHOGRJtnupKeugd2VhdXKddKU_X-Dz7FkdLOYC9F9OUvWp8A0Bjq5lKgUV-TJr95OczN4OE1wUC4vv778H$
}
2020-10-07 17:31:08,499:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/chall-v3/7731898080/6Z0fhg HTTP/1.1" 200 185
2020-10-07 17:31:08,499:DEBUG:acme.client:Received response:
HTTP 200
content-length: 185
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898080>;rel="up"
location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/6Z0fhg
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:08 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0101rxnXL7EW-AnZx6Ig-ue7BlmYeLWmIJKGuRB324iFOwA

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/6Z0fhg",
  "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
}
2020-10-07 17:31:08,499:DEBUG:acme.client:Storing nonce: 0101rxnXL7EW-AnZx6Ig-ue7BlmYeLWmIJKGuRB324iFOwA
2020-10-07 17:31:08,500:DEBUG:acme.client:JWS payload:
{}
2020-10-07 17:31:08,502:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/3DZ_Cw:
{
  "protected": "eyJub25jZSI6ICIwMTAxcnhuWEw3RVctQW5aeDZJZy11ZTdCbG1ZZUxXbUlKS0d1UkIzMjRpRk93QSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGwtdjMvNzczMTg5ODA4MS8zRFpfQ3ciLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDEuYXBpLmx$
  "payload": "e30",
  "signature": "dtbg06m5d4KSIWyYw7i360izesh1fPNBqxrDynDrdlq5B03Zdbfg4FhXlMG559XLjhf62W77YZBwvyzDTIVOY_XxncnGh_m_3F56FBPb3sjJScwbcjRH9SU-oVKUo5GU1aZ1THiOfar2Tv1Plh2NW5F0iBlqr2ngt6nBGbFznumWInGK_UJZeR5vgXGeSSB4B4f8MqENGoyBe61y9Li85Y_mb6yK-FfGSaH$
}
2020-10-07 17:31:08,691:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/chall-v3/7731898081/3DZ_Cw HTTP/1.1" 200 185
2020-10-07 17:31:08,692:DEBUG:acme.client:Received response:
HTTP 200
content-length: 185
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898081>;rel="up"
location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/3DZ_Cw
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:08 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0102D9I63WMuo1nhxeut8Ln6WZSF8RbccWH7mHbaz4EyGE4

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/3DZ_Cw",
  "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
}
2020-10-07 17:31:08,692:DEBUG:acme.client:Storing nonce: 0102D9I63WMuo1nhxeut8Ln6WZSF8RbccWH7mHbaz4EyGE4
2020-10-07 17:31:09,694:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:09,696:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898080:
{
  "protected": "eyJub25jZSI6ICIwMTAyRDlJNjNXTXVvMW5oeGV1dDhMbjZXWlNGOFJiY2NXSDdtSGJhejRFeUdFNCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvNzczMTg5ODA4MCIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J$
  "payload": "",
  "signature": "Bq96_opl4CyEVoaL07sbHKlPZzjTOij7CZjTGukCf1b4nItASydKlBZom7EgQVoRuAZlYNYKKNATm4du6tpFZKe4O9ZQPN8XX3d13nUPDPLZ6DM4mUQyC8wLe_SX52kbQIsMqKacjrHGbI0u_SyoR2VLuITGN2RVzqrxnIvutj5wUMOjr5SGSbKkxiOJUAkm6KxfYCG8cAS7XmQn7-cu9YGaNpEueROQ17W$
}
2020-10-07 17:31:09,861:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898080 HTTP/1.1" 200 794
2020-10-07 17:31:09,862:DEBUG:acme.client:Received response:
HTTP 200
content-length: 794
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:09 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0102odhp4xHTcbGfb39y-mzrmQEyBPe7qXX-pJ4MlROOrdM

{
  "identifier": {
    "type": "dns",
    "value": "mail.anadigi.net"
  },
  "status": "pending",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/6Z0fhg",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/ac4Mng",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/l8PVyQ",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    }
  ]
}

georgep · October 7, 2020, 5:52pm

2020-10-07 17:31:09,862:DEBUG:acme.client:Storing nonce: 0102odhp4xHTcbGfb39y-mzrmQEyBPe7qXX-pJ4MlROOrdM
2020-10-07 17:31:09,863:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:09,865:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898081:
{
  "protected": "eyJub25jZSI6ICIwMTAyb2RocDR4SFRjYkdmYjM5eS1tenJtUUV5QlBlN3FYWC1wSjRNbFJPT3JkTSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvNzczMTg5ODA4MSIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J$
  "payload": "",
  "signature": "CH36misN8W5sqhidG3uoZaG6OYcC3dlVMvlKZ82gDzAbU6wbIP-Oq9uprYj94mNcA_El_q7uASsiosTU38HpjyHTj65hDSQp0WLn-Nuv4XB-OoCqhX1QRKDlA2GIJwgaSeuCe08tQ6_NTcANmQcLENBOvQUenuBBdJnKfnLfxE3UzNz-MwUD3wodToSShSaaak3Q0lGP_t7r7JwqO5RYOOQAqfVs50hSViT$
}
2020-10-07 17:31:10,027:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898081 HTTP/1.1" 200 795
2020-10-07 17:31:10,028:DEBUG:acme.client:Received response:
HTTP 200
content-length: 795
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:09 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0102d6lWSusPn33hymG6n_n5ANZn3G1tGFmgX1DOO-bgMfk

{
  "identifier": {
    "type": "dns",
    "value": "mail2.anadigi.net"
  },
  "status": "pending",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/3DZ_Cw",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/kIctug",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/U8shgg",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    }
  ]
}
2020-10-07 17:31:10,028:DEBUG:acme.client:Storing nonce: 0102d6lWSusPn33hymG6n_n5ANZn3G1tGFmgX1DOO-bgMfk
2020-10-07 17:31:13,032:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:13,034:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898080:
{
  "protected": "eyJub25jZSI6ICIwMTAyZDZsV1N1c1BuMzNoeW1HNm5fbjVBTlpuM0cxdEdGbWdYMURPTy1iZ01mayIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvNzczMTg5ODA4MCIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J$
  "payload": "",
  "signature": "JEW-3hkDZe2BZx7iuUrQv3djoPJOQ-Me4aZat_6QywzAYEoWaWljG29t_L7gu3_oe6c6ZYWx6aiFTjNJe0N1gx-CZZHdT-PX_gBYF65Zd00stHHS2WzUgpX7YOsBBZiWHVHE9YYWS-nW70QGb2mqGP9udNwfeiNQjx88wsRKHkzW25sjb-rY53FzNPIPlpzp-mNrd17nbrVK9W2nneljRtPXQOxOLYSSaSD$
}
2020-10-07 17:31:13,205:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898080 HTTP/1.1" 200 794
2020-10-07 17:31:13,206:DEBUG:acme.client:Received response:
HTTP 200
content-length: 794
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:13 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0102glD1GdIkFMK5oZheWJApGHuA3cXtIlB1tT4pGVq61xg

{
  "identifier": {
    "type": "dns",
    "value": "mail.anadigi.net"
  },
  "status": "pending",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/6Z0fhg",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/ac4Mng",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/l8PVyQ",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    }
  ]
}
2020-10-07 17:31:13,206:DEBUG:acme.client:Storing nonce: 0102glD1GdIkFMK5oZheWJApGHuA3cXtIlB1tT4pGVq61xg
2020-10-07 17:31:13,207:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:13,209:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898081:
{
  "protected": "eyJub25jZSI6ICIwMTAyZ2xEMUdkSWtGTUs1b1poZVdKQXBHSHVBM2NYdElsQjF0VDRwR1ZxNjF4ZyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvNzczMTg5ODA4MSIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J$
  "payload": "",
  "signature": "dMMcx95B50vr-C11SWZce6HNkIyGoyG4ZyPj5Htv1AHf5WfLSzURusC-UGrrm8gIYCZnkjVKSbj5j71ZswHOt6Q6WiU_LXltA8qKjx4iI5mxLJ5KaWMN71yn0LgNEFMP2HOXb32ppLQxsQFo9GoLZs-BdEQFanFj3SyWg6Xnm-YS69REfEvGEACNlrvxEKapc-NH6pGBl2Nfc2P8a5_dB5YZQDY8efYXbuk$
}
2020-10-07 17:31:13,380:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898081 HTTP/1.1" 200 795
2020-10-07 17:31:13,381:DEBUG:acme.client:Received response:
HTTP 200
content-length: 795
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:13 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0101hMCUfQmiCG3QWzX2ci8IyEvzL758yIAX_JF5kdIDio0

{
  "identifier": {
    "type": "dns",
    "value": "mail2.anadigi.net"
  },
  "status": "pending",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/3DZ_Cw",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/kIctug",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/U8shgg",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    }
  ]
}
2020-10-07 17:31:13,382:DEBUG:acme.client:Storing nonce: 0101hMCUfQmiCG3QWzX2ci8IyEvzL758yIAX_JF5kdIDio0
2020-10-07 17:31:16,384:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:16,386:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898080:
{
  "protected": "eyJub25jZSI6ICIwMTAxaE1DVWZRbWlDRzNRV3pYMmNpOEl5RXZ6TDc1OHlJQVhfSkY1a2RJRGlvMCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvNzczMTg5ODA4MCIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J$
  "payload": "",
  "signature": "BsCaj7DpOEcmTP6s6wmWnKEiPfQDQ2WqIMdgZHyJFw3xEbOf7S7MxkqJK9LU0w2hmByaWRisZb2RRKxeWyZiAzrHolYcS5w_0A3xOoOI1ThjEh-gbLfPwUPQhZ5wg_wWMALhos-Hd-gSOyCJN1MOeFGvAXpd6WJvtDscwV2WYzS_c6I2TyhfHPzwtoCH82lrN4KuEOAwDwiF_yjW0BYLhiypU74nR9PM3pr$
}
2020-10-07 17:31:16,551:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898080 HTTP/1.1" 200 794
2020-10-07 17:31:16,552:DEBUG:acme.client:Received response:
HTTP 200
content-length: 794
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:16 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0102v2hckAE-QNYWP1BIG5oqVU75eDZMyDWGdC5LW4NEsTo

{
  "identifier": {
    "type": "dns",
    "value": "mail.anadigi.net"
  },
  "status": "pending",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/6Z0fhg",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/ac4Mng",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/l8PVyQ",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs"
    }
  ]
}
2020-10-07 17:31:16,552:DEBUG:acme.client:Storing nonce: 0102v2hckAE-QNYWP1BIG5oqVU75eDZMyDWGdC5LW4NEsTo
2020-10-07 17:31:16,552:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:16,555:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898081:
{
  "protected": "eyJub25jZSI6ICIwMTAydjJoY2tBRS1RTllXUDFCSUc1b3FWVTc1ZURaTXlEV0dkQzVMVzRORXNUbyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvNzczMTg5ODA4MSIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J$
  "payload": "",
  "signature": "bWPOu6R84RVyTyo25Hjz1KZY2h71kvGyJUyMJUrQM2d6SRsIuYTA_RkltfIbgv0vTxWbMQ4uILw5ubtHUlz6DfAkiwNIkpY_hpBIA2NaTUqdcHVKcrXb-Qe2a0xw4Aw0AvPIBzg4sd6D18zQIGDKQTBxVBPfL66g13BLkqPOiXBu9Lh8VVFwstMFn4vmeE6EBD-3xh2dagZDmBN9v2mcuHkiFd6IL8AOyzm$
}
2020-10-07 17:31:16,731:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898081 HTTP/1.1" 200 795
2020-10-07 17:31:16,732:DEBUG:acme.client:Received response:
HTTP 200
content-length: 795
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:16 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 01010wlVhZSItF7WHZ_0VxCXbWuxlKGLWmMUgOv_tCe4w_w

{
  "identifier": {
    "type": "dns",
    "value": "mail2.anadigi.net"
  },
  "status": "pending",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/3DZ_Cw",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/kIctug",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/U8shgg",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g"
    }
  ]
}
2020-10-07 17:31:16,732:DEBUG:acme.client:Storing nonce: 01010wlVhZSItF7WHZ_0VxCXbWuxlKGLWmMUgOv_tCe4w_w
2020-10-07 17:31:19,736:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:19,738:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898080:
{
  "protected": "eyJub25jZSI6ICIwMTAxMHdsVmhaU0l0RjdXSFpfMFZ4Q1hiV3V4bEtHTFdtTVVnT3ZfdENlNHdfdyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvNzczMTg5ODA4MCIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J$
  "payload": "",
  "signature": "TsPHKCu0nhsNuIXNepfNYo3A984Fy3ODGPQlylnsV5el9aLpEbjy-JtCLj-kbYe03RxsxoP6TXjLPAsmPbwIRQkWBunVgwerDU2h7L2_X44OlHwsTK-IMEXG07rw7Uq55d_QpiYYZT5R62lYuIdtfkYVtEFbklKefb-M0jhaeOqC1xYQabEpLOwyzej6nRr_w2CMjIN7yKRFNj-vjVGWQqW4PG0ZG-CuIn5$
}
2020-10-07 17:31:19,908:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898080 HTTP/1.1" 200 1029
2020-10-07 17:31:19,909:DEBUG:acme.client:Received response:
HTTP 200
content-length: 1029
cache-control: public, max-age=0, no-cache
content-length: 1029
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:19 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0101t2Q3cG7qOgX9MBy82QACZiytuOHOYKCOQPtXmZ_KwXk

{
  "identifier": {
    "type": "dns",
    "value": "mail.anadigi.net"
  },
  "status": "invalid",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "During secondary validation: Fetching http://mail.anadigi.net/.well-known/acme-challenge/LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs: Timeout during connect (likely firewall problem)",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898080/6Z0fhg",
      "token": "LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs",
      "validationRecord": [
        {
          "url": "http://mail.anadigi.net/.well-known/acme-challenge/LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs",
          "hostname": "mail.anadigi.net",
          "port": "80",
          "addressesResolved": [
            "5.196.126.91"
          ],
          "addressUsed": "5.196.126.91"
        }
      ]
    }
  ]
}
2020-10-07 17:31:19,909:DEBUG:acme.client:Storing nonce: 0101t2Q3cG7qOgX9MBy82QACZiytuOHOYKCOQPtXmZ_KwXk
2020-10-07 17:31:19,910:DEBUG:acme.client:JWS payload:

2020-10-07 17:31:19,912:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/7731898081:
{
  "protected": "eyJub25jZSI6ICIwMTAxdDJRM2NHN3FPZ1g5TUJ5ODJRQUNaaXl0dU9IT1lLQ09RUHRYbVpfS3dYayIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvNzczMTg5ODA4MSIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J$
  "payload": "",
  "signature": "EGTTVMj_5h1Vbsom18U_ywgWLAW7VLoZlJM3fWBq4GmpHdbJjQzn7NQfc8qR1z7F8pnMMdaWuLtjUBbaYAlvJ2GdZXs_vvJnbi9vqmDcgglY1oMEmhOKqa5iquzR1xf2xew8gDP4LuMjrUndEAvdjx1UNkQyDRK_DB4qo3bxoIViStCrOLlntDipCq6hPO_aWWhHZKMuU3gG7drpK0mleutVuziQCHmEUBq$
}
2020-10-07 17:31:20,078:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/authz-v3/7731898081 HTTP/1.1" 200 1033
2020-10-07 17:31:20,078:DEBUG:acme.client:Received response:
HTTP 200
content-length: 1033
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
boulder-requester: 17431926
date: Wed, 07 Oct 2020 15:31:20 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0102XyO3VHabKq0A3EZNfA3fKZcvx_5rNGemiDt7UayE58w

{
  "identifier": {
    "type": "dns",
    "value": "mail2.anadigi.net"
  },
  "status": "invalid",
  "expires": "2020-10-14T15:31:07Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:connection",
        "detail": "During secondary validation: Fetching http://mail2.anadigi.net/.well-known/acme-challenge/RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g: Timeout during connect (likely firewall problem)",
        "status": 400
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/7731898081/3DZ_Cw",
      "token": "RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g",
      "validationRecord": [
        {
          "url": "http://mail2.anadigi.net/.well-known/acme-challenge/RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g",
          "hostname": "mail2.anadigi.net",
          "port": "80",
          "addressesResolved": [
            "5.196.141.19"
          ],
          "addressUsed": "5.196.141.19"
        }
      ]
    }
  ]
}
2020-10-07 17:31:20,079:DEBUG:acme.client:Storing nonce: 0102XyO3VHabKq0A3EZNfA3fKZcvx_5rNGemiDt7UayE58w
2020-10-07 17:31:20,079:WARNING:certbot._internal.auth_handler:Challenge failed for domain mail.anadigi.net
2020-10-07 17:31:20,079:WARNING:certbot._internal.auth_handler:Challenge failed for domain mail2.anadigi.net
2020-10-07 17:31:20,080:INFO:certbot._internal.auth_handler:http-01 challenge for mail.anadigi.net
2020-10-07 17:31:20,080:INFO:certbot._internal.auth_handler:http-01 challenge for mail2.anadigi.net
2020-10-07 17:31:20,081:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server:

Domain: mail.anadigi.net
Type:   connection
Detail: During secondary validation: Fetching http://mail.anadigi.net/.well-known/acme-challenge/LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs: Timeout during connect (likely firewall problem)

Domain: mail2.anadigi.net
Type:   connection
Detail: During secondary validation: Fetching http://mail2.anadigi.net/.well-known/acme-challenge/RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g: Timeout during connect (likely firewall problem)

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address an$
2020-10-07 17:31:20,081:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

2020-10-07 17:31:20,081:DEBUG:certbot._internal.error_handler:Calling registered functions
2020-10-07 17:31:20,082:INFO:certbot._internal.auth_handler:Cleaning up challenges
2020-10-07 17:31:20,082:DEBUG:certbot._internal.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/LB0RCvfTzpadvQ01dDQGwMVxifT9dhQvzQsT4gPoiLs
2020-10-07 17:31:20,082:DEBUG:certbot._internal.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/RxKuBxkKSPbusqNAu__IWCHlVlECCV5zw-X3lG9To9g
2020-10-07 17:31:20,083:DEBUG:certbot._internal.plugins.webroot:All challenges cleaned up
2020-10-07 17:31:20,083:WARNING:certbot._internal.renewal:Attempting to renew cert (mail.anadigi.net) from /etc/letsencrypt/renewal/mail.anadigi.net.conf produced an unexpected error: Some challenges have failed.. Skipping.
2020-10-07 17:31:20,085:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/_internal/renewal.py", line 462, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1178, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 116, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/renewal.py", line 320, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 351, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 398, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
AuthorizationError: Some challenges have failed.

2020-10-07 17:31:20,085:ERROR:certbot._internal.renewal:All renewal attempts failed. The following certs could not be renewed:
2020-10-07 17:31:20,085:ERROR:certbot._internal.renewal:  /etc/letsencrypt/live/mail.anadigi.net/fullchain.pem (failure)
2020-10-07 17:31:20,087:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 9, in <module>
    load_entry_point('certbot==1.7.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1357, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1261, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/renewal.py", line 487, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
Error: 1 renew failure(s), 0 parse failure(s)
2020-10-07 17:31:20,088:ERROR:certbot._internal.log:1 renew failure(s), 0 parse failure(s)

rg305 · October 7, 2020, 7:50pm

It seems that you might have an IPS or other GeoLocation blocking inline device that is stopping some of the LE servers from reaching your HTTP site.
If you have web logs you should see multiple IPs hitting that same long URI at very close timestamps.
If you only see one IP, then there is definitely something blocking the other IPs.

georgep · October 8, 2020, 5:56am

5 stars for divining the problem and it is now resolved - all certificates now renewed ... BUT ...

I mentioned that no changes have been made to the firewall - including its (very) selective use of pfBlocke / MaxMindr to exclude the worst villains. Disabling pfBlocker allows LE to complete as before. Which begs the question - what changed?

Has LE started to use servers that are on the banned lists maintained by the pfBlocker database or has pfBlocker somehow added LE's servers to its list of undesirable servers?

pfBlocker has been in place here and with the same configuration since before LE was first used.

I understand that LE doesn't publicly list its servers - so it is not possible to open the firewall to them - but the impact here of disabling pfBlocker in order to allow LE to work is >1,000 fail2ban events across our servers inside the first 30 minutes so I would quite like to at least resolve which pfBlocker lists are involved.

Many thanks

George - FBCS, FIoD

rg305 · October 8, 2020, 6:06am

LE uses IPs that can and do change over time to validate HTTP auths from multiple locations on the planet.
Any list that blocks gets updated (adds/deletes)
The moon spun around the Earth
The Earth inched ever so closer to another spin around the Sun
Fall happened
Schools started back (in person)
Political debates happened
Eddie Van Halen died (R.I.P)
We met

Take your pick - there is hardly anything that doesn't change ~in this everchanging world in which we live in...~

rg305 · October 8, 2020, 6:19am

I think you could "redo" your IP blocking and have just as good a security posture.

To maintain utmost security, It would require the use of a dedicated/isolated web server.

The idea is simple:
Allow all port 80 (HTTP) connections in and direct them all (regardless of the number of external IPs) to the same single dedicated/isolated web server.
From there you can do one of two things:

handle all the LE auth requests then and there (making it like your cert "repository") [GOOD]
Pros: All-in-one stop. Only one place to look and fix cert issues.
Cons: The cert repository is directly accessible via HTTP from the entire planet (all eggs in one less than ideally secured basket)
You have to sync/distribute all the certs to their respective systems.
(reverse)Proxy only the LE auth requests to their respective internal servers (via SNI) [BETTER]
Pros: Each internal server can run it's own ACME client (certbot|acme.sh|PoshACME) without direct HTTP accessibility* from the Internet.
Cons: The added administrative overhead/costs associated with the additional dedicated server (can be virtual)

And simply redirect ALL other requests to HTTPS - which is where you handle the IP blocks/drops/and rolls.

JimPas · October 8, 2020, 6:30am

Hey George! It would be nice to mark Rudy's post (#4) as the solution.

rg305 · October 8, 2020, 7:04am

I got better than that!
I got:

[you don't see that everyday]
and Hi Jim!

georgep · October 8, 2020, 7:52am

Hi rg305,

Nice, flippant answer - I take it from your comment that you are a young student. You are talking to a Fellow of the British Computer Society (a rare beast) with almost 50 years experience in software development and delivery of rather critical systems under his belt.

No criticism intended - you resolved a problem that had me tearing my hair out for a month. Well done. But the problem doesn't end with the simple resolution of "open your firewall to the winds". That's experience speaking.

I've seen plenty of changes over the decades (been responsible for quite a few) but have usually managed to let users know about what affects them.

Just a thought ...

It's truly remarkable that our system has worked fine with its existing IP blockers in place (that do not block entire geographic regions - just known spam/malware/DOS servers) and suddenly, within the last certificate renewal cycle - so, certainly within the past 30+days - everything LE related stopped working - ie; almost ALL of LE's servers were blocked for appearance on a "naughty" list.

To answer your earlier point, our earlier HTTP logs do show multiple successful rapid accesses from LE servers requesting acme-challenge responses - eg;

66.133.109.36 - - [07/Oct/2020:12:54:00 +0200] "GET /.well-known/acme-challenge/9fz9eo7HJeezR-5muRzos4IANPBWTdqtsd4jpNXOD0E HTTP/1.1" 200 87
18.196.96.172 - - [07/Oct/2020:12:54:00 +0200] "GET /.well-known/acme-challenge/9fz9eo7HJeezR-5muRzos4IANPBWTdqtsd4jpNXOD0E HTTP/1.1" 200 87

the "200" response (unless I have completely lost my marbles) indicating a successful file delivery.

RDNS on these servers shows them to be:

The very next log entry is:

64.78.149.164 - - [07/Oct/2020:16:52:25 +0200] "GET /.well-known/acme-challenge/vwZbiUywkz2zKSo_L01V9BXa7hCUJw5geXrEeURGBkU HTTP/1.1" 301 -

which resolves as:

outbound2.letsencrypt.org

... which provoked a "301" (Permanently moved") response from our server (no configuration change) followed by a "404" (Not found) response.

So, it seems that some acme-challenge requests were getting through - but the responses from the certbot software were erratic, to say the least. What was not happening was any run of more than two successful acme-challenge requests - indicating that you are correct in identifying that some firewall blocking was happening. The questions remain - Why? and How so consistently that they stopped all certificate renewals for a month?

The log from this morning's successful application of new certificates is:

64.78.149.164 - - [08/Oct/2020:05:38:51 +0200] "GET /.well-known/acme-challenge/dxO5T54ZqM9CZkJJaZRTActnb-DaFSWVqpQMoUuaT0I HTTP/1.1" 200 87
52.28.236.88 - - [08/Oct/2020:05:38:51 +0200] "GET /.well-known/acme-challenge/dxO5T54ZqM9CZkJJaZRTActnb-DaFSWVqpQMoUuaT0I HTTP/1.1" 200 87
3.128.26.105 - - [08/Oct/2020:05:38:51 +0200] "GET /.well-known/acme-challenge/dxO5T54ZqM9CZkJJaZRTActnb-DaFSWVqpQMoUuaT0I HTTP/1.1" 200 87
34.209.232.166 - - [08/Oct/2020:05:38:52 +0200] "GET /.well-known/acme-challenge/dxO5T54ZqM9CZkJJaZRTActnb-DaFSWVqpQMoUuaT0I HTTP/1.1" 200 87
3.128.26.105 - - [08/Oct/2020:05:38:52 +0200] "GET /.well-known/acme-challenge/a_lbX8pBlSJf5kXU5USxzIryhZdansgN_w8AzFYvaqE HTTP/1.1" 200 87
64.78.149.164 - - [08/Oct/2020:05:38:52 +0200] "GET /.well-known/acme-challenge/a_lbX8pBlSJf5kXU5USxzIryhZdansgN_w8AzFYvaqE HTTP/1.1" 200 87
18.196.96.172 - - [08/Oct/2020:05:38:52 +0200] "GET /.well-known/acme-challenge/a_lbX8pBlSJf5kXU5USxzIryhZdansgN_w8AzFYvaqE HTTP/1.1" 200 87
34.209.232.166 - - [08/Oct/2020:05:38:52 +0200] "GET /.well-known/acme-challenge/a_lbX8pBlSJf5kXU5USxzIryhZdansgN_w8AzFYvaqE HTTP/1.1" 200 87

which resolve as:

a run of 9 acme-challenge requests for two keys.

Now ... NONE of those servers should appear on simple GeoIP blocklists but it looks as if some of the AWS server IPs being used have recently been used by spammers or malware distributors - and have therefore ended up on "naughty" lists.

So my question remains - what changed? Has LE changed to spinning up AWS instances at random rather than running it's own servers? I suspect it has because I can only find two "outboundX.letsencrypt,org" DNS records but you are clearly issuing plenty of acme-challenges from random cloud server instances.

Again, just a thought, but might it be an idea to check the reputation of any IP you are issued by a cloud provider against common blocklists before hanging something as important as an acme-challenge server behind it?

I ask because at the moment the only choice you are giving users of LE that have sensible reputation based firewall blocks in place is to completely remove those blocks (and thereby open their systems up to the scale of spam / malware / DOS abuse that we have seen this morning) OR see the certificate renewal process fail repeatedly and consistently to the point that certificates expire.

Anyway - it appears LE/EFF has a problem ... some of the servers being used by LE have managed to end up on IP blocklists for (presumably former) repeated abuses. That's not something I'd like to happen to any of my servers so I'm sure it's at least worth looking into by LE. I repeat - we do not use generic blocklists (eg; we don't uniformly exclude all of Asia from our systems) but only exclude IP addresses that appear on lists the equivalent of DNSBL.

The 301 and 404 errors triggered by entirely normal acme-challenge requests also deserve someone taking a look at because I repeat ... NOTHING changed to the server config here - suggesting that (unless you think the current Apache2 software has a habit of randomly redirecting requests) the certbot software is occasionally doing very odd things indeed.

Anyway - consider problem resolved. I hope this thread helps anyone else running a half-way sensible firewall and wondering why LE will no longer renew certificates.

Thanks for the quick (and correct) response - even if it leaves the underlying problem unresolved.

georgep · October 8, 2020, 7:56am

Hi Jim - I'd love to ... but see my longer post below. I believe he has cured the symptom but left an illness in circulation. One that's likey to bite anyone else with a half-way sensible firewall in place.

Please feel free to tell me I am wrong - and provide an alternative rational explanation for why a "reputation based" firewall blocklist stops all LE activity ... which springs back into life as soon as "naughty" machines are allowed in (along with every other ne'er-do-well on the planet).

George

rg305 · October 8, 2020, 8:08am

Then your spidey senses are way off - I'm retired.

I got plenty of that too.
Many years working with a lot of three letter "secure businesses" [if you catch my drift].

They change their IPs at will.
ISP like to rotate bad IPs to new customers (to force them clean again).
[that is the nature of the beast]
If you can't do HTTP, there is always DNS auth.
There are plenty ways to skin this cat - one has to fit into your happy meal and let you sleep at night.

Yes, 200 is the expected; but you only show two IPs - they do one main plus two out of three varied.
So you only got two out of four - that's an F on any curve.

certbot doesn't respond to HTTP requests (unless you used it with --standalone).
That is a function of your normal web service.

Of course they don't.
You are checking only the ones that got 200 - which came from your web server which implies they already passed the IPblocking.

They are implementing many more (let's call them random) secondary checks.
This has been advertised/notified/talked about/explained for ages.
There are tons of posts on this forum telling people NOT to block HTTP for that very reason.

In a sense, they are doing both actually.

That has been, let's say "asked and answered" already.

If you could be more specific about which IP(s) were found on which list(s), I'm sure I can find the right person to get that info to soon enough.

rg305 · October 8, 2020, 8:11am

I think you don't realize a couple of things:

I am a volunteer here - I don't work for LE and can't fix the problem(s) you have brought up.
I did resolve your problem on this topic and it should be marked as closed (with whatever answer you choose that does that best).
I don't get any brownie points for closing tickets, so do what you like or feel. You are only "hurting" potential future readers of this thread/topic (when you fail to do your part)

rg305 · October 8, 2020, 8:38am

Didn't catch the reference to Live and Let Die (song for 1973 James Bond movie)
Exactly which part of my response(s) did you consider as being

and made you think I was

I mean even an old dog can learn a new trick and I am the first to say that I am (still) far from perfect.

georgep · October 8, 2020, 9:09am

Rudy - flame off please. No insults intended anywhere on my part. I realise all LE staff are volunteers and respect the work you do and effort you all put in. Please accept my apologies for thinking you a student - you did say you were "In person" back at school.

I'm not trying to score points here or have a "whose experience is greater than whose?" contest. I am attempting to

Alert other users with similar system configurations to my own (ie; multiple servers behind an "enterprise grade" firewall that uses "reputation based" blocklists to keep unwanted visitors out) that the reason LE might stop working for them is the answer you gave - for which I gave you 5 stars and thanked you multiple times.
Alert LE that they are promoting a "simple" way of moving the web from open HTTP to 'secure' HTTPS by installing a simple program and running one command. Look hard at the Letsencrypt and certbot websites - NOWHERE will you see warnings to avoid blocking HTTP or doing any of the "stuff" I have had to do or look into this morning.

Please try to separate your resolution of the problem I was experiencing from the broader issues here.

I take your word for it that the issue "... has been advertised/notified/talked about/explained for ages" but I believe I am like the vast majority of LE users in just having time to read the (overly simplistic IMHO) encouraging information at letsencrypt.org and certbot.eff.org and just installing the thing. Nobody can track every open source project on the planet - nor should they. If something is significant, shout it out on the project's front page. Strange - can't see anything about opening firewalls to the four winds on the LE home page. Sorry, don't have time to track every discussion thread concerning the project.

I merely and meekly expect LE to work - as stated. As it has in my case for several years.

From a broader system perspective, it is simply unacceptable to tell people (again, read the LE and certbot documentation) that http-01 challenges are the simple (and only recommended - look deep and see what is written about the fast-changing [because it simply doesn't work! - and I'm not just referring here to the configuration instructions given but to the security implications behind the existing mechanisms, which are, to be fair, openly disclosed] landscape surrounding DNS based challenges) ... and then have that very same http-01 challenge mechanism fail, NOT because of anything a user has done but because LE/EFF in full knowledge of how ISPs and cloud provider cycle damaged IP addresses decide anyway to use servers for critical functions that they KNOW will break user's systems.

And the answer?

Don't, for heaven's sake, take responsibility on yourselves - just tell the users it's all their fault for not reading some arcane, deeply buried discussion document that tells then to disable their firewalls.

Am I impressed? Take a guess.

LE was sponsored by the EFF to encourage adoption of HTPS. You can tell me to use a DNS based challenge method - I have the skills to do it, our own DNS servers to play with and all the means necessary to automate the update process - been there, done it, got the T-shirt and so can confidently assert it's a useless, non-working mechanism).

Which brings us back to the BIG point.

If LE and certbot (an EFF product) aims to encourage simple adoption of HTTPS it is the responsibility of LE and EFF to deliver a solution that "just works" as advertised on their web sites.

You and I should not be having this discussion. I should not have had to spend hundreds of hours scouring server logs and looking through code to find out why a piece of software provided by the EFF was telling me my firewall was blocking requests when it clearly was NOT - it was correctly blocking random IP addresses that LE/EFF had chosen to use in the full knowledge that they would be blocked.

And not just a few - BUT the MAJORITY of challenge issuing servers so guaranteeing that it becomes EXTREMELY unlikely that any user's system sitting behind a half-way decent firewall would ever get lucky enough to be challenged by sufficient unblocked LE servers to allow it certificates to be renewed.

So - 5 *s - well done and all that - but, sheesh (not at you,Rudy, at your colleagues who decide how to operate LE), at least build a product that puts out sensible error messages - or better yet stop causing the problem in the first place.

As for me matching lists of tens of thousands of servers correctly blocked for bad behaviour against the few LE is (by your words) currently using and may change at any moment

what do you think that might achieve? (no need to answer because it solves nothing)
I have better things to do with my time (especially having just wasted nearly a month fighting LE's software) than track down just a few rogue IP addresses that you already know are rogue.

Like it or not, the only solution to this problem lies in the hands of the people running LE.

If they're happy to let the situation roll on without change, so be it. Let this thread be a warning to anyone else running a firewall and wondering why LE won't work any more.

I hope it saves them a month and a lot of hair.

I'm not going to spend time answering your other reposts - but will, as you clearly believe it necessary - speak with the good folks in Champaign, Ohio and let them know that Apache has taken to randomly issuing 301 and 404 responses to identical requests made against an unchanged site config that immediately before issued a normal 200 response to an identical request. I'm sure they'll believe me (more importantly, you) that the problem lies in their code.

Closing this thread as requested.

George

rg305 · October 8, 2020, 9:45am

I think your words may go mostly unheard in this topic and they do deserve to be heard.
You raise valid points and do present a reasonable case and an "easy fix" to such problems:
LE should review the IPs being provided to them against "certain" major blocklists prior to moving them into production or ensure they have been sufficiently "cleaned" before doing so.
[sorry if I'm paraphrasing here]
That said, I really think you should open a separate topic just for that reason.
I mean: This is a forum (not a wiki).
It is meant for this very reason = discussion.
You have a valid topic of discussion [which, again, has been all but lost in this unrelated topic].
If I failed to mention it - my flames were off, I don't think I attacked you in any way.
I only tried to make you see me/understand where I'm coming from better.
As for the "school thing" I have one kid left in high school and my wife teaches - both of which had to go back "in person" recently. The other ones in the university are doing it online (for the moment).
But we digress...

So please do take the time to open another topic specifically to address your concerns about the secondary validation IPs and the trouble it has caused you - they need to hear it and it needs to be discussed more "directly".

I would like an answer to this last question, if you care to take the time (you can even PM me if you rather not to post anything here - publicly):

flippant?
if that was the comment about "5 stars for divining the problem"

That was "said" to an old friend (@JimPas) and I really meant exactly what I said - I liked it!
And you don't see that everyday - I sure don't - It actually put a smile on my face (for real)
[I mean I would have even clicked the like button if I hadn't run out of likes for today ]

Anyways, we are always here and happy to help encrypt the world.
Cheers from Miami

georgep · October 8, 2020, 1:36pm

Hi Rudy - I suspect we have both been around the Internet (BIX? Usenet? Arpanet?) long enough to understand that communication too often turns into unintended miscommunication.

You paraphrased well. LE needs to either (a) put its new AWS servers to use to take some innocuous load for a month or two (eg; take some of the simple website load) until the assigned IP becomes clean or (b) go through the process of contacting all the DNS BL holders saying "Hey, we're the EFF - please delete IP xxx.xxx.xxx.xxx from your list".

(a) is easier than (b) - even if the IP was just left dormant and not doing anything "naughty" for a while it should eventually fall off the blocklists. I understand that EFF is a donation funded organisation and that money is always tight but - step back - the cost of a tiny AWS instance (that issues an IP address) pales against the value of just this one user's time spent on this nonsense and as soon as the IP is clean the AWS instance can be scaled to whatever size preserving the IP and put into production causing nobody harm.

The important thing is to ensure that only servers with squeaky clean IP addresses are put into production as acme-challenge servers.

I don't think my situation/environment is that uncommon. OK - I'm an old white-haired IT guy who has the chops to run a server farm behind a firewall that uses DNS blacklists to minimise the amount of crud that hits the systems.

But one of your early comments was that (my turn to paraphrase) change happens ... which in this example takes the form of a small business (the precise target market I suspect EFF is aiming certbot at) signing up for a virtual or real rack server on-line and ticking the box that asks "Do you want a firewall with that?"

Viewed technically, the only difference then between that environment and the one I operate is a slight difference in scale. Oh, and the fact that I can at least open up my firewall when I need to - where the innocent small business person has absolutely zero control over the (probably functionally so similar as to be considered identical) hosting company firewall.

So, what are their options? Give up on HTTPS altogether (unlikely as if they want any kind of SEO placement let alone e-commerce these days HTTPS is de-rigeur) or go hunting for other ACME solutions -> which probably involves some expensive "professional" help -> by which time this story is already far away from the EFF's original intent to make use of HTTPS simple and cheap.

As you say, the solution is far from mind-troubling. As I say, it needs to be done and ASAP.

I'm happy to write a quick article about the issue and linking back to here - what subject / topic area would you suggest?

flippant?

I asked "what changed?" as a serious question because I wanted to understand what had altered in the design / implementation / operation of LE and certbot so that I could decide whether to adjust my systems and persist with it or, myself, head off to look at ACME alternatives.

Recall my opening description that I have four servers operating numerous certificates that have been happily running and updating LE almost since Le's inception - and several of the websites (including a couple comprising a few million lines of PHP that I wrote on top of a somewhat (English understatement!) complex database that haven't seen a config change in over a decade - other than the initial implementation of LE - and now none of them will renew nor even when stripped down to basics install afresh certificates using certbot/LE and you may begin to understand my eagerness to understand just what was going on.

The response you gave was - how can I describe it when I am English, you are American, two nations forever separated by a common language? "Jocular"? Jokey"? "Off-hand"? I have many American friends but we all know each other well so converse in a kind of mid-Atlantic jargon and know each other well enough to simply ask for explanation when something is not clear.

Anyway - I was hoping for a serious answer to my serious question that would help me push the penny up the hill.

I got a 'jokey' response that took me nowhere.

Understand that this was against a background of having spent much of the past month tearing certbot and my servers' logs apart ... line by line ... with absolutely zero starting understanding or available documentation covering the way that certbot is designed to work and how it is implemented operationally. Also, as my certificates entered the expiry / renewal period one-by-one and not all at once this problem started as a small camp-fire then turned into a raging forest fire as the certificate on my email server actually expired yesterday causing the loss of all access to email (and a queue of incoming emails that will only arrive over the coming days) and you might understand why "flippant" - cf; https://www.thefreedictionary.com/flippant to read the first definition as "marked by inappropriate levity; frivolous or offhand". No offence is meant or taken from use of "flippant" in English use and, if you read the above explanation, you will I think understand it is an appropriate word.

Let me know where I should best suggest whoever releases LE / certbot into the wild that they pay a bit more intention to both the project's mission statement and think through the impact of decisions on (according to the LE website just now) even 1% of 225 million LE certificate holders and I'll give it my best shot.

Regards from Bordeaux, SW France

rg305 · October 8, 2020, 1:55pm

OK, George, I do see your point and view there ("flippant").

As for the topic "placement" I would not worry too much about that as it can easily be moved afterwards to any other category. I would start it within "Issuance Tech" and go from there.

I was in Paris on my Birthday back in summer of 2018 - it was a very good time and very well spent

georgep · October 8, 2020, 2:03pm

Thanks - will do

Paris is a lovely city - one that can easily (and safely) be walked from one interesting corner to he opposite then all around the middle - it's quite compact. Not quite right at the moment though but what city is?

If you cross the puddle again I suggest a trip to Bordeaux - becoming almost as popular a tourist destination as Paris - best to avoid August when Paris empties and Bordeaux fills. The whole of SW France / Aquitaine is a "bucket list" destination if you have never been. We liked it so much we retired here and live like locals.

system · November 7, 2020, 2:03pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.