Root domain redirected via DNS to Hetzner, cert issuing woes

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: homeworldlore.net

I ran this command: certbot certonly (and others)

It produced this output:

File "/usr/lib/python3/dist-packages/acme/client.py", line 602, in _check_response
    raise messages.Error.from_json(jobj)
acme.messages.Error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Service busy; retry later.
2024-11-13 17:46:32,863:ERROR:certbot._internal.log:An unexpected error occurred:
2024-11-13 17:46:32,863:ERROR:certbot._internal.log:Service busy; retry later.

My web server is (include version): Apache2: 2.4.58 on Hetzner

The operating system my web server runs on is (include version): Ubuntu 24.04 lts on Hetzner

My hosting provider, if applicable, is: Hetzner.com

My Domain Host: Inleed.net - runs CloudLinux afaik

I can login to a root shell on my machine (yes or no, or I don't know): Yes, on both

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): This needs more explanation: The domain is managed by inleed.net and I have other sites/domains there and do all my DNS configuration there. I created DNS A entries redirecting the homeworldlore.net domain to a Hetzner VPS IPv4 address. Not IPV6.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.9.0

I could of course have hit the rate limit at 10 per hour, as I can see here Rate Limits - Let's Encrypt but I should stress that I am only trying to register ONE certificate for ONE domain so not sure how that applies. Been at it for hours and many attempts.

I suspect the DNS redirection from the server that is the domain host to the Hetzner VPS might be a problem, but it would be nice to get a verification if or, if not. The Control Panel on the domain web host - inleed.net - claims there is no certificate for www.homeworldlore.net.
There is no wildcard certificate for the domain either. I should add that there is a subdomain with certificate called mw.homeworldlore.net that I keep at the domain host shared server and that works fine.

Not sure what else to add. please feel free to ask me and I shall answer as best i can and run all the commands you like via SSH to get more info.

Nice... :slight_smile:

I really should edit some in the 1st post, but ah well, from that link: crt.sh | homeworldlore.net

I can see that last successful certificate for the site was issued on the Domain Web Host - inleed.net - on the 30 of September: crt.sh | 14744833123

What I am doing is moving the site from that web host to the Hetzner VPS. Hence the new cert requests.

But this is from today

image

Looking at this specific error, it looks like the Let's Encrypt systems are temporary overloaded. Not much you can do to fix/prevent this specific "Service busy" error.

See also I need help in ssl certificate - #4 by jcjones which was posted a few minutes ago about the same error in a different thread.

2 Likes

Seen it... I need help in ssl certificate - #4 by jcjones

Ok, will try again later or tomorrow.

1 Like

So giving it a shot again but no joy...

certbot --apache -d homeworldlore.net -d www.homeworldlore.net

Requesting a certificate for homeworldlore.net and www.homeworldlore.net

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: homeworldlore.net
  Type:   unauthorized
  Detail: 2001:67c:750::19: Invalid response from https://homeworldlore.net/.well-known/acme-challenge/EMAlj6v-5kSsi-VVtX9PUZeAdDWNgzOqwo9i69mv5OM: 404

  Domain: www.homeworldlore.net
  Type:   unauthorized
  Detail: 2001:67c:750::19: Invalid response from https://www.homeworldlore.net/.well-known/acme-challenge/ZUdEcb03KFC1fXXOm84tmR0G9WGolz5cSlW0d12aTX8: 404

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

then I went for the simpler

root@hwlore-bsa-ubuntu-4gb-nbg1-1:~# certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Apache Web Server plugin (apache)
2: Runs an HTTP server locally which serves the necessary validation files under
the /.well-known/acme-challenge/ request path. Suitable if there is no HTTP
server already running. HTTP challenge only (wildcards not supported).
(standalone)
3: Saves the necessary validation files to a .well-known/acme-challenge/
directory within the nominated webroot path. A seperate HTTP server must be
running and serving files from the webroot path. HTTP challenge only (wildcards
not supported). (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 1

Which names would you like to activate HTTPS for?
We recommend selecting either all domains, or all domains in a VirtualHost/server block.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: homeworldlore.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Requesting a certificate for homeworldlore.net

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: homeworldlore.net
  Type:   unauthorized
  Detail: 2001:67c:750::19: Invalid response from https://homeworldlore.net/.well-known/acme-challenge/_MAUZ2KlJigaSEyxPNIamSUMy0bN4rbai9avpjYb428: 404

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

Logs

2024-11-14 06:47:44,095:DEBUG:certbot._internal.error_handler:Calling registered functions
2024-11-14 06:47:44,096:INFO:certbot._internal.auth_handler:Cleaning up challenges
2024-11-14 06:47:44,217:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot==2.9.0', 'console_scripts', 'certbot')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1894, in main
    return config.func(config, plugins)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1450, in run
    new_lineage = _get_and_save_cert(le_client, config, domains,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 143, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 517, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 428, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2024-11-14 06:47:44,219:ERROR:certbot._internal.log:Some challenges have failed.

I noticed the timestamp does not match my local time. Correcting before trying again.

Still server issues on your side?

time corrected

root@hwlore-bsa-ubuntu-4gb-nbg1-1:~# sudo timedatectl set-timezone Europe/Berlin
root@hwlore-bsa-ubuntu-4gb-nbg1-1:~# timedatectl
               Local time: Thu 2024-11-14 07:52:19 CET
           Universal time: Thu 2024-11-14 06:52:19 UTC
                 RTC time: Thu 2024-11-14 06:52:20
                Time zone: Europe/Berlin (CET, +0100)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

and that made no difference

See Let's Debug.

Your IPv6 address responds with something from a "Litespeed" webserver while the IPv4 address responds as "Apache". Looking at the answers on the questionnaire, the latter is probably correct, but the former not. And Let's Encrypt prefers IPv6, as do many other systems.

1 Like

Nice catch, but I already mentioned that...

[MultipleIPAddressDiscrepancy](https://letsdebug.net/homeworldlore.net/2281416#MultipleIPAddressDiscrepancy-Warning)

Warning

homeworldlore.net has multiple IP addresses in its DNS records. While they appear to be accessible on the network, we have detected that they produce differing results when sent an ACME HTTP validation request. This may indicate that some of the IP addresses may unintentionally point to different servers, which would cause validation to fail.

[Address=2001:67c:750::19,Address Type=IPv6,Server=LiteSpeed,HTTP Status=301,Number of Redirects=1,Final HTTP Status=404] vs [Address=167.235.28.25,Address Type=IPv4,Server=Apache/2.4.58 (Ubuntu),HTTP Status=404]

Which brings me to what I suspected with ipv6... being not defined in DNS, or rather having the wrong entry. When I try to get what that should translate to (ipv6 is garbage) I am not able to get the corresponding IPv6 for the ipv4 address.

What is 167.235.28.25 in ipv6 ?
Hmmm

I will try that....

or this

root@hwlore-bsa-ubuntu-4gb-nbg1-1:~# ip -6 addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 fe80::9400:3ff:fec8:96e1/64 scope link
valid_lft forever preferred_lft forever

Nothing, it's IPv4. IPv6 is a different protocol entirely (and let's forget some transition stuff here, they are not in use any longer). If you want or need IPv6 connectivity, you should find out what the IPv6 address of the server is and configure that specific address in DNS.

2 Likes

Well, it is not me wanting it, its LetsEncrypt... from what I understand of your posts...

Let's Encrypt prefers IPv6, but uses IPv4 without a problem if IPv6 is not available. However, if DNS claims IPv6 is available and it doesn't work.. Or is a completely different server entirely such as in your case, well, sure, then it doesn't work.

1 Like

Yeah, well trying to fix that.

Got the ipv6 for the relevant server and adding it to dns and will test again a bit later.

But it is a jungle of settings.

  • I had to order the ipv6 explicitly if I wanted to have any chance of having it in DNS
  • Then I had to edit a configuration file (yaml syntax) on server which I do not know if it is correctly done yet because the file do not look anything like the example in the referred documentation. The docs example is also from Ubuntu 20 and I am running Ubuntu 24.
  • Also seems that Hetzner runs a reverse proxy and I do not know if that has to be active or not for the IPv6 address.
  • Have edited the DNS entries with new IPv6 address but still can't ping it using an external web service for that since there seem to be issues pinging IPv6 from your own computer. Maybe it just needs a bit more time.

And still can't edit my posts here.... :face_with_symbols_over_mouth:

Maybe I just have to skip the DNS for IPv6, but I would very much like it to work, because, well it is the correct thing to do and I am very much in to doing things the CORRECT way.

And just tried again just for the hell of it.

~#certbot certonly
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Apache Web Server plugin (apache)
2: Runs an HTTP server locally which serves the necessary validation files under
the /.well-known/acme-challenge/ request path. Suitable if there is no HTTP
server already running. HTTP challenge only (wildcards not supported).
(standalone)
3: Saves the necessary validation files to a .well-known/acme-challenge/
directory within the nominated webroot path. A seperate HTTP server must be
running and serving files from the webroot path. HTTP challenge only (wildcards
not supported). (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 1

Which names would you like to activate HTTPS for?
We recommend selecting either all domains, or all domains in a VirtualHost/server block.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: homeworldlore.net
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Requesting a certificate for homeworldlore.net

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/homeworldlore.net/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/homeworldlore.net/privkey.pem
This certificate expires on 2025-02-12.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

Hell yeah...

1 Like

And then I had to edit apache files before getting it to work... but eventually sorted it out...

I guess I should have installed CloudPanel to make things simpler before messing with all this.

Yes, because you used the certonly subcommand :slight_smile: So it's to be expected that you needed to manually edit the Apache configuration files.

That said, it seems that the currently configured IPv6 address, 2a01:4f8:1c1c:2b42::1, is not reachable from my point of view.

1 Like

Not from mine either. Worse, their cert renewal will fail because they redirect the HTTP Challenge to HTTPS.

@SecCon You need to fix your IPv6 or remove the AAAA record. Let's Encrypt will try IPv4 after an IPv6 timeout for only the initial HTTP challenge request. You redirect that to HTTPS and Let's Encrypt challenge will fail as your IPv6 times out. See: IPv6 Support - Let's Encrypt

Note that anyone else trying to use your IPv6 will also have problems. You need to fix that.

Here's the sequence you have with failing IPv6

# The initial HTTP challenge to you
curl -I6 -m5 http://homeworldlore.net/.well-known/acme-challenge/Test404
curl: (28) Failed to connect to homeworldlore.net port 80 after 2501 ms: 
Connection timed out

# Let's Encrypt Server will retry with IPv4. 
# It succeeds but you redirect to HTTPS
curl -I4 -m5 http://homeworldlore.net/.well-known/acme-challenge/Test404
HTTP/1.1 302 Found
Server: Apache/2.4.58 (Ubuntu)
Location: https://homeworldlore.net/.well-known/acme-challenge/Test404

# The request using IPv6 fails and LE does not retry.  Challenge fails
curl -I6 -m5 https://homeworldlore.net/.well-known/acme-challenge/Test404
curl: (28) Failed to connect to homeworldlore.net port 443 after 2501 ms: 
Connection timed out

You can prove this will fail by yourself by running

sudo certbot renew --dry-run

This will not affect existing production certs. It is only a test

2 Likes