Timeout creating new certificate with mailcow

andhotz · March 25, 2024, 1:47pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: www.hotzweb.ch

I ran this command: Since setup of my new mailcow, the built in acme client tries to get a certificate, but always fails

It produced this output: Here's the log:
acme-mailcow-1 | Mon Mar 25 14:06:40 CET 2024 - Initializing, please wait...
acme-mailcow-1 | Mon Mar 25 14:06:40 CET 2024 - Using existing domain rsa key /var/lib/acme/acme/key.pem
acme-mailcow-1 | Mon Mar 25 14:06:40 CET 2024 - Using existing Lets Encrypt account key /var/lib/acme/acme/account.pem
acme-mailcow-1 | Mon Mar 25 14:06:40 CET 2024 - Detecting IP addresses...
acme-mailcow-1 | Mon Mar 25 14:06:49 CET 2024 - OK: 82.136.105.63, 0000:0000:0000:0000:0000:0000:0000:0000
acme-mailcow-1 | Mon Mar 25 14:06:49 CET 2024 - Found A record for www.hotzweb.ch: 82.136.105.63
acme-mailcow-1 | (skipping check, returning 0)
acme-mailcow-1 | Mon Mar 25 14:06:49 CET 2024 - Confirmed A record 82.136.105.63
acme-mailcow-1 | Mon Mar 25 14:06:49 CET 2024 - Certificate /var/lib/acme/www.hotzweb.ch/cert.pem missing or changed domains 'www.hotzweb.ch' - start obtaining
acme-mailcow-1 | Mon Mar 25 14:06:49 CET 2024 - Checking resolver...
acme-mailcow-1 | Mon Mar 25 14:06:49 CET 2024 - Resolver OK
acme-mailcow-1 | Mon Mar 25 14:06:49 CET 2024 - Using command acme-tiny --account-key /var/lib/acme/acme/account.pem --disable-check --csr /var/lib/acme/www.hotzweb.ch/acme.csr --acme-dir /var/www/acme/
acme-mailcow-1 | Parsing account key...
acme-mailcow-1 | Parsing CSR...
acme-mailcow-1 | Found domains: www.hotzweb.ch
acme-mailcow-1 | Getting directory...
acme-mailcow-1 | Directory found!
acme-mailcow-1 | Registering account...
acme-mailcow-1 | Already registered! Account ID: https://acme-v02.api.letsencrypt.org/acme/acct/1635442977
acme-mailcow-1 | Creating new order...
acme-mailcow-1 | Order created!
acme-mailcow-1 | Verifying www.hotzweb.ch...
acme-mailcow-1 | Traceback (most recent call last):
acme-mailcow-1 | File "/usr/bin/acme-tiny", line 8, in
acme-mailcow-1 | sys.exit(main())
acme-mailcow-1 | ^^^^^^
acme-mailcow-1 | File "/usr/lib/python3.11/site-packages/acme_tiny.py", line 195, in main
acme-mailcow-1 | signed_crt = get_crt(args.account_key, args.csr, args.acme_dir, log=LOGGER, CA=args.ca, disable_check=args.disable_check, directory_url=args.directory_url, contact=args.contact, check_port=args.check_port)
acme-mailcow-1 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
acme-mailcow-1 | File "/usr/lib/python3.11/site-packages/acme_tiny.py", line 153, in get_crt
acme-mailcow-1 | raise ValueError("Challenge did not pass for {0}: {1}".format(domain, authorization))
acme-mailcow-1 | ValueError: Challenge did not pass for www.hotzweb.ch: {'identifier': {'type': 'dns', 'value': 'www.hotzweb.ch'}, 'status': 'invalid', 'expires': '2024-04-01T13:06:52Z', 'challenges': [{'type': 'http-01', 'status': 'invalid', 'error': {'type': 'urn:ietf:params:acme:error:connection', 'detail': '82.136.105.63: Fetching http://www.hotzweb.ch/.well-known/acme-challenge/cIiNzgg-UX1qwz0ob7sSlAKCapDbT9PcAwmcj3dqI9A: Timeout during connect (likely firewall problem)', 'status': 400}, 'url': 'https://acme-v02.api.letsencrypt.org/acme/chall-v3/330522413377/2_sLMg', 'token': 'cIiNzgg-UX1qwz0ob7sSlAKCapDbT9PcAwmcj3dqI9A', 'validationRecord': [{'url': 'http://www.hotzweb.ch/.well-known/acme-challenge/cIiNzgg-UX1qwz0ob7sSlAKCapDbT9PcAwmcj3dqI9A', 'hostname': 'www.hotzweb.ch', 'port': '80', 'addressesResolved': ['82.136.105.63'], 'addressUsed': '82.136.105.63', 'resolverAddrs': ['A:10.1.12.89:26534', 'AAAA:10.1.12.85:25533']}], 'validated': '2024-03-25T13:06:54Z'}]}
acme-mailcow-1 | Mon Mar 25 14:07:07 CET 2024 - Failed to obtain certificate /var/lib/acme/www.hotzweb.ch/cert.pem for domains 'www.hotzweb.ch'
acme-mailcow-1 | OK
acme-mailcow-1 | Mon Mar 25 14:07:07 CET 2024 - Some errors occurred, retrying in 30 minutes...
acme-mailcow-1 | OK

The connection from the internet goes throug an ipfire firewall with portforwarding to the server. Port 80 and 443 are open ond firewall and server, and are accessible from the internet. If i access the requested challenge (http://www.hotzweb.ch/.well-known/acme-challenge/cIiNzgg-UX1qwz0ob7sSlAKCapDbT9PcAwmcj3dqI9A) from outside, I get an answer immediately

My web server is (include version): Not sure, from the log I think it is nginx?

The operating system my web server runs on is (include version): CentOS 7

I can login to a root shell on my machine: YES

MikeMcQ · March 25, 2024, 2:33pm

Welcome @andhotz

Tests from the Let's Encrypt staging system are able to reach you. Have you checked your firewall logs to be sure it is not blocking the IP addresses of the Let's Encrypt production system? The IP's used by staging and production are different.

andhotz · March 25, 2024, 3:47pm

Thanks for testing and your hint, but the log shows no blocking. Such would occur only if your server is in some dubious region of the world anyway. However, I disabled geoblocking completely and still no success. So the question remains: If your test say everything is OK and I can access the requested file from a different provider, why the timeout?

MikeMcQ · March 25, 2024, 4:11pm

The test I showed uses the Let's Encrypt Staging system but your failure shows timeouts from the Let's Encrypt production system. As I noted, these use different IPs

The LE server farms are in various parts of the world. I don't know which you consider dubious (see recent changes here)

Do you block by other criteria than just geo? Have you tried disabling all blocks?

It may also be a comms routing problem between the LE server farm(s) and your destination. These can be exceedingly difficult to debug. And, they generally are closer to your end of the connection. LE issues about 4 million certs per day so problems near their server farm would be painfully obvious to the monitoring systems.

Do you see any evidence of inbound HTTP requests to your system as a result of your cert request? You should be seeing 2-3 requests from a production request and 2-5 from staging tests. As the rollout continues there will be 3-5 from both. You should be seeing at least two requests.

Be careful about retrying too often. There is a rate limit of 5 failures / hour. The error will reflect if you are blocked as a result (for an hour).

andhotz · March 25, 2024, 4:23pm

To be sure nothing blocks, I disabled geoblocking completely, and there are no other general blocking rules in place. The intrusion detection System reports a DROP_CTINVALID message every now and then, but none of these messages is even near the times, when your system tries to contact my server. However I will scan my firewall for stale and/or silent rules possibly blocking in the next few hours.

My maicow tries to get a certificate every 30 Minutes, so I assume this will do no harm.

andhotz · March 25, 2024, 4:25pm

And to be sure it is not the cause of the problem, I just disabled intrusion prevention completely for the next hours.

andhotz · March 25, 2024, 4:55pm

GOOD News: After disabling everything, the newest try was a success, so obviously some rule silently blocked your production server. That leaves me with finding out which it was

Thank you for helping.

system · April 24, 2024, 4:56pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.