I am stunned. I have two domains, w-t.me and wikitranslate.org. Prior to the issue I was creating certs for both domains individually. This worked like charm. Now I came up with the idea to redirect w-t.me to wikitranslate.org. While doing so I changed the virtual host for w-t.me to continue being able to create new certs. This works with
That is why I am stunned: The certificate creation of wikitranslate.org now fails even though I have not changed anything about this domain. Same stuff, same document root, same content, same everything.
This is the error log for certbot renew --dry-run (certbot version 2.6.0)
Processing /etc/letsencrypt/renewal/wikitranslate.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Simulating renewal of an existing certificate for wikitranslate.org and www.wikitranslate.org
Performing the following challenges:
http-01 challenge for wikitranslate.org
http-01 challenge for www.wikitranslate.org
Using the webroot path /var/www/html/wiki for all unmatched domains.
Waiting for verification...
Challenge failed for domain wikitranslate.org
http-01 challenge for wikitranslate.org
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: wikitranslate.org
Type: connection
Detail: 172.104.251.189: Fetching https://wikitranslate.org/.well-known/acme-challenge/0FUQjBn7nbwQgrEt7DK5m4gg51rYT-2Eoby8jx7cJUA: Timeout during connect (likely firewall problem)
Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
Cleaning up challenges
Failed to renew certificate wikitranslate.org with error: Some challenges have failed.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following simulated renewals succeeded:
/etc/letsencrypt/live/w-t.me/fullchain.pem (success)
The following simulated renewals failed:
/etc/letsencrypt/live/wikitranslate.org/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
Once I disable the virtual hosts for w-t.me, the cert creation obviously fails for w-t.me, but it starts working for wikitranslate.org. I'd understand if w-t.me would fail for whatever reason, but why should wikitranslate.org fail?
Honestly, I have no idea why this is failing—any suggestion would be appreciated.
Hmmm. Well, I haven't looked at the timeout yet but your redirects for that domain lose track of the acme-challenge URI. You can see this in the detailed info from Let's Debug test site. Look at the HTTPCheck section at the top. This should be seeing a 404 Not Found but instead gets sent to a home page
Later in that test Let's Debug uses the Let's Encrypt staging servers and also gets the timeout. I would fix the redirect issue first and address the timeout after if it still exists then.
Interestingly, the error from Let's Encrypt shows the IPv4 address yet your IPv6 seems to be fine in my tests (and Let's Debug). LE would use the IPv6 and only fallback to IPv4 for certain problems.
I could not get IPv6 to fail from my own servers even with user-agent looking like Let's Encrypt validation servers. Still, I'd fix the redirects first.
Well, I have a standard ErrorDocument in place for the site: ErrorDocument 404 https://wikitranslate.org/. Not sure why this should be an issue. If I disable the virtual hosts for w-t.me it works event with the ErrorDocument. Anyhow, I just removed the ErrorDocuments, and it still fails.
The error seems clear.
Something is blocking LE from reaching your IP via HTTPS.
I, personally, would NOT redirect the ACME challenge requests from HTTP to HTTPS; As that is simply unnecessary [and can only lead to problems - not help to solve any].
Changing back to the virtual host I posted above works for w-t.me and a certificate is being created.
My issue is why it no longer works for wikitranslate.org. For this domain, nothing changed. I am sure why certbot would even want to touch w-t.me for the cert creation for wikitranslate.org.
I have restarted the Apache server several times after making the changes suggested. Anyhow, the creation of certs is currently broken for both domains. I will now revert to the least broken setup, i.e. working for w-t.me and failing for wikitranslate.org.
VirtualHost configuration:
*:80 is a NameVirtualHost
default server 172.104.251.189 (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost 172.104.251.189 (/etc/apache2/sites-enabled/000-default.conf:1)
alias li1828-189.members.linode.com
port 80 namevhost w-t.me (/etc/apache2/sites-enabled/w-t.me.conf:1)
alias www.w-t.me
port 80 namevhost wikitranslate.org (/etc/apache2/sites-enabled/wikitranslate.org.conf:1)
alias www.wikitranslate.org
*:443 is a NameVirtualHost
default server 172.104.251.189 (/etc/apache2/sites-enabled/000-default.tls.conf:2)
port 443 namevhost 172.104.251.189 (/etc/apache2/sites-enabled/000-default.tls.conf:2)
alias li1828-189.members.linode.com
port 443 namevhost w-t.me (/etc/apache2/sites-enabled/w-t.me.tls.conf:4)
port 443 namevhost www.w-t.me (/etc/apache2/sites-enabled/w-t.me.tls.conf:37)
port 443 namevhost wikitranslate.org (/etc/apache2/sites-enabled/wikitranslate.org.tls.conf:4)
port 443 namevhost www.wikitranslate.org (/etc/apache2/sites-enabled/wikitranslate.org.tls.conf:126)
I am not sure why changing the virtual host for domain A would break the one for domain B, but there is probably a reason. Perhaps someday I will fully grok this. Until then, many thanks to the community members for answering and helping me out of my misery. Much appreciated.
LetsEncrypt caches successful validations (for a domain+account combination) for a period of time. Often times when people are troubleshooting a problem like this, some validations succeed and become cached. This unfortunately has a side effect in which people will think they have fully or partially fixed a problem, but the server is actually still broken and misconfigured.
IMHO, there are two techniques that may be easier in the future:
Configure the server to proxypass the traffic against /.well-known/acme-challenge onto a higher port (like 8081). Then run certbot in standalone mode with --http-01-port=8081. LetsEncrypt will reach out on port 80, your server sends traffic to 8081, and Certbot will handle everything from there. The only troubleshooting you really need to do is to ensure the proxypasses work, which can be done by spinning up a test server on the commandline.
Redirect all traffic against /.well-known/acme-challenge on all sites to a dedicated authorization domain - for example acme.w-t.me, and then ONLY use that domain for Certbot challenges. You can then troubleshoot to ensure both domains handle the redirect and the acme-challenge correctly.
While this may sound like it's overcomplicated, I find that it is actually much easier -- because using these techniques both domains will handle traffic the same way, instead of one domain forwarding traffic and the other domain responding to acme-challenges for both it and other domains. Many errors are often made when trying to get one domain to behave differently based on context.
Thanks a lot for providing some supplemental information on testing. I did not expect --dry-run to render outdated results, but it explains my observed weird behavior. Good do know, though, testing gets hard here, I guess.
Thank you for suggesting circumventing such a situation and hardening the setup. Much appreciated! I'm unsure if I grok fully, but it is definitely worth a shot.
