This is mostly for anyone that runs into this issue. But if anyone has any tips, please feel free to chime in.
We’re using dehydrated in an automated fashion where new custom domains get a cert issued automatically. We ran into an issue recently where our version of dehydrated was hitting a 301 Moved Permanently when at the Creating fullchain.pem... step.
+ Challenge is valid!
+ Requesting certificate...
+ Checking certificate...
+ Done!
+ Creating fullchain.pem...
+ ERROR: An error occurred while sending get-request to http://cert.int-x3.letsencrypt.org/ (Status 301)
Details:
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
But here we are with a few domains at the rate limit because this sequence would fire repeatedly and cause LE to issue certs but the fullchain.pem would never get created. Now we cannot get the automated sequence to fire and issue a valid cert because of the rate limits. I am off to try to use the issued certs to manually create the fullchain.pem. But I am not hopeful.
If you run into this issue. Please patch or upgrade dehydrated.
If you still have the private keys, you should be able to get the certificates from crt.sh
If you have the certs, you can recreate the fullchain by downloading the intermediate from here and concatenating it to the end of the cert.
If you don’t have the key, it sounds like the rate limit you ran into was probably the duplicate certificates limit, so you might be able to issue a new cert with a slightly different set of domains (eg separating www and non-www into different certificates or combining them into one).
Semi-related (let me know if I should open a separate topic), but does anyone know more about the underlying redirects that caused this problem with dehydrated? Is it new that requests like http://cert.int-x3.letsencrypt.org perform a redirect?
For what it’s worth, that URL doesn’t seem to be redirecting for me now, but I know over the weekend I had been observing the staging environment performing redirects off and on for http://cert.stg-int-x1.letsencrypt.org/ (redirecting to https://letsencrypt.org/cert.stg-int-x1/). However, those redirects seemed somewhat intermittent, sometimes I’d get a 200 OK response, and other times a 301 redirect.
All of this is obviously solved by the upgrade to the dehydated library to follow redirects, I was just curious about the underlying change and whether the introduction of redirects was documented anywhere or if any other libraries besides dehydrated might be affected. Thanks!
curl needs the -L option to follow redirects, but most HTTP client libraries follow redirects automatically so changing this would only affect a few clients.
Additionally, there is no reason for ACME clients to use the http://cert.*.letsencrypt.org/ URLs, as the certificate is also provided from the main API server and referenced in a Link header when the API server returns a certificate. But, dehydrated does things a bit differently than most clients, retrieving the intermediate certificate from the AIA URL embedded in the certificate itself instead of the Link header returned by the CA. Few other clients do this, so dehydrated may indeed be the only one affected.
