Can't get certificates, suddenly

I recently migrated about 50 domains from one system to another. The server fetched about 45 certificates correctly then suddenly stopped working. This was on 30/10 and now a week later nothing has changed. I thought I maybe hit a weekly rate limit, but it doesn’t seem so. I created another setup with the exact same configuration and there certificates gets fetched just fine. It feels like I’m being ip rate limited.

Just recently tried with this new domain which still gives the same result.

I’m running caddy with on demand tls, so if you hit the domain, It’ll try again.

My domain is: testing.signifly.com

It produced this output:

[testing.signifly.com] acme: Obtaining bundled SAN certificate
[testing.signifly.com] acme: Obtaining bundled SAN certificate
[testing.signifly.com] acme: Obtaining bundled SAN certificate
[testing.signifly.com] acme: Obtaining bundled SAN certificate
[testing.signifly.com] acme: Obtaining bundled SAN certificate
[testing.signifly.com] acme: Obtaining bundled SAN certificate

My web server is (include version): caddy 1.0.3

Hi @mxrlkn

your configuration is buggy.

You have redirects http -> https, but your https doesn’t work - https://check-your-website.server-daten.de/?q=testing.signifly.com

Domainname Http-Status redirect Sec. G
http://testing.signifly.com/ 188.166.201.213 301 https://testing.signifly.com/ Html is minified: 103,23 % 0.047 A
https://testing.signifly.com/ 188.166.201.213 -10 7.483 P
SecureChannelFailure - The request was aborted: Could not create SSL/TLS secure channel.
http://testing.signifly.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 188.166.201.213 Inline-JavaScript (∑/total): 0/0 Inline-CSS (∑/total): 0/0 301 https://testing.signifly.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de Html is minified: 101,53 % 0.046 A
Visible Content: Moved Permanently .
https://testing.signifly.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de -10 7.780 P
SecureChannelFailure - The request was aborted: Could not create SSL/TLS secure channel.
Visible Content:

Grade P:

P https://testing.signifly.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
-10
Error creating a TLS-Connection: IANA TLS Alert No. 80, internal_error. An internal error unrelated to the peer or the correctness of the protocol (such as a memory allocation failure) makes it impossible to continue. SSL_ERROR_INTERNAL_ERROR_ALERT (Mozilla) / ERR_SSL_PROTOCOL_ERROR (Chrome)

So it’s impossible that Letsencrypt validates the file (if your caddy uses something like webroot).

Minimal two options:

  • remove the redirect http -> https
  • create a self signed certificate and use that with https, so you have a minimal working https configuration

Is there a better log with the exact error?

Sadly that’s the only logs I got. I’ve made an issue about that.

This is how the logs look when it actually works:

[mydomain.com] acme: Obtaining bundled SAN certificate
[mydomain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxxxx
[mydomain.com] acme: use tls-alpn-01 solver
[mydomain.com] acme: Trying to solve TLS-ALPN-01
[mydomain.com] The server validated our request
[mydomain.com] acme: Validations succeeded; requesting certificates
[mydomain.com] Server responded with a certificate.

So it doesn’t even get to the AuthURL part.

Caddy should remove that http -> https redirect when it requests a certificate automatically. But I’ve removed it now.

I have the same configuration with the http -> https redirect on another server, and there it works fine though.

Creating a self signed certificate is a bit tricky. I have many certificates on this system which works fine, currently live. So it is a working https configuration.

That’s another validation method, tls-alpn-01. Check, if the testing.signifly.com uses tls-alpn or another validation method.

If tls-alpn is used, the redirect isn’t relevant.

Caddy first gets the AuthURL, then it tries TLS-ALPN-01, if that fails, it tries with HTTP-01. But it doesn’t even reach that stage.

If caddy got to that stage, testing.signifly.com would try TLS-ALPN-01 and then HTTP-01.

@JuergenAuer Do you have any idea of what the issue could be? I’m really struggling with this one. How it can work fine and then suddenly stop makes no sense to me.

The solution seems to be in the direction of first NOT redirecting the challenge requests to HTTPS.
How that is done it a bit different/specific to each web server.
I see it’s using:
Server: Caddy
Can you show the config in use?

:80/caddyping {
	proxyprotocol 188.166.0.0/16
	status 200 /
}

:443 {
	proxyprotocol 188.166.0.0/16

	tls {
		ask http://caddy-api/query
	}

	proxy / proxyservice:80 {
		transparent
		websocket
	}
}

This is what I changed it to after removing the http -> https redirect.
This config works on my test setup.

[[ I’m not a Caddy expert ]]
But I found an example online and modified it (to what I think should work):

http:// {
    # keep challenge requests in http
    browse /.well-known/acme-challenge
}

What code was used to redirect?

:80 {
  proxyprotocol x.x.x.x/16

  redir 301 {
    if {path} not /caddyping
    https://{host}{uri}
  }

  status 200 /caddyping
}

:443 {
  proxyprotocol x.x.x.x/16

  tls {
    ask http://caddy-api/query
  }

  proxy / proxyservice:80 {
    transparent
    websocket
  }
}

I’ll try your suggestion.

That needs to also include /.well-known/acme-challenge (in the exclude)
I don’t speak Caddy, so I can’t be certain about the syntax…
But it should be doable.

Perhaps with “and

  if {path} not /caddyping and {path} not /.well-known/acme-challenge

or “not in

  if {path} not in [/caddyping, /.well-known/acme-challenge]

or nested ifs

  if {path} not /caddyping 
    if {path} not /.well-known/acme-challenge
      https://{host}{uri} 

Best of luck.
Let us know which (if any) solved this problem

I’ve finally gotten some logs and I believe It’s a caddy issue or bad config or something. I’ll report back when I figure out the solution.

1 Like

It was a JWS verification error. I don’t know the solution yet, but if anybody has the same problem you can follow along here:


Thanks for all the help and quick responses!

1 Like