Gitlab cert renewal fails

Help
1

Please fill out the fields below so we can help you better.

My domain is: gitlab.foo.net

I ran this command: certbot renew

It produced this output: Attempting to renew cert from /etc/letsencrypt/renewal/gitlab.foo.net.conf produced an unexpected error: Failed authorization procedure. gitlab.foo.net (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://gitlab.foo.net/.well-known/acme-challenge/Xezz1O962IT62NyBIPxdnJLtklWN-ByPMvxi8uzFSpw: Timeout. Skipping.

My web server is (include version): embedded nginx (gitlab-ce 9.3.4-ce.0)

The operating system my web server runs on is (include version): ubuntu 16.04

My hosting provider, if applicable, is: NA

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

This one seems to come up a lot.
We’re running gitlab on a server and it’s got a working letsencrypt cert. Renewal is about 10 days away.
Domain changed to “foo” for privacy/security/whatever.

letsencrypt is configured:
/etc# grep letsencrypt /etc/gitlab/gitlab.rb

set up letsencrypt

nginx[‘custom_gitlab_server_config’] = "location ^~ /.well-known { root /var/www/letsencrypt; }"
nginx[‘ssl_certificate’] = "/etc/letsencrypt/live/gitlab.foo.net/fullchain.pem"
nginx[‘ssl_certificate_key’] = “/etc/letsencrypt/live/gitlab.foo.net/privkey.pem”

Further, nginx is redirecting port 80 to 443 in gitlab.rb:
nginx[‘redirect_http_to_https’] = true
nginx[‘redirect_http_to_https_port’] = 80

When I run the renewal command I get the error above, namely “Fetching http://gitlab.foo.net/.well-known/acme-challenge/Xezz1O962IT62NyBIPxdnJLtklWN-ByPMvxi8uzFSpw: Timeout. Skipping.”

I created a test file in .well-known and using wget -S from the command line I can get using http (aka, it follows the redirect). It seems that certbot is not able to follow the redirect, I’m just not sure why.
I also tried shutting down gitlab and renewing with -standalone, but that also failed for different reasons. I’d rather have this automated so I don’t have to deal with it again, and I’m pulling my hair out figuring out why this isn’t working.

Thanks :slight_smile:

2

@SwartzCr, I think we’ve seen something like four reports in the past two days involving nginx and a “Timeout” failure to connect. I don’t know if they have something in common, but maybe you could look into this? You can probably find them all by searching the forum for nginx timeout “failed authorization procedure”.

3

I looked through those reports myself.
I have a theory - will certbot follow a 80->443 redirect? According to this reply, it will not:

If that's true, then it explains why certbot is returning the error (yet doesn't really solve my problem).
Edit: Double checking, I see that http-01 is in the error I pasted above.
I am using the webroot plugin (originally set up using the link here

I'm wondering if should instead be using the nginx plugin?
I tried specifying --preferred-challenges when running the renew but it will only let me use http-01.

4

Hi @dan-tcell,

The Let’s Encrypt CA will follow an 80 → 443 redirect when validating an HTTP-01 challenge. It won’t follow any redirects when validating a TLS-SNI-01 challenge because the TLS-SNI-01 challenge doesn’t even use the HTTP protocol.

HTTP-01 challenges must always start out on port 80, but can be redirected to another port or even another computer (using HTTP or HTTPS).

TLS-SNI-01 challenge always start out on port 443 and must always succeed or fail immediately, depending on the configuration of the server that they negotiate a TLS session with.

The webroot plugin only supports HTTP-01, while the nginx plugin only supports TLS-SNI-01.

Using the webroot plugin is normally file. We could make some test files to try to figure out why it can’t connect (particularly if you tell us your real domain name).

5

If it truly follows the 80-443 redirect, then it would be good. It doesn't seem like it does.
I've even tried excluding the webroot dir per another thread here:

replaced: location ^~ /.well-known { root /var/www/letsencrypt; }
with: location ^~ /.well-known { root /var/www/letsencrypt; rewrite ^/.well-known/acme-challenge/(.*) https://gitlab.foo.net$1 permanent;}

It made no difference.

With security groups in place you won't be able to reach the real URL anyway, sorry.
Happy to try test files. I've already tried that:

dan$ curl -v http://gitlab.foo.net/.well-known/readme.txt

  • Trying 54.153.34.252...
  • TCP_NODELAY set
  • Connected to gitlab.foo.net (54.153.34.252) port 80 (#0)

GET /.well-known/readme.txt HTTP/1.1
Host: gitlab.foo.net
User-Agent: curl/7.51.0
Accept: /

< HTTP/1.1 301 Moved Permanently
< Server: nginx
< Date: Wed, 05 Jul 2017 20:52:55 GMT
< Content-Type: text/html
< Content-Length: 178
< Connection: keep-alive
< Location: https://gitlab.foo.net:443/.well-known/readme.txt
<

301 Moved Permanently

301 Moved Permanently

nginx * Curl_http_done: called premature == 0 * Connection #0 to host gitlab.foo.net left intact ip-192-168-10-14:~ dan$ wget -Sv http://gitlab.foo.net/.well-known/readme.txt --2017-07-05 13:53:09-- http://gitlab.foo.net/.well-known/readme.txt Resolving gitlab.foo.net... 54.153.34.252 Connecting to gitlab.foo.net|54.153.34.252|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 301 Moved Permanently Server: nginx Date: Wed, 05 Jul 2017 20:53:08 GMT Content-Type: text/html Content-Length: 178 Connection: keep-alive Location: https://gitlab.foo.net:443/.well-known/readme.txt Location: https://gitlab.foo.net:443/.well-known/readme.txt [following] --2017-07-05 13:53:09-- https://gitlab.foo.net/.well-known/readme.txt Connecting to gitlab.foo.net|54.153.34.252|:443... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Server: nginx Date: Wed, 05 Jul 2017 20:53:09 GMT Content-Type: text/plain Content-Length: 4 Last-Modified: Mon, 03 Jul 2017 23:36:13 GMT Connection: keep-alive ETag: "595ad4ed-4" Strict-Transport-Security: max-age=31536000 Accept-Ranges: bytes Length: 4 [text/plain] Saving to: ‘readme.txt’

readme.txt 100%[==========================================================================================>] 4 --.-KB/s in 0s

2017-07-05 13:53:09 (71.0 KB/s) - ‘readme.txt’ saved [4/4]

6

Hmmm, if I wouldn't be able to reach the real URL, how come the certificate authority can reach it? Did you try to whitelist an IP address that you saw the certificate authority validating from or something?

7

Wait a minute - your server is connecting to mine (as the client). If I’m blocking outside traffic, you won’t be able to get there, and that’s why it is timing out.
Edit - just saw your reply come in at the same time. It was still failing then I noticed my redirect was causing an issue so I reverted that change.

Moral of the story: certbot uses INCOMING http to validate the request. You have to allow inbound http (or https if you are redirecting). When testing with curl or wget as I was doing, make sure you test from a host outside your network/VPC/security groups, aka a public host.

Now a different problem - you still do not publish whitelists of IPs I believe, so I need to manually edit my security groups each time I want to renew. Am I wrong about that?

8

That makes sense! The inbound connections are required to prove your control over the domain name.

If you don’t want to permit that, you can also use the DNS-01 challenge type and make changes to your DNS zone to prove control over the domain name. There are a lot of messages in other threads on the forum about how that works.

9

By the way, to @SwartzCr, I learned that the prevalence of the "Timeout" message is because it was introduced in a Boulder update as a new error detail message meant to be more specific than previous errors about failing to get validation data. So, there's no reason to think that all of the "Timeout" errors are actually due to the same cause.

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.