Inability to renew gitlab certificate

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: gitlab.rnl.tecnico.ulisboa.pt and git.rnl.tecnico.ulisboa.pt

I ran this command: gitlab-ctl reconfigure

It produced this output:
here was an error running gitlab-ctl reconfigure:

letsencrypt_certificate[gitlab.rnl.tecnico.ulisboa.pt] (letsencrypt::http_authorization line 6) had an error: RuntimeError: acme_certificate[staging] (letsencrypt::http_authorization line 43) had an error: RuntimeError: ruby_block[create certificate for gitlab.rnl.tecnico.ulisboa.pt] (letsencrypt::http_authorization line 110) had an error: RuntimeError: [gitlab.rnl.tecnico.ulisboa.pt] Validation failed, unable to request certificate, Errors: [{url: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/14814552253/2u-7Ng, status: invalid, error: {"type"=>"urn:ietf:params:acme:error:dns", "detail"=>"DNS problem: server failure at resolver looking up A for git.rnl.tecnico.ulisboa.pt; DNS problem: server failure at resolver looking up AAAA for git.rnl.tecnico.ulisboa.pt", "status"=>400}} ]

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is: in-house

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

1 Like

Welcome @First_Thunder

I can reproduce your problem and these are sometimes difficult to correct. But, I would start by fixing your glue records as shown below

pt to ulisboa.pt: The glue address(es) for ns2.tecnico.ulisboa.pt (2001:690:2100:1::2) differed from its authoritative address(es) (2001:690:2100:1::53:2). See RFC 1034, Sec. 4.2.2.
As reported by: git.rnl.tecnico.ulisboa.pt | DNSViz

I can reproduce DNS query problems using: https://unboundtest.com
This queries in a similar way to Let's Encrypt

Not all queries fail but some do. The LE queries may follow any one of the paths through your DNS tree and sometimes it might get lucky and "miss" the problem branch. Other times not. One example failure (note the SERVFAIL):
https://unboundtest.com/m/AAAA/git.rnl.tecnico.ulisboa.pt/XDSFSKFT

3 Likes

The odd thing is that this worked for literal years without ever being a problem. Maybe some change above me exposed this fragility?

Hard to say. Could be combination of things. But, the wrong glue records should be fixed regardless. And, will likely help.

Could be the DNS provider has just gotten slower. The slow responses cause problems even for routine test queries from my own system.

dig +trace gitlab.rnl.tecnico.ulisboa.pt
...
;; communications error to 2001:690:21c0:a::150#53: timed out
;; communications error to 2001:690:21c0:a::150#53: timed out
;; communications error to 2001:690:21c0:a::150#53: timed out
gitlab.rnl.tecnico.ulisboa.pt. 86400 IN A       193.136.164.27
rnl.tecnico.ulisboa.pt. 86400   IN      NS      ns.rnl.tecnico.ulisboa.pt.
rnl.tecnico.ulisboa.pt. 86400   IN      NS      ns2.tecnico.ulisboa.pt.
rnl.tecnico.ulisboa.pt. 86400   IN      NS      ns2.rnl.tecnico.ulisboa.pt.
rnl.tecnico.ulisboa.pt. 86400   IN      NS      ns1.tecnico.ulisboa.pt.
;; Received 279 bytes from 193.136.128.1#53(ns1.tecnico.ulisboa.pt) in 138 ms

3 Likes

Hmmm. So the issue appears to be at the level of ns.tecnico.ulisboa.pt instead of ns.rnl.tecnico.ulisboa.pt. If it is at that slightly higher level, I've got some people to annoy lol

2 Likes

Refer them to the dnsviz I linked earlier. Points out the performance problems too not just the glue issue :slight_smile:

4 Likes

Also, AAAA records are expected not to be found, as for now at least, we haven't configured ipv6 on our gitlab instance

But LE will look for them anyway because it prefers IPv6. Same with CAA record. You don't need CAA record but LE must look to ensure there isn't one that would prevent it from issuing a cert.

We sometimes see DNS systems that handle "found" records okay but handle "not found" conditions poorly. Not saying this is what is happening here. Just background info.

If your commenting about the dig +trace those IPv6 derive from the DNS setup itself. If those servers don't support IPv6 they shouldn't have AAAA records.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.