Ubuntu running GitLab and OpenProject failed renew due to Failed Authorization Procedure

Help
1

I’m not really sure where to start or what to do. I hope I’ve provided enough information. Everything was running great until the certificate expired and when I tried to renew, it wouldn’t.

My domain is: gitlab.omniteklabs.com and project.omniteklabs.com
The former is running the latest version of GitLab Omnibus and the latter is running the latest version of OpenProject.

I ran this command: sudo certbot renew

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/gitlab.omniteklabs.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for gitlab.omniteklabs.com
http-01 challenge for project.omniteklabs.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (gitlab.omniteklabs.com) from /etc/letsencrypt/renewal/gitlab.omniteklabs.com.conf produced an unexpected error: Failed authorization procedure. gitlab.omniteklabs.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://gitlab.omniteklabs.com/.well-known/acme-challenge/qUO8nY4rrNtOZJh0eiF8lSLUMrvVecxBBvGPHTp9acs: Connection refused, project.omniteklabs.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://project.omniteklabs.com/.well-known/acme-challenge/uztDz1GH0zDUq2wfjdvihjPzpdH2t2uAjpqXESCy9S8: Connection refused. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gitlab.omniteklabs.com/fullchain.pem (failure)

-------------------------------------------------------------------------------

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gitlab.omniteklabs.com/fullchain.pem (failure)
-------------------------------------------------------------------------------
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: gitlab.omniteklabs.com
   Type:   connection
   Detail: Fetching
   http://gitlab.omniteklabs.com/.well-known/acme-challenge/qUO8nY4rrNtOZJh0eiF8lSLUMrvVecxBBvGPHTp9acs:
   Connection refused

   Domain: project.omniteklabs.com
   Type:   connection
   Detail: Fetching
   http://project.omniteklabs.com/.well-known/acme-challenge/uztDz1GH0zDUq2wfjdvihjPzpdH2t2uAjpqXESCy9S8:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version): Server version: Apache/2.4.18 (Ubuntu), Server built: 2017-09-18T15:09:02

The operating system my web server runs on is (include version):
Linux gitlab 4.13.0-37-generic #42~16.04.1-Ubuntu SMP Wed Mar 7 16:03:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): if the gitlab config file counts.

Here’s a copy of the virtual host file for the gitlab subdomain
https://pastebin.com/SRqy2nNT

Here’s a copy of the virtual host file for the project subdomain
https://pastebin.com/Xi0hzdwK

2

Hi @cadtek91,

You should review your firewall and/or port forwarding conf because I’m also getting a connection refused error trying to reach your sites.

$ curl -IkL http://gitlab.omniteklabs.com
curl: (7) Failed to connect to gitlab.omniteklabs.com port 80: Connection refused

$ curl -IkL http://project.omniteklabs.com
curl: (7) Failed to connect to project.omniteklabs.com port 80: Connection refused

Cheers,
sahsanu

3

Okay, so I looked into and apparently Apache was not starting up like it should. It had something to do with permissions in the sites-enabled folder. Both sites come up now.

So now when I do sudo certbot renew --dry-run it comes back with this

omnitek@gitlab:~$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/gitlab.omniteklabs.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for project.omniteklabs.com
http-01 challenge for gitlab.omniteklabs.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (gitlab.omniteklabs.com) from /etc/letsencrypt/renewal/gitlab.omniteklabs.com.conf produced an unexpected error: Failed authorization procedure. gitlab.omniteklabs.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://gitlab.omniteklabs.com/.well-known/acme-challenge/GEx_VbMqa9zrVH7z3qSF0WHaZkbliFY7HbXC2yUkYtg: "<!DOCTYPE html>
<html class="devise-layout-html">
<head prefix="og: http://ogp.me/ns#">
<meta charset="utf-8">
<meta content="IE". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gitlab.omniteklabs.com/fullchain.pem (failure)

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/gitlab.omniteklabs.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: gitlab.omniteklabs.com
   Type:   unauthorized
   Detail: Invalid response from
   http://gitlab.omniteklabs.com/.well-known/acme-challenge/GEx_VbMqa9zrVH7z3qSF0WHaZkbliFY7HbXC2yUkYtg:
   "<!DOCTYPE html>
   <html class="devise-layout-html">
   <head prefix="og: http://ogp.me/ns#">
   <meta charset="utf-8">
   <meta content="IE"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
4

@cadtek91, check whether the webroot in /etc/letsencrypt/renewal/gitlab.omniteklabs.com.conf is the same as in your web server conf. If it is the same, put a test file in /path/to/your/webroot/.well-known/acme-challenge/ and try to reach it from a public network.

Example:

echo "this is a test" > /path/to/your/webroot/.well-known/acme-challenge/test

And try to reach it using your browser:

http://gitlab.omniteklabs.com/.well-known/acme-challenge/test

or from command line:

curl -ikL http://gitlab.omniteklabs.com/.well-known/acme-challenge/test

Cheers,
sahsanu

5

It was not the same, however I updated the letsencrypt conf to the same DocuemtRoot from the virtualhost conf

It’s now

omnitek@gitlab:~$ cat /etc/letsencrypt/renewal/gitlab.omniteklabs.com.conf
# renew_before_expiry = 30 days
cert = /etc/letsencrypt/live/gitlab.omniteklabs.com/cert.pem
privkey = /etc/letsencrypt/live/gitlab.omniteklabs.com/privkey.pem
chain = /etc/letsencrypt/live/gitlab.omniteklabs.com/chain.pem
fullchain = /etc/letsencrypt/live/gitlab.omniteklabs.com/fullchain.pem
version = 0.17.0
archive_dir = /etc/letsencrypt/archive/gitlab.omniteklabs.com

# Options and defaults used in the renewal process
[renewalparams]
installer = None
authenticator = webroot
account = e968d255a3d5e159a99f8795ef23cd32
[[webroot_map]]
gitlab.omniteklabs.com = /opt/gitlab/embedded/service/gitlab-rails/public
project.omniteklabs.com = /opt/openproject/public

I did a dry run after I changed that, and apparently that worked… figures it was something really small like that.

omnitek@gitlab:~$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/gitlab.omniteklabs.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for project.omniteklabs.com
http-01 challenge for gitlab.omniteklabs.com
Waiting for verification...
Cleaning up challenges

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/gitlab.omniteklabs.com/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/gitlab.omniteklabs.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
1 Like
6

@cadtek91, I’m glad you get it working :+1:

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.