GitLab on Ubuntu 20.04 SSL certificate renew

OK, I've reviewed the config and we will need to rerun certbot and then check the logs files:
cat /var/log/gitlab/nginx/gitlab_error.log | grep well-known
cat /var/log/letsencrypt/letsencrypt.log

First run the certbot and than cat /var/log/gitlab/nginx/gitlab_error.log | grep well-known cat /var/letsencrypt/letsencrypt.log ?

Yes.
#1 run certbot
#2 cat error log
#3 cat LE log

Here the result of all:

root@git-01:~# sudo certbot certonly --webroot --webroot-path=/var/www/letsencrypt -d git-01.antares3000.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for git-01.antares3000.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/git-01.antares3000.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/git-01.antares3000.com/privkey.pem
This certificate expires on 2021-10-05.
These files will be updated when the certificate renews.
Certbot has set up a scheduled task to automatically renew this certificate in the background.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
root@git-01:~# cat /var/log/gitlab/nginx/gitlab_error.log | grep well-known
2021/07/07 09:09:32 [error] 2424#0: *107 open() "/var/www/letsencrypt/.well-knowntest-file-1234" failed (2: No such file or directory), client: 64.62.250.100, server: git-01.antares3000.com, request: "HEAD /.well-knowntest-file-1234 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:09:32 [error] 2424#0: *108 open() "/var/www/letsencrypt/.well-knowntest-file-1234" failed (2: No such file or directory), client: 64.62.250.100, server: git-01.antares3000.com, request: "GET /.well-knowntest-file-1234 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:09:33 [error] 2424#0: *109 open() "/var/www/letsencrypt/.well-known/test-file-1234" failed (2: No such file or directory), client: 64.62.250.100, server: git-01.antares3000.com, request: "HEAD /.well-known/test-file-1234 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:09:33 [error] 2424#0: *110 open() "/var/www/letsencrypt/.well-known/test-file-1234" failed (2: No such file or directory), client: 64.62.250.100, server: git-01.antares3000.com, request: "GET /.well-known/test-file-1234 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:10:12 [error] 2424#0: *111 open() "/var/www/letsencrypt/.well-known/test-file-1234" failed (2: No such file or directory), client: 195.32.120.223, server: git-01.antares3000.com, request: "GET /.well-known/test-file-1234 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:13:29 [error] 2424#0: *117 open() "/var/www/letsencrypt/.well-known/test-file-1234" failed (2: No such file or directory), client: 195.32.120.223, server: git-01.antares3000.com, request: "GET /.well-known/test-file-1234 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:34:40 [error] 2424#0: *134 open() "/var/www/letsencrypt/.well-known/test-file-1234" failed (2: No such file or directory), client: 64.62.250.100, server: git-01.antares3000.com, request: "GET /.well-known/test-file-1234 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:34:48 [error] 2424#0: *135 open() "/var/www/letsencrypt/.well-known/test-file-1234" failed (2: No such file or directory), client: 195.32.120.223, server: git-01.antares3000.com, request: "GET /.well-known/test-file-1234 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:35:30 [error] 2424#0: *138 open() "/var/www/letsencrypt/.well-known/test-file-1" failed (2: No such file or directory), client: 64.62.250.100, server: git-01.antares3000.com, request: "GET /.well-known/test-file-1 HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:35:30 [error] 2424#0: *139 open() "/var/www/letsencrypt/.well-known/test-file-" failed (2: No such file or directory), client: 64.62.250.100, server: git-01.antares3000.com, request: "GET /.well-known/test-file- HTTP/1.1", host: "git-01.antares3000.com"
2021/07/07 09:35:32 [error] 2424#0: *140 open() "/var/www/letsencrypt/.well-known/test-file-234" failed (2: No such file or directory), client: 64.62.250.100, server: git-01.antares3000.com, request: "GET /.well-known/test-file-234 HTTP/1.1", host: "git-01.antares3000.com"
root@git-01:~# cat /var/letsencrypt/letsencrypt.log
cat: /var/letsencrypt/letsencrypt.log: No such file or directory

Ok, now i have the SSL certificate, i have to tell gitlab to take that in the new repo

OK I'm not 100% sure but it seems that creating the subdirectories fixed the problem.

Sorry:

should have been:
cat /var/log/letsencrypt/letsencrypt.log

[it's very very late/early here - LOL]

You may now need to restart/reload nginx for it to use the newly created cert.

ok, now it works! :slight_smile:

I have modified the gitlab.rb file to add the new repo of the SSL certificate
restart gitlab and nginx and now works.

Thanks for your help!

[goodnight/goodmoring :smiley:]

Another question... for the next time, what i have to do for renew the certificate?

2 Likes

Check for either a cron job or a systemd timer that may have already been setup during the certbot install with:
crontab -l
OR
crontab -u root -l
and
systemctl list-timers | grep certbot

If neither is found let me know.

Hi,

i have tested the automatic renewal of SSL certificate by certbot with this command:
sudo certbot renew --dry-run

But with the firewall enabled with this configuration:
Cattura

It doesn't work, instead with the firewall disabled it works, how can i resolve this issue?

The HTTP ACME challenge requests are now being redirected to HTTPS.
But your firewall rules only allow HTTPS from two IPs.
You need to handle to the ACME challenge requests in HTTP.
OR
Allow HTTPS from Anywhere.

I don't know who your HTTPS site is supposed to be served to, so I will help you handle the ACME challenge requests in HTTP.

Please show the HTTP server block now in use.

I don't know how to do that..

Here:


nginx: the configuration file /var/opt/gitlab/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /var/opt/gitlab/nginx/conf/nginx.conf test is successful
# configuration file /var/opt/gitlab/nginx/conf/nginx.conf:
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

user gitlab-www gitlab-www;
worker_processes 2;
error_log stderr;
pid nginx.pid;

daemon off;

events {
  worker_connections 10240;
}

http {
  log_format gitlab_access '$remote_addr - $remote_user [$time_local] "$request_method $filtered_request_uri $server_protocol" $status $body_bytes_sent "$filtered_http_referer" "$http_user_agent" $gzip_ratio';
  log_format gitlab_mattermost_access '$remote_addr - $remote_user [$time_local] "$request_method $filtered_request_uri $server_protocol" $status $body_bytes_sent "$filtered_http_referer" "$http_user_agent" $gzip_ratio';

  server_names_hash_bucket_size 64;

  sendfile on;
  tcp_nopush on;
  tcp_nodelay on;

  keepalive_timeout 65;

  gzip on;
  gzip_http_version 1.1;
  gzip_comp_level 2;
  gzip_proxied no-cache no-store private expired auth;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json;

  include /opt/gitlab/embedded/conf/mime.types;

  proxy_cache_path proxy_cache keys_zone=gitlab:10m max_size=1g levels=1:2;
  proxy_cache gitlab;

  map $http_upgrade $connection_upgrade {
      default upgrade;
      ''      close;
  }

  # Remove private_token from the request URI
  # In:  /foo?private_token=unfiltered&authenticity_token=unfiltered&rss_token=unfiltered&...
  # Out: /foo?private_token=[FILTERED]&authenticity_token=unfiltered&rss_token=unfiltered&...
  map $request_uri $temp_request_uri_1 {
    default $request_uri;
    ~(?i)^(?<start>.*)(?<temp>[\?&]private[\-_]token)=[^&]*(?<rest>.*)$ "$start$temp=[FILTERED]$rest";
  }

  # Remove authenticity_token from the request URI
  # In:  /foo?private_token=[FILTERED]&authenticity_token=unfiltered&rss_token=unfiltered&...
  # Out: /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&rss_token=unfiltered&...
  map $temp_request_uri_1 $temp_request_uri_2 {
    default $temp_request_uri_1;
    ~(?i)^(?<start>.*)(?<temp>[\?&]authenticity[\-_]token)=[^&]*(?<rest>.*)$ "$start$temp=[FILTERED]$rest";
  }

  # Remove rss_token from the request URI
  # In:  /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&rss_token=unfiltered&...
  # Out: /foo?private_token=[FILTERED]&authenticity_token=[FILTERED]&rss_token=[FILTERED]&...
  map $temp_request_uri_2 $filtered_request_uri {
    default $temp_request_uri_2;
    ~(?i)^(?<start>.*)(?<temp>[\?&]rss[\-_]token)=[^&]*(?<rest>.*)$ "$start$temp=[FILTERED]$rest";
  }

  # A version of the referer without the query string
  map $http_referer $filtered_http_referer {
    default $http_referer;
    ~^(?<temp>.*)\? $temp;
  }

  # Enable vts status module.
  vhost_traffic_status_zone;

  upstream gitlab-workhorse {
    server unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket;
  }

  include /var/opt/gitlab/nginx/conf/gitlab-http.conf;





  include /var/opt/gitlab/nginx/conf/nginx-status.conf;


}

# configuration file /opt/gitlab/embedded/conf/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /var/opt/gitlab/nginx/conf/gitlab-http.conf:
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

## GitLab
## Modified from https://gitlab.com/gitlab-org/gitlab-foss/blob/master/lib/support/nginx/gitlab-ssl & https://gitlab.com/gitlab-org/gitlab-foss/blob/master/lib/support/nginx/gitlab
##
## Lines starting with two hashes (##) are comments with information.
## Lines starting with one hash (#) are configuration parameters that can be uncommented.
##
##################################
##        CHUNKED TRANSFER      ##
##################################
##
## It is a known issue that Git-over-HTTP requires chunked transfer encoding [0]
## which is not supported by Nginx < 1.3.9 [1]. As a result, pushing a large object
## with Git (i.e. a single large file) can lead to a 411 error. In theory you can get
## around this by tweaking this configuration file and either:
## - installing an old version of Nginx with the chunkin module [2] compiled in, or
## - using a newer version of Nginx.
##
## At the time of writing we do not know if either of these theoretical solutions works.
## As a workaround users can use Git over SSH to push large files.
##
## [0] https://git.kernel.org/cgit/git/git.git/tree/Documentation/technical/http-protocol.txt#n99
## [1] https://github.com/agentzh/chunkin-nginx-module#status
## [2] https://github.com/agentzh/chunkin-nginx-module
##
###################################
##         configuration         ##
###################################

## Redirects all HTTP traffic to the HTTPS host
server {
  listen *:80;

  server_name git-01.antares3000.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice


  location / {
    return 301 https://git-01.antares3000.com:443$request_uri;
  }

  # health checks configuration
  include /var/opt/gitlab/nginx/conf/gitlab-health.conf;

  access_log  /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
  error_log   /var/log/gitlab/nginx/gitlab_error.log;
}

server {
  listen *:443 ssl http2;


  server_name git-01.antares3000.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice

  ## Increase this if you want to upload large attachments
  ## Or if you want to accept large git objects over http
  client_max_body_size 0;

  ## Strong SSL Security
  ## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
  ssl_certificate /etc/letsencrypt/live/git-01.antares3000.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/git-01.antares3000.com/privkey.pem;

  # GitLab needs backwards compatible ciphers to retain compatibility with Java IDEs
  ssl_ciphers 'ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:!aNULL:!eNULL:!EXPORT:!DES:!MD5:!PSK:!RC4';
  ssl_protocols  TLSv1.2 TLSv1.3;
  ssl_prefer_server_ciphers on;
  ssl_session_cache  builtin:1000  shared:SSL:10m;
  ssl_session_timeout  5m;


  ## Real IP Module Config
  ## http://nginx.org/en/docs/http/ngx_http_realip_module.html

  ## HSTS Config
  ## https://www.nginx.com/blog/http-strict-transport-security-hsts-and-nginx/
  add_header Strict-Transport-Security "max-age=31536000";

  # Rails sets a default policy of strict-origin-when-cross-origin, so
  # hide that and just send the one we've configured for nginx
  proxy_hide_header Referrer-Policy;
  add_header Referrer-Policy strict-origin-when-cross-origin;

  ## Individual nginx logs for this GitLab vhost
  access_log  /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
  error_log   /var/log/gitlab/nginx/gitlab_error.log;

  if ($http_host = "") {
    set $http_host_with_default "git-01.antares3000.com";
  }

  if ($http_host != "") {
    set $http_host_with_default $http_host;
  }

  gzip on;
  gzip_static on;
  gzip_comp_level 2;
  gzip_http_version 1.1;
  gzip_vary on;
  gzip_disable "msie6";
  gzip_min_length 250;
  gzip_proxied no-cache no-store private expired auth;
  gzip_types text/plain text/css application/x-javascript text/xml application/xml application/xml+rss text/javascript application/json;

  ## https://github.com/gitlabhq/gitlabhq/issues/694
  ## Some requests take more than 30 seconds.
  proxy_read_timeout      3600;
  proxy_connect_timeout   300;
  proxy_redirect          off;
  proxy_http_version 1.1;

  proxy_set_header Host $http_host_with_default;
  proxy_set_header X-Real-IP $remote_addr;
  proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  proxy_set_header Upgrade $http_upgrade;
  proxy_set_header Connection $connection_upgrade;
  proxy_set_header X-Forwarded-Proto https;
  proxy_set_header X-Forwarded-Ssl on;

  location ~ (/api/v\d/jobs/\d+/artifacts$|\.git/git-receive-pack$|\.git/gitlab-lfs/objects|\.git/info/lfs/objects/batch$) {
    proxy_cache off;
    proxy_pass http://gitlab-workhorse;
    proxy_request_buffering off;
  }

  location /-/grafana/ {
    proxy_pass http://localhost:3000/;
  }


  # health checks configuration
  include /var/opt/gitlab/nginx/conf/gitlab-health.conf;

  location / {
    proxy_cache off;
    proxy_pass  http://gitlab-workhorse;
  }

  location /assets {
    add_header X-Content-Type-Options nosniff;
    proxy_cache gitlab;
    proxy_pass  http://gitlab-workhorse;
  }

  error_page 404 /404.html;
  error_page 500 /500.html;
  error_page 502 /502.html;
  location ~ ^/(404|500|502)(-custom)?\.html$ {
    root /opt/gitlab/embedded/service/gitlab-rails/public;
    internal;
  }

  location ^~ /.well-known { root /var/www/letsencrypt; }
}

# configuration file /var/opt/gitlab/nginx/conf/gitlab-health.conf:
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

location /error.txt {
  # return code here is ignored by the error_page directive
  return 500 'nginx returned $status when communicating with gitlab-workhorse\n';
}

location /error.json  {
  # return code here is ignored by the error_page directive
  return 500 '{"error":"nginx returned $status when communicating with gitlab-workhorse","status":$status}\n';
}

location = /-/health {
  proxy_cache off;
  proxy_pass  http://gitlab-workhorse;
  error_page 404 500 502 /error.txt;
}
location = /-/readiness {
  proxy_cache off;
  proxy_pass  http://gitlab-workhorse;
  error_page 404 500 502 /error.json;
}
location = /-/liveness {
  proxy_cache off;
  proxy_pass  http://gitlab-workhorse;
  error_page 404 500 502 /error.json;
}

# configuration file /var/opt/gitlab/nginx/conf/nginx-status.conf:
server  {
    listen *:8060;
    server_name localhost;
    location /nginx_status {
      stub_status;
      server_tokens off;
      access_log off;
      allow 127.0.0.1;
      deny all;
    }
    location /metrics {
      vhost_traffic_status_display;
      vhost_traffic_status_display_format prometheus;
      server_tokens off;
      access_log off;
      allow 127.0.0.1;
      deny all;
    }

    location /rails-metrics {
      proxy_cache off;
      proxy_pass  http://gitlab-workhorse/-/metrics;
      server_tokens off;
      access_log off;
      allow 127.0.0.1;
      deny all;
    }
}

The configuration shows:

# configuration file /var/opt/gitlab/nginx/conf/gitlab-http.conf:
# This file is managed by gitlab-ctl. Manual changes will be
# erased! To change the contents below, edit /etc/gitlab/gitlab.rb
# and run `sudo gitlab-ctl reconfigure`.

## Redirects all HTTP traffic to the HTTPS host
server {
  listen *:80;

  server_name git-01.antares3000.com;
  server_tokens off; ## Don't show the nginx version number, a security best practice


  location / {
    return 301 https://git-01.antares3000.com:443$request_uri;
  }

  # health checks configuration
  include /var/opt/gitlab/nginx/conf/gitlab-health.conf;

  access_log  /var/log/gitlab/nginx/gitlab_access.log gitlab_access;
  error_log   /var/log/gitlab/nginx/gitlab_error.log;
}

It is redirecting all HTTP to HTTPS.
And states that all changes will be erased/overwritten on reboot.
One should edit the file /etc/gitlab/gitlab.rb instead.
Being unfamiliar with that file and its' content, I'm hesitant in asking you to post it here.
Maybe you can send it to me via PM for review and modification.
Being unfamili

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.