Certbot(1) failing behind a home NAT firewall

Hello Friends:

I’ll first show you my error and then describe what I performed for setup.

When running this command to get a new certificate:

user$ sudo certbot certonly -a manual -d gitlab.k8sbox.io --email jdoe@example.com

I’m getting this error:

Challenge failed for domain gitlab.k8sbox.io
http-01 challenge for gitlab.k8sbox.io
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: gitlab.k8sbox.io
   Type:   connection
   Detail: Fetching
   http://gitlab.k8sbox.io/.well-known/acme-challenge/j3jALHY9pxJTuDhKOfYZI9QUouQop8LlA1ifly6k6bo:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

THE SETUP:

  • The webserver is implemented by GitLab software (not by me), and runs nginx. (Forgive me that I’m not a web administrator). :relaxed:

  • The webserver is in my house behind a Cable-Modem-Router. I have a static WAN IP-Address (i.e. my ISP provider never changes it).

  • DNS A-Records exist for both gitlab.k8sbox.io and www.gitlab.k8sbox.io, both of which point to said ISP static WAN IP-Address. Note: The GitLab web service will likely be down if you click the above. (I shut it down when not in use).

  • Next, within my Cable-Modem-Router, I have ports 80 and 443 port-forwarded to the home webserver (… again, that’s what runs GitLab / nginx). That home webserver, which runs Fedora-30, does not have firewall packages installed or enabled.

  • Following the interactive certbot(1) CLI instructions, I created the following on the home (GitLab / nginx) webserver before pressing enter:

user$ mkdir /var/opt/gitlab/nginx/www/.well-known/acme-challenge/j3jALHY9pxJTuDhKOfYZI9QUouQop8LlA1ifly6k6bo/

user$ echo <long-ciphertext> > /var/opt/gitlab/nginx/www/.well-known/acme-challenge/j3jALHY9pxJTuDhKOfYZI9QUouQop8LlA1ifly6k6bo/index.html

  • And finally, using Firefox, Opera and Chrome, I can successfully browse to http://gitlab.k8sbox.io/.well-known/acme-challenge/j3jALHY9pxJTuDhKOfYZI9QUouQop8LlA1ifly6k6bo/index.html and see the ciphertext content I placed in it (and without logging into GitLab).

Did I miss something? Why could not certbot(1) complete the transaction?

Thank you in advance!

1 Like

This bit isn’t working for me. I can’t open a TCP connection on either 80 or 443 to your current IP.

The error from Let’s Encrypt also reflects that.

Does your ISP block any ports? It seems that there’s some reports that they do: https://duckduckgo.com/?q=rcn+blocked+ports . In that case, it makes sense that you can access your own services from inside your LAN, but external visitors cannot.

1 Like

Hello. I actually disabled port-forwarding after I posted this, so this is why you can’t get to it (sorry). I was, however, able to telnet gitlab.k8sbox.io 80 | 443 before turning it down. I should have mentioned that. For good measure, I’ll try again in a few moments. You can, too in a about a minute.

1 Like

Try again now. I just re-enabled port-forwarding. Sorry. I haven’t locked things down (security wise), so I’m a little paranoid. :relaxed:

That said, maybe because I’m on the RCN network I can somehow get to it. I just tried from an outside cloud server and I can’t get to port 80; but can get to port 443. Can you confirm the same? Thank you!

1 Like

443/tcp opened up, but 80/tcp still times out. That, along with the search results, suggests pretty strongly that your ISP is filtering it.

You can probably still get your certificate, you just need to use the TLS-ALPN challenge, which occurs over port 443 rather than port 80.

At the moment, Certbot does not support it, but acme.sh does (and some others do as well).

You’d do something like:

acme.sh --issue --alpn -d gitlab.k8sbox.io \
--pre-hook "service nginx stop" --post-hook "service nginx restart"

Unfortunately it competes with nginx for port 443, so the above command stops nginx momentarily as the certificate is retrieved. There’s some ways to do this without downtime but they’re a bit complicated.

2 Likes

Hi @nmvega

your port 443 answers via telnet. Your port 80 not. But port 80 is required, if you want to use http validation.

1 Like

Hi @JuergenAuer @_az

First thank you both. You guys are correct about port 80. I should have tried externally to RCN and didn’t think of it.

@_az I’m trying to understand your workaround. Reading it carefully now. :slight_smile:

I think I understand the above.

Qusetion: Do I need to provide the GitLab / nginx document-root to acme.sh; for example /var/opt/gitlab/nginx/www/.well-known/acme-challenge/ or even the entire path created for certbot?

I’ll read the GitHub documentation. :slight_smile:

--alpn starts its own temporary standalone TLS server on port 443, which is why nginx is stopped while the process runs.

As such, acme.sh doesn’t need to know about any document roots.

You can read about how the TLS-ALPN (and other) challenges work here: https://letsencrypt.org/docs/challenge-types/#tls-alpn-01

2 Likes

Only /var/opt/gitlab/nginx/www/

But this only matters for http-01 using webroot, not for tls-alpn.

1 Like

@_az
@JuergenAuer

Guys, thank you so much not only for the super rapidity in replying, but for debugging, for offering solutions and for writing answers that were so clear to follow (which makes a difference). I really appreciate it! :blush:

Using @_az acme.sh solution, I was able to generate certificates manually:

/root/.acme.sh/gitlab.k8sbox.io
/root/.acme.sh/gitlab.k8sbox.io/ca.cer
/root/.acme.sh/gitlab.k8sbox.io/gitlab.k8sbox.io.conf
/root/.acme.sh/gitlab.k8sbox.io/gitlab.k8sbox.io.cer
/root/.acme.sh/gitlab.k8sbox.io/fullchain.cer
/root/.acme.sh/gitlab.k8sbox.io/gitlab.k8sbox.io.csr.conf
/root/.acme.sh/gitlab.k8sbox.io/gitlab.k8sbox.io.key
/root/.acme.sh/gitlab.k8sbox.io/gitlab.k8sbox.io.csr

Now I have to determine what (from the manually generated above) and where to copy into GitLab’s, to get it to work with HTTPS. I suspect it’s here (below), but that’s a just stab-in-the-dark because it looks correct LoL:

user$ find /etc/gitlab/ssl/
/etc/gitlab/ssl/
/etc/gitlab/ssl/gitlab.k8sbox.io.crt        <------- Here and
/etc/gitlab/ssl/gitlab.k8sbox.io.key        <------- Here
/etc/gitlab/ssl/gitlab.k8sbox.io.key-staging

Warm Regards!

2 Likes

@9peppe Thank you as well. I didn’t forget you. Being a new user, the site doesn’t allow me to name more than two users. :slight_smile:

1 Like

Use fullchain.cer.

You can tell acme.sh to perform this installation automatically for you like so:

acme.sh --install-cert -d gitlab.k8sbox.io \
--key-file       /etc/gitlab/ssl/gitlab.k8sbox.io.key  \
--fullchain-file /etc/gitlab/ssl/gitlab.k8sbox.io.crt \
--reloadcmd     "service nginx reload"

It will do that every time the certificate renews.

As long as you used the --[pre,post]-hook that I suggested in the original command, all of this should be totally automatic at every renewal (and assuming you don’t unforward your ports).

(Now that I check more carefully, I’m not sure whether Gitlab’s nginx can be controlled in that way exactly - depends how you installed it. But you probably get the idea).

2 Likes

Hi @nmvega Looks like your port 80 is still timing out and port 443 is refusing to talk.
RIp

1 Like

@_az

Thank you again.

BTW, I don’t mind stopping and starting GitLab when renewing certifications for now.

So cobbling everything together, can I repeatedly run acme.sh with the below options (noticing that I replaced --issue with --renew-all)? Meaning, after the original --issue, can I run the below for both subsequent cases – when the certifications haven’t yet expired as well as when they have? (One command for both cases – simpler). :slight_smile:

Also, might there be a way to specify the notification eMail address (and preferably two of them) on the below command line? From what I’ve read (i.e. it’s manpage), the command wants to consult ACCOUNT_EMAIL= in ~/.acme.sh/account.conf, and even then I believe it only accepts one address; though I’m not sure about that last part.

# NOTE: This is run as root because it needs port 443.
#
DOMAIN='gitlab.k8sbox.io'

/home/user/.acme.sh/acme.sh \
       --renew-all \  # --issue
       --install-cert \
       --alpn \
       --domain ${DOMAIN} \
       --key-file /etc/gitlab/ssl/${DOMAIN}.key  \
       --fullchain-file /etc/gitlab/ssl/${DOMAIN}.crt \
       --webroot /tmp/www/

BTW @_az You caught a bug. I was copying to the destination as .cer not .crt (would have taken me forever to catch).

Thank you again.

You don’t need to make your own script.

acme.sh deals with renewal automatically - it saves your --issue and --install-cert commands for later.

When you first installed acme.sh, it setup an automatic cron job on your server (check crontab -l of your root user).

The other issue is that you cannot combine --issue and --install-cert like that. They must be two separate commands, as they do different things.

So, imagining that you had a totally fresh Gitlab environment, you would:

  1. Install acme.sh
  2. Run the acme.sh --issue ... command with all the correct hooks to stop and start nginx.
  3. Run the acme.sh --install-cert ... command.
  4. Walk away. There’s no need to do anything else, acme.sh’s cronjob will deal with renewal for you (that’s the idea in --pre-hook, --post-hook, --reloadcmd - to be totally non-interactive and automated).
2 Likes

Yes. You can do:

acme.sh --update-account --accountemail me@example.com

Not sure about multiple emails to be honest. Let’s Encrypt does support it, but acme.sh doesn’t seem to.

1 Like

@_az

That’s what I thought (about the multiple eMails).

I get the end-to-end picture now (thank you). I will wipe my ~/.acme.sh/ directory and cron(1) job, and perform a fresh install as root (adding my eMail address along the way); because I initially did this as my own user, but root is needed for port 443. No big deal.

Then I’ll perform the steps you mentioned in sequence and call it a day – letting cron(1) do the rest.

Side Note: The reason I crafted a manual command above is because this server is actually a LXC O/S container nested within a bare-metal server (one of many LXC O/S containers), and won’t always be on. This is part of a larger setup in my personal R&D Lab. So I’ll likely have to manually renew the certificates anyway, but I can do that by simply running the command installed into cron(1) – as you mentioned.

I got so much accomplished today with this thread and your guidance. I really appreciate it. Thank you very much!