Issuing Certificate Failed due to timeout, but domain/port are reachable

Blackton · August 27, 2023, 3:30am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: evenstar.me

I ran this command: sudo /usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh [myemail]@hotmail.com

It produced this output:

This script will:

Need a working DNS record pointing to this machine(for hostname )
Install additional dependencies in order to request Let’s Encrypt certificate (acme.sh)
Configure and reload nginx or apache2, whichever is used
Configure the coturn server to use Let's Encrypt certificate and add required deploy hooks
Configure renew of certificate

% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1032 0 1032 0 0 10750 0 --:--:-- --:--:-- --:--:-- 10750
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 216k 100 216k 0 0 2252k 0 --:--:-- --:--:-- --:--:-- 2252k
[Sat 26 Aug 2023 11:05:55 PM EDT] Installing from online archive.
[Sat 26 Aug 2023 11:05:55 PM EDT] Downloading https://github.com/acmesh-official/acme.sh/archive/master.tar.gz
[Sat 26 Aug 2023 11:05:55 PM EDT] Extracting master.tar.gz
[Sat 26 Aug 2023 11:05:55 PM EDT] It is recommended to install socat first.
[Sat 26 Aug 2023 11:05:55 PM EDT] We use socat for standalone server if you use standalone mode.
[Sat 26 Aug 2023 11:05:55 PM EDT] If you don't use standalone mode, just ignore this warning.
[Sat 26 Aug 2023 11:05:55 PM EDT] Installing to /opt/acmesh/.acme.sh
[Sat 26 Aug 2023 11:05:55 PM EDT] Installed to /opt/acmesh/.acme.sh/acme.sh
[Sat 26 Aug 2023 11:05:55 PM EDT] No profile is found, you will need to go into /opt/acmesh/.acme.sh to use acme.sh
[Sat 26 Aug 2023 11:05:55 PM EDT] Installing cron job
17 0 * * * "/opt/acmesh/.acme.sh"/acme.sh --cron --home "/opt/acmesh/.acme.sh" > /dev/null
[Sat 26 Aug 2023 11:05:55 PM EDT] Good, bash is found, so change the shebang to use bash as preferred.
[Sat 26 Aug 2023 11:05:57 PM EDT] OK
[Sat 26 Aug 2023 11:05:57 PM EDT] Install success!
[Sat 26 Aug 2023 11:05:58 PM EDT] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Sat 26 Aug 2023 11:05:58 PM EDT] Single domain='evenstar.me'
[Sat 26 Aug 2023 11:05:58 PM EDT] Getting domain auth token for each domain
[Sat 26 Aug 2023 11:05:59 PM EDT] Getting webroot for domain='evenstar.me'
[Sat 26 Aug 2023 11:05:59 PM EDT] Verifying: evenstar.me
[Sat 26 Aug 2023 11:05:59 PM EDT] Pending, The CA is processing your order, please just wait. (1/30)
[Sat 26 Aug 2023 11:06:03 PM EDT] Pending, The CA is processing your order, please just wait. (2/30)
[Sat 26 Aug 2023 11:06:06 PM EDT] Pending, The CA is processing your order, please just wait. (3/30)
[Sat 26 Aug 2023 11:06:10 PM EDT] Pending, The CA is processing your order, please just wait. (4/30)
[Sat 26 Aug 2023 11:06:13 PM EDT] evenstar.me:Verify error:18.221.195.49: Fetching https://blog-dev.domain.com/blog/.well-known/acme-challenge/vb4yX78MYwMN68-YOiBPXZ36dXI5iEsAZhokVUnwqKA: Timeout during connect (likely firewall problem)
[Sat 26 Aug 2023 11:06:13 PM EDT] Please add '--debug' or '--log' to check more details.
[Sat 26 Aug 2023 11:06:13 PM EDT] See: How to debug acme.sh · acmesh-official/acme.sh Wiki · GitHub
Issuing the certificate from Let's Encrypt failed, continuing ...

My web server is (include version): n/a

The operating system my web server runs on is (include version): Linux Mint Mate 20.3

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Command 'certbot' not found

Comments: per the Jitsi installation instructions on github (Self-Hosting Guide - Debian/Ubuntu server | Jitsi Meet), I had previously run the following commands:

sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw allow 10000/udp
sudo ufw allow 22/tcp
sudo ufw allow 3478/udp
sudo ufw allow 5349/tcp
sudo ufw enable

After the failed certification issuance, I verified that these ports were open in terminal by running "telnet evenstar.me 80" (and also for ports 443, 10000, etc.).

I also tried pinging evenstar.me successfully.

I've also tried the command "sudo ufw allow 'Nginx Full'" but that made no difference.

I also went into my router and disabled the firewall, but this didn't make a difference either.

I've been trying to get this simple certificate issuance to work for almost four hours now, and I've read and tried everything I can find, but nothing seems to make any difference. It seems like there are no problems with the DNS, port forwarding, firewall, etc., but why can't Let's Encrypt get through to me?

rg305 · August 27, 2023, 5:12am

Hi @Blackton, and welcome to the LE community forum

The error message is pretty clear:
[LE can't reach your IP on port 80]

That said, I do understand why you would think that it should.
And I can reach it from my corner of the Internet too.

I think you are missing something in the other firewall.
I see ufw has been set correctly.
But AWS also has a firewall feature.
You need to check that you are not blocking certain networks.
Perhaps you are blocking all "internal" networks [172.16.0.0/12]
But the entry has been incorrectly entered as something much bigger.

Blackton · August 27, 2023, 5:34am

OK, thank you for responding. I am not using AWS or anything else like that, BTW, so I assume that should not apply here? I had my firewall turned off completely actually, and it still didn't work. And like you said, you could get to it yourself. Where would I check to see if I am blocking "internal" networks?

9peppe · August 27, 2023, 9:16am

Are you hosting at home in IPv4?

Did you configure your port forwarding in your router interface?

MikeMcQ · August 27, 2023, 10:07am

Is the redirect to blog-dev.domain.com intentional?

rg305 · August 27, 2023, 12:06pm

Seems like it is on an AWS system:

Name:     evenstar.me
Address:  18.221.195.49

Name:     blog-dev.evenstar.me
Address:  18.221.195.49

Name:     ec2-18-221-195-49.us-east-2.compute.amazonaws.com
Address:  18.221.195.49

MikeMcQ · August 27, 2023, 12:28pm

Also, are you using some sort of URL Redirect service at NameCheap?

If so, that won't work with HTTPS. You need to use regular DNS and setup an A record to point to your public IP. And/or an AAAA record for IPv6

rg305 · August 27, 2023, 12:39pm

Your [AWS] site is using a self-signed cert for *.domain.com.
It also forwards all connections to:
https://blog-dev.domain.com/blog/

That is NOT something that can work [over the Internet].
You don't own the domain "domain.com".
[it will never point to the IP of your (AWS) system]

You need to edit that web server config and replace "domain.com" with your actual domain name.
Then, you might be able to get a cert for it.

rg305 · August 27, 2023, 12:45pm

For anyone testing, the responses seem to take the user agent into consideration:

default curl agent returns 301 to domain.com:

curl -Iik https://evenstar.me/.well-known/acme-challenge/Test_File-1234                                       HTTP/2 301
server: nginx
date: Sun, 27 Aug 2023 12:42:17 GMT
content-type: text/html
content-length: 162
location: https://blog-dev.domain.com/blog/.well-known/acme-challenge/Test_File-1234

LE default user agent returns 406 error:

curl -Iik https://evenstar.me/.well-known/acme-challenge/Test_File-1234 -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
HTTP/2 406
server: nginx
date: Sun, 27 Aug 2023 12:42:27 GMT
content-type: text/html
content-length: 156

Blackton · August 27, 2023, 7:31pm

Yes, I am hosting at home in IPv4. And yes, I configured port forwarding. I think I mentioned that I can ping evenstar.me from terminal and it returns my public IP, although now I do see that aws stuff there too when I ping, so not sure what that's about. I guess I figured that was something to do with how it was routed through my ISP or something? I'm using Verizon.

Blackton · August 27, 2023, 7:41pm

Hmm, it seems that when I reply to a specific response, it just sticks my reply at the very bottom or all the responses, so I guess I just give one combined reply at this point.

Regarding the redirect to `blog-dev.domain.com', I had been assuming that that was part of the verification process that Let's Encrypt does for the certificate. I was not personally intending any redirect though, so if that's not coming from Let's Encrypt, I'm not sure what's going on.

As I just mentioned above, I am not using AWS, so I'm not sure where that AWS stuff is coming from. My DNS (NameCheap) seems to be properly routing to my public IP as far as I can tell. So I was assuming that was something to do with how it was getting routed by the ISP or something, although I guess that doesn't really make sense that the ISP would need to going through AWS. I'm very confused about this myself actually.

I am going to paste in the contents of that script though, as maybe there is some useful information in there.

Blackton · August 27, 2023, 7:42pm

install-letsencrypt-cert.sh contents:

#!/bin/bash

set -e

echo "-------------------------------------------------------------------------"
echo "This script will:"
echo "- Need a working DNS record pointing to this machine(for hostname ${DOMAIN})"
echo "- Install additional dependencies in order to request Let’s Encrypt certificate (acme.sh)"
echo "- Configure and reload nginx or apache2, whichever is used"
echo "- Configure the coturn server to use Let's Encrypt certificate and add required deploy hooks"
echo "- Configure renew of certificate"
echo ""

EMAIL=$1

if [ -z "$EMAIL" ]; then
  echo "You need to agree to the ACME server's Subscriber Agreement (https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf) "
  echo "by providing an email address for important account notifications"

  echo -n "Enter your email and press [ENTER]: "
  read EMAIL
fi

DOMAIN=$2
if [ -z "$DOMAIN" ]; then
  DEB_CONF_RESULT=$(debconf-show jitsi-meet-web-config | grep jitsi-meet/jvb-hostname)
  DOMAIN="${DEB_CONF_RESULT##*:}"
fi
# remove whitespace
DOMAIN="$(echo -e "${DOMAIN}" | tr -d '[:space:]')"


export HOME=/opt/acmesh
curl https://get.acme.sh | sh -s email=$EMAIL

# Checks whether nginx or apache is installed
NGINX_INSTALL_CHECK="$(dpkg-query -f '${Status}' -W 'nginx' 2>/dev/null | awk '{print $3}' || true)"
NGINX_FULL_INSTALL_CHECK="$(dpkg-query -f '${Status}' -W 'nginx-full' 2>/dev/null | awk '{print $3}' || true)"
NGINX_EXTRAS_INSTALL_CHECK="$(dpkg-query -f '${Status}' -W 'nginx-extras' 2>/dev/null | awk '{print $3}' || true)"
OPENRESTY_INSTALL_CHECK="$(dpkg-query -f '${Status}' -W 'openresty' 2>/dev/null | awk '{print $3}' || true)"
APACHE_INSTALL_CHECK="$(dpkg-query -f '${Status}' -W 'apache2' 2>/dev/null | awk '{print $3}' || true)"

RELOAD_CMD=""
if [ "$NGINX_INSTALL_CHECK" = "installed" ] || [ "$NGINX_INSTALL_CHECK" = "unpacked" ] \
   || [ "$NGINX_FULL_INSTALL_CHECK" = "installed" ] || [ "$NGINX_FULL_INSTALL_CHECK" = "unpacked" ] \
   || [ "$NGINX_EXTRAS_INSTALL_CHECK" = "installed" ] || [ "$NGINX_EXTRAS_INSTALL_CHECK" = "unpacked" ]; then
    RELOAD_CMD="systemctl force-reload nginx.service"
elif [ "$OPENRESTY_INSTALL_CHECK" = "installed" ] || [ "$OPENRESTY_INSTALL_CHECK" = "unpacked" ] ; then
    RELOAD_CMD="systemctl force-reload openresty.service"
elif [ "$APACHE_INSTALL_CHECK" = "installed" ] || [ "$APACHE_INSTALL_CHECK" = "unpacked" ] ; then
    RELOAD_CMD="systemctl force-reload apache2.service"
else
    RELOAD_CMD="echo 'No webserver found'"
fi

RELOAD_CMD+=" && /usr/share/jitsi-meet/scripts/coturn-le-update.sh ${DOMAIN}"

ISSUE_FAILED_CODE=0
ISSUE_CERT_CMD="/opt/acmesh/.acme.sh/acme.sh -f --issue -d ${DOMAIN} -w /usr/share/jitsi-meet --server letsencrypt"
eval "${ISSUE_CERT_CMD}" || ISSUE_FAILED_CODE=$?

INSTALL_CERT_CMD="/opt/acmesh/.acme.sh/acme.sh -f --install-cert -d ${DOMAIN} --key-file /etc/jitsi/meet/${DOMAIN}.key --fullchain-file /etc/jitsi/meet/${DOMAIN}.crt --reloadcmd \"${RELOAD_CMD}\""
if [ ${ISSUE_FAILED_CODE} -ne 0 ] ; then
    # it maybe this certificate already exists (code 2 - skip, no need to renew)
    if [ ${ISSUE_FAILED_CODE} -eq 2 ]; then
        eval "$INSTALL_CERT_CMD"
    else
        echo "Issuing the certificate from Let's Encrypt failed, continuing ..."
        echo "You can retry later by executing:"
        echo "/usr/share/jitsi-meet/scripts/install-letsencrypt-cert.sh $EMAIL"
    fi
else
    eval "$INSTALL_CERT_CMD"
fi

MikeMcQ · August 27, 2023, 7:56pm

Why do you say that? Is your public IP 18.221.195.49 ?

That redirect to blog-dev is not Let's Encrypt that is what that IP is doing.

My guess is that you have a URL Redirect enabled in your NameCheap DNS config and this ends up pointing to this 18... IP on AWS to provide the redirection. Instead, you need to set an A record to your public IP as I noted earlier. Check your NameCheap again and let us know what you see.

Blackton · August 27, 2023, 8:05pm

Hmm, I am looking inside this acme.sh script that is called by that install-letsencrypt-cert.sh script that I just posted, but the acme.sh contents are kind of a lot to post here. I have been trying to look at some of the environment variables inside of this though. I checked "$ACME_DIR" and it seems to not even be defined, so I'm wondering if that could be part of the issue here, that this thing is relying on scripts that rely on environment variables that were never set or not set properly? If so, I have no idea about any of this, or even how those scripts got onto my system. I assumed it had do with the Let's Encrypt install putting them there...

Blackton · August 27, 2023, 8:06pm

I'm really confused about what's going on right now, so if anyone has some direction for what I should check/try rather than me just blindly looking in script files and guessing about environment variables, I would appreciate it!

rg305 · August 27, 2023, 8:10pm

Please begin at the beginning.

First: You need a working HTTP site.

What is the public IP of your site?
Try showing us the output of:
curl ifconfig.io

Blackton · August 27, 2023, 9:48pm

Yes, thank you!

curl ifconfig.io shows this IP address: 96.245.194.124

Blackton · August 27, 2023, 9:52pm

Is that supposed to be showing me my public IP? It seems to be different from my public IP, but I just checked and it seems that my public IP has also changed from when I checked it last night, so I just updated that within my DNS panel.

When I check my public IP, it's now showing as 13.57.130.190

Note that I did not realize that (non-static) IP addresses would change so quickly. I thought they rarely change or often don't change at all, but just that they could change.

It seems that even with the new IP address set in the DNS that this is still failing to get the certificate. So I don't think the issue was that the public IP had changed right between the time I checked it and when I tried to get the certificate.

On that note though, is my certificate going to become invalid every time my IP address gets changed by my ISP?

Bruce5051 · August 27, 2023, 10:11pm

That is the Public IP Address from where the curl ifconfig.io was issued from.

But evenstar.me Public Address is presently 18.221.195.49 as show here https://dnsspy.io/scan/evenstar.me

Bruce5051 · August 27, 2023, 10:23pm

For https://pool-96-245-194-124.phlapa.fios.verizon.net/ I see this, which looks like Jitsi

And the certificate being served for https://pool-96-245-194-124.phlapa.fios.verizon.net/ is a self-signed certificate for the domain name evenstar.me

$ openssl s_client -showcerts -servername pool-96-245-194-124.phlapa.fios.verizon.net -connect pool-96-245-194-124.phlapa.fios.verizon.net:443 < /dev/null
CONNECTED(00000003)
depth=0 O = localdomain, OU = david-MacBookPro, CN = evenstar.me, emailAddress = webmaster@david-MacBookPro.localdomain
verify error:num=18:self-signed certificate
verify return:1
depth=0 O = localdomain, OU = david-MacBookPro, CN = evenstar.me, emailAddress = webmaster@david-MacBookPro.localdomain
verify return:1
---
Certificate chain
 0 s:O = localdomain, OU = david-MacBookPro, CN = evenstar.me, emailAddress = webmaster@david-MacBookPro.localdomain
   i:O = localdomain, OU = david-MacBookPro, CN = evenstar.me, emailAddress = webmaster@david-MacBookPro.localdomain
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 27 02:17:19 2023 GMT; NotAfter: Aug 24 02:17:19 2033 GMT
-----BEGIN CERTIFICATE-----
MIIFrzCCA5egAwIBAgIUB8rOZGwB5rWneIGQA3Ik0mhQXQUwDQYJKoZIhvcNAQEL
BQAwfjEUMBIGA1UECgwLbG9jYWxkb21haW4xGTAXBgNVBAsMEGRhdmlkLU1hY0Jv
b2tQcm8xFDASBgNVBAMMC2V2ZW5zdGFyLm1lMTUwMwYJKoZIhvcNAQkBFiZ3ZWJt
YXN0ZXJAZGF2aWQtTWFjQm9va1Byby5sb2NhbGRvbWFpbjAeFw0yMzA4MjcwMjE3
MTlaFw0zMzA4MjQwMjE3MTlaMH4xFDASBgNVBAoMC2xvY2FsZG9tYWluMRkwFwYD
VQQLDBBkYXZpZC1NYWNCb29rUHJvMRQwEgYDVQQDDAtldmVuc3Rhci5tZTE1MDMG
CSqGSIb3DQEJARYmd2VibWFzdGVyQGRhdmlkLU1hY0Jvb2tQcm8ubG9jYWxkb21h
aW4wggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQDxMsARO1TX9ZC4Xntu
rzJ9nh8AIcM98Rew7RyrDNKE8vPlQFSeQ6UPq4jCaO8JjcXljgeIgI0hROBMfFG7
D7NkttWytzBuX/zdfVZlFG9Aim3UA6hZoEa/XbbAc9l3YRgUrowrt/1y5eN0Yyi0
aLQbzLgBoWIljP0yLLWeRaiWk5NgF/gNaXJs5y5b+VoFseU26kDSKEGZKEDzZwxj
nFAG3qtnKjib64+h20BOrTONMR5XHFTYTBf+J1Ru5iPBYEOoUt5USmW2O5FyaanH
3sFpuRNH4zGfpKiQ1mq0uClSzfy9cxP8G4epT08VCZ1CAhsK/zS0FukmZic8d+iN
+WJ4bDDgLOCY++AQQRVn9DnzUO6qgCIUdH+FLpMf3p0gcQHWROg9m1QyemeoyD6T
XRC6gdOVGgI9bO/l9qcDmSzhJSSAeKwIc6ek54zTTtVQvYcTaie9vjaAEf4PyVKH
hK1jqKSGZazNa+LIl0LxF+a0fmRXKzb/pPv/KDZA2lwGUHS6FQ1+9m+24ZLRtIso
JjnsqznmWtdsKlbD2NQL2Wi6pueRK6OeyCodo0pnDdYT0IgpQS5b79iaxh7Q34pN
xy4EP1U4lB3i/xVFn6vaGa+jqRFZvS/5SQV2NvofleD1isoo/bNDvhyIzlDLEXNN
5q1DK1ElnVxbqP9UQESG6P7J6QIDAQABoyUwIzAhBgNVHREEGjAYgglsb2NhbGhv
c3SCC2V2ZW5zdGFyLm1lMA0GCSqGSIb3DQEBCwUAA4ICAQDKCI54V3rSqt/kkFVC
lSQPXboCio2tkSyHAdcGi3ntMXS4fP7AZtY5+xuMg8vKAc1dCZ9xqeHqXcWK+bAe
0vh0H1IxM+WuxWmki8eGHIPIjM67eJdMcxQssf/phW8dhghPK/W7ATaFnMLKzhnP
MfFia5ifDZstStsauRldzOMCckgLLkOVf7dNOAqkzLLX+THrBbgtlQPj7/j5nHNr
QJVFyGun6r/VdSQufsYsYVs8hMlfzBsWA4janXYqPzqYmYHLvqhxlA6SApRXC4mC
EY8z0Z32OznSYbeSXUsxyrrT8gUN1qfit8muXL8Se7GTBIhUJrit+debSAPBeEQ1
GKt51nJ7GYbGBFec7X7XDw3DnGW1jespsVzXGIWeGovZNIvEXVu7Vco+Ap27PxwQ
XH+f3txQuGjKm4Ok4ykC3CCvVuYyNI+tMa9YP9AWC6rq0Y6Lwndv3CoVSyZ7inFQ
ySJ0atIjHIG7cvHXBF42Bb560hF2sFs0mB0fSesetZIGZfrGcLeDrqO+Zn+G92GM
fexeBT+svDrMVaCrgBH9HRX/6eqYcgoePDDQzsxJtKQ+PqvZDwB+ihSpv7/DfkWA
695n218UJnVXmO5P1YyvL28d4yeXQnAW2dCmSgZTajb+kAUvLfodWeIlj3LLr2py
AnDC4tZ0f9itSu1tYd79U4ILOw==
-----END CERTIFICATE-----
---
Server certificate
subject=O = localdomain, OU = david-MacBookPro, CN = evenstar.me, emailAddress = webmaster@david-MacBookPro.localdomain
issuer=O = localdomain, OU = david-MacBookPro, CN = evenstar.me, emailAddress = webmaster@david-MacBookPro.localdomain
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 2275 bytes and written 425 bytes
Verification error: self-signed certificate
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 18 (self-signed certificate)
---
DONE