Could not get a certificate using Certbоt, asks to install the root certificate of the Unified Internet Access Gateway on ubuntu lts 20.04

My domain is: moodle.kipk.edu.kz

I ran this command: certbot --nginx --server https://acme-staging-v02.api.letsencrypt.org/directory -d moodle.kipk.edu.kz -d www.moodle.kipk.edu.kz

It produced this output: Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
cancel): kipk@mail.kz
An unexpected error occurred:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 485, in wrap_socket
cnx.do_handshake()
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1915, in do_handshake
self._raise_ssl_error(self._ssl, result)
File "/usr/lib/python3/dist-packages/OpenSSL/SSL.py", line 1647, in _raise_ssl_error
_raise_current_error()
File "/usr/lib/python3/dist-packages/OpenSSL/_util.py", line 54, in exception_from_error_queue
raise exception_type(errors)
OpenSSL.SSL.Error: [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 666, in urlopen
httplib_response = self._make_request(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 377, in _make_request
self._validate_conn(conn)
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 1001, in validate_conn
conn.connect()
File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 366, in connect
self.sock = ssl_wrap_socket(
File "/usr/lib/python3/dist-packages/urllib3/util/ssl
.py", line 370, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3/dist-packages/urllib3/contrib/pyopenssl.py", line 491, in wrap_socket
raise ssl.SSLError("bad handshake: %r" % e)
ssl.SSLError: ("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])",)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 720, in urlopen
retries = retries.increment(
File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 436, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))

During handling of the above exception, another exception occurred:

requests.exceptions.SSLError: HTTPSConnectionPool(host='acme-staging-v02.api.letsencrypt.org', port=443): Max retries exceeded with url: /directory (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')])")))
Please see the logfiles in /var/log/letsencrypt for more details.

My web server is (include version): nginx

The operating system my web server runs on is (include version): Ubuntu 20.04.6 LTS

My hosting provider, if applicable, is: nginx version: nginx/1.18.0 (Ubuntu)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.40.0

That is unusual for your setup. You were consistently getting Let's Encrypt certs all last year and as recently as Jan20 this year. What changed in your system?

I also see you just got a cert from ZeroSSL. Did you switch to using acme.sh client and used its default CA setting?

Do you still want help getting a Let's Encrypt cert?

If you still want to try Certbot I would upgrade your very old version. Ubuntu easily supports using the snap install for Certbot. Follow the instructions carefully below. Or, if you switched to acme.sh you can change the default CA to Let's Encrypt or just use --server letsencrypt on the issue command.

2 Likes

The provider changed, and our white IP address, and after that cerbot stopped working, you can check if your IP address is blocked on 82.200.198.106, so using cerbot is very convenient for me, I would like to continue working with it, and switching to ZeroSSL is a temporary measure

As you can see by the error, it's not a blocking thing: your server apparently receives the servers certificate fine. However, it claims it cannot verify this certificate for some reason. There might be multiple issues:

  • Your server somehow suddenly doesn't recognise the ISRG Root X1 root certificate any longer;
  • Your server somehow tries to connect to an incorrect IP address for the hostname acme-staging-v02.api.letsencrypt.org which sends a non-trusted certificate (either e.g. self-signed or for a different host altogether).

What does the following command give as output?:

openssl s_client -connect acme-staging-v02.api.letsencrypt.org:443

1 Like

administrator@moodle:~$ sudo openssl s_client -connect acme-staging-v02.api.letsencrypt.org:443

CONNECTED(00000003)
depth=2 C = KZ, ST = Nur-Sultan, O = State Technical Service, OU = HQ, CN = Unified State Internet Access Gateway, emailAddress = support@sts.kz
verify return:1
depth=1 C = KZ, ST = Astana, L = Astana, O = State Technical Service, OU = HQ, CN = USIAG Intermediate January, emailAddress = support@sts.kz
verify return:1
depth=0 CN = acme-staging-v02.api.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:CN = acme-staging-v02.api.letsencrypt.org
   i:C = KZ, ST = Astana, L = Astana, O = State Technical Service, OU = HQ, CN = USIAG Intermediate January, emailAddress = support@sts.kz
 1 s:C = KZ, ST = Astana, L = Astana, O = State Technical Service, OU = HQ, CN = USIAG Intermediate January, emailAddress = support@sts.kz
   i:C = KZ, ST = Nur-Sultan, O = State Technical Service, OU = HQ, CN = Unified State Internet Access Gateway, emailAddress = support@sts.kz
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFOTCCBCGgAwIBAgIIEY8IgR0TqYAwDQYJKoZIhvcNAQELBQAwgaIxCzAJBgNV
BAYTAktaMQ8wDQYDVQQIEwZBc3RhbmExDzANBgNVBAcTBkFzdGFuYTEgMB4GA1UE
ChMXU3RhdGUgVGVjaG5pY2FsIFNlcnZpY2UxCzAJBgNVBAsTAkhRMSMwIQYDVQQD
ExpVU0lBRyBJbnRlcm1lZGlhdGUgSmFudWFyeTEdMBsGCSqGSIb3DQEJARYOc3Vw
cG9ydEBzdHMua3owHhcNMjMxMjI2MTA0ODA5WhcNMjQwMzI1MTA0ODA4WjAvMS0w
KwYDVQQDEyRhY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcwggEi
MA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDfxsNLXWmyMt5zHqN6N4Foa7Jq
vviYCERMl0vlLtIt4QGXyfMVDAVfOsbisaN5PZiX6Hx4t2YdSo8NEdN1vVphQqqc
d/8qNFeLbJm/l7ZE9VclUOpzZvEfs4vP/JnZaObALrVdhCaf6JyuKG1qFuPske9h
jo1nzZqc2iV+Ddwu0nhoF0QI62Nu3gSmovdl4loTxUS9LDo2j1WkV0CL2URq68sx
lxZUh4CCj2adZ6KkqdKStX2sBpYLHIw7QDBDR/VMamFxBptsvHeVbMJwuNsYmLbc
RZbMWeNZiW6SJwbGIDspEPviAsps150Yh/JIMBtcAuUnuvz5vFbgwlc9UP1HAgMB
AAGjggHjMIIB3zAOBgNVHQ8BAf8EBAMCBaAwDAYDVR0TAQH/BAIwADBRBgNVHREE
SjBIgiRhY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmeCIGluY2lk
ZW50LXN0YWdpbmcubGV0c2VuY3J5cHQub3JnMIIBagYJYIZIAYb4QgENBIIBWwpN
TU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU1NTU0K
TU1NTU1NTU1NTU1NTU1NTU1NTU1NV1gwMFhXTU1NTU1NTU1NTU1NTU1NTU1NTU1N
Ck1NTU1NTU1NTU1NTU1NTU1NV0trbzouICAuO29rS1dNTU1NTU1NTU1NTU1NTU1N
TQpNTU1NTU1NTU1NTmtkb2xjOycuLi4sbG9vYywuIC4nO2Nsb2RrTk1NTU1NTU1N
TU0KTU1NTU1NTU1NTU8uIC4sOzpsZE9YV01NTU1XS2tkbDosJy4gLk9NTU1NTU1N
TU1NCk1NTU1NTU1NTU1PLiBsV01NTU1NTU1NTU1NTU1NTU1NTVdjIC5PTU1NTU1N
TU1NTQpNTU1NTU1NTU1NTy4gbE1NTVdOWE5OWFhYWE5OWE5XTU1NbCAuT01NTU1N
TU1NTU0KTU1NMA0GCSqGSIb3DQEBCwUAA4IBAQCTvnYZCVCaJosuf/bNTygeghhG
jw95QQDLSGNpEpqhVSuomp9uUb4PIvcpy5nBG1ZL/eSoCmUzH5Jsvh3qjq/m9/xt
YglCiLPRLE3EQ3k84T1/NN6l45vhRpsuL0b3iEY+vcaiMSCYvib/OvaMlBneCWKL
VxuSku3eu5EIRoT6WGK2Lo22CEyNcWHpftofdxb2zIFMR2u7/qjUxgPCCHxCwnYz
WF+If7vtqaL1RI3sinz94CF3RNKTh6xP+PrBIsm9spT8xCwHLKqV7r1CD6q0M2c8
OlPclql9+w3gcgW0Q/k1HUDEw1NjzZtHzYLEiQLYo0a4bdXVxti9UpS2ICnb
-----END CERTIFICATE-----
subject=CN = acme-staging-v02.api.letsencrypt.org

issuer=C = KZ, ST = Astana, L = Astana, O = State Technical Service, OU = HQ, CN = USIAG Intermediate January, emailAddress = support@sts.kz

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 3210 bytes and written 408 bytes
Verification: OK
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: A0E0FD0686D54B99214B59E8A3802528E3D5852668C8F370294E43C46D82049D
    Session-ID-ctx:
    Resumption PSK: 1F46E44EAF0FE5D7B249DC134C784D8148254BDDDF3A3ECCE4C8F76181DE5A1E9147CA00DB778BE2462616BAD6D8BA6B
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - f8 b1 95 64 69 ee ee 20-b3 ef 40 d0 17 ea 3d 24   ...di.. ..@...=$
    0010 - d1 ef 6e 5a 13 65 62 94-6d 73 63 cb b1 af c4 2d   ..nZ.eb.msc....-
    0020 - cc 02 5e c0 a4 8e f6 b1-6e 53 13 bf 93 38 73 05   ..^.....nS...8s.
    0030 - 9e 7e 79 b7 24 94 e2 42-a9 36 ef 99 67 4c a6 30   .~y.$..B.6..gL.0
    0040 - f7 26 3b 40 c6 d8 e3 92-83 51 0b cb 3e 8f 09 66   .&;@.....Q..>..f
    0050 - eb 50 cc 75 17 1c 0f cd-33 c1 7b 9f b7 26 12 a1   .P.u....3.{..&..
    0060 - 88 86 04 05 e5 a5 e9 cd-3a c1 c2 1b 65 6e d3 40   ........:...en.@
    0070 - 15 88 46 0d b7 97 87 4b-4e 98 11 41 7f 36 bb 61   ..F....KN..A.6.a
    0080 - 4f eb 43 08 90 66 3f 6a-67 ec 02 bc 5b d3         O.C..f?jg...[.

    Start Time: 1707659572
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK
---
Post-Handshake New Session Ticket arrived:
SSL-Session:
    Protocol  : TLSv1.3
    Cipher    : TLS_AES_256_GCM_SHA384
    Session-ID: 9E27F663ACEFF656C0E9CC7982F50C3F67D516196C414AE0615F6F5CFD04D9AD
    Session-ID-ctx:
    Resumption PSK: 7B42806AB732DECCF4B3064B644C4D7DEFF67C1B1D5D32C89C082CDAC1012788372B0D37E891E6337E6244F817A7E2E9
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 86400 (seconds)
    TLS session ticket:
    0000 - 57 02 e8 79 dd 40 5c d2-98 7d 79 a2 6b 39 5b 68   W..y.@\..}y.k9[h
    0010 - 78 22 c2 1c 1f 87 46 d6-a4 f3 d9 c5 76 aa 43 d7   x"....F.....v.C.
    0020 - 29 6a f1 02 95 15 22 5f-c8 a3 e2 f0 1f 8f f0 6e   )j...."_.......n
    0030 - a5 7e 2c c1 f8 34 8b e8-7a f5 79 fa 42 65 20 42   .~,..4..z.y.Be B
    0040 - b7 81 60 4f ee 30 16 16-71 56 92 ef 6f e1 70 ec   ..`O.0..qV..o.p.
    0050 - 0f e1 0d 79 17 85 47 31-c9 c7 c9 3f aa d1 ef f8   ...y..G1...?....
    0060 - f6 f1 e5 d9 1e 45 8b d0-67 20 af ed 0b 7e ed 14   .....E..g ...~..
    0070 - ce 12 63 25 80 fe 09 49-bc a9 c8 e7 08 2e 7a 4d   ..c%...I......zM
    0080 - 13 4a e3 fe 14 ef bf 60-d3 0f f0 a0 8c 13         .J.....`......

    Start Time: 1707659572
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
    Max Early Data: 0
---
read R BLOCK

Well, the good news is: HTTPS is working succesfully! It's blocking access to a server you didn't intend to connect to!

The bad news is: what the heck is that thing you're actually connecting to? "State Technical Service"? "Unified State Internet Access Gateway"? Some Kazakhstan state institution trying to read your HTTPS data using a Man-in-the-Middle attack?

Another website mentions this STS/USIAG thingy: https://sts.kz/en/2020/11/09/Более-590-миллионов-атак-отражено-инстру/ as a state wide "firewall" thingy stopping attacks.. Not a good thing if your goverment can read everything you do.. :roll_eyes:

In a somewhat similar thread I suggested using Tor to bypass this goverment firewall: Help me resolve this problem - #23 by Osiris

5 Likes

Yes, our organization is connected to a Single Internet Access Gateway (EDI), this is a mandatory information security requirement for government institutions, the most interesting thing is that we have been connected for the third year and there have been no problems before, everything worked without problems, it may be in this EDI

You probably didn't install the mitm root certificate. Check the documentation for your setup.

NB, I am sure you are aware of the outcry that happens every time the Kazakhstan government tries to get a root certificate publicly trusted. That's just not going to happen. You'll have to add the cert yourself.

5 Likes

We managed to install all the root certificates, and certbot is working, thanks for the help!

3 Likes

Here is a small instruction on how to solve this problem
1 Download the certificate from the official website sts.kz
2 Move the root certificate to your server in the folder with the certificate extension change from .cer to .pem
sudo cp /home/user/Unified_State_Internet_Access_Gateway.cer /usr/share/ca-certificates/Unified_State_Internet_Access_Gateway.pem
3 Add the name of your certificate to the
sudo nano /etc/ca-certificates.conf configuration file

# This file lists certificates that you wish to use or to ignore to be
# installed in /etc/ssl/certs.
# update-ca-certificates(8) will update /etc/ssl/certs by reading this file.
#
# This is autogenerated by dpkg-reconfigure ca-certificates.
# Certificates should be installed under /usr/share/ca-certificates
# and files with extension '.crt' is recognized as available certs.
#
# line begins with # is comment.
# line begins with ! is certificate filename to be deselected.
#
###################YESHDI Certificate
Unified_State_Internet_Access_Gateway.pem
#################################
  1. Update the root certificates using
    the
    sudo update-ca-certificat
  2. Restart Nginx
    sudo systemctl restart nginx

Please note that this will let the goverment of Kazakhstan read ANY communication between the client and the server, even when using HTTPS/SSL/TLS!!!

5 Likes

So... you should encrypt the packets before using the SSL/TLS connection!
Encrypt the encryption!
Possible for private situations.
Not possible for "normal" public Internet sites :frowning:

2 Likes

Tor :wink:

It's meant for exactly these kind of situations!

2 Likes

Yes, this certificate must be installed only if you have a SINGLE INTERNET ACCESS GATEWAY connected. If you do not install this certificate, then the Internet will not work properly. And yes, this is relevant only for government agencies in Kazakhstan.

1 Like

Indeed, only reason I didn't go medieval on the root certificate's ass only because using a private root to perform TLS inspection is pretty common in corporate environments.

3 Likes