Urllib3 not working

Help
#1

Hi! Yes the urllib is throwing fits for me also. I have regenerated the certificates with

certbot renew --preferred-chain "ISRG Root X1"

and edited the fullchain file according to the descriptions but the client still refuses to connect. Also checked what is being served with this

openssl s_client -connect sentry.example.com:443 -servername sentry.example.com

Before I edited the fullchain file but after I regenerated the certs with "preferred-chain" I got continuous errors like this on the server side

SSL_do_handshake() failed (SSL: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:SSL alert number 45) while SSL handshaking

And on the client side

>>> [2021-10-09 23:11:25,894] ERROR Sentry responded with an error: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)> (url: https://sentry.example.com/api/3/store/)
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/urllib/request.py", line 1350, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/local/lib/python3.7/http/client.py", line 1281, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1327, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1276, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1036, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 976, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/raven/utils/http.py", line 38, in connect
    sock, ca_certs=ca_certs, cert_reqs=ssl.CERT_REQUIRED)
  File "/usr/local/lib/python3.7/ssl.py", line 1238, in wrap_socket
    suppress_ragged_eofs=suppress_ragged_eofs
  File "/usr/local/lib/python3.7/ssl.py", line 423, in wrap_socket
    session=session
  File "/usr/local/lib/python3.7/ssl.py", line 870, in _create
    self.do_handshake()
  File "/usr/local/lib/python3.7/ssl.py", line 1139, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/lib/python3.7/site-packages/raven/transport/http.py", line 43, in send
    ca_certs=self.ca_certs,
  File "/usr/local/lib/python3.7/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.7/urllib/request.py", line 525, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.7/urllib/request.py", line 543, in _open
    '_open', req)
  File "/usr/local/lib/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.7/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
  File "/usr/local/lib/python3.7/urllib/request.py", line 1352, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_ssl.c:1091)>

I am using the raven client locally to trigger the error. I have removed the last certificate from the fullchain.pem but then that only gives me this (client side)

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1091)>

And server side

SSL_do_handshake() failed (SSL: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca:SSL alert number 48) while SSL handshaking

I guess I could fork and change the code for the raven client to ask urllib to not verify the certificates, but that feels like the wrong way to solve this. The browser works just fine and no other errors beside this. Appreciate any help regarding this!

Www.ssllabs.com is showing a missing Certificate
#2

I've upgraded requests, certifi but nothing really changed. I saw in the code in urllib that it imported a method via six and when I upgraded that one I think I got a few calls through actually. I have upgraded the staging env and will check further tomorrow. Thanks for the rubber duck debugging and the information in the other threads which lead me in the right direction!

1 Like
#3

Please show the version of openssl used:
openssl version

1 Like
#4

The nginx container is running OpenSSL 1.1.1d 10 Sep 2019

The client runs OpenSSL 1.1.1k 25 Mar 2021

I could not see that it made any difference on my staging env overnight. Will throw a few more glances on it later today, thank for the help :+1:

1 Like
#5

Using the urlib library causes the SSL problem even in the very latest 3.9 version of Python. I got over the problem by using the python request library.

1 Like
#6

urllib3 seems to be using the certifi library by default. You might need to update that package.

If you'd run the command python -m certifi it should return the path of the certificate root bundle used.

Edit: That command fails on my Gentoo system with a "No module named certifi.main; 'certifi' is a package and cannot be directly executed" error.. :stuck_out_tongue:

Running python -c "import certifi; print(certifi.where())" does work though.

3 Likes
#7

Yes I updated to the latest certifi but i cannot get around using urllib3 since it is used by the client app. will try and upgrade sentry-sdk and see if that could solve it.

#8

Please provide the actual hostname, so we can take a look.

2 Likes
#9

Certainly, it is https://sentry.wikblad.com .

#10

Your certificate still sends the certificate chain which includes the ISRG Root X1-signed-by-DST Root CA X3:

Certificate chain
 0 s:CN = sentry.wikblad.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

How did you install the certificate into your webserver?

Also, urllib3 seems to work perfectly on my laptop:

osiris@erazer ~ $ python
Python 3.9.6 (default, Oct  9 2021, 21:05:43) 
[GCC 10.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib3
>>> http = urllib3.PoolManager()
>>> url = "https://sentry.wikblad.com"
>>> resp = http.request('GET', url)
>>> resp.status
200
>>>

What's the clients code?

2 Likes
#11

Also, I've moved the posts into a separate thread.

2 Likes
#12

I checked and the sentry-sdk latest version does not seem to include breaking stuff so upgraded that and tried to trigger an custom even which worked. That is progress at least. Sorry for not being more comprehensive in my replies, kids are breaking things left to right here.

I have updated and pushed this new change to staging will try and see if I can capture something in the wild.

#13

I generated it with

certbot renew --force-renewal --preferred-chain "ISRG Root X1" as I thought was how to do it?

In the nginx conf

ssl_certificate /etc/letsencrypt/live/sentry.wikblad.com/fullchain.pem; # managed by Certbot #
ssl_certificate_key /etc/letsencrypt/live/sentry.wikblad.com/privkey.pem; # managed by Certbot
Www.ssllabs.com is showing a missing Certificate
#14

Client code would be this?

#15

Which version of certbot are you using? Perhaps it's new enough to recognise the --preferred-chain option, but too old for it to actually have any effect.

Also, about the urllib3 code: I meant the code actually used in the application. Although I must say, I have no idea what "sentry-sdk" actually is.. So I can't help you with additional guidance for that.

You see, my urllib3 example showed it can work perfectly. But for some reason yours doesn't.

2 Likes
#16

Yes totally understand, I upgraded to certbot 1.20. The code is inside a handshake section and the module is loaded dynamically through interfaces which means I'd need to attach a debugger to get to the bottom of it. I just could not do it at the time.

However, by upgrading all the adjacent libraries it does not crash anymore at least but I need to verify it against the prod environment to know for sure. Once I have that I will post which versions I upgraded to, platform etc.

Thanks again!

2 Likes
#17

Hi again, was able to solve this by updating my packages on the client side to

requests==2.26.0
six==1.16.0
sentry_sdk==1.4.3
certifi==2021.10.8

Also upgraded certbot to 1.20.

I am running the prod env on Python 3.7.12
I think the biggest impact was certifi upgrade tbh.

Amazing support from the community, appreciate it!

2 Likes
#18

But you already updated that package earlier and it didn't work you said?

1 Like
#19

Yes, in hindsight that was probably an incorrect conclusion. I tried to verify if it worked or not by sending manual events to sentry but they never showed up and the logs were littered with exceptions due to the other services which had not yet received that upgrade. Also, the events themselves did not show up, but that I think was because the event themselves were never tagged with a specific environment which the view was filtering on.

The day after I saw that user transactional data, which monitors the user experience, page load etc had started to trickle in and that made me realise my error. Transactional data is aggregated in the views and not reported on an item per item basis which is why I saw it first the day after.

1 Like
#20

I have no idea what all of that means, but I'm glad it's working.

3 Likes