Renew failure - unknown error (_ssl.c:3057)

Help
#1

My domain is: balex.tech

I ran this command: sudo certbot renew

It produced this output:
Processing /etc/letsencrypt/renewal/balex.tech.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Attempting to renew cert (balex.tech) from /etc/letsencrypt/renewal/balex.tech.conf produced an unexpected error: unknown error (_ssl.c:3057). Skipping.
All renewal attempts failed. The following certs could not be renewed:
** /etc/letsencrypt/live/balex.tech/fullchain.pem (failure)**

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/balex.tech/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

My web server is (include version): nginx/1.10.3 (Ubuntu)

The operating system my web server runs on is (include version): ubuntu 16.04.6 LTS

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0

Thanks

1 Like
#2

@_az any thoughts on what this could be?

It looks like an unhandled error from Python's own internal TLS implementation when attempting to connect to the ACME endpoint.

(don't know which version, so it's hard to identify what line 3057 refers to)

In turn, Python's implementation can be imported by urllib3, which is imported by requests.

It presumably shouldn't be possible to crash Python's TLS implementation in a way that doesn't generate an intelligible Python exception, but apparently it is!

@balex3000, do you know which version of Python your copy of Certbot is running under?

3 Likes
#3

... OK,

https://www.python2.net/questions-1482129.htm

suggests it's Python 3.8—I found some Python 3.8 source packages for Ubuntu 16.04 but I didn't find anything compelling at that location in their Modules/_ssl.c.

3 Likes
#4

Can you direct me how to verify python version of my certbot copy?

1 Like
#5

Yes,

python3 --version

and

dpkg -l python3

might have relevant output for that.

(Unusually, I suspect this is a bug in Python rather than in Certbot. That's almost never the case!)

2 Likes
#6

Python 3.5.2

Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-================================-=====================-=====================-======================================================================
ii python3 3.5.1-3 amd64 interactive high-level object-oriented language (default python3 versi

1 Like
#7

I'm kind of confused about how you can get Python 3.5 on Ubuntu 16.04.6. Was this system upgraded from an earlier version of Ubuntu?

Sorry, I was confusing 16.04 and 18.04. On 16.04 Python 3.5 is available, while on 18.04 it's not. It makes sense that you have 3.5.

2 Likes
#8

No, it was initially created as 16.04.6. Definitely needed to choose much more updated version but this is what we have for now.

2 Likes
#9

Hmmm, with 0.31.0 you could also be running under Python 2. Could you try these?

python --version

dpkg -l python

3 Likes
#10

$ python --version
The program 'python' can be found in the following packages:

  • python-minimal
  • python3
    Try: sudo apt install <selected package

$ dpkg -l python
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-================================-=====================-=====================-======================================================================
un python <none <none (no description available)

1 Like
#11

OK, it looks like in Python 3.5.1 that line is

static PyObject *
_ssl__SSLContext_load_dh_params(PySSLContext *self, PyObject *filepath)
/*[clinic end generated code: output=1c8e57a38e055af0 input=c8871f3c796ae1d6]*/
{
    FILE *f;
    DH *dh;

    f = _Py_fopen_obj(filepath, "rb");
    if (f == NULL)
        return NULL;

    errno = 0;
    PySSL_BEGIN_ALLOW_THREADS
    dh = PEM_read_DHparams(f, NULL, NULL, NULL);
    fclose(f);
    PySSL_END_ALLOW_THREADS
    if (dh == NULL) {
        if (errno != 0) {
            ERR_clear_error();
            PyErr_SetFromErrnoWithFilenameObject(PyExc_OSError, filepath);
        }
        else {
            _setSSLError(NULL, 0, __FILE__, __LINE__);
        }
        return NULL;
    }
    if (SSL_CTX_set_tmp_dh(self->ctx, dh) == 0)
        _setSSLError(NULL, 0, __FILE__, __LINE__);
    DH_free(dh);
    Py_RETURN_NONE;
}

@_az, I'm almost tempted to ask the Let's Encrypt staff to ask the CDN staff whether they can think of what could cause the DH parameter to be NULL with no errno set. It seems like that's what happens that triggers an unspecified error from _ssl.c, but I don't know why.

3 Likes
#12

@balex3000 I don't think you're doing anything wrong at all; what's more, Ubuntu 16.04 (for all that I confused it with 18.04) is still under support from Canonical, and is a supported version of Ubuntu for Certbot too. (Lots of people show up on the forum running OS versions that are no longer supported by the operating system developer, which is much more of a problem, since they're no longer receiving security updates or bug fixes at all. But that's not your situation!)

It looks like there might be a subtle bug in Python that for some reason prevents it from connecting to Let's Encrypt's API on your system (while allowing almost everyone to connect almost all of the time!). I'm still not sure what that could be, but maybe we can figure it out.

Can I assume that you don't have anything blocking your outbound network connections? Like if you run

curl https://acme-v02.api.letsencrypt.org/directory

on this same system, you see something more or less like the following, rather than any kind of error?

{
  "KzBl6npZxPM": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
3 Likes
#13

$ curl https://acme-v02.api.letsencrypt.org/directory
{
"PmeqUTutt44": "Adding random entries to the directory",
"keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
"meta": {
"caaIdentities": [
"letsencrypt.org"
],
"termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
"website": "https://letsencrypt.org"
},
"newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
"newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
"newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
"revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"

1 Like
#14

I would like to add that I installed certbot on the server in August 2020 and the renewal worked perfectly until certificate expiration date on 6/6/2021. I suppose it stopped working sometime earlier. I can try to search logs when it stopped working.

1 Like
#15

Do you find it odd for an SSL client to be reading DH params from disk in the first place?

At least, I can't get a bp on _ssl__SSLContext_load_dh_params to trigger on that version of Python and Certbot installed from the old PPA.

I'd be interested to see the stack trace from /var/log/letsencrypt/letsencrypt.log, to see who the caller is.

Without getting super deep into this, I suggest installing the Certbot snap, which will hopefully not suffer from whatever is happening here.

1 Like
#16

Certbot should have started trying to renew 30 days before the expiry date, so that would have been in early May.

Another area of concern should also be server software. They too get bug fixes and security updates. Nginx - the version installed on @balex3000's server had a sevurity issue that fixed by a later version.
Version 1.20.1 is the latest stable version of nginx. They've had many bug fixes, security patches, features and improvements since the version @balex3000 is running. The latest stable version is 1.20.1 and the latest mainline version is v1.21.0. They do keep legacy versions as well (on this download page).

nginx mainline, stable and legacy versions download page

Changes made to every version are here https://unit.nginx.org/CHANGES.txt

4 Likes
#17

Hmm...
I wonder what apt update would find (and fix) and when it was last run.

2 Likes
#18

The code you showed is server-side code that loads Diffie-Hellman parameters for (later) use in a handshake. It's not related to any actual TLS connection, but is usually done before initializing a TLS server socket.

(This is done when you want to support non-ECC groups: Instead of generating random groups at runtime or using compiled-in defaults, you configure the library to load known-good groups such as ffdhe2048 from a file)

I would guess that OpenSSL reports an error if you try to load a corrupted file. There seems to be logic on the cpython side that ignores non-existing files, so that should not cause issues.

Note that it makes no sense to call this as a client, as the server chooses the group parameters in SSLv3/TLS1.0/1.1/1.2 (and 1.3 only uses defined named groups).

PS: In the original python 3.5.2 source - wasn't able to find the Ubuntu xenial source code, as it's too old - the function directly above is _ssl__SSLContext_load_verify_locations_impl, which makes much more sense, as that's likely called on a client connection. An incorrect/corrupt trust store could cause issues here.

2 Likes
#19

I did apt-get update several times while trying to deal with the problem.
It didn't help

1 Like
#20

That's good.
But how about?:
apt-get upgrade

3 Likes