Renew certificate verification failed after delete certification

sadeshwar94 · October 8, 2021, 5:05am

Hi there, recently I have noticed some of the users cannot access GitLab server due to an error showing it tortoiseGit progress panel

unable to access "https://gitlab.gpms.jp/" : SSL certificate problem: certificates has expired

which is weird, because I have updated the SSL certificate two weeks ago. therefore i tried to renew it again as following,

I ran this command:

certbot delete --cert-name gitlab.gpms.jp
reboot
letsencrypt certonly --standalone -d gitlab.gpms.jp

It produced this error output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
An unexpected error occurred:
SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
Please see the logfiles in /var/log/letsencrypt for more details.

letsencrypt log as follows

2021-10-08 13:46:43,557:DEBUG:certbot.main:certbot version: 0.27.1
2021-10-08 13:46:43,557:DEBUG:certbot.main:Arguments: ['--standalone', '-d', 'gitlab.gpms.jp']
2021-10-08 13:46:43,557:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-10-08 13:46:43,569:DEBUG:certbot.log:Root logging level set at 20
2021-10-08 13:46:43,569:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2021-10-08 13:46:43,570:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
2021-10-08 13:46:43,729:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7fd5aeae49d0>
Prep: True
2021-10-08 13:46:43,729:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7fd5aeae49d0> and installer None
2021-10-08 13:46:43,729:INFO:certbot.plugins.selection:Plugins selected: Authenticator standalone, Installer None
2021-10-08 13:46:43,749:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None), uri=u'https://acme-v02.api.letsencrypt.org/acme/acct/42495813', new_authzr_uri=None, terms_of_service=None), 737031eae76c65ea24d48b2d6f04fd8b, Meta(creation_host=u'v133-18-196-72', creation_dt=datetime.datetime(2018, 9, 21, 6, 46, 55, tzinfo=<UTC>)))>
2021-10-08 13:46:43,750:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2021-10-08 13:46:43,753:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2021-10-08 13:46:44,048:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/letsencrypt", line 9, in <module>
    load_entry_point('certbot==0.27.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1364, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1238, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 648, in _init_le_client
    return client.Client(config, acc, authenticator, installer, acme=acme)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 247, in __init__
    acme = acme_from_config_key(config, self.account.key, self.account.regr)
  File "/usr/lib/python2.7/site-packages/certbot/client.py", line 50, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/lib/python2.7/site-packages/acme/client.py", line 744, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/lib/python2.7/site-packages/acme/client.py", line 1078, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python2.7/site-packages/acme/client.py", line 1027, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
2021-10-08 13:46:44,050:ERROR:certbot.log:An unexpected error occurred:

My web server is (include version):

nginx version: nginx/1.12.2

Operating system : CentOS Linux release 7.2.1511 (Core)

I can login to a root shell on my machine (yes or no, or I don't know): yes

Cerbot version: certbot 0.27.1

anyone could help us with how to install the SSL certificate again for the domain which I had been using previously.

Thank you

rg305 · October 8, 2021, 10:46am

Hi @sadeshwar94 welcome to the LE community forum

There are a few things that can be improved.

certbot 0.27.1 can be updated - the latest version is now 1.20.0
The site isn't serving a chain:

    openssl s_client -connect gitlab.gpms.jp:443 -servername gitlab.gpms.jp |head
    depth=0 CN = gitlab.gpms.jp
    verify error:num=18:self signed certificate
    verify return:1
    depth=0 CN = gitlab.gpms.jp
    verify return:1
    CONNECTED(00000005)
    ---
    Certificate chain
     0 s:CN = gitlab.gpms.jp
       i:CN = gitlab.gpms.jp
    ---

On this system, was certbot ever installed using pip?
If so, that may need to be reverted.

sadeshwar94 · October 11, 2021, 8:29am

@rg305 thank you for your prompt reply.

As you mentioned, I have tried to update certbot, but I was able to update the version until 1.11.0, which epel releases for CentOS 7 only until 1.11.0. Therefore I have to update the entire server, which is kinda risky at the moment.

but for the time being, I need to find a different method to install certbot 1.20.0, if you could guide me any other way. that would be much helpful.

im not sure what you talking about, it this happening because I tried to delete the certificate and re-install the certificate?
in this system cerbot were not installed using pip.

Thank you again

Osiris · October 11, 2021, 8:33am

For CentOS 7 you can also use certbot-1.11.0-2 from EPEL, which has the working --preferred-chain feature backported into it. Notice the -2 suffix. Not sure on how to install it though, I'm not familiar with CentOS/EPEL.

rg305 · October 11, 2021, 11:00am

There are also other ACME client software, and even a docker image of certbot.
[if you can't/won't be using snaps]
No, all certs are provided with their chain files. The web server must be incorrectly configured.
OK; but that is also an option.
See: https://certbot.eff.org/

MikeMcQ · October 11, 2021, 2:59pm

@sadeshwar94 Also see this post from Felix about the Centos 7 update for Certbot

FelixSchwarz · October 11, 2021, 8:23pm

# yum update --enablerepo=epel-testing certbot

sadeshwar94 · October 12, 2021, 8:27am

@rg305

thank you so much, everyone, I have checked the other ACME client software but it seems, still, need to upgrade the other packages of the server. and it seems python3 needed to be installed on the server in order to run the latest version of certbot.

sadeshwar94 · October 12, 2021, 8:31am

@MikeMcQ @FelixSchwarz @Osiris
Thank you for your answers and opinios,

still, I can not issue the certificate even after I run the # yum update --enablerepo=epel-testing certbot command. the package got installed but still shows the same version and the errors of the lentencrypt logs still remain the same.

FelixSchwarz · October 12, 2021, 8:47am

Can you run rpm -q certbot so we can see exactly which version you have?
Can you send the exact commands you used + logs/output? Initially you tried with certbot 0.27 and I hope you are seeing something different now.

sadeshwar94 · October 12, 2021, 9:25am

certbot-1.11.0-2.el7.noarch

Commands an output as follows,

letsencrypt certonly --standalone -d gitlab.gpms.jp

error output

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
An unexpected error occurred:
SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
Please see the logfiles in /var/log/letsencrypt for more details.`

cat /var/log/letsencrypt/letsencrypt.log

letsencrypt error.log as following,

2021-10-12 18:17:11,913:DEBUG:certbot._internal.main:certbot version: 1.11.0
2021-10-12 18:17:11,913:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/letsencrypt
2021-10-12 18:17:11,913:DEBUG:certbot._internal.main:Arguments: ['--standalone', '-d', 'gitlab.gpms.jp']
2021-10-12 18:17:11,913:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-10-12 18:17:11,927:DEBUG:certbot._internal.log:Root logging level set at 20
2021-10-12 18:17:11,927:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2021-10-12 18:17:11,928:DEBUG:certbot._internal.plugins.selection:Requested authenticator standalone and installer None
2021-10-12 18:17:11,929:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot._internal.plugins.standalone:Authenticator
Initialized: <certbot._internal.plugins.standalone.Authenticator object at 0x7f7edb57c090>
Prep: True
2021-10-12 18:17:11,929:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot._internal.plugins.standalone.Authenticator object at 0x7f7edb57c090> and installer None
2021-10-12 18:17:11,929:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator standalone, Installer None
2021-10-12 18:17:11,948:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None, external_account_binding=None), uri=u'https://acme-v02.api.letsencrypt.org/acme/acct/42495813', new_authzr_uri=None, terms_of_service=None), 737031eae76c65ea24d48b2d6f04fd8b, Meta(creation_host=u'v133-18-196-72', register_to_eff=None, creation_dt=datetime.datetime(2018, 9, 21, 6, 46, 55, tzinfo=<UTC>)))>
2021-10-12 18:17:11,949:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2021-10-12 18:17:11,955:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2021-10-12 18:17:12,230:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/letsencrypt", line 9, in <module>
    load_entry_point('certbot==1.11.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1421, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 1277, in certonly
    le_client = _init_le_client(config, auth, installer)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/main.py", line 659, in _init_le_client
    return client.Client(config, acc, authenticator, installer, acme=acme)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 255, in __init__
    acme = acme_from_config_key(config, self.account.key, self.account.regr)
  File "/usr/lib/python2.7/site-packages/certbot/_internal/client.py", line 43, in acme_from_config_key
    return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
  File "/usr/lib/python2.7/site-packages/acme/client.py", line 831, in __init__
    directory = messages.Directory.from_json(net.get(server).json())
  File "/usr/lib/python2.7/site-packages/acme/client.py", line 1168, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python2.7/site-packages/acme/client.py", line 1118, in _send_request
    response = self.session.request(method, url, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/sessions.py", line 596, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg/requests/adapters.py", line 497, in send
    raise SSLError(e, request=request)
SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)
2021-10-12 18:17:12,232:ERROR:certbot._internal.log:An unexpected error occurred:
2021-10-12 18:17:12,232:ERROR:certbot._internal.log:SSLError: ("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",)`

i hope this would give you some kind of information to solve.
Thank you.

FelixSchwarz · October 12, 2021, 10:05am

I suspect you might have installed some Python packages manually into /usr/lib/python2.7/site-packages/ with pip. That way you create a "Frankenstein" system which is completely broken in various regards.

You can check your installation for unmanaged Python packages with this command:

LANG=C rpm -qf /usr/lib/python2.7/site-packages/* | grep 'is not owned by any package'

You should not get any output but at least /usr/lib/python2.7/site-packages/requests-2.11.1-py2.7.egg does not come from an EPEL repository.

So you need to clean up your system.

delete/remove all items which do not originate from CentOS or Fedora EPEL
reinstall python3-requests and possibly other overridden packages

Some background for your error:

upstream requests (installed by pip) uses the certifi database which contains also the expired root certificate. openssl 1.0.2 does not like that and throws certificate verification errors.
RHEL did patch the requests package to use the system-wide CA database (package ca-certificates). That database does not contain the expired certificate anymore so openssl is happy.
So if you only use distro packages everything works fine.

In the future please do NOT use pip install as root user without activated virtualenv. Even though some tutorials tell you to do that you are on the way to "hell".

system · November 11, 2021, 10:06am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.