Beginner Questions

curl -I http://ocsp.int-x3.letsencrypt.org/

HTTP/1.1 200 OK
Server: nginx
Content-Length: 0
Cache-Control: max-age=14832
Expires: Sat, 16 Feb 2019 10:42:43 GMT
Date: Sat, 16 Feb 2019 06:35:31 GMT
Connection: keep-alive

That looks right.

hmm…

Try:
certbot certificates -vvv

Then show the log file.
[usually /var/log/letsencrypt/letsencrypt.log]

certbot certificates -vvv

2019-02-16 01:39:53,457:DEBUG:certbot.main:certbot version: 0.30.2
2019-02-16 01:39:53,458:DEBUG:certbot.main:Arguments: ['-vvv']
2019-02-16 01:39:53,458:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-02-16 01:39:53,507:DEBUG:certbot.log:Root logging level set at -10
2019-02-16 01:39:53,508:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-02-16 01:39:53,525:DEBUG:certbot.ocsp:Querying OCSP for /etc/letsencrypt/live/fellsbiker.com/cert.pem
2019-02-16 01:39:53,526:DEBUG:certbot.ocsp:openssl ocsp -no_nonce -issuer /etc/letsencrypt/live/fellsbiker.com/chain.pem -cert /etc/letsencrypt/live/fellsbiker.com/cert.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile /etc/letsencrypt/live/fellsbiker.com/chain.pem -verify_other /etc/letsencrypt/live/fellsbiker.com/chain.pem -trust_other -header Host ocsp.int-x3.letsencrypt.org
2019-02-16 01:39:53,531:DEBUG:certbot.ocsp:Error while running openssl ocsp -no_nonce -issuer /etc/letsencrypt/live/fellsbiker.com/chain.pem -cert /etc/letsencrypt/live/fellsbiker.com/cert.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile /etc/letsencrypt/live/fellsbiker.com/chain.pem -verify_other /etc/letsencrypt/live/fellsbiker.com/chain.pem -trust_other -header Host ocsp.int-x3.letsencrypt.org.
[OSCP usage options - redacted for brevity]
2019-02-16 01:39:53,531:INFO:certbot.ocsp:OCSP check failed for /etc/letsencrypt/live/fellsbiker.com/cert.pem (are we offline?)

This is the command that “failed”:

openssl ocsp -no_nonce -issuer /etc/letsencrypt/live/fellsbiker.com/chain.pem -cert /etc/letsencrypt/live/fellsbiker.com/cert.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile /etc/letsencrypt/live/fellsbiker.com/chain.pem -verify_other /etc/letsencrypt/live/fellsbiker.com/chain.pem -trust_other -header Host ocsp.int-x3.letsencrypt.org

What version of openssl are you using?
openssl version

[and try the command above from the prompt]

Hmmm, well if i run openssl version, it tells me 0.9.8zh, which I assume is what comes with the OS. But the version that installs through the package manager with certbot is newer, 1.0.2q. Seems like certbot is using the wrong openssl. Might be a bug with the certbot that comes with homebrew package manager.

0.9.8zh is very old/out-dated.

I found a problem with running:

openssl ocsp -no_nonce -issuer /etc/letsencrypt/live/fellsbiker.com/chain.pem -cert /etc/letsencrypt/live/fellsbiker.com/cert.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile /etc/letsencrypt/live/fellsbiker.com/chain.pem -verify_other /etc/letsencrypt/live/fellsbiker.com/chain.pem -trust_other -header Host ocsp.int-x3.letsencrypt.org

There is an “=” missing between Host and ocsp.int-x3.letsencrypt.org

It should be:

openssl ocsp -no_nonce -issuer /etc/letsencrypt/live/fellsbiker.com/chain.pem -cert /etc/letsencrypt/live/fellsbiker.com/cert.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile /etc/letsencrypt/live/fellsbiker.com/chain.pem -verify_other /etc/letsencrypt/live/fellsbiker.com/chain.pem -trust_other -header Host=ocsp.int-x3.letsencrypt.org

Try that from the prompt.

As written, I get:

Missing = in header key=value
ocsp: Use -help for summary.

With the added “=” I get:

Response verify OK
/etc/letsencrypt/live/Mysite/cert.pem: good
        This Update: Feb 16 05:00:00 2019 GMT
        Next Update: Feb 23 05:00:00 2019 GMT

No luck with your fixed command. Even if I run it on my openssl 1.0.2, instead of the 0.9.8. Either way I get the same “unable to load certificate” error.

Please show:
ls -l /etc/letsencrypt/live/fellsbiker.com/cert.pem
ls -l /etc/letsencrypt/live/fellsbiker.com/chain.pem
ls -l /usr/lib/python3/dist-packages/certbot/ocsp.py
ls -l /opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/ocsp.py

Ok I just realized that if openssl is accessing the cert files then that command is going to need to be run with sudo. I run the command as sudo both ways, meaning the default 0.9.8 version and the 1.0.2 version. Now I get the OCSP usage options, as if theres something else wrong with the commands options or something

Then try:
sudo certbot certificates

I was using sudo with that command. And it still fails the same way it was before,

OCSP check failed for /etc/letsencrypt/live/fellsbiker.com/cert.pem (are we offline?)

(we are not offline)

going in a circle...

sudo ls -l /etc/letsencrypt/live/fellsbiker.com/cert.pem

lrwxr-xr-x 1 root wheel 38 Feb 13 05:57 /etc/letsencrypt/live/fellsbiker.com/cert.pem -> ../../archive/fellsbiker.com/cert1.pem

sudo ls -l /etc/letsencrypt/live/fellsbiker.com/chain.pem

lrwxr-xr-x 1 root wheel 39 Feb 13 05:57 /etc/letsencrypt/live/fellsbiker.com/chain.pem -> ../../archive/fellsbiker.com/chain1.pem

sudo ls -l /usr/lib/python3/dist-packages/certbot/ocsp.py

ls: /usr/lib/python3/dist-packages/certbot/ocsp.py: No such file or directory

sudo ls -l /opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/ocsp.py

ls: /opt/eff.org/certbot/venv/lib/python2.7/site-packages/certbot/ocsp.py: No such file or directory

find / -name ocsp.py

I downloaded your public cert and chain file.
And ran the openssl against it (with the “=”) and it came back fine:

openssl ocsp -no_nonce -issuer chain.pem -cert cert.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile chain.pem -verify_other chain.pem -trust_other -header Host=ocsp.int-x3.letsencrypt.org
Response verify OK
cert.pem: good
        This Update: Feb 13 10:00:00 2019 GMT
        Next Update: Feb 20 10:00:00 2019 GMT

We already knew the cert was valid and the problem was with my certbot/related software though, right? Also my locate database was not up to date and it’s taking longer to update than expected. I’ll have answers for you very soon.

This is now confirmed.
But would need to also test it with OpenSSL v0.9.8 to be sure that doesn't play into the mix.

[just double-checking any & all possibilities]

I'm leaning towards some file dependency not having been properly updated properly or multiple versions of files that are picked up via PATH statement and using an older/wrong file.

locate ocsp.py

/usr/local/Cellar/certbot/0.31.0/libexec/lib/python3.7/site-packages/asn1crypto/ocsp.py
/usr/local/Cellar/certbot/0.31.0/libexec/lib/python3.7/site-packages/certbot/ocsp.py
/usr/local/Cellar/certbot/0.31.0/libexec/lib/python3.7/site-packages/cryptography/hazmat/backends/openssl/ocsp.py
/usr/local/Cellar/certbot/0.31.0/libexec/lib/python3.7/site-packages/cryptography/x509/ocsp.py

That “looks” correct and updated…

So then maybe the version of OpenSSL is to blame.

which openssl
find / -name openssl

What O/S and version are you on?
Can you update OpenSSL?