Update Script Failing... Wait, no come back!

My domain is: quantum-equities.com

I ran this command: /usr/local/bin/letsencrypt-renew
#-----------------------------------------------------
DATE=$(date +%Y-%m-%d)
DIR=/etc/pki/tls

setenforce 0
systemctl stop httpd

CERT="${DIR}/certs/quantum-equities.com_cert-${DATE}.pem"
CHAIN="${DIR}/chains/quantum-equities.com_chain-${DATE}.pem"
CSR=/etc/letsencrypt/csr-quantum-equities.com.csr
DIR2=/etc/letsencrypt/live/quantum-equities.com
FULLCHAIN="${DIR}/chains/quantum-equities.com_fullchain-${DATE}.pem"
OUT=/tmp/certbot-QE.out

certbot certonly --csr ${CSR} --fullchain-path ${FULLCHAIN} --chain-path ${CHAIN} --cert-path ${CERT} --apache > ${OUT} 2>&1

if [ ! -f ${FULLCHAIN} -o ! -f ${CHAIN} -o ! -f ${CERT} ]; then
cat /tmp/certbot-QE.out | mail -s ā€œTLS Cert Update Fail for quantum-equities.comā€ postmaster@quantum-equities.com
fi

link into place

cert only (apache needs)

ln -sf ${CERT} ${DIR2}/cert.pem

cert with chain (stunnel needs)

ln -sf ${FULLCHAIN} ${DIR2}/fullchain.pem

chain only (apache needs)

ln -sf ${CHAIN} ${DIR2}/chain.pem

setenforce 1

reload the services

systemctl reload httpd
#systemctl restart stunnel4
#-----------------------------------------------------

It produced this output:
#-----------------------------------------------------
++ date +%Y-%m-%d

  • DATE=2018-05-21
  • DIR=/etc/pki/tls
  • setenforce 0
  • systemctl stop httpd
  • CERT=/etc/pki/tls/certs/quantum-equities.com_cert-2018-05-21.pem
  • CHAIN=/etc/pki/tls/chains/quantum-equities.com_chain-2018-05-21.pem
  • CSR=/etc/letsencrypt/csr-quantum-equities.com.csr
  • DIR2=/etc/letsencrypt/live/quantum-equities.com
  • FULLCHAIN=/etc/pki/tls/chains/quantum-equities.com_fullchain-2018-05-21.pem
  • OUT=/tmp/certbot-QE.out
  • certbot certonly --csr /etc/letsencrypt/csr-quantum-equities.com.csr --fullchain-path /etc/pki/tls/chains/quantum-equities.com_fullchain-2018-05-21.pem --chain-path /etc/pki/tls/chains/quantum-equities.com_chain-2018-05-21.pem --cert-path /etc/pki/tls/certs/quantum-equities.com_cert-2018-05-21.pem --apache
  • ā€˜[ā€™ ā€˜!ā€™ -f /etc/pki/tls/chains/quantum-equities.com_fullchain-2018-05-21.pem -o ā€˜!ā€™ -f /etc/pki/tls/chains/quantum-equities.com_chain-2018-05-21.pem -o ā€˜!ā€™ -f /etc/pki/tls/certs/quantum-equities.com_cert-2018-05-21.pem ā€˜]ā€™
  • cat /tmp/certbot-QE.out
  • mail -s ā€˜TLS Cert Update Fail for quantum-equities.comā€™ postmaster@quantum-equities.com
  • ln -sf /etc/pki/tls/certs/quantum-equities.com_cert-2018-05-21.pem /etc/letsencrypt/live/quantum-equities.com/cert.pem
  • ln -sf /etc/pki/tls/chains/quantum-equities.com_fullchain-2018-05-21.pem /etc/letsencrypt/live/quantum-equities.com/fullchain.pem
  • ln -sf /etc/pki/tls/chains/quantum-equities.com_chain-2018-05-21.pem /etc/letsencrypt/live/quantum-equities.com/chain.pem
  • CERT=/etc/pki/tls/certs/quantum-sci.com_cert-2018-05-21.pem
  • CHAIN=/etc/pki/tls/chains/quantum-sci.com_chain-2018-05-21.pem
  • CSR=/etc/letsencrypt/csr-quantum-sci.com.csr
  • DIR2=/etc/letsencrypt/live/quantum-sci.com
  • FULLCHAIN=/etc/pki/tls/chains/quantum-sci.com_fullchain-2018-05-21.pem
  • OUT=/tmp/certbot-QS.out
  • CERT=/etc/pki/tls/certs/delphi-real-estate.com_cert-2018-05-21.pem
  • CHAIN=/etc/pki/tls/chains/delphi-real-estate.com_chain-2018-05-21.pem
  • CSR=/etc/letsencrypt/csr-delphi-real-estate.com.csr
  • DIR2=/etc/letsencrypt/live/delphi-real-estate.com
  • FULLCHAIN=/etc/pki/tls/chains/delphi-real-estate.com_fullchain-2018-05-21.pem
  • OUT=/tmp/certbot-DRE.out
  • setenforce 1
  • systemctl reload httpd
    Job for httpd.service invalid.
    #-----------------------------------------------------

My web server is (include version): Apache 2.4.6

The operating system my web server runs on is (include version): CentOS 7.5

My hosting provider, if applicable, is: Self

I can login to a root shell on my machine (yes or no, or I donā€™t know): Yes

Iā€™m using a control panel to manage my site (no, or provide the name and version of the control panel): N/A

apachectl -t
journalctl -u httpd -f

I am not sure why you stop httpd. Certbot with --apache is capable of dealing with a running service.

True. Fixed that now to systemctl start httpd

apachectl -t

journalctl -u httpd -f

May 21 22:15:06 quantum.darkmatter.org systemd[1]: Starting The Apache HTTP Serverā€¦
May 21 22:15:06 quantum.darkmatter.org httpd[16019]: AH00526: Syntax error on line 95 of /etc/httpd/conf.d/vhosts.conf:
May 21 22:15:06 quantum.darkmatter.org httpd[16019]: SSLCertificateFile: file ā€˜/etc/letsencrypt/live/quantum-equities.com/cert.pemā€™ does not exist or is empty
May 21 22:15:06 quantum.darkmatter.org systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
May 21 22:15:06 quantum.darkmatter.org kill[16022]: kill: cannot find process ā€œā€
May 21 22:15:06 quantum.darkmatter.org systemd[1]: httpd.service: control process exited, code=exited status=1
May 21 22:15:06 quantum.darkmatter.org systemd[1]: Failed to start The Apache HTTP Server.
May 21 22:15:06 quantum.darkmatter.org systemd[1]: Unit httpd.service entered failed state.
May 21 22:15:06 quantum.darkmatter.org systemd[1]: httpd.service failed.

cat /tmp/certbot-QE.out
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Error while running apachectl configtest.

AH00526: Syntax error on line 95 of /etc/httpd/conf.d/vhosts.conf:
SSLCertificateFile: file ā€˜/etc/letsencrypt/live/quantum-equities.com/cert.pemā€™ does not exist or is empty

Could not choose appropriate plugin: The apache plugin is not working; there may be problems with your existing configuration.
The error was: MisconfigurationError(ā€œError while running apachectl configtest.\n\nAH00526: Syntax error on line 95 of /etc/httpd/conf.d/vhosts.conf:\nSSLCertificateFile: file ā€˜/etc/letsencrypt/live/quantum-equities.com/cert.pemā€™ does not exist or is empty\nā€,)
The apache plugin is not working; there may be problems with your existing configuration.
The error was: MisconfigurationError(ā€œError while running apachectl configtest.\n\nAH00526: Syntax error on line 95 of /etc/httpd/conf.d/vhosts.conf:\nSSLCertificateFile: file ā€˜/etc/letsencrypt/live/quantum-equities.com/cert.pemā€™ does not exist or is empty\nā€,)

#------------------------------------------------------
/var/log/letsencrypt/letsencrypt.log

2018-05-21 22:15:06,460:DEBUG:certbot.main:certbot version: 0.23.0
2018-05-21 22:15:06,460:DEBUG:certbot.main:Arguments: [ā€™ā€“csrā€™, ā€˜/etc/letsencrypt/csr-quantum-equities.com.csrā€™, ā€˜ā€“fullchain-pathā€™, ā€˜/etc/pki/tls/chains/quantum-equities.com_fullchain-2018-05-21.pemā€™, ā€˜ā€“chain-pathā€™, ā€˜/etc/pki/tls/chains/quantum-equities.com_chain-2018-05-21.pemā€™, '$
2018-05-21 22:15:06,460:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-05-21 22:15:06,477:DEBUG:certbot.log:Root logging level set at 20
2018-05-21 22:15:06,477:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-05-21 22:15:06,478:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2018-05-21 22:15:06,555:ERROR:certbot.util:Error while running apachectl configtest.

AH00526: Syntax error on line 95 of /etc/httpd/conf.d/vhosts.conf:
SSLCertificateFile: file ā€˜/etc/letsencrypt/live/quantum-equities.com/cert.pemā€™ does not exist or is empty

2018-05-21 22:15:06,556:DEBUG:certbot.plugins.disco:Misconfigured PluginEntryPoint#apache: Error while running apachectl configtest.

AH00526: Syntax error on line 95 of /etc/httpd/conf.d/vhosts.conf:
SSLCertificateFile: file ā€˜/etc/letsencrypt/live/quantum-equities.com/cert.pemā€™ does not exist or is empty
Traceback (most recent call last):
File ā€œ/usr/lib/python2.7/site-packages/certbot/plugins/disco.pyā€, line 126, in prepare
self._initialized.prepare()
File ā€œ/usr/lib/python2.7/site-packages/certbot_apache/configurator.pyā€, line 206, in prepare
self.config_test()
File ā€œ/usr/lib/python2.7/site-packages/certbot_apache/configurator.pyā€, line 2034, in config_test
raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apachectl configtest.

AH00526: Syntax error on line 95 of /etc/httpd/conf.d/vhosts.conf:
SSLCertificateFile: file ā€˜/etc/letsencrypt/live/quantum-equities.com/cert.pemā€™ does not exist or is empty

2018-05-21 22:15:06,557:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x7efc74756990>
Prep: Error while running apachectl configtest.

Did Certbot actually succeed to generate the certificate?

ls -laHZ /etc/letsencrypt/live/quantum-equities.com/cert.pem

If the file is valid, are there any denies in the /var/log/audit/audit.log ?

No it doesnā€™t get the cert:
ls: cannot access /etc/letsencrypt/live/quantum-equities.com/cert.pem: No such file or directory

certbot seems to be failing on some Apache config check, which fails because thereā€™s not cert.

Iā€™ve removed --apache from the certbot command, and now itā€™s been stuck on the certbot command for two minutes.

At least itā€™s not failing immediatelyā€¦

Well, yeah. Youā€™re going to have to repair your httpd configuration by hand, since Certbot wonā€™t be able to run against a broken config.

Iā€™m not sure itā€™s in me at the moment to try tackle this script (sorry), but it might be helpful to have a look at the older /var/log/letsencrypt/ logs, before the configuration was broken, to see what failed initially.

Replacing symlinks inside /etc/letsencrypt seems a bit fragile as well.

It's probably waiting for interactive input, like for you to provide an authenticator (i.e. it'll never finish).

What should I say? Itā€™s supposed to be an automatic script. just gives a newline.

You need to decide how Certbot is meant to authenticate your domain.

When you were providing --apache, it was rewriting your configuration file in order to perform the HTTP challenge.

When you removed --apache, it no longer has an authenticator to use, so it just stalls.

An alternative authenticator may be to use --authenticator webroot -w /path/to/your/domains/webroot.

Or keep using -a apache, but repair your Apache config by hand first (which youā€™ll need to do in any case).

It might help to go back to the standard Certbot flow if you can. Are you using the custom --csr flow because you need it to re-use the private key? I canā€™t see any other benefit in this custom script.

I guess Iā€™m too tired for this. Adding the webroot gives a huge letsencrypt.log file that ends with:
FailedChallenges: Failed authorization procedure. quantum-equities.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://quantum-equities.com/.well-known/acme-challenge/Bad90npzaLN6AJYP3bKnTJGFS-eyYdrFBn2HkKeudZ$

Maybe itā€™s requiring the webserver to be running for the challenge? But it canā€™t be for a --csr request, at least thatā€™s what Iā€™d understood.

Yes using csr to keep private key for other DNS security functions.

Hi,

From my port scan results.. your server is either not listening to request or filtered the request (on port 80 and 443)

The csr flag is just let you specify the private key & domain included within the certificate (instead of certbot handle this part).

You can use servers other than the one handling this website, however in that case you would need to manually generate verification (copy verification files and put in the corresponding server / folder as well as update the certificate)

In short, using CSR flag just gave you more control in this process (like key reused.. obtain ECDHA certificate ), it will not let you pass the verification and get a certificate without pass corresponding challenges.

Thank you

Right. Because the webserver canā€™t start, because there is no cert.

Understand what --csr is about. The problem is certbot canā€™t validate my site because of no cert.

Iā€™m now copying /etc/pki/tls/chains/localhost.crt to /etc/pki/tls/certs/quantum-equities.com_cert-2018-05-21.pem (synlinked from /etc/letsencrypt/live/quantum-equities.com/cert.pem), to try and fool it into working.

Edit: Ok, apparently Apache must be running in order for any verification to work. Yet I canā€™t make Apache run with fake certs. So Iā€™ll have to try and manually make a certbot --csr.

Wow. I know that this is possible because Iā€™ve done it before. But I canā€™t even make this work:
# certbot certonly -d quantum-equities.com,www.quantum-equities.com,mail.quantum-equities.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log

How would you like to authenticate with the ACME CA?
-------------------------------------------------------------------------------
1: Apache Web Server plugin - Beta (apache)
2: Spin up a temporary webserver (standalone)
3: Place files in webroot directory (webroot)
-------------------------------------------------------------------------------
Select the appropriate number [1-3] then [enter] (press 'c' to cancel): 2
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
[('PEM routines', 'PEM_read_bio', 'no start line')]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/crypto_util.py", line 309, in _load_cert_or_req
    return load_func(typ, cert_or_req_str)
Error: [('PEM routines', 'PEM_read_bio', 'no start line')]
An unexpected error occurred:
Error: [('PEM routines', 'PEM_read_bio', 'no start line')]
Please see the logfiles in /var/log/letsencrypt for more details.

And no the server is not nginx, itā€™s apache.

2018-05-22 09:44:55,416:DEBUG:certbot.main:certbot version: 0.23.0
2018-05-22 09:44:55,416:DEBUG:certbot.main:Arguments: ['-d', 'quantum-equities.com,www.quantum-equities.com,mail.quantum-equities.com']
2018-05-22 09:44:55,416:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-05-22 09:44:55,430:DEBUG:certbot.log:Root logging level set at 20
2018-05-22 09:44:55,430:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-05-22 09:44:55,430:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2018-05-22 09:44:55,605:DEBUG:certbot_apache.configurator:Apache version is 2.4.6
2018-05-22 09:44:56,002:DEBUG:certbot.plugins.selection:Multiple candidate plugins: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x7f691dea6210>
Prep: True

* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7f691dde07d0>
Prep: True

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f691dde0ad0>
Prep: True
2018-05-22 09:45:02,166:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7f691dde07d0> and installer None
2018-05-22 09:45:02,166:INFO:certbot.plugins.selection:Plugins selected: Authenticator standalone, Installer None
2018-05-22 09:45:02,189:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, contact=(u'mailto:colony.three@protonmail.ch',), agreement=u'https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf', key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7f6920b02750>)>)), uri=u'https://acme-v01.api.letsencrypt.org/acme/reg/24597921', new_authzr_uri=u'https://acme-v01.api.letsencrypt.org/acme/new-authz', terms_of_service=u'https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf'), efe906d6eb95cf90fccee2da3cdba202, Meta(creation_host=u'quantum.darkmatter.org', creation_dt=datetime.datetime(2017, 11, 20, 20, 45, 20, tzinfo=<UTC>)))>
2018-05-22 09:45:02,194:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2018-05-22 09:45:02,197:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2018-05-22 09:45:03,251:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 658
2018-05-22 09:45:03,252:DEBUG:acme.client:Received response:
HTTP 200
content-length: 658
expires: Tue, 22 May 2018 16:45:03 GMT
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
pragma: no-cache
cache-control: max-age=0, no-cache, no-store
date: Tue, 22 May 2018 16:45:03 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: mwCgqTgLN2PZIqFTh2O_K1jMdjT1_fn8QiYEv8yTeOE

{
  "ZxZM0V5LRsY": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "terms-of-service": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
}
2018-05-22 09:45:03,255:ERROR:certbot.crypto_util:[('PEM routines', 'PEM_read_bio', 'no start line')]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/crypto_util.py", line 309, in _load_cert_or_req
    return load_func(typ, cert_or_req_str)
Error: [('PEM routines', 'PEM_read_bio', 'no start line')]
2018-05-22 09:45:03,255:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 9, in <module>
    load_entry_point('certbot==0.23.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1266, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1150, in certonly
    should_get_cert, lineage = _find_cert(config, domains, certname)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 283, in _find_cert
    action, lineage = _find_lineage_for_domains_and_certname(config, domains, certname)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 310, in _find_lineage_for_domains_and_certname
    return _find_lineage_for_domains(config, domains)
  File "/usr/lib/python2.7/site-packages/certbot/main.py", line 254, in _find_lineage_for_domains
    ident_names_cert, subset_names_cert = cert_manager.find_duplicative_certs(config, domains)
  File "/usr/lib/python2.7/site-packages/certbot/cert_manager.py", line 165, in find_duplicative_certs
    return _search_lineages(config, update_certs_for_domain_matches, (None, None))
  File "/usr/lib/python2.7/site-packages/certbot/cert_manager.py", line 380, in _search_lineages
    rv = func(candidate_lineage, rv, *args)
  File "/usr/lib/python2.7/site-packages/certbot/cert_manager.py", line 153, in update_certs_for_domain_matches
    candidate_names = set(candidate_lineage.names())
  File "/usr/lib/python2.7/site-packages/certbot/storage.py", line 851, in names
    return crypto_util.get_names_from_cert(f.read())
  File "/usr/lib/python2.7/site-packages/certbot/crypto_util.py", line 357, in get_names_from_cert
    csr, OpenSSL.crypto.load_certificate, typ)
  File "/usr/lib/python2.7/site-packages/certbot/crypto_util.py", line 337, in _get_names_from_cert_or_req
    loaded_cert_or_req = _load_cert_or_req(cert_or_req, load_func, typ)
  File "/usr/lib/python2.7/site-packages/certbot/crypto_util.py", line 309, in _load_cert_or_req
    return load_func(typ, cert_or_req_str)
Error: [('PEM routines', 'PEM_read_bio', 'no start line')]
2018-05-22 09:45:03,256:ERROR:certbot.log:An unexpected error occurred:

Of course Apache isnā€™t running because it canā€™t find a cert for quantum-equities.com . Yet I canā€™t get a cert because Apache isnā€™t runningā€¦

So edit your Apache config to not require the cert (and therefore not serve SSL). Or give it a self-signed cert. Either will work.

The normal case that Certbot was designed to work with for --apache is an Apache configuration that has an HTTP virtual host and no HTTPS virtual host. Certbot will then obtain a certificate and install it by creating a new HTTPS virtual host based on the contents of the existing HTTP one and using the newly-issued certificate. This avoids the problem of needing a certificate in order to configure HTTPS.

If you do have an existing, working HTTPS virtual host, Certbot can also work with that. The case that doesn't work is trying to create the HTTPS virtual host by yourself ahead of time without having a certificate, because then indeed Apache can't start.

Thanks but Iā€™ve tried that.

May 22 10:22:33 quantum.darkmatter.org systemd[1]: Starting The Apache HTTP Server...
-- Subject: Unit httpd.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has begun starting up.
May 22 10:22:33 quantum.darkmatter.org systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
May 22 10:22:33 quantum.darkmatter.org kill[65359]: kill: cannot find process ""
May 22 10:22:33 quantum.darkmatter.org systemd[1]: httpd.service: control process exited, code=exited status=1
May 22 10:22:33 quantum.darkmatter.org systemd[1]: Failed to start The Apache HTTP Server.
-- Subject: Unit httpd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit httpd.service has failed.
-- 
-- The result is failed.
May 22 10:22:33 quantum.darkmatter.org systemd[1]: Unit httpd.service entered failed state.
May 22 10:22:33 quantum.darkmatter.org systemd[1]: httpd.service failed.
May 22 10:22:33 quantum.darkmatter.org polkitd[738]: Unregistered Authentication Agent for unix-process:65352:8063362 (system bus name :1.131, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)

Could this be a certbot bug? I ought to be able to get a cert with option 2, even without the httpd server.

The context of your error in /var/log/letsencrypt makes it look like you have an existing certificate from Certbot in /etc/letsencrypt/archive that has been replaced with an empty or corrupted file. Is that possible? Can you run certbot certificates and/or ls -l /etc/letsencrypt/archive/*?

# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/quantum-equities.com.conf produced an unexpected error: expected /etc/letsencrypt/live/quantum-equities.com/cert.pem to be a symlink. Skipping.
verifying the signature of the cert located at /etc/letsencrypt/live/quantum-sci.com/cert.pem has failed.                 Details: [('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'data too large for modulus')]
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/crypto_util.py", line 232, in verify_renewable_cert_sig
    OpenSSL.crypto.verify(chain, cert.signature, cert.tbs_certificate_bytes, hash_name)
Error: [('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'data too large for modulus')]
Renewal configuration file /etc/letsencrypt/renewal/quantum-sci.com.conf produced an unexpected error: verifying the signature of the cert located at /etc/letsencrypt/live/quantum-sci.com/cert.pem has failed.                 Details: [('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'data too large for modulus')]. Skipping.
verifying the signature of the cert located at /etc/letsencrypt/live/delphi-real-estate.com/cert.pem has failed.                 Details: Unable to load certificate
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/certbot/crypto_util.py", line 230, in verify_renewable_cert_sig
    cert = x509.load_pem_x509_certificate(cert.read(), default_backend())
  File "/usr/lib64/python2.7/site-packages/cryptography/x509/base.py", line 43, in load_pem_x509_certificate
    return backend.load_pem_x509_certificate(data)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/multibackend.py", line 341, in load_pem_x509_certificate
    return b.load_pem_x509_certificate(data)
  File "/usr/lib64/python2.7/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 1175, in load_pem_x509_certificate
    raise ValueError("Unable to load certificate")
ValueError: Unable to load certificate
Renewal configuration file /etc/letsencrypt/renewal/delphi-real-estate.com.conf produced an unexpected error: verifying the signature of the cert located at /etc/letsencrypt/live/delphi-real-estate.com/cert.pem has failed.                 Details: Unable to load certificate. Skipping.
Renewal configuration file /etc/letsencrypt/renewal/quantum-equities.com-0001.conf produced an unexpected error: expected /etc/letsencrypt/live/quantum-equities.com/cert.pem to be a symlink. Skipping.
Renewal configuration file /etc/letsencrypt/renewal/quantum-sci.com-0001.conf produced an unexpected error: expected /etc/letsencrypt/live/quantum-sci.com-0001/cert.pem to be a symlink. Skipping.
Renewal configuration file /etc/letsencrypt/renewal/delphi-real-estate.com-0001.conf produced an unexpected error: expected /etc/letsencrypt/live/delphi-real-estate.com-0001/cert.pem to be a symlink. Skipping.

-------------------------------------------------------------------------------

The following renewal configuration files were invalid:
  /etc/letsencrypt/renewal/quantum-equities.com.conf
  /etc/letsencrypt/renewal/quantum-sci.com.conf
  /etc/letsencrypt/renewal/delphi-real-estate.com.conf
  /etc/letsencrypt/renewal/quantum-equities.com-0001.conf
  /etc/letsencrypt/renewal/quantum-sci.com-0001.conf
  /etc/letsencrypt/renewal/delphi-real-estate.com-0001.conf
-------------------------------------------------------------------------------

It doesnā€™t seem wise to delete everything in my live directory.

And certbot doesnā€™t seem to like symlinks. But under Posix standards the correct place for certs is /etc/pki.

This is such a mess. Can I just wipe /etc/letsencrypt and start over? Not only am I unable to renew, I canā€™t even do it manually.

Yes, you can do that and remove any reference in your Apache configuration to files that are there.

Certbot uses the structure of /etc/letsencrypt/live and /etc/letsencrypt/archive for renewals. If you delete or rename anything there, renewals will break. This is suggested in a README in each subdirectory of /etc/letsencrypt/live, but Certbot should give a more useful error message in this case instead of crashing.