Auto renew failing; website down

Help
1

My domain is: https://www.iconcierge.net.au/

I ran this command:

certbot renew

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/apps.itourism.com.au.conf

Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',)
Attempting to renew cert (apps.itourism.com.au) from /etc/letsencrypt/renewal/apps.itourism.com.au.conf produced an unexpected error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',). Skipping.

Processing /etc/letsencrypt/renewal/blank_iconcierge.net.au.conf

Cert not yet due for renewal

Processing /etc/letsencrypt/renewal/cairnstoursandtravel.com.conf

Cert not yet due for renewal

Processing /etc/letsencrypt/renewal/cairnstoursandtravel.conf

Cert not yet due for renewal

Processing /etc/letsencrypt/renewal/iconcierge.net.au-0001.conf

Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',)
Attempting to renew cert (iconcierge.net.au-0001) from /etc/letsencrypt/renewal/iconcierge.net.au-0001.conf produced an unexpected error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',). Skipping.

Processing /etc/letsencrypt/renewal/iconcierge.net.au-0002.conf

Cert is due for renewal, auto-renewing...
Could not choose appropriate plugin: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',)
Attempting to renew cert (iconcierge.net.au-0002) from /etc/letsencrypt/renewal/iconcierge.net.au-0002.conf produced an unexpected error: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.',). Skipping.

Processing /etc/letsencrypt/renewal/iconcierge.net.au.conf

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 67, in _reconstitute
renewal_candidate = storage.RenewableCert(full_path, config)
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 463, in init
self._check_symlinks()
File "/usr/lib/python3/dist-packages/certbot/storage.py", line 522, in _check_symlinks
"expected {0} to be a symlink".format(link))
certbot.errors.CertStorageError: expected /etc/letsencrypt/live/iconcierge.net.au/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/iconcierge.net.au.conf is broken. Skipping.

Processing /etc/letsencrypt/renewal/visitorcentre.com.au.conf

Cert not yet due for renewal
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/apps.itourism.com.au/fullchain.pem (failure)
/etc/letsencrypt/live/iconcierge.net.au-0001/fullchain.pem (failure)
/etc/letsencrypt/live/iconcierge.net.au-0002/fullchain.pem (failure)

The following certs are not due for renewal yet:
/etc/letsencrypt/live/blank_iconcierge.net.au/fullchain.pem expires on 2020-03-05 (skipped)
/etc/letsencrypt/live/cairnstoursandtravel.com/fullchain.pem expires on 2020-04-11 (skipped)
/etc/letsencrypt/live/cairnstoursandtravel/fullchain.pem expires on 2020-03-04 (skipped)
/etc/letsencrypt/live/visitorcentre.com.au/fullchain.pem expires on 2020-03-09 (skipped)
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/apps.itourism.com.au/fullchain.pem (failure)
/etc/letsencrypt/live/iconcierge.net.au-0001/fullchain.pem (failure)
/etc/letsencrypt/live/iconcierge.net.au-0002/fullchain.pem (failure)

Additionally, the following renewal configurations were invalid:
/etc/letsencrypt/renewal/iconcierge.net.au.conf (parsefail)

3 renew failure(s), 1 parse failure(s)

My web server is (include version): Linux

The operating system my web server runs on is (include version): Ubuntu, 4.4.0-165.generic

My hosting provider, if applicable, is: Vultr(?)

I can login to a root shell on my machine (yes or no, or I don't know): No, but I have an admin that can sudo

I'm using a control panel to manage my site: putty, 0.70

The version of my client is: certbot 0.31.0

I am thinking of just add the URL to a working certificate, but they may just cause even worse issues, so I might hear some people out here first! Any input appreciated.

-LuB

1 Like
2

I wonder if this may help me...

1 Like
3

Bump, I have had no luck :frowning:

1 Like
4

Please show the output of:
certbot certificates
and also:
ls -l /etc/letsencrypt/renewal/
ls -l /etc/letsencrypt/live/*

1 Like
5

Command:

certbot certificates

Output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/iconcierge.net.au.conf produced an unexpected error: expected /etc/letsencrypt/live/iconcierge.net.au/cert.pem to be a symlink. Skipping.
Revocation status for /etc/letsencrypt/live/apps.itourism.com.au/cert.pem is unknown
Revocation status for /etc/letsencrypt/live/iconcierge.net.au-0001/cert.pem is unknown
Revocation status for /etc/letsencrypt/live/iconcierge.net.au-0002/cert.pem is unknown

Found the following certs:
Certificate Name: apps.itourism.com.au
Domains: apps.itourism.com.au
Expiry Date: 2019-10-07 06:04:03+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/apps.itourism.com.au/fullchain.pem
Private Key Path: /etc/letsencrypt/live/apps.itourism.com.au/privkey.pem
Certificate Name: blank_iconcierge.net.au
Domains: iconcierge.net.au
Expiry Date: 2020-03-05 19:06:23+00:00 (VALID: 51 days)
Certificate Path: /etc/letsencrypt/live/blank_iconcierge.net.au/fullchain.pem
Private Key Path: /etc/letsencrypt/live/blank_iconcierge.net.au/privkey.pem
Certificate Name: cairnstoursandtravel.com
Domains: cairnstoursandtravel.com
Expiry Date: 2020-04-11 10:47:05+00:00 (VALID: 88 days)
Certificate Path: /etc/letsencrypt/live/cairnstoursandtravel.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/cairnstoursandtravel.com/privkey.pem
Certificate Name: cairnstoursandtravel
Domains: www.cairnstoursandtravel.com cairnstoursandtravel.com cairnstoursandtravel.com.au cairnstravelandtours.com cairnstravelandtours.com.au www.cairnstoursandtravel.com.au www.cairnstravelandtours.com www.cairnstravelandtours.com.au
Expiry Date: 2020-03-04 09:18:09+00:00 (VALID: 50 days)
Certificate Path: /etc/letsencrypt/live/cairnstoursandtravel/fullchain.pem
Private Key Path: /etc/letsencrypt/live/cairnstoursandtravel/privkey.pem
Certificate Name: iconcierge.net.au-0001
Domains: *.iconcierge.net.au apps.itourism.com.au
Expiry Date: 2020-01-12 01:17:49+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/iconcierge.net.au-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/iconcierge.net.au-0001/privkey.pem
Certificate Name: iconcierge.net.au-0002
Domains: *.iconcierge.net.au apps.itourism.com.au
Expiry Date: 2020-01-12 01:18:41+00:00 (INVALID: EXPIRED)
Certificate Path: /etc/letsencrypt/live/iconcierge.net.au-0002/fullchain.pem
Private Key Path: /etc/letsencrypt/live/iconcierge.net.au-0002/privkey.pem
Certificate Name: visitorcentre.com.au
Domains: visitorcentre.com.au australianvisitorcentres.com.au australiawidetours.com australiawidetours.com.au ausvc.com.au bestaustraliantours.com.au bestofaustraliatravelcentres.com.au bestofvictoria.com.au lastminutetickets.net.au thesvc.com.au www.australianvisitorcentres.com.au www.australiawidetours.com www.australiawidetours.com.au www.ausvc.com.au www.bestaustraliantours.com.au www.bestofaustraliatravelcentres.com.au www.bestofvictoria.com.au www.lastminutetickets.net.au www.thesvc.com.au www.visitorcentre.com.au
Expiry Date: 2020-03-09 04:13:47+00:00 (VALID: 54 days)
Certificate Path: /etc/letsencrypt/live/visitorcentre.com.au/fullchain.pem
Private Key Path: /etc/letsencrypt/live/visitorcentre.com.au/privkey.pem

The following renewal configurations were invalid:
/etc/letsencrypt/renewal/iconcierge.net.au.conf

Command:

ls -l /etc/letsencrypt/renewal/

Output:

total 32
-rw-r--r-- 1 root root 562 Jul 9 2019 apps.itourism.com.au.conf
-rw-r--r-- 1 root root 621 Dec 6 20:06 blank_iconcierge.net.au.conf
-rw-r--r-- 1 root root 579 Jan 12 11:47 cairnstoursandtravel.com.conf
-rw-r--r-- 1 root root 559 Dec 5 10:18 cairnstoursandtravel.conf
-rw-r--r-- 1 root root 550 Oct 14 02:17 iconcierge.net.au-0001.conf
-rw-r--r-- 1 root root 550 Oct 14 02:18 iconcierge.net.au-0002.conf
-rw-r--r-- 1 root root 525 Oct 14 02:13 iconcierge.net.au.conf
-rw-r--r-- 1 root root 559 Dec 10 05:13 visitorcentre.com.au.conf

Command:

ls -l /etc/letsencrypt/live/*

Output:

-rwxrwxrwx 1 abraham www-data 740 Jul 1 2019 /etc/letsencrypt/live/README

/etc/letsencrypt/live/apps.itourism.com.au:
total 4
lrwxrwxrwx 1 abraham www-data 44 Jul 9 2019 cert.pem -> ../../archive/apps.itourism.com.au/cert1.pem
lrwxrwxrwx 1 abraham www-data 45 Jul 9 2019 chain.pem -> ../../archive/apps.itourism.com.au/chain1.pem
lrwxrwxrwx 1 abraham www-data 49 Jul 9 2019 fullchain.pem -> ../../archive/apps.itourism.com.au/fullchain1.pem
lrwxrwxrwx 1 abraham www-data 47 Jul 9 2019 privkey.pem -> ../../archive/apps.itourism.com.au/privkey1.pem
-rwxrwxrwx 1 abraham www-data 692 Jul 9 2019 README

/etc/letsencrypt/live/blank_iconcierge.net.au:
total 4
lrwxrwxrwx 1 root root 47 Dec 6 20:06 cert.pem -> ../../archive/blank_iconcierge.net.au/cert3.pem
lrwxrwxrwx 1 root root 48 Dec 6 20:06 chain.pem -> ../../archive/blank_iconcierge.net.au/chain3.pem
lrwxrwxrwx 1 root root 52 Dec 6 20:06 fullchain.pem -> ../../archive/blank_iconcierge.net.au/fullchain3.pem
lrwxrwxrwx 1 root root 50 Dec 6 20:06 privkey.pem -> ../../archive/blank_iconcierge.net.au/privkey3.pem
-rwxrwxrwx 1 abraham www-data 692 Aug 8 06:24 README

/etc/letsencrypt/live/cairnstoursandtravel:
total 4
lrwxrwxrwx 1 root root 44 Dec 5 10:18 cert.pem -> ../../archive/cairnstoursandtravel/cert4.pem
lrwxrwxrwx 1 root root 45 Dec 5 10:18 chain.pem -> ../../archive/cairnstoursandtravel/chain4.pem
lrwxrwxrwx 1 root root 49 Dec 5 10:18 fullchain.pem -> ../../archive/cairnstoursandtravel/fullchain4.pem
lrwxrwxrwx 1 root root 47 Dec 5 10:18 privkey.pem -> ../../archive/cairnstoursandtravel/privkey4.pem
-rwxrwxrwx 1 abraham www-data 692 Aug 7 01:10 README

/etc/letsencrypt/live/cairnstoursandtravel.com:
total 4
lrwxrwxrwx 1 root root 48 Jan 12 11:47 cert.pem -> ../../archive/cairnstoursandtravel.com/cert4.pem
lrwxrwxrwx 1 root root 49 Jan 12 11:47 chain.pem -> ../../archive/cairnstoursandtravel.com/chain4.pem
lrwxrwxrwx 1 root root 53 Jan 12 11:47 fullchain.pem -> ../../archive/cairnstoursandtravel.com/fullchain4.pem
lrwxrwxrwx 1 root root 51 Jan 12 11:47 privkey.pem -> ../../archive/cairnstoursandtravel.com/privkey4.pem
-rwxrwxrwx 1 abraham www-data 692 Jul 16 00:23 README

/etc/letsencrypt/live/iconcierge.net.au:
total 4
lrwxrwxrwx 1 root root 41 Oct 14 02:13 cert_old.pem -> ../../archive/iconcierge.net.au/cert7.pem
lrwxrwxrwx 1 root root 42 Oct 14 02:13 chain_old.pem -> ../../archive/iconcierge.net.au/chain7.pem
lrwxrwxrwx 1 root root 46 Oct 14 02:13 fullchain_old.pem -> ../../archive/iconcierge.net.au/fullchain7.pem
lrwxrwxrwx 1 root root 44 Oct 14 02:13 privkey_old.pem -> ../../archive/iconcierge.net.au/privkey7.pem
-rwxrwxrwx 1 abraham www-data 692 Jul 9 2019 README

/etc/letsencrypt/live/iconcierge.net.au-0001:
total 4
lrwxrwxrwx 1 root root 46 Oct 14 02:17 cert.pem -> ../../archive/iconcierge.net.au-0001/cert1.pem
lrwxrwxrwx 1 root root 47 Oct 14 02:17 chain.pem -> ../../archive/iconcierge.net.au-0001/chain1.pem
lrwxrwxrwx 1 root root 51 Oct 14 02:17 fullchain.pem -> ../../archive/iconcierge.net.au-0001/fullchain1.pem
lrwxrwxrwx 1 root root 49 Oct 14 02:17 privkey.pem -> ../../archive/iconcierge.net.au-0001/privkey1.pem
-rw-r--r-- 1 root root 692 Oct 14 02:17 README

/etc/letsencrypt/live/iconcierge.net.au-0002:
total 4
lrwxrwxrwx 1 root root 46 Oct 14 02:18 cert.pem -> ../../archive/iconcierge.net.au-0002/cert1.pem
lrwxrwxrwx 1 root root 47 Oct 14 02:18 chain.pem -> ../../archive/iconcierge.net.au-0002/chain1.pem
lrwxrwxrwx 1 root root 51 Oct 14 02:18 fullchain.pem -> ../../archive/iconcierge.net.au-0002/fullchain1.pem
lrwxrwxrwx 1 root root 49 Oct 14 02:18 privkey.pem -> ../../archive/iconcierge.net.au-0002/privkey1.pem
-rw-r--r-- 1 root root 692 Oct 14 02:18 README

/etc/letsencrypt/live/visitorcentre.com.au:
total 4
lrwxrwxrwx 1 root root 44 Dec 10 05:13 cert.pem -> ../../archive/visitorcentre.com.au/cert6.pem
lrwxrwxrwx 1 root root 45 Dec 10 05:13 chain.pem -> ../../archive/visitorcentre.com.au/chain6.pem
lrwxrwxrwx 1 root root 49 Dec 10 05:13 fullchain.pem -> ../../archive/visitorcentre.com.au/fullchain6.pem
lrwxrwxrwx 1 root root 47 Dec 10 05:13 privkey.pem -> ../../archive/visitorcentre.com.au/privkey6.pem
-rwxrwxrwx 1 abraham www-data 692 Jul 29 05:36 README

1 Like
6

:confounded: quite intimidatedā€¦

1 Like
7

/etc/letsencrypt/live/iconcierge.net.au/cert.pem
No longer exists.
What is there now has "_old" in the name:

Which appears to be a consequence of:

We should have a look at that file.

There are also three expired certs:

[which can probably be deleted - ensure they are replaced with a valid one]

Something seems to have gone terribly wrong on October 14:

2 Likes
8

Thanks rg.

We should have a look at that file.

Contents of iconcierge.net.au.conf:

# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/iconcierge.net.au
cert = /etc/letsencrypt/live/iconcierge.net.au/cert.pem
privkey = /etc/letsencrypt/live/iconcierge.net.au/privkey.pem
chain = /etc/letsencrypt/live/iconcierge.net.au/chain.pem
fullchain = /etc/letsencrypt/live/iconcierge.net.au/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = b63e7e5e7fa17181161d3afb1499ec1e
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = manual

Got co-workers and clients all over me on this, hoping I get it fixed today!

1 Like
9

It looks to me like there might be two problems at hand:

  1. A broken renewal configuration:
    certbot.errors.CertStorageError: expected /etc/letsencrypt/live/iconcierge.net.au/cert.pem to be a symlink Renewal configuration file /etc/letsencrypt/renewal/iconcierge.net.au.conf is broken
  2. These renewal configurations are specified as manual
    The error was: PluginError(ā€˜An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.ā€™,).

When #1 is fixed, you may still have to contend with #2. Certbot cannot perform a ā€œmanualā€ renewal non-interactively unless it is provided with an authentication script. So, you may need to choose a different plugin.

Is this a configuration you inherited from someone you can contact?

You might want to make a backup of the data, try performing new certificate issuance (i.e. not using these renewal configuraitons) with a different authentication plugin against our Staging environment. Once successful, switch to our prod environment to get brand new certificates using new certbot configurations.

4 Likes
10

Is this a configuration you inherited from someone you can contact?

Yeah, I took over a position and all of this was existing. I have actually just reached out to the past employee who built all of this, so hoping for some help; we'll see.

You might want to make a backup of the data, try performing new certificate issuance (i.e. not using these renewal configuraitons) with a different authentication plugin against our Staging environment. Once successful, switch to our prod environment to get brand new certificates using new certbot configurations.

I will look into this, thanks heaps mate!

1 Like
11

It looks like only some of these configurations are manual - (after making a backup) you might be able to change the manual ones to use the same plugin and plugin-options as the others. This won't fix the broken symlink, but for the domain/configurations which are not otherwise broken, it could get you some new certificates.

Again, be careful if you are testing multiple times to use our Staging URI because our production API will apply rate-limits.

4 Likes
12

Okay, I am failing hard :frowning:

I need to get apps.itourism.com.au running so I decided to add it to another certificate in the short term while I fix the other outstanding problems.

I added it to the vistorcentre.com.au cert, and it added successfully , but now returns a 403 forbiddenā€¦

SSL is such a headache sometimes!

2 Likes
13

It really is! That's why we believe so strongly in free and automatic certificates. I'm sorry it isn't working well for you this time around, but I'm excited for when your configurations are all fixed up.

5 Likes
14

Iā€™ll get there. Working through it slowly and I do appreciate the help very much!

4 Likes
15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.