Probably not. The idea about the webroot plugin is that you can use your existing configuration to serve the challenge file.
I'm not sure I understand the question here. It's rather simple. The webroot plugin has to point to the "DocumentRoot" of your VirtualHost and it needs to be able to serve files under the /.well-known/acme-challenge/ path. That's it.
No, the installer plugin is separate from the webroot authenticator plugin.
An expired certificate is not a problem for the validation servers. It even ignores self-signed certificates.
OK its fixed now. I rebooted the system. After that .well-known didn't give a 404. After I installed the new certs and reverted everything back to the way it used to be. I tested the certbot again, and it failed. Wget shows its once again failing if the folder name is .well-known but its OK if its .well-known2. Its a mystery whats causing this, but its probably related to my system and not letsencrypt. I am wondering if its some apparmor side effect. Anyway, thanks for the responses. I will have to deal with this issue again in 90 days, sigh
Hah! I found the issue. fail2ban had a regex for /var/www/jstb/.well-known/security.txt . Something I must have added , having looked at cyber attackers doing GETs for various files in my apache logfiles. At that time I didn't realise that .well-known was related to letsencrypt, and added that as a regex to ban cyber attackers. Not sure why the reboot helped, maybe it unbanned the IP I was using to test.