@bmw, could I ask if you have any hypotheses about this? I’m really confused about what’s happened here.
@Vince42, you have (at least the remnants of) two copies of Certbot installed on your system. The files in /opt/eff.org/certbot are from certbot-auto while the packages you see apt come from Ubuntu’s official repos and are not installed by certbot-auto.
Related to this, what’s the output of:
command -v certbotcertbot --version/opt/eff.org/certbot/venv/bin/certbot --version
Outside of that, it unfortunately seems that none of us here are familiar with how virtualmin uses Certbot. With this in mind, you may be able to get additional help by trying to get support from virtualmin. Knowing how they configure Certbot could easily shed some light on this issue.
To try and help more though, are there more logs in /var/log/letsencrypt than what you included above? That was only from 2 days and seeing logs where Certbot 0.34.0+ run or a certificate was issued and a duplicate configuration file was created would help.
1 Like
@bmw Sorry for the late reply: I have been travelling.
command -v certbot: /usr/bin/certbot
certbot --version: certbot 0.23.0
/opt/eff.org/certbot/venv/bin/certbot --version: certbot 0.37.1
I already pointed the guys from Virtualmin to this thread and I hope that they will read it in parallel and inform me about their findings, which I would contribute here then.
I have plenty of logs, but I am unsure how to identify, which log files would really be helpful. I just anonymized and uploaded the latest log file, as the others are handled by logrotate and would have some more work implied as of unzipping etc.
So it does seem that you have multiple copies of Certbot installed on your system.
I think both of these installations are running regularly. The Ubuntu package containing Certbot in /usr/bin comes with a cron job/systemd timer to run Certbot every 12 hours. The installation in /opt/eff.org comes from certbot-auto which has also apparently been run recently as it has upgraded itself to the version of Certbot we released on Thursday. Removing one of these installations and fixing up whatever is regularly calling it to use the other installation is a big first step to solving this problem I think.
As for which logfiles would be helpful, they would match:
grep 'POST request to .*cert' /path/to/file
Ideally it’d be nice to see one like that which contains “certbot version: 0.23.0” and another which contains “certbot version: 0.3”.
1 Like
@bmw Thank you very much for your analysis.
I will ask the Virtualmin guys, which package they are using and remove the other one.
Regarding the helpful lines from the log files:
letsencrypt.log.10:2019-06-18 23:56:16,205:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/035f38ab842532e55bad3ae6f64a9dff1151:
letsencrypt.log.10:2019-06-19 00:06:11,878:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/0300ed252cc15aa8b1a1b38cdb349d106b95:
letsencrypt.log.10:2019-06-19 00:16:12,997:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/036a55af85cac6a970b2dcb2c78c41735da1:
letsencrypt.log.10:2019-06-19 00:16:22,694:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/033281878612c9b6645c7c3fb98f976eee7f:
letsencrypt.log.10:2019-06-19 00:26:12,682:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03be7edecf8c507efcc13c694cb38e16cc62:
letsencrypt.log.10:2019-06-19 00:26:21,721:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03b305a6ece4a9d62622cf849a44e3e53f65:
letsencrypt.log.10:2019-06-19 00:26:31,234:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/039fe11bff19dcd0be5bff51d0959af8ba40:
letsencrypt.log.10:2019-06-19 00:31:15,492:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/039e68e308d1a2966aa4b0f88ee4d1f1a8d8:
letsencrypt.log.10:2019-06-19 00:31:25,756:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/030cc31a02b644c3abe3923a2bf370aa42fc:
letsencrypt.log.10:2019-06-19 10:56:14,205:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03c17970b395894f505510dd37cc3e2efb43:
letsencrypt.log.11:2019-06-12 21:26:48,424:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/034d366f596ad355cfdc0a176d0e46f5c667:
letsencrypt.log.2:2019-08-18 00:16:28,056:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03b4c0964fbc29f4244e054372fdc98fc338:
letsencrypt.log.2:2019-08-18 00:26:24,124:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/030265992015f666596cb7a2f28cdb9a5bec:
letsencrypt.log.2:2019-08-18 00:26:34,174:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/036d561bdad404de664e583d529a90c2872d:
letsencrypt.log.2:2019-08-18 00:41:10,583:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/0303dd25284275ba04ad1e62008cf80f479e:
letsencrypt.log.2:2019-08-18 00:41:21,813:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/039008cb65f444282ad9921a28e6a2c030d8:
letsencrypt.log.3:2019-08-05 02:21:32,777:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/035dd994fdba77bdca29c735d5aceb876fac:
letsencrypt.log.5:2019-07-25 14:01:14,397:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03168efa18d9655b48a730aec7755059581d:
letsencrypt.log.5:2019-07-25 22:46:19,540:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03680783a1da9b433f6e65ee74ea4df87d3b:
letsencrypt.log.6:2019-07-19 00:11:15,028:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03e1b96f898cba78b0eaac96ef38cbd512ff:
letsencrypt.log.6:2019-07-19 00:21:11,977:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03e0fc9d1f2740309750a71c42031926f2a5:
letsencrypt.log.6:2019-07-19 00:21:21,206:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/0381d808e53a5613e7aba4f526f51d5b361e:
letsencrypt.log.6:2019-07-19 00:31:12,480:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/030b1dc5c12e33aacebe1e321019ce129e20:
letsencrypt.log.6:2019-07-19 00:31:22,075:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03b4678e1cb13201fd5a4c98b5a7794e774f:
letsencrypt.log.6:2019-07-19 00:31:32,160:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03f2203457de58b8deb7ec8c7941dc18e7d8:
letsencrypt.log.6:2019-07-19 00:36:10,128:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03826cce418a0ddfc2ae4195f3381d58ddeb:
letsencrypt.log.6:2019-07-19 00:36:20,166:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03c36c4fbaf222d6c4bf878d4e4fef14e277:
letsencrypt.log.6:2019-07-19 11:01:11,664:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/036e4dd918b36b18fde1569c56145f8f78b1:
letsencrypt.log.7:2019-07-12 21:31:36,936:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03cfa547a6518aa5791f4e2b4329c45b830b:
letsencrypt.log.8:2019-07-06 02:12:22,306:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/0390afd806032cf9d0ee1a6ed6eeca3a5377:
letsencrypt.log.8:2019-07-06 02:17:26,387:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03869a162aee8f8723d8cfec1f54e791e3c4:
letsencrypt.log.8:2019-07-06 02:19:43,819:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/0318d23cc828ac55192a3bebfdf9e5fced3d:
letsencrypt.log.8:2019-07-06 02:20:40,232:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/038ac9e2844ca50f534b2ab3486e2ee04079:
letsencrypt.log.9:2019-06-25 13:56:15,831:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/cert/03a8d7ab2e400eeec12f257cfb53f7fcc8f1:
I downloaded all log files from the server and are able to provide them in case it should be needed.
One certificate renewal failed on 2019-08-18, but I do not want to blow this topic up - just FYI.
Thanks for the update!
If after cleaning up your 2nd Certbot installation you’re still having issues that the Virtualmin maintainers can’t resolve, please let us know.
1 Like
Hi @Vince42
Did you solve that?
We are experiencing the same issue and I think started several months ago.
The worst thing is that the certificate is going to expire today!!
The problems are the same that exposed.
Running certbot certificates
root@panel:~# certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Found the following certs:
Certificate Name: api.example.com-0001
Domains: api.example.com
Expiry Date: 2019-11-08 17:17:19+00:00 (VALID: 61 days)
Certificate Path: /etc/letsencrypt/live/api.example.com-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/api.example.com-0001/privkey.pem
Certificate Name: api.example.com-0002
Domains: api.example.com
Expiry Date: 2019-11-08 00:19:08+00:00 (VALID: 60 days)
Certificate Path: /etc/letsencrypt/live/api.example.com-0002/fullchain.pem
Private Key Path: /etc/letsencrypt/live/api.example.com-0002/privkey.pem
Certificate Name: api.example.com
Domains: api.example.com
Expiry Date: 2019-11-08 17:17:24+00:00 (VALID: 61 days)
Certificate Path: /etc/letsencrypt/live/api.example.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/api.example.com/privkey.pem
As you can see, there are duplicates 0001 and 0002. Also, when I enter api.example.com the chrome browser says it expires today! So it is not the same. Virtualmin is saving the SSL keys under the home directory ( SSL certificate file /home/api/ssl.cert)
Also, we have this line of cronjob in webmin
test -x /usr/bin/certbot -a ! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew
We have never put that there, and I don't if it was there since the beginning of webmin installation.
We have other machines with webmin and that cron job doesn't exists.
What can we do?
If you are reading this and know how to solve it, please reply asap.
Thanks.
root@panel:~# grep -Ri sslcertificatefile /etc/apache2
/etc/apache2/sites-enabled/0-example.com.conf:SSLCertificateFile /home/example/ssl.cert
/etc/apache2/sites-enabled/panel.example.com-le-ssl.conf:SSLCertificateFile /etc/letsencrypt/live/panel.example.com/fullchain.pem
/etc/apache2/sites-enabled/api.example.com.conf:SSLCertificateFile /home/api/ssl.cert
/etc/apache2/sites-available/default-ssl.conf: # SSLCertificateFile directive is needed.
/etc/apache2/sites-available/default-ssl.conf: SSLCertificateFile /etc/ssl/certs/ssl-cert-snakeoil.pem
/etc/apache2/sites-available/default-ssl.conf: # the referenced file can be the same as SSLCertificateFile
/etc/apache2/sites-available/0-example.com.conf:SSLCertificateFile /home/example/ssl.cert
/etc/apache2/sites-available/panel.example.com-le-ssl.conf:SSLCertificateFile /etc/letsencrypt/live/panel.example.com/fullchain.pem
/etc/apache2/sites-available/api.example.com.conf:SSLCertificateFile /home/api/ssl.cert
root@panel:~#
That is strange. panel.example.com is the webmin installation (we followed this guide months ago How To Install Webmin on Ubuntu 18.04 | DigitalOcean)
However, the subdomain api cert is reading /home/api/ssl.cert and don't the ones from foldere /etc/letsencrpy
Until someone knows how to properly fix this, I have manually edited apache config
In virtualmin / Services / Configure SSL Website / Edit directives
Changed:
SSLCertificateFile /home/api/ssl.cert
SSLCertificateKeyFile /home/api/ssl.key
SSLCACertificateFile /home/api/ssl.ca
to
SSLCertificateFile /etc/letsencrypt/live/api.example.com-0002/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/api.example.com-0002/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/api.example.com-0002/chain.pem
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.