Ubuntu 16.06 Xenial Certbot Still Using Acmev1 (last package version is certbot version 0.31.0)?

adamovic · May 3, 2020, 7:54am

Hi guys,

Months ago I’ve received info that I shall migrate to AcmeV2.

My webserver is custom build Tomcat 8.5.5 using Tomcat Native.
I have even written a tutorial how I did it (back in 2016): https://mladenadamovic.wordpress.com/2016/09/06/configure-tomcat-with-ssl-on-ubuntu-minimal/

It looked like the way to solve is to upgrade certbot (I had a version from the year 2016).
So I followed the instructions from:

But it seems this didn’t solve the problem.
I have received this email again.

#apt-get update
...
# apt-get install certbot
Reading package lists... Done
Building dependency tree       
Reading state information... Done
certbot is already the newest version (0.31.0-1+ubuntu16.04.1+certbot+1).

With legacy reasons, I have this in my cron:

# m h  dom mon dow   command
5   1  1   *   *     /root/renew_cert_numbeo.sh

root@condor1796 ~ # cat renew_cert_numbeo.sh
#!/bin/bash

mkdir -p /tmp/letsencrypt/public_html
certbot certonly -n --force-renewal --webroot --webroot-path /tmp/letsencrypt/public_html -d numbeo.com -d www.numbeo.com \
        -d es.numbeo.com -d  pt.numbeo.com -d  fr.numbeo.com -d  ru.numbeo.com -d  ja.numbeo.com -d  de.numbeo.com -d nl.numbeo.com \
        -d it.numbeo.com -d zh.numbeo.com -d ar.numbeo.com \
     --agree-tos --email mladen.adamovic@gmail.com

/root/fix_letsencrypt_chmod.sh
if [ $? != 0 ]; then

   date | mail -s "Lets encrypt renew certificate fails for numbeo.com" mladen.adamovic@gmail.com
else
   /etc/init.d/tomcat restart
fi

This is in my systemctl:

root@condor1796 ~ # systemctl list-timers
NEXT                         LEFT          LAST                         PASSED              UNIT                         ACTIVATES
Sun 2020-05-03 06:34:09 CDT  3h 47min left Sat 2020-05-02 06:36:54 CDT  20h ago             apt-daily-upgrade.timer      apt-daily-upgrade.service
Sun 2020-05-03 09:18:52 CDT  6h left       Sat 2020-05-02 09:18:52 CDT  17h ago             systemd-tmpfiles-clean.timer systemd-tmpfiles-clean.service
Sun 2020-05-03 12:58:26 CDT  10h left      Sun 2020-05-03 02:29:37 CDT  17min ago           motd-news.timer              motd-news.service
Sun 2020-05-03 13:25:59 CDT  10h left      Sat 2020-05-02 21:45:54 CDT  5h 0min ago         apt-daily.timer              apt-daily.service
Sun 2020-05-03 20:21:43 CDT  17h left      Sun 2020-05-03 02:29:20 CDT  17min ago           certbot.timer                certbot.service
n/a                          n/a           Sat 2019-11-23 07:22:56 CST  5 months 9 days ago ureadahead-stop.timer        ureadahead-stop.service

6 timers listed.

in cron.* there is a certbot as well:

root@condor1796 ~ # cat /etc/cron.*/certbot | tail -n 1
0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 'sleep int(rand(43200))' && certbot -q renew

How to fix this so that this is working as intended?

Email error message:

According to our records, the software client you're using to get Let's 
Encrypt TLS/SSL certificates issued or renewed at least one HTTPS certificate 
in the past two weeks using the ACMEv1 protocol. Here are the details of one 
recent ACMEv1 request from each of your account(s):

Client IP address:  2a01:4f8:150:1229::2 

User agent:  CertbotACMEClient/0.31.0 (certbot; Ubuntu 16.04.6 LTS) Authenticator/webroot Installer/None (renew; flags: n) Py/3.5.2 

Hostname(s):  "[numbeo.com](http://numbeo.com/)","[ar.numbeo.com](http://ar.numbeo.com/)","[de.numbeo.com](http://de.numbeo.com/)","[es.numbeo.com](http://es.numbeo.com/)","[fr.numbeo.com](http://fr.numbeo.com/)","[it.numbeo.com](http://it.numbeo.com/)","[ja.numbeo.com](http://ja.numbeo.com/)","[nl.numbeo.com](http://nl.numbeo.com/)","[pt.numbeo.com](http://pt.numbeo.com/)","[ru.numbeo.com](http://ru.numbeo.com/)","[www.numbeo.com](http://www.numbeo.com/)","[zh.numbeo.com](http://zh.numbeo.com/)" 

Request time:  2020-05-01 12:24:18 UTC

adamovic · May 3, 2020, 7:59am

More details if I run

root@condor1796 ~ # /usr/bin/certbot --dry-run renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/numbeo.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator webroot, Installer None
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ar.numbeo.com
http-01 challenge for de.numbeo.com
http-01 challenge for es.numbeo.com
http-01 challenge for fr.numbeo.com
http-01 challenge for it.numbeo.com
http-01 challenge for ja.numbeo.com
http-01 challenge for nl.numbeo.com
http-01 challenge for numbeo.com
http-01 challenge for pt.numbeo.com
http-01 challenge for ru.numbeo.com
http-01 challenge for www.numbeo.com
http-01 challenge for zh.numbeo.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/numbeo.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/numbeo.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

_az · May 3, 2020, 8:05am

Check if any of your renewal configurations contain references to the ACME v1 server:

sudo grep -R acme-v01 /etc/letsencrypt/renewal

Also check that your python3-acme version is up to date:

dpkg-query -l python3-acme

bmw · May 7, 2020, 6:38pm

I think _az is correct that there is a reference acme-v01 in the files under /etc/letsencrypt/renewal. The line probably looks like:

server = https://acme-v01.api.letsencrypt.org/directory

Deleting that line from your configuration file should solve the problem.

As for why this happened, I think there was a problem with the ACMEv2 migration we did in Certbot a long time ago for certificates that were initially issued using letsencrypt 0.4.2 or older and have since then only used the renew subcommand to renew your certificates. If that’s what’s going on, it was a bug in Certbot and not due to any misconfiguration on your end. Thanks for using Certbot for such a long time and sorry for the trouble.

adamovic · May 8, 2020, 6:00am

Hi guys,

Thank you very much for trying to solve this issue. I appreciate it, by knowing that it is free.

Unfortunately, there is no reference to acme-v01

root@condor1796 /etc/letsencrypt/renewal # grep -R acme-v01 /etc/letsencrypt/renewal
root@condor1796 /etc/letsencrypt/renewal # grep -R server /etc/letsencrypt/renewal
/etc/letsencrypt/renewal/numbeo.com.conf:server = https://acme-v02.api.letsencrypt.org/directory
r

The only reference of v01 in /lets/encrypt directory is
accurance of directory

/etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/

Probably it is safe to delete this directory?

I'll let you know if this email message reappears.

mnordhoff · May 8, 2020, 6:32am

It's probably not safe: Because Let's Encrypt uses one account database regardless of the API version you're accessing, older Certbot installations will have the real account files in /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/ and symlinks in /etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/.

At worst, /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/ is critically important. At best, it's harmless. Deleting it won't fix anything, and will probably break things.

bmw · May 13, 2020, 7:12pm

Looking at certificate transparency and Let’s Encrypt’s server logs, something strange is going on here.

I found that on May 1st, two certificates were issued for the same set of numbeo.com subdomains. One certificate can be seen at https://crt.sh/?id=2748786245 and the other at https://crt.sh/?id=2751598083.

Both of these certificates were issued using Certbot 0.31.0 on Ubuntu 16.04. There are some differences between the runs though. In the order they occurred:

Run 1: Used ACMEv2, had --force-renewal or --renew-by-default set on the command line, and used the certonly subcommand.
Run 2: Used ACMEv1, did not have --force-renewal or --renew-by-default on the command line, and used the renew subcommand.

I suspect the ACMEv2 issuance came from the custom script you provided above.

Where did the ACMEv1 issuance come from though and why did it happen? The IP address Certbot connected from is the one provided your email from Let’s Encrypt. The certificate currently being used is the older certificate that was issued using ACMEv2 rather than the later certificate that was issued with ACMEv1.

Some data you could provide that may help answer these questions are:

Output of sudo openssl x509 -in /etc/letsencrypt/live/numbeo.com/cert.pem -noout -text
Output of sudo certbot certificates
Any (recent) log files in /var/log/letsencrypt that reference acme-v01

adamovic · May 14, 2020, 11:05am

Thank you.

As requested:

root@condor1796 ~ # sudo openssl x509 -in /etc/letsencrypt/live/numbeo.com/cert.pem -noout -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:e9:a6:60:fd:39:ab:1c:df:89:00:6e:76:24:c4:be:84:dd
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, O=Let's Encrypt, CN=Let's Encrypt Authority X3
        Validity
            Not Before: May  1 05:05:17 2020 GMT
            Not After : Jul 30 05:05:17 2020 GMT
        Subject: CN=numbeo.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:9d:17:be:8d:d2:d7:77:a3:4a:49:5a:27:a7:14:
                    b8:c8:2c:4c:68:88:28:33:17:36:eb:5f:5d:55:f1:
                    da:d1:ce:63:1d:b7:bd:53:d4:06:3e:ff:24:99:ef:
                    43:60:6a:07:f7:39:b7:66:ff:3f:a9:e3:bd:97:6c:
                    d3:c3:17:46:32:86:81:01:2b:ae:b5:3c:97:be:c6:
                    d9:12:ef:d5:4d:17:46:4a:97:be:48:47:98:22:c3:
                    39:35:56:5d:70:1d:38:a2:0a:3f:5c:88:90:0e:49:
                    a9:90:b7:57:39:3e:fa:2a:32:5d:13:3b:61:1c:92:
                    26:c2:64:0a:18:d7:66:bb:8a:e1:ef:5d:7a:26:d5:
                    0c:ea:67:0f:8a:d6:fe:32:3f:0f:92:12:aa:90:3f:
                    85:d0:55:46:db:01:b7:fb:b7:28:e5:3e:78:bb:8c:
                    1b:09:c0:ad:4e:45:e9:48:6c:c2:25:e1:32:e2:2e:
                    60:90:b2:1b:a7:a0:22:23:19:bc:d8:c9:b1:ba:af:
                    78:75:51:76:c1:16:fb:9f:45:96:44:fc:d9:83:ab:
                    80:9f:7e:d2:f6:20:f3:f4:13:fd:b3:2d:17:58:ea:
                    e5:27:56:98:d1:e8:d5:24:ae:17:2f:00:fd:23:42:
                    48:16:50:6b:8d:b6:ba:33:00:84:6c:cb:57:50:9f:
                    25:f3
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                91:00:A0:A5:50:4A:21:18:F7:D3:0B:71:73:05:CA:08:FE:C3:BE:51
            X509v3 Authority Key Identifier: 
                keyid:A8:4A:6A:63:04:7D:DD:BA:E6:D1:39:B7:A6:45:65:EF:F3:A8:EC:A1

            Authority Information Access: 
                OCSP - URI:http://ocsp.int-x3.letsencrypt.org
                CA Issuers - URI:http://cert.int-x3.letsencrypt.org/

            X509v3 Subject Alternative Name: 
                DNS:ar.numbeo.com, DNS:de.numbeo.com, DNS:es.numbeo.com, DNS:fr.numbeo.com, DNS:it.numbeo.com, DNS:ja.numbeo.com, DNS:nl.numbeo.com, DNS:numbeo.com, DNS:pt.numbeo.com, DNS:ru.numbeo.com, DNS:www.numbeo.com, DNS:zh.numbeo.com
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : B2:1E:05:CC:8B:A2:CD:8A:20:4E:87:66:F9:2B:B9:8A:
                                25:20:67:6B:DA:FA:70:E7:B2:49:53:2D:EF:8B:90:5E
                    Timestamp : May  1 06:05:17.683 2020 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:41:9B:8A:CC:1A:21:0D:EF:1D:02:0A:AD:
                                00:FD:7E:59:43:52:F4:30:14:72:1B:38:16:1E:BA:B7:
                                0F:4E:43:D7:02:20:7B:EF:70:61:AF:8D:D6:CC:08:A3:
                                19:D7:12:CA:E9:97:D8:62:6D:35:BC:A9:B7:E1:8B:E8:
                                03:D5:73:39:A0:F5
                Signed Certificate Timestamp:
                    Version   : v1(0)
                    Log ID    : 6F:53:76:AC:31:F0:31:19:D8:99:00:A4:51:15:FF:77:
                                15:1C:11:D9:02:C1:00:29:06:8D:B2:08:9A:37:D9:13
                    Timestamp : May  1 06:05:17.932 2020 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:44:02:20:4E:53:F6:D8:2F:E6:10:4A:15:DC:76:8A:
                                01:E2:C8:C9:40:7E:11:3B:14:FC:F7:49:EF:8F:0C:D8:
                                5F:AD:AC:BF:02:20:27:74:69:6A:08:3E:89:C7:6C:64:
                                D0:EC:5B:78:C0:F8:97:41:8E:BD:95:C2:28:70:06:1C:
                                77:71:1E:E5:5E:BE
    Signature Algorithm: sha256WithRSAEncryption
         30:01:14:db:c7:b2:8b:92:6c:1e:04:1b:64:b2:7f:2a:17:c0:
         12:e4:d5:b2:75:be:30:08:8d:c9:a1:98:97:0b:45:11:09:c2:
         ac:37:d9:4c:84:f2:10:c4:fb:02:71:c8:e8:f8:e0:3e:a8:f8:
         66:4b:3c:6a:13:91:5d:92:c7:54:10:c0:fb:f7:b1:62:c3:59:
         e5:0a:a9:8b:16:1f:b5:91:2f:8f:31:98:e0:b0:94:b9:6d:f0:
         0b:b5:3a:17:d7:0d:a5:df:bb:f0:27:d8:c5:43:b9:a1:56:9a:
         2b:38:f3:85:1c:56:b4:85:c4:38:87:38:44:03:59:ec:cb:1f:
         55:3b:e3:9a:2a:ce:ae:25:0a:6d:f1:1e:f6:bd:79:99:ae:35:
         30:10:e6:8a:de:17:51:3c:09:c5:e0:a1:d3:ae:08:81:20:84:
         88:be:ea:fc:a7:ed:8e:45:77:0d:a7:4c:53:ff:65:81:c0:f7:
         28:96:e0:be:79:4e:d2:c8:43:a6:69:59:d3:f4:a0:29:5c:f3:
         7d:8b:f6:70:e2:3b:a1:80:5b:bc:af:a5:ee:ed:60:6a:45:8f:
         ed:e3:1b:04:67:7f:9a:b7:0c:db:46:fb:bc:7f:cc:e4:85:ff:
         76:3f:bb:fe:83:8e:48:8f:de:87:22:a5:48:30:cb:72:66:f5:
         6b:e8:ad:91
root@condor1796 ~ # sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: numbeo.com
    Domains: numbeo.com ar.numbeo.com de.numbeo.com es.numbeo.com fr.numbeo.com it.numbeo.com ja.numbeo.com nl.numbeo.com pt.numbeo.com ru.numbeo.com www.numbeo.com zh.numbeo.com
    Expiry Date: 2020-07-30 05:05:17+00:00 (VALID: 76 days)
    Certificate Path: /etc/letsencrypt/live/numbeo.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/numbeo.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

root@condor1796 ~ # grep "acme-v01" /var/log/letsencrypt/letsencrypt.log
root@condor1796 ~ # ls -l /var/log/letsencrypt/
total 2112
-rw-r--r-- 1 root root   5507 May 14 05:44 letsencrypt.log
-rw-r--r-- 1 root root 224614 Apr  1  2019 letsencrypt.log.10
-rw-r--r-- 1 root root  28346 Mar  8 18:44 letsencrypt.log.10.gz
-rw-r--r-- 1 root root  16603 Mar  1 01:05 letsencrypt.log.11.gz
-rw-r--r-- 1 root root   1373 Feb 24 01:35 letsencrypt.log.12.gz
-rw-r--r-- 1 root root   1397 May 11 04:07 letsencrypt.log.1.gz
-rw-r--r-- 1 root root 220829 Dec  1 01:05 letsencrypt.log.2
-rw-r--r-- 1 root root  55926 May  3 02:58 letsencrypt.log.2.gz
-rw-r--r-- 1 root root 207577 Nov  1  2019 letsencrypt.log.3
-rw-r--r-- 1 root root   1067 Apr 25 22:42 letsencrypt.log.3.gz
-rw-r--r-- 1 root root 220522 Oct  1  2019 letsencrypt.log.4
-rw-r--r-- 1 root root   1430 Apr 20 02:36 letsencrypt.log.4.gz
-rw-r--r-- 1 root root 212445 Sep  1  2019 letsencrypt.log.5
-rw-r--r-- 1 root root   1071 Apr 11 20:36 letsencrypt.log.5.gz
-rw-r--r-- 1 root root 224533 Aug  1  2019 letsencrypt.log.6
-rw-r--r-- 1 root root  17128 Apr  6 02:14 letsencrypt.log.6.gz
-rw-r--r-- 1 root root 224551 Jul  1  2019 letsencrypt.log.7
-rw-r--r-- 1 root root   1060 Mar 28 20:46 letsencrypt.log.7.gz
-rw-r--r-- 1 root root 224751 Jun  1  2019 letsencrypt.log.8
-rw-r--r-- 1 root root   1401 Mar 23 02:53 letsencrypt.log.8.gz
-rw-r--r-- 1 root root 229033 May  1  2019 letsencrypt.log.9
-rw-r--r-- 1 root root   1202 Mar 15 01:46 letsencrypt.log.9.gz

gokaraoke · May 15, 2020, 10:47am

Hello,

I'm seeing this exact same issue on Ubuntu 16.04.6 LTS. I received an email yesterday saying:

According to our records, the software client you're using to get Let's
Encrypt TLS/SSL certificates issued or renewed at least one HTTPS certificate
in the past two weeks using the ACMEv1 protocol.

The list of the notified domains was on +20 containers that run Ubuntu 16.04.6 LTS with straighforward certbot setup and

certbot=0.31.0-1+ubuntu16.04.1+certbot+1
python3-acme=0.31.0-1+ubuntu16.04.1+certbot+1
python3-acme=0.31.0-2+ubuntu16.04.6+certbot+2 on other containers

I don't see acme-v01 anywhere in /etc/letsencrypt/renewal or inside the crons. I also have both acme-v01 and acme-v02 accounts created under /etc/letsencrypt/accounts/ and /etc/letsencrypt/accounts/acme-v02.api.letsencrypt.org/directory is a symlink for /etc/letsencrypt/accounts/acme-v01.api.letsencrypt.org/directory

All the API calls as well inside : /var/log/letsencrypt/letsencrypt.log are reporting to https://acme-v02.api.letsencrypt.org/, except ( 2020-05-15 08:47:57,084:DEBUG:certbot.main:Picked account: ) that is reporting two links only for acme-v01
new_authzr_uri='https://acme-v01.api.letsencrypt.org/acme/new-authz', uri='https://acme-v01.api.letsencrypt.org/acme/reg/ACCOUNT_ID

certbot renew --dry-run -v as well is showing acme-v02 links.

dpkg list: https://pastebin.com/raw/4t7gvZzR

I'm kinda concerned about why I'm receiving these LE alerts while the LE clients are all on the latest version and the logs are confirming it's acme-v02 too. Please lemme know if you need any information and if there is anything I can do to fix this.

Thank you,

bmw · May 18, 2020, 9:28pm

@adamovic, what about the output of:

sudo find /var/log/letsencrypt -type f -mtime -180 -exec zgrep -H "acme-v01" {} \;

Also, is the IP address “2a01:4f8:150:1229::2” from your initial post the server you’re running these commands on?

Part of the reason I’m asking this is while the cert issued using ACMEv1 used your same ACME account, the IP address Certbot connected from was a different IPv4 address. I’m not sure if this is because Certbot was run on a different server or if it’s just a difference in the environments Certbot was run in on the same server.

@gokaraoke, I recommend running a similar find/zgrep command on any of your containers running Certbot related to the domains and IPs in the email(s) from Let’s Encrypt you received to see if you can’t find a log of the problem.

adamovic · May 19, 2020, 6:53am

Hi bmw,

I didn’t figure out, the 2a01:4f8:150:1229::2 is IPv6 address of another server of my control. That server doesn’t invoke my custom script, but perhaps it has some hooks as I might used it before (a few years ago).

It also had currently installed the last version of certbot

root@newTralev ~ # sudo apt-get install certbot
Reading package lists... Done
Building dependency tree       
Reading state information... Done
certbot is already the newest version (0.31.0-1+ubuntu16.04.1+certbot+1).
0 upgraded, 0 newly installed, 0 to remove and 25 not upgraded.

This server is unable to renew certificates as it is not authorized, dry-run shows:
Attempting to renew cert (numbeo.com) from /etc/letsencrypt/renewal/numbeo.com.conf produced an unexpected error: Failed authorization procedure. de.numbeo.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://de.numbeo.com/.well-known/acme-challenge/LPevIRsLOwGG9fPSV6cymRzngcMAzAm4lkslY6far2A [209.126.

So shall conclude that the problem is kind of irrelevant as it occurs due to hooks on an old server?

I don’t know what would be appropriate commands to fix this old server (which is currently not in use)?

Thanks

bmw · May 19, 2020, 5:08pm

What I suspect happened here is:

The server at 2a01:4f8:150:1229::2 has the same account under /etc/letsencrypt/accounts as your other server as well as a certificate for the exact same domains.
Something is configured on the server at 2a01:4f8:150:1229::2 to automatically run certbot renew. This may be your own scripts, however, I think it is (also) probably the systemd timer included in the Certbot .deb package.
The server at 2a01:4f8:150:1229::2 made use of authz reuse to issue a certificate for those domains despite not being able to prove control of that domain itself.
The server at 2a01:4f8:150:1229::2 used the ACMEv1 endpoint due to the problem described above.

To fix the issue, if the server is not in use, I’d either:

Use sudo certbot delete to delete the certificate so the default systemd timer in the Certbot package won’t try to renew it anymore.
Uninstall the certbot package and optionally delete /etc/letsencrypt to remove all account and certificate information from the server.

adamovic · May 20, 2020, 11:45am

Thank you, Brad. It’s shame on me that I haven’t checked IP address before posting.

It was totally unexpected that it’s a different server, also I was confused by IPv6 which I never use, and I haven’t used that server for that domain for a very long time.

bmw · May 20, 2020, 8:05pm

Not a problem! Happy to help.

system · June 19, 2020, 8:05pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.