Certbot renew not working?

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: studio.smkfmartialarts.com

I ran this command: certbot renew

It produced this output:
2023-01-08 19:57:38,870:DEBUG:certbot._internal.main:certbot version: 1.31.0
2023-01-08 19:57:38,871:DEBUG:certbot._internal.main:Location of certbot entry point: /bin/certbot
2023-01-08 19:57:38,871:DEBUG:certbot._internal.main:Arguments:
2023-01-08 19:57:38,871:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-01-08 19:57:38,895:DEBUG:certbot._internal.log:Root logging level set at 30
2023-01-08 19:57:38,897:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/studio.smkfmartialarts.com.conf
2023-01-08 19:57:38,919:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7ff315cdc410> and installer <certbot._internal.cli.cli_utils._Default object at 0x7ff315cdc410>
2023-01-08 19:57:38,958:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2023-01-08 19:57:38,971:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2023-01-08 19:57:38,972:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/studio.smkfmartialarts.com/cert2.pem is signed by the certificate's issuer.
2023-01-08 19:57:38,973:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/studio.smkfmartialarts.com/cert2.pem is: OCSPCertStatus.GOOD
2023-01-08 19:57:38,976:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2023-01-24 15:05:02 UTC.
2023-01-08 19:57:38,976:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2023-01-08 19:57:38,976:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2023-01-08 19:57:40,729:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.54

So the log says it's auto-renewing, but when I run: certbot certificates, it hasn't renewed..

Found the following certs:
Certificate Name: studio.smkfmartialarts.com
Serial Number: 36d57c8f035ae1a3ad1774acd5920843280
Key Type: RSA
Domains: studio.smkfmartialarts.com www.studio.smkfmartialarts.com
Expiry Date: 2023-01-24 15:05:02+00:00 (VALID: 15 days)
Certificate Path: /etc/letsencrypt/live/studio.smkfmartialarts.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/studio.smkfmartialarts.com/privkey.pem

My web server is (include version):Apache version is 2.4.54

The operating system my web server runs on is (include version): Amazon Linux - amzn2-ami-kernel-5.10-hvm-2.0.20221004.0-x86_64-gp2

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): ni

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot version: 1.31.0

2

The Certbot log should contain much more lines after the last line you've posted here at 2023-01-08 19:57:40,729: the actual error/problem doesn't show currently. Please provide the entire log so we can see what's going on.

6 Likes
3

What shows?:
sudo apachectl -t -D DUMP_VHOSTS

4 Likes
4

The log ends at the last line shown... This is what you see at the terminal:

etsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/studio.smkfmartialarts.com.conf

Killed

5

pachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443 is a NameVirtualHost
default server ip-172-31-6-237.us-west-1.compute.internal (/etc/httpd/conf.d/ssl.conf:56)
port 443 namevhost ip-172-31-6-237.us-west-1.compute.internal (/etc/httpd/conf.d/ssl.conf:56)
port 443 namevhost studio.smkfmartialarts.com (/etc/httpd/conf/httpd-le-ssl.conf:2)
alias www.studio.smkfmartialarts.com
*:80 is a NameVirtualHost
default server studio.smkfmartialarts.com (/etc/httpd/conf/httpd.conf:159)
port 80 namevhost studio.smkfmartialarts.com (/etc/httpd/conf/httpd.conf:159)
alias www.studio.smkfmartialarts.com
port 80 namevhost studio.smkfmartialarts.com (/etc/httpd/conf/httpd-le-ssl.conf:16)
alias www.studio.smkfmartialarts.com

6

Not sure what's up with your server, but it sounds very similar as your previous thread Certbot killed immediately after starting?

6 Likes
7

Yes, it's almost identical. The certs were renewed last time but it's failing again... So I don't quite understand why...

8

Without a proper log, we neither.

Although:

It's not a good idea to have the same hostname configured in two separate virtualhosts.

5 Likes
9

This is the entirety of the log file:

!cat

cat /var/log/letsencrypt/letsencrypt.log

2023-01-08 20:22:08,357:DEBUG:certbot._internal.main:certbot version: 1.31.0
2023-01-08 20:22:08,357:DEBUG:certbot._internal.main:Location of certbot entry point: /bin/certbot
2023-01-08 20:22:08,357:DEBUG:certbot._internal.main:Arguments:
2023-01-08 20:22:08,357:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-01-08 20:22:08,376:DEBUG:certbot._internal.log:Root logging level set at 30
2023-01-08 20:22:08,378:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/studio.smkfmartialarts.com.conf
2023-01-08 20:22:08,399:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7fa411d46f10> and installer <certbot._internal.cli.cli_utils._Default object at 0x7fa411d46f10>
2023-01-08 20:22:08,438:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2023-01-08 20:22:08,450:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2023-01-08 20:22:08,451:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/studio.smkfmartialarts.com/cert2.pem is signed by the certificate's issuer.
2023-01-08 20:22:08,452:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/studio.smkfmartialarts.com/cert2.pem is: OCSPCertStatus.GOOD
2023-01-08 20:22:08,455:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2023-01-24 15:05:02 UTC.
2023-01-08 20:22:08,455:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2023-01-08 20:22:08,455:DEBUG:certbot._internal.plugins.selection:Requested authenticator apache and installer apache
2023-01-08 20:22:10,013:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.54
[root@ip-172-31-6-237 httpd]#

It ends at the "version is 2.4.54" line...

10

Well, if Certbot gets killed, that would explain the lack of both the incomplete log and lack of renewal.

5 Likes
11

Got rid of the one in httpd-le-ssl.conf, now I get:

apachectl -t -D DUMP_VHOSTS

VirtualHost configuration:
*:80 studio.smkfmartialarts.com (/etc/httpd/conf/httpd.conf:159)
*:443 is a NameVirtualHost
default server ip-172-31-6-237.us-west-1.compute.internal (/etc/httpd/conf.d/ssl.conf:56)
port 443 namevhost ip-172-31-6-237.us-west-1.compute.internal (/etc/httpd/conf.d/ssl.conf:56)
port 443 namevhost studio.smkfmartialarts.com (/etc/httpd/conf/httpd-le-ssl.conf:2)
alias www.studio.smkfmartialarts.com

12

But what could be killing it?

13

I don't think it's from lack of memory..

987936 total, 827600 free, 89900 used, 70436 buff/cache

14

The linux kernel seems to be killing cerbot via the oom_killer:

Jan 8 20:32:34 ip-172-31-6-237 kernel: Out of memory: Killed process 29738 (certbot) total-vm:1099292kB, anon-rss:828008kB, file-rss:4kB, shmem-rss:0kB, UID:0 pgtables:1960kB oom_score_adj:0

15

There's really nothing else running except for the usual daemons...

Jan 8 20:34:02 ip-172-31-6-237 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 1747] 0 1747 35657 129 315392 0 0 systemd-journal
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 1764] 0 1764 29189 65 131072 0 0 lvmetad
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2556] 0 2556 14937 115 131072 0 -1000 auditd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2589] 81 2589 14601 175 151552 0 -900 dbus-daemon
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2594] 32 2594 16820 135 167936 0 0 rpcbind
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2597] 999 2597 3158 43 73728 0 0 lsmd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2602] 0 2602 7258 253 102400 0 0 systemd-logind
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2613] 0 2613 25401 118 159744 0 0 gssproxy
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2620] 998 2620 23522 212 212992 0 0 rngd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2630] 996 2630 30090 137 135168 0 0 chronyd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2847] 0 2847 24668 518 221184 0 0 dhclient
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 2896] 0 2896 24668 510 204800 0 0 dhclient
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3041] 0 3041 22588 263 208896 0 0 master
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3043] 89 3043 22629 258 204800 0 0 qmgr
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3103] 0 3103 113489 420 483328 0 0 rsyslogd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3119] 0 3119 6974 50 98304 0 0 atd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3144] 0 3144 2640 31 65536 0 0 agetty
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3146] 0 3146 30328 32 65536 0 0 agetty
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3222] 0 3222 1068 26 49152 0 0 acpid
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 17135] 0 17135 27710 259 245760 0 -1000 sshd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 5707] 0 5707 6173 158 94208 0 0 crond
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 5794] 0 5794 11457 99 126976 0 -1000 systemd-udevd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 18155] 0 18155 37131 328 323584 0 0 sshd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 18182] 1000 18182 37131 345 315392 0 0 sshd
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 18183] 1000 18183 31184 231 77824 0 0 bash
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 1361] 0 1361 59953 269 303104 0 0 sudo
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 1362] 0 1362 47622 116 212992 0 0 su
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 1363] 0 1363 31218 267 81920 0 0 bash
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 1394] 0 1394 59953 268 307200 0 0 sudo
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 1395] 0 1395 47622 116 212992 0 0 su
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 1396] 0 1396 31251 307 81920 0 0 bash
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3446] 0 3446 311871 1692 184320 0 0 amazon-ssm-agen
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 3513] 0 3513 335825 2994 229376 0 0 ssm-agent-worke
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 20196] 89 20196 22613 254 204800 0 0 pickup
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 29703] 0 29703 31085 117 77824 0 0 log4j-cve-2021-
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 30277] 0 30277 275383 207592 2023424 0 0 certbot
Jan 8 20:34:02 ip-172-31-6-237 kernel: [ 30343] 0 30343 28661 17 53248 0 0 sleep
Jan 8 20:34:02 ip-172-31-6-237 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=certbot,pid=30277,uid=0
Jan 8 20:34:02 ip-172-31-6-237 kernel: Out of memory: Killed process 30277 (certbot) total-vm:1101532kB, anon-rss:830368kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1976kB oom_score_adj:0

16

What happened to the alias name for www? Isn't causing the certbot failure just noting it.

5 Likes
17

Probably removed a little bit too much.

And with regard to the killing: seems to be a memory shortage after all.

5 Likes
18

Yes, I concur... but top shows right now:
987936 total, 798176 free, 90296 used, 99464 buff/cache

So there's like 800MB of memory free. How much does certbot need?

19

Sounds like Augeas might be running out of memory when parsing your Apache configuration, which is surprising to me because it's a C library that shouldn't have a big memory footprint.

Unless your Apache configuration contains some large files, I don't know why it would use so much.

If you only have 1gb RAM and Certbot is using ~900MB, it's not surprising you get oomkilled.

If you have a document root for your domain, you can try renew using that, without involving the Apache plugin:

certbot renew --cert-name studio.smkfmartialarts.com -a webroot -w /path/to/the/webroot/for/that/domain
5 Likes
20

Is there a way to startup certbot with an oom score of like -1000?