It definitely used Apache
Something must have changed since...
Maybe the Java app is now using a lot more memory ???
1 Like
tried it with Java killed, leaving like 600Mb, still died...
I have to step out.
Maybe someone else can help while I'm away.
If not, I'll get back to you later.
1 Like
Also, nothing really had changed... same apache config since getting the certs, haven't installed anything new on EC2 since... just uploading war file and testing continuously...
But now I needed a subdomain, so I wanted to expand the certs... hence the issue at hand..
Edit: nevermind, I didnt't see the "Killed" output.
Try check dmesg -w for Certbot being killed.
2 Likes
Indeed... out of memory it says, but how much does it need?! It had over 600Mb!
[38084.348222] Out of memory: Killed process 4992 (certbot) total-vm:934524kB, anon-rss:673596kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1656kB oom_score_adj:0
[38315.290042] [ 6650] 0 6650 233796 168540 1699840 0 0 certbot
[38315.308616] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=certbot,pid=6650,uid=0
[38315.321281] Out of memory: Killed process 6650 (certbot) total-vm:935184kB, anon-rss:674160kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1660kB oom_score_adj:0
Killed java but still the same...
[44478.018904] Out of memory: Killed process 6330 (certbot) total-vm:930700kB, anon-rss:669696kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1644kB oom_score_adj:0
Readout from top:
KiB Mem : 987936 total, 670520 free, 244296 used, 73120 buff/cache
Can you install the time package and then run Certbot like:
\time -v certbot
and tell me what it says for
Maximum resident set size
2 Likes
Is that the correct flag for time?
time -v certbot -v
bash: -v: command not found
real 0m0.001s
user 0m0.001s
sys 0m0.000s
time is also a bash built-in, so you will need to install the actual time package from yum and then call \time with the \ at the front.
1 Like
Ah, I am silly. The anon-rss of Certbot is already present in the output you already posted. That's interesting, it should not be so big.
How many virtual hosts does your Apache configuration does?
1 Like
which time
/bin/time
\time -v certbot -v
Root logging level set at 10
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requested authenticator None and installer None
Apache version is 2.4.54
Command terminated by signal 9
Command being timed: "certbot -v"
User time (seconds): 3.92
System time (seconds): 0.49
Percent of CPU this job got: 64%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.83
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 669336
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 222
Minor (reclaiming a frame) page faults: 183409
Voluntary context switches: 1119
Involuntary context switches: 644
Swaps: 0
File system inputs: 38688
File system outputs: 32
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
Some other things you might try, just to see if they help:
- Rebooting the server (Just to see if it fixes things, obviously it's a poor workaround if you might need to reboot every time your certificate needs renewal)
- Uninstalling the certbot from yum and using the pip installation method instead
2 Likes
and the letsencrypt.log says:
cat /var/log/letsencrypt/letsencrypt.log
2022-10-17 18:49:09,776:DEBUG:certbot._internal.main:certbot version: 1.11.0
2022-10-17 18:49:09,776:DEBUG:certbot._internal.main:Location of certbot entry point: /bin/certbot
2022-10-17 18:49:09,776:DEBUG:certbot._internal.main:Arguments: ['-v']
2022-10-17 18:49:09,776:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-10-17 18:49:09,804:DEBUG:certbot._internal.log:Root logging level set at 10
2022-10-17 18:49:09,804:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2022-10-17 18:49:09,810:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2022-10-17 18:49:10,693:DEBUG:certbot_apache._internal.configurator:Apache version is 2.4.54
Yes, I've restarted EC2 a few times....
Uninstalled via yum and re-installed via yum (like the first time when it worked)
So I should remove it again and use:
pip3 install -g certbot ?
Only had one, now two with the subdomain.
Out of curiosity, does Certbot still get killed if you run:
certbot certonly -d foo.bar.nonexistent.com \
--apache-server-root /tmp/non-existent \
--webroot -w /tmp \
--dry-run
1 Like
I'm sorry, just removed the certbot installed by yum..
Installed one using pip....
yum remove certbot python2-certbot-apache
python3 -m venv /opt/certbot/
/opt/certbot/bin/pip install --upgrade pip
/opt/certbot/bin/pip install certbot
And now when I do:
certbot --expand -d studio.smkfmartialarts.com -d www.studio.smkfmartialarts.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Certbot doesn't know how to automatically configure the web server on this system. However, it can still get a certificate for you. Please run "certbot certonly" to do so. You'll need to manually configure your web server to use the resulting certificate.
Can you tell me the proper command line to expand my current cert with the new subdomain running off a java war file, with port 80 being redirected by apache?
Running:
certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Found the following certs:
Certificate Name: smkfmartialarts.com
Serial Number: 41176e95bd2ff9***********************
Key Type: RSA
Domains: smkfmartialarts.com www.smkfmartialarts.com
Expiry Date: 2022-12-27 17:38:55+00:00 (VALID: 70 days)
Certificate Path: /etc/letsencrypt/live/smkfmartialarts.com/fullchain.pem
Private Key Path: /etc/letsencrypt/live/smkfmartialarts.com/privkey.pem
I see:
cat /var/log/letsencrypt/letsencrypt.log
2022-10-17 19:15:29,647:DEBUG:certbot._internal.main:certbot version: 1.31.0
2022-10-17 19:15:29,647:DEBUG:certbot._internal.main:Location of certbot entry point: /bin/certbot
2022-10-17 19:15:29,647:DEBUG:certbot._internal.main:Arguments: ['--expand', '-d', 'studio.smkfmartialarts.com', '-d', 'www.studio.smkfmartialarts.com']
2022-10-17 19:15:29,647:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2022-10-17 19:15:29,661:DEBUG:certbot._internal.log:Root logging level set at 30
2022-10-17 19:15:29,662:DEBUG:certbot._internal.plugins.selection:Requested authenticator None and installer None
2022-10-17 19:15:29,663:DEBUG:certbot._internal.plugins.selection:No candidate plugin
So maybe I need to maybe?:
certbot --expand --apache -d studio.smkfmartialarts.com -d www.studio.smkfmartialarts.com