OSError: [Errno 12] Cannot allocate memory

[gmgj]

I have included the requested information after the ==== below.
The short story is I think I have found a bug. I think I report that as an issue on the github site [Issues · certbot/certbot · GitHub]. But I am reporting here to get some guidance. I have processed 4 lists of 90 domains that request certificates. While processing the 5 list, I started getting errors. My prior requests for help on how to approach my situation.

How to handle www.domains and plain domains?

M…

Missing a SSLCertificateFile or Alias are wrong?

I…

I can do screen share and phone conversations. My silly wild ass guess is that after creating 360 or so certificates, with now 720 or so apache virtual hosts, is that certbot was not tested with this many files to sort and process. In this case I have a lot of output files.
One of the /var/log/letsencrypt/letsencrypt.log files got to 4,229 KB
The current log is 1,109kb

What should I do?

=======
My domain is:
Example domain that works
https://www.pbaclouda2019.com/

I am in the process of requesting certificates. I have a little php that requests 30 at a time. 90 in a day
I have 6 lists with 90 or so site names in each list. I processed the first 4 lists fine. While working on the 5 list, I started getting errors. I rebooted the server and tried again with the same result.

... examples from 1 list
http://www.DrKPeterHuberBlog.com
http://www.DrStevenLintOnline.com
http://www.DrKentSalholmBlog.com
http://www.DrCassidyBoelkBlog.com
http://www.DrPhilipMuenchBlog.com
http://www.DrMasonConnollyBlog.com
http://www.DrRobertTownsendBlog.com

I ran this command: in a php script

$cmd1 = 'certbot -n --apache --agree-tos --redirect -d ' . $ourhost . ' -d ' . $ourhostwww;

that turns into this example

certbot -n --apache --agree-tos --redirect -d drroberttownsendblog.com -d drroberttownsendblog.com

It produced this output:

Unable to run the command: apache2ctl configtest
Encountered exception during recovery:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/util.py", line 72, in run_script
universal_newlines=True)
File "/usr/lib/python3.6/subprocess.py", line 729, in init
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 2212, in config_test
util.run_script(self.option("conftest_cmd"))
File "/usr/lib/python3/dist-packages/certbot/util.py", line 77, in run_script
raise errors.SubprocessError(msg)
certbot.errors.SubprocessError: Unable to run the command: apache2ctl configtest

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/error_handler.py", line 108, in _call_registered
self.funcs-1
File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 323, in _cleanup_challenges
self.auth.cleanup(achalls)
File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 2312, in cleanup
self.restart()
File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 2174, in restart
self.config_test()
File "/usr/lib/python3/dist-packages/certbot_apache/configurator.py", line 2214, in config_test
raise errors.MisconfigurationError(str(err))
certbot.errors.MisconfigurationError: Unable to run the command: apache2ctl configtest
Unable to run the command: apache2ctl configtest
Cleaning up challenges
Unable to run the command: apache2ctl configtest
Encountered exception during recovery:
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/certbot/util.py", line 72, in run_script
universal_newlines=True)
File "/usr/lib/python3.6/subprocess.py", line 729, in init
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1295, in _execute_child
restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory

My web server is (include version):
Apache/2.4.29 (Ubuntu)

The operating system my web server runs on is (include version):
Ubuntu 18.04 LTS

My hosting provider, if applicable, is: Rackspace

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site :no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot --version certbot 0.31.0

[JuergenAuer]

Hi @gmgj

certbot -n --apache

if I know it correct, there are older threads:

If you use the Apache-plugin, there is a lot of parsing required. If it is possible, switch to webroot.

[mnordhoff]

How much memory does the system have?

Can you see how much memory Certbot is using?

If Certbot is being executed from a PHP web application or something, is some sort of memory limitation being enforced against it?

With hundreds of virtual hosts, I wouldn’t be surprised if Certbot’s Apache plugin was slow, but I’d be surprised if it used enough memory to get killed on a reasonable system.

$cmd1 = 'certbot -n --apache --agree-tos --redirect -d ’ . $ourhost . ’ -d ’ . $ourhostwww;

That looks like it risks command injection vulnerabilities.

[gmgj]

Thanks for the replies

1. see the following cmd and op

free -m
total used free shared buff/cache available
Mem: 1987 441 850 6 695 1356
Swap: 0 0 0

cat /proc/meminfo
MemTotal: 2035188 kB
MemFree: 873072 kB
MemAvailable: 1391232 kB
Buffers: 102248 kB
Cached: 493364 kB
SwapCached: 0 kB
Active: 692308 kB
Inactive: 248848 kB
Active(anon): 351712 kB
Inactive(anon): 464 kB
Active(file): 340596 kB
Inactive(file): 248384 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 0 kB

2. I do not know how to check how much memory certbot is using in this circumstance. When something is throwing memory errors, its tricky. If you have suggestions, I will be happy to try them

3. I have run the same php script over 12 times, creating certificates. It did not start throwing errors until I had created over 350 certificates.

4. I had a knowledgeable system admin from my hosting provider (Rackspace) check the system for errors. they looked in the system log looked for zombie processes etc and did not find anything.

I executed 2 requests exactly like the php script runs from the command line. see the example below
certbot --apache -d drtoddsimpsonblog.com -d www.drtoddsimpsonblog.com

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Error running command [‘apache2ctl’, ‘-t’, ‘-D’, ‘DUMP_INCLUDES’] for runtime parameters!

The apache plugin is not working; there may be problems with your existing configuration.

The error was: MisconfigurationError(‘Error accessing loaded Apache parameters: %s’, [‘apache2ctl’, ‘-t’, ‘-D’, ‘DUMP_INCLUDES’])

[gmgj]

Thank you. I included some new information in the reply to mnordhoff. I will lookup how to use webroot. I think part of my responsibilities as an opensource user is to report and help people fix bugs. In this case, I am not knowledgeable enough to know if I have found a bug, or I created an error myself that I should not have. My feeling is that, if at this point, I cannot request a single certificate from the command line, that it should be reported. If you agree, I could use some guidance. Can I delete the /var/log/letsencrypt/letsencrypt.log (its 1,555 KB ) or do something so that I can isolate my case to make it easier to identify and fix the problem? I can make myself available for screen share (at a mutually agreeable time etc if that will help. edit - What about apache information, and system information? I use to develop scripts to send to my users that executed system check commands and included log files that helped me fix problems. Is there anything like that in this situation?

[gmgj]

Unfortunately, I do not have a test site with 1 ip and 20 or so domains pointing to that ip. During my testing, I found that I did not have to shutdown apache. I generated at least 350 certificates in batches of 30 without shutting apache down. Now I am considering that my problem is that I should shutdown apache. Any comments?

[mnordhoff]

For the Apache and webroot plugins, Apache has to be running.

[gmgj]

On page https://certbot.eff.org/docs/using.html#getting-certificates-and-choosing-plugins, under --webroot, it says “If you’re running a local webserver for which you have the ability to modify the content being served, and you’d prefer not to stop the webserver during the certificate issuance process, you can use the webroot plugin …”. is a little confusing. On this page https://www.linuxbabe.com/security/letsencrypt-webroot-tls-certificate , it takes about having to shutdown a standalone webserver. The problem with a standalone web server is that you need to stop your running web server (Apache, Nginx or other) to release port 80 and port 443

I added the following note to my web page https://garyjohnson53.wordpress.com/2019/05/26/a-letsencrypt-example/ on using certbot. I shutdown the Apache server when I place the Virtual Hosts File in the etc/apache2/sites-enabled directory. Then I start up apache so that it will recognize the new Virtual Hosts. You keep the apache server running while you are requesting certificates.

Does that sound like a reasonable addition to the documentation? Also, a more complete definition of local webserver/ standalone might be in order. The letsencrypt community has made this process really easy. I thank you all. I make these comments only to help clarify and amplify all of the wonderful ways you have made this possible.

See https://garyjohnson53.wordpress.com/2013/10/27/in-cacert-org-i-trust/. Letsencrypt is fulfilling the mission that Cacert started in 2005. That is AWESOME.