Error: Inconsistency detected by ld.so: dl-close.c


#1

I ran the following commands on the production server:

# git clone https://github.com/letsencrypt/letsencrypt
# cd letsencrypt
# ./letsencrypt-auto certonly --webroot -w /www/mydomain.tld -d mydomain.tld -d www.mydomain.tld
...
Setting up libssl1.0.0:i386 (1.0.1-4ubuntu5.31) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Inconsistency detected by ld.so: dl-close.c: 759: _dl_close: Assertion `map->l_init_called' failed!
E: Sub-process /usr/bin/dpkg returned an error code (127)
Failed to install a working "virtualenv" command, exiting

If I trying to run any command I get the error:
Inconsistency detected by ld.so: dl-close.c: 759: _dl_close: Assertion `map->l_init_called’ failed!

# ls
Inconsistency detected by ld.so: dl-close.c: 759: _dl_close: Assertion `map->l_init_called' failed!
# reboot
Inconsistency detected by ld.so: dl-close.c: 759: _dl_close: Assertion `map->l_init_called' failed!
# apt-get install something
Inconsistency detected by ld.so: dl-close.c: 759: _dl_close: Assertion `map->l_init_called' failed!

I can’t run any command, I can’t create second connection to the host, so if I lose this connection, I can not connect the server and solve this.
What should I do to get the server working?


#2

This looks like an old linux bug - letsencrypt usually updates your system first. Personally I’d power cycle the server to get it back up, then update the system and apply all patches


#3

Unfortunately I don’t have any image of the server, and I can’t run any command on the server except ‘pwd’ and ‘cd’.
But ‘/bin/pwd’ is causing the same error:

# pwd
/www/
# /bin/pwd
Inconsistency detected by ld.so: dl-close.c: 759: _dl_close: Assertion `map->l_init_called' failed!

#4

Can you physically get to the server though, to power cycle it ?

Alternatively you could try the Magic SysRq key … Used in sequence, some of these actions can be used to cleanly end processes, flush data to disk, unmount all file systems, and restart your computer. To perform this process, press and hold the Alt + SysRq key combination and – while holding the Alt and SysRq keys down — type the following keys in order, pausing for several seconds in between each key: reisub

See https://en.wikipedia.org/wiki/Magic_SysRq_key

Just make sure you are roing it on the server, not your linux pc ( if connected that way ) … otherwise it will boot your linux pc :wink:


#5

I was trying to connect to the server (to make second SSH connection) but get the error: “Connection closed.”, so the current connection is my last chance to solve this issue, I don’t want to loose it.

I’m able to reboot the VPS through the Control Panel. But will it be working after that?


#6

I don’t know the exact details of your VPS system, so can’t say 100%, however 99% of the time, yes it will come back up perfectly OK. On the other hand, you haven’t got a whole lot to lose, as it’s not working or doing anything at the moment unfortunately.


#7

Right now all the websites are running well, the only problem is that no one command is running.
I’ll try to reboot.


#8

Nope, the server does not start.

VPS’s VNC shows the same error and additional: “Kernel panic - not syncing: Attempted to kill init!”

How could this small script kills whole server…


#9

It isn’t the script ( and in many ways has nothing do to with the LE script). It is an old linux bug that was not patched on your server, and needs patching. If you can access the main terminal via VNC then you can reboot in a recovery mode and patch it. exactly how depends on the OS you are using, I’m assuming ubuntu of some version, if so then you can enter recovery mode at the GRUB boot time.


#10

Yes, this is Ubuntu. I’m very appreciated for your help. Unfortunately I have neither the ability to run VNC in recovery mode, nor to get access to a GRUB. VNC on the start shows the error about kernel panic.


#11

What version of Ubuntu ? and what is the setup ? I’m guessing a VPS hosted somewhere “in the cloud” and you have a “control panel” which provides a certain amount of “console / KVM / VNC” access to the virtual machine ? Can you use the “console / VNC” to add any key strokes at boot time ?


#12

Ubuntu-14.04-amd64
The control panel is VM Manager, by ISPsystem. It gives the only ability to run VNC in new browser window. When the VNC is started the kernel panic error is occurs immediately.

Screenshot https://www.dropbox.com/s/tzekpq7kymqrlx7/2015-12-07%2020_26_37-VNC-vm3875093.png?dl=0


#13

That does appear to be a limited VNC, looking to see if I can see a way around it for you


#14

does http://doc.ispsystem.com/index.php/Recovery_mode help ?


#15

Thank you for your help! I don’t have access to DCImanager. I asked my hosting support to help with the issue.


#16

good luck, it should be recoverable easily enough once someone can get to the boot / grub prompt


#17

The server has started just now. Thank you!


#18

Great :smiley: worth doing an apt-get update and upgrade to ensure all packages are patched ( which LE should have done befpre it started trying to do anything else)