Running certbot just returns with no action

OK I installed the -dbg library with apt-get install and then ran

sudo gdb python
(gdb) run /usr/bin/certbot --version

Program received signal SIGILL, Illegal instruction.
0xb6886de0 in _armv7_neon_probe () from /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
(gdb)

how do I go about running the backtrace to find the specific calls?

Type backtrace at that prompt (it can also be abbreviated bt).

Program received signal SIGILL, Illegal instruction.
0xb6886de0 in _armv7_neon_probe () from /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
(gdb) bt
#0  0xb6886de0 in _armv7_neon_probe () from /usr/lib/arm-linux-gnueabihf/libcrypto.so.1.0.0
#1  0xb68834f4 in OPENSSL_cpuid_setup () at armcap.c:71
#2  0xb6fdeffc in call_init (l=<optimized out>, argc=3, argv=0xbefff7f4, env=0xbefff804) at dl-init.c:78
#3  0xb6fdf0d8 in _dl_init (main_map=main_map@entry=0x37b428, argc=3, argv=0xbefff7f4, env=0xbefff804) at dl-init.c:126
#4  0xb6fe3658 in dl_open_worker (a=<optimized out>) at dl-open.c:577
#5  0xb6fdee94 in _dl_catch_error (objname=0xb6fdee94 <_dl_catch_error+112>, objname@entry=0xbeffcf14, errstring=0xb6ff2510, 
errstring@entry=0xbeffcf18, mallocedp=0xbeffcf14, mallocedp@entry=0xbeffcf13, operate=0xbeffcf13, args=args@entry=0xbeffcf1c)
at dl-error.c:187
#6  0xb6fe2d48 in _dl_open (file=0x37d358 "/usr/lib/python2.7/lib-dynload/_hashlib.arm-linux-gnueabihf.so", mode=-2147483646, 
caller_dlopen=0x10a6b8 <_PyImport_GetDynLoadFunc+272>, nsid=-2, argc=3, argv=0xbefff7f4, env=0xbefff804) at dl-open.c:661
#7  0xb6f7fba8 in dlopen_doit (a=0xbeffd168) at dlopen.c:66
#8  0xb6fdee94 in _dl_catch_error (objname=0xb6fdee94 <_dl_catch_error+112>, errstring=0xb6ff2510, mallocedp=0x3738e4, 
operate=0x3738e0, args=0xbeffd168) at dl-error.c:187
#9  0xb6f802a8 in _dlerror_run (operate=0xb6f7fb28 <dlopen_doit>, args=args@entry=0xbeffd168) at dlerror.c:163
#10 0xb6f7fc74 in __dlopen (file=<optimized out>, mode=<optimized out>) at dlopen.c:87
#11 0x0010a6b8 in _PyImport_GetDynLoadFunc ()
#12 0x00109ef0 in _PyImport_LoadDynamicModule ()
#13 0x00067838 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Well, that got us a lot further!

It looks like this is a somewhat widespread problem

https://duckduckgo.com/?q=armv7_neon_probe+sigill

I havenā€™t looked at enough of these other threads to understand whether there is a clearly-established solution to it yet. Itā€™s a bug related to CPU detection, which might ultimately be a kernel bug or an OpenSSL bug. Unfortunately several of those threads are from a few years ago (even though Iā€™m pretty sure that the underlying problem is related), which is discouraging because I would have expected the problem to have been fixed by now.

While trying to figure out what kind of CPU you have and what instructions it can perform, the crypto code here from OpenSSL is apparently experimentally trying different things. There is an error called SIGILL (Illegal instruction) that the kernel will send the program if the CPU complains that it doesnā€™t know how to do something it was requested to do. Apparently OpenSSL is supposed to then detect this error and conclude ā€œOK, I guess this CPU canā€™t do thatā€, but for some reason because of some logic error or incomplete research about platforms or whatever, it apparently wasnā€™t actually expecting to receive the error at this particular point in the test (for example, it might have assumed that all CPUs of a certain kind can definitely do a certain thing, yet maybe some of them canā€™t?).

According to the first ddg result (the launchpad bug), this may be gdb catching an expected SIGILL before openssl can.

Run gdb again as before, but after running bt to get a backtrace, run continue to try and keep going. When it catches another signal, run bt and then continue again, and repeat until it will go no further or it repeatedly crashes in a loop in the same function(s). Then paste the full log from all that here.

Hopefully that will reveal a more useful error.

1 Like

thank you guys for the support, i do appreciate it

here is the log from continuing

(gdb) continue
Continuing.

Program received signal SIGILL, Illegal instruction.
_armv7_tick () at armv4cpuid.S:17
17	armv4cpuid.S: No such file or directory.
(gdb) continue
Continuing.

Program received signal SIGILL, Illegal instruction.
0xb63eff84 in ?? () from /usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_openssl.arm-linux-gnueabihf.so
(gdb) continue
Continuing.

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.
(gdb) continue
The program is not being run.
(gdb)

OK, it seems like we want you to also install the debug version of python-crypto, which might be called python-crypto-dbg, unless they also have an :armhf at the end of this package name too.

i installed the python-crypto-dbg package, there didnā€™t seem to be a :armhf version available

i ran

sudo gdb python
(gdb) run /usr/bin/certbot --version

and it returned 149 lines of output before returning

(gdb)
(gdb) continue
Continuing.

Program received signal SIGILL, Illegal instruction.
_armv7_tick () at armv4cpuid.S:17
17	armv4cpuid.S: No such file or directory.
(gdb) continue
Continuing.

Program received signal SIGILL, Illegal instruction.
0xb63eff84 in ?? () from /usr/lib/python2.7/dist-packages/cryptography/hazmat/bindings/_openssl.arm-linux-gnueabihf.so
(gdb) continue
Continuing.

Program terminated with signal SIGILL, Illegal instruction.
The program no longer exists.
(gdb) continue
The program is not being run.
(gdb) 

do you need me to paste the 149 lines from the run command?

Unfortunately, it looks like my suggested package didnā€™t install a debug version of that library.

@bmw, can I get your opinion about this? It looks like a weird ARM-specific problem that now seems to be somewhere in the python-crypto bindings. :frowning:

i installed the debug version listed below

pi@raspberrypi:~ $ apt-cache search python-crypto
bittornado - bittorrent client (and tracker) with console and curses interfaces
bittornado-gui - bittorrent client with GUI interface
python-crypto - cryptographic algorithms and protocols for Python
python-crypto-dbg - cryptographic algorithms and protocols for Python (debug extension)
python-crypto-doc - cryptographic algorithms and protocols for Python (documentation)
python-cryptography - Python library exposing cryptographic recipes and primitives (Python 2)
python-cryptography-doc - Python library exposing cryptographic recipes and primitives (documentation)
python-cryptography-vectors - Test vectors for python-cryptography (Python 2)
python3-cryptography-vectors - Test vectors for python-cryptography (Python 3)

Oh, I got mixed up here between python-crypto and python-cryptography, which are two different libraries. However, python-cryptography doesnā€™t seem to have a debug version of the package.

Did you install certbot from jessie-backports?

Debian Backports armhf is not compatible with the CPU used in the Raspberry Pi 1. Enabling it would install packages that would crash like this.

It might be possible to use certbot-auto but itā€™s possible that it could try to install packages from backports too. (Iā€™m not sure.)

1 Like

i did install it with jessie-backports. if true, it seems this is my issue.

would upgrading to a newer pi fix my problem? or is the problem with the arm cpus on all pis?

Raspbian is an unofficial rebuild of Debian that only exists because the official ARM builds of Debian only support ARMv7, while the original Raspberry Pi version 1 used an ARMv6 processor. The Raspberry Pi v2 uses an ARMv7 processor and the v3 uses an ARMv8/ARM64 processor, both of which are compatible with official Debian armhf. But Raspbian is still itā€™s own thing and deviates from Debian in certain ways, so even though Debian backports is compatible with the processor in the later Pis it is still officially unsupported.

That being said, lots of people on this forum use the certbot from backports on their later model Pis quite successfully, so that is an option. :smile:

I looked at the certbot code and while certbot-auto does need backports for older versions of Debian it does not need it for Jessie. So you should be able to uninstall all the packages from backports and install certbot-auto on your rpi1 instead. It is important to make sure all the backports packages get uninstalled because even one dependency could get used by certbot-auto and lead to crashing.

thank you for the information

i think as an academic exercise ill try to get it running on the rpi1, but ultimately i am due for an upgradeā€¦ ill order a newer rpiv3 and reinstall my webserver on it - it will be good practice!

thanks for your help, if i have any more problems i wonā€™t hesitate to ask

1 Like

Huh, I didn't know about this kind of issue. Thanks for pointing this out and saving a lot of further investigation. Is there a sensible way that operating systems could handle this and tell users about this problem rather than afflicting random processes with SIGILL?

@schoen The arm version is encoded into binaries and can be retrieved with e.g. readelf -a /path/to/binary | grep Tag_CPU_arch. The kernel or ld-linux.so could possibly check that and abort before executing. Or gcc/clang/ld could add some code to every binary that aborts if (current_arch_version < build_arch_version) (similar to how all Windows binaries detect and warn when they are executed under MS-DOS).

At the very least a dpkg postinstall script could examine just installed binaries and warn about them.

Raspbian really should have used an architecture string besides armhf since they are not compatible with the Debian architecture of that name. Then dpkg would refuse to install incompatible packages and we wouldnā€™t be in this mess. :slightly_frowning_face:

The fact that package managers prevent this pretty much everywhere but Raspbian (and only because they lie to dpkg) is probably why more attention hasnā€™t been paid to it.

@dirtyelf One more option I forgot last night: Raspbian 9 Stretch was just released the other day. certbot was included with the stretch release, so if you can upgrade or reinstall, it should be installable right away.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.