Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

1.5 cross-compiled to mips32 openwrt 18.06 illegal instruction WITH_SYS_TREE=yes #1209

Closed
rngadam opened this issue Mar 25, 2019 · 21 comments
Closed

1.5 cross-compiled to mips32 openwrt 18.06 illegal instruction WITH_SYS_TREE=yes #1209

rngadam opened this issue Mar 25, 2019 · 21 comments

Comments

@rngadam
Copy link

rngadam commented Mar 25, 2019 •
edited

built from source, tried disabling most things:

../microrouter-dockcross/dockcross-linux-mips make \
	WITH_WEBSOCKETS=no \
	WITH_SRV=no \
	WITH_UUID=no \
	WITH_TLS=no \
	WITH_TLS_PSK=no \
	WITH_THREADING=no \
	WITH_DOCS=no \
	WITH_SHARED_LIBRARIES=no \
	WITH_EPOLL=no \
	WITH_SOCKS=no \
	WITH_DOCS=no \
	WITH_EC=no \
	WITH_BRIDGE=no \
	CFLAGS="-mips32 -DDEBUG -g" \
	LDFLAGS="-Wl,-dynamic-linker,/lib/ld-musl-mips-sf.so.1"

crashes on start:

./mosquitto -v
1553474137: mosquitto version 1.5.8 starting
1553474137: Using default config.
1553474137: Opening ipv4 listen socket on port 1883.
1553474137: Opening ipv6 listen socket on port 1883.
Illegal instruction

platform info

uname

root@GL-USB150:~# uname -a
Linux GL-USB150 4.9.120 #0 Thu Aug 16 07:51:15 2018 mips GNU/Linux

openwrt

root@GL-USB150:~# cat /etc/openwrt_release 
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='18.06.1'
DISTRIB_REVISION='r7258-5eb055306f'
DISTRIB_TARGET='ar71xx/generic'
DISTRIB_ARCH='mips_24kc'
DISTRIB_DESCRIPTION='OpenWrt 18.06.1 r7258-5eb055306f'
DISTRIB_TAINTS='busybox'

cpuinfo

root@GL-USB150:~# cat /proc/cpuinfo 
system type		: Atheros AR9330 rev 1
machine			: GL.iNet GL-USB150
processor		: 0
cpu model		: MIPS 24Kc V7.4
BogoMIPS		: 265.42
wait instruction	: yes
microsecond timers	: yes
tlb_entries		: 16
extra interrupt vector	: yes
hardware watchpoint	: yes, count: 4, address/irw mask: [0x0ffc, 0x0ffc, 0x0ffb, 0x0ffb]
isa			: mips1 mips2 mips32r1 mips32r2
ASEs implemented	: mips16
shadow register sets	: 1
kscratch registers	: 0
package			: 0
core			: 0
VCED exceptions		: not available
VCEI exceptions		: not available
@rngadam
Copy link
Author

rngadam commented Mar 25, 2019

tried downgrading to tag v1.5 (there's an opkg for that version that works):

/usr/xcc/mips-unknown-linux-gnu/bin/mips-unknown-linux-gnu-gcc -mips32 -DDEBUG -g  -I. -I.. -I../lib  -DVERSION="\"1.5\"" -DWITH_BROKER -DWITH_PERSISTENCE -DWITH_MEMORY_TRACKING -DWITH_SYS_TREE -c conf.c -o conf.o
conf.c: In function 'config__read_file_core':
conf.c:1293:12: error: 'i' undeclared (first use in this function)
        for(i=0; i<config->listener_count; i++){

@rngadam
Copy link
Author

rngadam commented Mar 25, 2019

I went a bit further with disabling things and got a working mosquitto:

../microrouter-dockcross/dockcross-linux-mips make \
	WITH_WEBSOCKETS=no \
	WITH_SRV=no \
	WITH_UUID=no \
	WITH_TLS=no \
	WITH_TLS_PSK=no \
	WITH_THREADING=no \
	WITH_DOCS=no \
	WITH_SHARED_LIBRARIES=no \
	WITH_EPOLL=no \
	WITH_SOCKS=no \
	WITH_DOCS=no \
	WITH_EC=no \
	WITH_BRIDGE=no \
	WITH_BROKER=no \
	WITH_PERSISTENCE=no \
	WITH_MEMORY_TRACKING=no \
	WITH_SYS_TREE=no \
	CFLAGS="-mips32 -DDEBUG -g" \
	LDFLAGS="-Wl,-dynamic-linker,/lib/ld-musl-mips-sf.so.1"

not sure yet what disabled feature made it work.

@rngadam
Copy link
Author

rngadam commented Mar 25, 2019

version with websockets working:

root@GL-USB150:~#  ./mosquitto -c mosquitto-simple.conf 
1553475600: mosquitto version 1.5.8 starting
1553475600: Config loaded from mosquitto-simple.conf.
1553475600: Opening websockets listen socket on port 9001.
1553475600: Opening ipv4 listen socket on port 1883.
1553475600: Opening ipv6 listen socket on port 1883.
1553475600: Warning: Mosquitto should not be run as root/administrator.

@ralight
Copy link
Contributor

ralight commented Mar 25, 2019

tried downgrading to tag v1.5 (there's an opkg for that version that works):

conf.c:1293:12: error: 'i' undeclared (first use in this function)

That is caused by WITH_BRIDGE=no. Yes it should work, and was fixed a while back.

Have you figured out the part that was causing the problem?

@karlp
Copy link
Contributor

karlp commented Mar 25, 2019

almost definitely a bad compiler for your target, or not enough/correct compiler options for your target/compiler pair. Try disabling mips16 generation if that's an option, it's often buggy. mosquitto is happily getting compiled for lots andlots of mips variants as part of the openwrt project.

dockcross seems to be only targetting mips hard float too, which won't work on all targets

@rngadam
Copy link
Author

rngadam commented Mar 25, 2019

@karlp I wouldn't think it's mips16 as the compiler has the -mips32 switch. and if the compiler was wrong, I would expect it to fail completely in all cases, not selectively based on features enabled like now?

@ralight As I mentioned, I've been able to get to a working version by disabling all features I could except for the one I really wanted (websockets). It has to be one of these:

	WITH_BROKER=no \
	WITH_PERSISTENCE=no \
	WITH_MEMORY_TRACKING=no \
	WITH_SYS_TREE=no \

Most likely WITH_PERSISTENCE? perhaps trying to write somewhere it's not supposed to?

My next steps should be to get a gdb installed and compile with debugging so I can figure out precisely which line fails. the hardware manufacturer repo does not have gdb as part of its openwrt 18.06 packages so not sure how to proceed yet (https://www.gl-inet.com/products/gl-usb150/) except to also cross-compile gdb...

@rngadam rngadam changed the title 1.5.8 cross-compiled to mips32 openwrt 18.06 illegal instruction 1.5 cross-compiled to mips32 openwrt 18.06 illegal instruction Mar 25, 2019
@ralight
Copy link
Contributor

ralight commented Mar 26, 2019

Karl is right though, this is highly unlikely to be anything other than a compilation problem. SIGILL occurs if the CPU instruction is invalid, which isn't something that I have any control over, it's just the compiler.

@karlp
Copy link
Contributor

karlp commented Mar 26, 2019

-mips32 just means "Equivalent to -march=mips32." it's completely orthogonal to "-mips16
-mno-mips16
Generate (do not generate) MIPS16 code. If GCC is targeting a MIPS32 or MIPS64 architecture, it makes use of the MIPS16e ASE." I don't know whether it defaults on/off for your combinations, just that it's an area I've seen a lot of failures with wrt mips building in int he openwrt environment. (It's so prevalent, that there's a special flag to ensure that mips16 generation is disabled for certain projects)

Is there any reason you're trying to use some magical docker cross compile thing instead of the openwrt buildroot that your platform expects/provides? gl.inet devices are supported in openwrt upstream, and they're regarded as a "good" vendor (so no vendor tree required), is there any reason you can't just use the plain 18.06 release, with the packaged version of mosquitto as is? (which is 1.5.7 at present, I'm only just back from holidays and haven't gotten to updating it yet)

@karlp
Copy link
Contributor

karlp commented Mar 26, 2019

THe bottom of http:https://downloads.openwrt.org/releases/18.06.2/targets/ar71xx/generic/ has downloads for the imagebuilder and the SDK, which will give you a much better chance of successfully building packages for your target. downloads.openwrt.org has historical releases too, if you want to find one that more closely matches what you're running. I've just linked the latest release.

@rngadam
Copy link
Author

rngadam commented Mar 27, 2019

@karlp my reason for building my own mosquitto is because the version that comes packaged does not support mqtt to websockets which is the main functionality I require.

I'll give a try to -mno-mips16

I don't have any idea how to build stuff using openwrt process. I was excited at even being able to do it using the cross-docker (much easier than last time I tried to cross-compile anything) so I'm not sure where to start with the openwrt build process.

https://openwrt.org/docs/guide-user/additional-software/imagebuilder looks way more involved than single shot compiles. I don't really want to build whole images, just one or two pieces of software I care about.

Although being able to build our own packages would be great, especially if we delve deeper into deploying some of our logic to microrouter.

@rngadam
Copy link
Author

rngadam commented Mar 27, 2019

-mno-mips16 still results in an executable that fails with Illegal instruction.

by process of elimination, I got it down to this flag: WITH_SYS_TREE=yes

I ended up with this configuration setup:

../microrouter-dockcross/dockcross-linux-mips make \
        WITH_WEBSOCKETS=yes \
        WITH_TLS=no \
        WITH_UUID=no \
        WITH_DOCS=no \
        WITH_SHARED_LIBRARIES=no \
        WITH_TLS_PSK=no \
        WITH_SYS_TREE=no \
        CFLAGS="-mips32 -I../libwebsockets/include/ -I./" \
        LDFLAGS="-Wl,-dynamic-linker,/lib/ld-musl-mips-sf.so.1 -L/work/libwebsockets/lib -lwebsockets"

...which runs fine.

@rngadam rngadam changed the title 1.5 cross-compiled to mips32 openwrt 18.06 illegal instruction 1.5 cross-compiled to mips32 openwrt 18.06 illegal instruction WITH_SYS_TREE=yes Mar 27, 2019
@ralight
Copy link
Contributor

ralight commented Mar 27, 2019

Could you try compiling and testing this example program? You'll need to rename it to .c, then compile with e.g. gcc exp.c -o exp -lm

exp.txt

@karlp
Copy link
Contributor

karlp commented Mar 27, 2019

The version that comes packaged does support websockets. I package it. I use it daily. But ok.

You don't have to use the image builder, I just listed that as an option. But use the SDK at least. You definitely have a problem with your cross compiler setup. Perhaps just https://openwrt.org/docs/guide-developer/crosscompile is enough if you really want to ignore things and do it by hand.

I think you're still missing the core that the docker mips toolchain is targetting hardfloat.

fwiw, here's the compiler options passed by the openwrt 1806 buildroot.

AR="mips-openwrt-linux-musl-gcc-ar" AS="ccache_cc -c -Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -iremap/home/karlp/src/smartgate_firmware/openwrt/build_dir/target-mips_24kc_musl/mosquitto-ssl/mosquitto-1.5.8:mosquitto-1.5.8 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro" LD=mips-openwrt-linux-musl-ld NM="mips-openwrt-linux-musl-gcc-nm" CC="ccache_cc" GCC="ccache_cc" CXX="ccache_cxx" RANLIB="mips-openwrt-linux-musl-gcc-ranlib" STRIP=mips-openwrt-linux-musl-strip OBJCOPY=mips-openwrt-linux-musl-objcopy OBJDUMP=mips-openwrt-linux-musl-objdump SIZE=mips-openwrt-linux-musl-size CFLAGS="-Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -iremap/home/karlp/src/smartgate_firmware/openwrt/build_dir/target-mips_24kc_musl/mosquitto-ssl/mosquitto-1.5.8:mosquitto-1.5.8 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CXXFLAGS="-Os -pipe -mno-branch-likely -mips32r2 -mtune=24kc -fno-caller-saves -fno-plt -fhonour-copts -Wno-error=unused-but-set-variable -Wno-error=unused-result -msoft-float -mips16 -minterlink-mips16 -iremap/home/karlp/src/smartgate_firmware/openwrt/build_dir/target-mips_24kc_musl/mosquitto-ssl/mosquitto-1.5.8:mosquitto-1.5.8 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro " CPPFLAGS="-I/home/karlp/src/smartgate_firmware/openwrt/staging_dir/target-mips_24kc_musl/usr/include -I/home/karlp/src/smartgate_firmware/openwrt/staging_dir/target-mips_24kc_musl/include -I/home/karlp/src/smartgate_firmware/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.3.0_musl/usr/include -I/home/karlp/src/smartgate_firmware/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.3.0_musl/include/fortify -I/home/karlp/src/smartgate_firmware/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.3.0_musl/include " LDFLAGS="-L/home/karlp/src/smartgate_firmware/openwrt/staging_dir/target-mips_24kc_musl/usr/lib -L/home/karlp/src/smartgate_firmware/openwrt/staging_dir/target-mips_24kc_musl/lib -L/home/karlp/src/smartgate_firmware/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.3.0_musl/usr/lib -L/home/karlp/src/smartgate_firmware/openwrt/staging_dir/toolchain-mips_24kc_gcc-7.3.0_musl/lib -znow -zrelro

1.5.8 is getting updated for openwrt "right now" so it should even be available as a binary for you tomorrow morning.

@rngadam
Copy link
Author

rngadam commented Mar 27, 2019

Could you try compiling and testing this example program? You'll need to rename it to .c, then compile with e.g. gcc exp.c -o exp -lm

exp.txt

./dockcross-linux-mips bash -c '$CC -mips32 -Wl,-dynamic-linker,/lib/ld-musl-mips-sf.so.1 exp.c -o exp -lm'

and crashes with Illegal instructions. I figured trying to compile for soft-float but that fails:

./dockcross-linux-mips bash -c '$CC -msoft-float -mips32 -Wl,-dynamic-linker,/lib/ld-musl-mips-sf.so.1 exp.c -o exp -lm'
In file included from /usr/xcc/mips-unknown-linux-gnu/mips-unknown-linux-gnu/sysroot/usr/include/features.h:434:0,
                 from /usr/xcc/mips-unknown-linux-gnu/mips-unknown-linux-gnu/sysroot/usr/include/bits/libc-header-start.h:33,
                 from /usr/xcc/mips-unknown-linux-gnu/mips-unknown-linux-gnu/sysroot/usr/include/stdio.h:28,
                 from exp.c:1:
/usr/xcc/mips-unknown-linux-gnu/mips-unknown-linux-gnu/sysroot/usr/include/gnu/stubs.h:8:33: fatal error: gnu/stubs-o32_soft.h: No such file or directory
 # include <gnu/stubs-o32_soft.h>

@rngadam
Copy link
Author

rngadam commented Mar 27, 2019

The version that comes packaged does support websockets. I package it. I use it daily. But ok.

seems to be a much older package included in the openwrt in my image:

Package mosquitto-nossl (1.5-3) installed in root is up to date.

based off this:

https://github.com/gl-inet/openwrt

filed an issue with dockcross since -msoft-float seems unsupported:

dockcross/dockcross#315

@karlp
Copy link
Contributor

karlp commented Mar 27, 2019

yes, as I said, that docker thing is very explicitly just targetting hardfloat. if you're working on openwrt, I strongly suggest you get used to it's tooling, you will save yourself (and others!) much heartache and time. Using a busted compiler will give you very random results.

Also, unless you have a special need to use gl-inets tree, just use openwrt itself. All their hardware is normally supported upstream, they probably only have a tree for initial contributions. If you insist on using third party trees you will very quickly be on your own when it comes to getting support for them.

And no, the -nossl build doesn't include websockets support.

@ralight
Copy link
Contributor

ralight commented Mar 27, 2019

Ok, so if the test program which just uses doubles, fabs() and exp() fails then it's not a mosquitto problem so this issue can be closed?

@rngadam
Copy link
Author

rngadam commented Mar 27, 2019 •
edited

I'm going to follow up in two ways:

@karlp
Copy link
Contributor

karlp commented Mar 27, 2019

yes, this can be closed. nothing to do with mosquitto.

(fwiw, mosquitto 1.5.8 has been pushed to both the 18.06 and master branches of openwrt though now)

@rngadam
Copy link
Author

rngadam commented Mar 28, 2019

@karlp agreed. thanks for your help.

@rngadam rngadam closed this as completed Mar 28, 2019
@MattLok
Copy link

MattLok commented May 14, 2019

@rngadam I believe I'm building on the same gl device as you and have been running into a ton of issues with soft/hardfloat. Did including the http:https://downloads.openwrt.org/releases/18.06.2/targets/ar71xx/generic/openwrt-sdk-18.06.2-ar71xx-generic_gcc-7.3.0_musl.Linux-x86_64.tar.xz allow you to get past the following error:
In file included from /usr/mips-linux-gnu/include/features.h:391:0, from /usr/mips-linux-gnu/include/stdlib.h:24, from _cgo_export.c:3: /usr/mips-linux-gnu/include/gnu/stubs.h:8:33: fatal error: gnu/stubs-o32_soft.h: No such file or directory
Thanks any help would be appreciated.

@lock lock bot locked as resolved and limited conversation to collaborators Aug 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ralight @karlp @rngadam @MattLok