regression of `rv-virt` kernel builds #12275

yf13 · 2024-05-02T22:12:28Z

On Ubuntu 22.04 with qemu-system-riscv64/32 (version: 6.2+dfsg-2ubuntu6.16) targets and gcc-riscv64-unknown-elf toolchain (GCC 10.2) and latest master branch.

The build from rv-virt/nsh and rv-virt/nsh64 can boot.
The build from both rv-virt/knsh32 and rv-virt/knsh64 can't boot. The ABC string is invibsile from console.

The command line to run nuttx qemu-system-riscv64 -M virt,aclint=on -semihosting -nographic -bios nuttx and it has been working well until I pulled latest master. Looks like this booting mode no longer works after patch #12178.

By using qemu-system-riscv64 -M virt,aclint=on -semihosting -nographic -kernel nuttx to run nuttx with default OpenSBI (v0.9) for rv64, got the exception riscv_exception: EXCEPTION: Illegal instruction. MCAUSE: 0000000000000002, EPC: 000000008020902c, MTVAL: 0000000014d79073.

The call stack looks like:

(gdb) bt
#0  0x000000008020902c in riscv_write_stime (value=18446744073709551615) at /home/yf/Projects/Nuttx/nuttx/arch/risc-v/src/common/riscv_mtimer.c:135
#1  riscv_mtimer_set_mtimecmp (value=18446744073709551615, priv=0x804072c0) at /home/yf/Projects/Nuttx/nuttx/arch/risc-v/src/common/riscv_mtimer.c:156
#2  riscv_mtimer_initialize (mtime=mtime@entry=33603576, mtimecmp=mtimecmp@entry=33570816, irq=irq@entry=21, freq=freq@entry=10000000) at /home/yf/Projects/Nuttx/nuttx/arch/risc-v/src/common/riscv_mtimer.c:349
#3  0x0000000080208a3e in up_timer_initialize () at /home/yf/Projects/Nuttx/nuttx/arch/risc-v/src/qemu-rv/qemu_rv_timerisr.c:67
#4  0x0000000080204670 in clock_initialize () at /home/yf/Projects/Nuttx/nuttx/sched/clock/clock_initialize.c:212
#5  0x00000000802049d8 in nx_start () at /home/yf/Projects/Nuttx/nuttx/sched/init/nx_start.c:671
#6  0x00000000802006e6 in qemu_rv_start (mhartid=<optimized out>, dtb=<optimized out>) at /home/yf/Projects/Nuttx/nuttx/arch/risc-v/src/qemu-rv/qemu_rv_start.c:180
#7  0x000000008020004a in _stext () at /home/yf/Projects/Nuttx/nuttx/arch/risc-v/src/qemu-rv/qemu_rv_head.S:74

maybe the RV_SSTC extension is unavailable in QEMU 6.2 shipped with Ubuntu 22.04?

Haven't tried -kernel nuttx for rv32 as the default firmware seems lacking in qemu-system-data package on Ubuntu 22.04.

The text was updated successfully, but these errors were encountered:

lupyuen · 2024-05-04T01:05:58Z

@inochisa Any idea why rv-virt:knsh64 crashes on startup? I tested on macOS with QEMU 7.0.0

https://gist.github.com/lupyuen/ce07dd0c21118cebe8e83a8d61439112

→ qemu-system-riscv64 -nographic -semihosting -M virt,aclint=on -cpu rv64 -kernel nuttx

OpenSBI v1.0
...
ABC[    0.000000] riscv_exception: EXCEPTION: Illegal instruction. MCAUSE: 0000000000000002, EPC: 000000008020853a, MTVAL: 0000000014d79073
[    0.000000] riscv_exception: PANIC!!! Exception = 0000000000000002
[    0.000000] _assert: Current Version: NuttX  12.5.1-RC0 0e67a79 May  4 2024 08:54:24 risc-v
[    0.000000] _assert: Assertion failed panic: at file: common/riscv_exception.c:119 task: Idle_Task process: Kernel 0x80200dec
[    0.000000] up_dump_register: EPC: 000000008020853a
[    0.000000] up_dump_register: A0: 00000000804072c0 A1: 0000000000000000 A2: 0000000000000000 A3: 0000000000000001
[    0.000000] up_dump_register: A4: 0000000000000000 A5: ffffffffffffffff A6: 0000000000000000 A7: fffffffffffffff8
[    0.000000] up_dump_register: T0: 0000000080200050 T1: 0000000000000007 T2: 0000000000001000 T3: 00000000804072f8
[    0.000000] up_dump_register: T4: 00000000804072f0 T5: 0000000000000027 T6: 0000000000000001
[    0.000000] up_dump_register: S0: 00000000804072c0 S1: 0000000000000015 S2: 0000000000989680 S3: 0000000002004000
[    0.000000] up_dump_register: S4: 000000000200bff8 S5: 0000000000000000 S6: 8000000a00006800 S7: 000000000000007f
[    0.000000] up_dump_register: S8: 0000000080017038 S9: 0000000080038ea0 S10: 0000000000000000 S11: 0000000000000000
[    0.000000] up_dump_register: SP: 0000000080406b50 FP: 00000000804072c0 TP: 000000008003e000 RA: 000000008020851e
[    0.000000] dump_stack: User Stack:
[    0.000000] dump_stack:   base: 0x80406010
[    0.000000] dump_stack:   size: 00003056
[    0.000000] dump_stack:     sp: 0x80406b50
[    0.000000] stack_dump: 0x80406b30: 80401b30 00000000 00000038 00000000 80400af8 00000000 8020851e 00000000
[    0.000000] stack_dump: 0x80406b50: 80400c70 00000000 80400ae8 00000000 80401b30 00000000 80401b2c 00000000
[    0.000000] stack_dump: 0x80406b70: 80400af8 00000000 8020835e 00000000 80400af8 00000000 802019b6 00000000
[    0.000000] stack_dump: 0x80406b90: 80400af8 00000000 80200fd8 00000000 80600000 00000000 00400000 00000000
[    0.000000] stack_dump: 0x80406bb0: 00000000 00000000 87000000 00000000 80200000 00000000 80406000 00000000
[    0.000000] stack_dump: 0x80406bd0: 00000000 00000000 80200706 00000000 00000000 00000000 00000001 00000000
[    0.000000] stack_dump: 0x80406bf0: 8003df30 00000000 8020004a 00000000 00000000 00000000 00000000 00000000
[    0.000000] dump_tasks:    PID GROUP PRI POLICY   TYPE    NPX STATE   EVENT      SIGMASK          STACKBASE  STACKSIZE      USED   FILLED    COMMAND
[    0.000000] dump_tasks:   ----   --- --- -------- ------- --- ------- ---------- ---------------- 0x80400a80      2048      1016    49.6%    irq
[    0.000000] dump_task:       0     0   0 FIFO     Kthread - Running            0000000000000000 0x80406010      3056       704    23.0%    Idle_Task
QEMU: Terminated

→ qemu-system-riscv64 --version
QEMU emulator version 7.0.0
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

inochisa · 2024-05-04T01:11:58Z

@yf13 @lupyuen Can you try opensbi v1.1. I think this may be caused by SSTC.

lupyuen · 2024-05-04T01:13:06Z

The RISC-V Exception:

EXCEPTION: Illegal instruction. MCAUSE: 0000000000000002, EPC: 000000008020853a, MTVAL: 0000000014d79073

Points to this code:

riscv_write_stime():
/private/tmp/nuttx/nuttx/arch/risc-v/src/common/riscv_mtimer.c:135
  WRITE_CSR(CSR_STIMECMP, value);
    80208538:	57fd                	li	a5,-1
    8020853a:	14d79073          	csrw	0x14d,a5

So CSR_STIMECMP is probably missing. How do we get Opensbi v1.1? Is it in the NuttX Doc?

lupyuen · 2024-05-04T01:17:02Z

@inochisa STIMECMP needs QEMU 7.2 I think?

qemu/qemu@e46e262

inochisa · 2024-05-04T01:18:30Z

The RISC-V Exception:

EXCEPTION: Illegal instruction. MCAUSE: 0000000000000002, EPC: 000000008020853a, MTVAL: 0000000014d79073

Points to this code:

riscv_write_stime():
/private/tmp/nuttx/nuttx/arch/risc-v/src/common/riscv_mtimer.c:135
  WRITE_CSR(CSR_STIMECMP, value);
    80208538:	57fd                	li	a5,-1
    8020853a:	14d79073          	csrw	0x14d,a5

So CSR_STIMECMP is probably missing. How do we get Opensbi v1.1? Is it in the NuttX Doc?

For now you can build from source, the newer qemu should ship with possible opensbi.

I will write a document for it. Or just disable the SSTC as temporary solution.

inochisa · 2024-05-04T01:19:43Z

@inochisa STIMECMP needs QEMU 7.2 I think?

qemu/qemu@e46e262

I think you are right. My system is arch and always has newest qemu. So I forgot this issue. Thanks.

lupyuen · 2024-05-04T01:21:03Z

OK lemme upgrade QEMU and retest. If it works, let's update the NuttX Docs for QEMU Version thanks!

@yf13 Please try QEMU 7.2 or newer thanks!

This fixes `rv-virt/knsh64` booting issue apache#12275. Signed-off-by: Yanfeng Liu <[email protected]>

lupyuen · 2024-05-04T01:56:38Z

Tested OK on QEMU 7.2.9: https://gist.github.com/lupyuen/b2520054b5ecc12a2f33909aaa46121c

@inochisa Could you update the NuttX Doc to say that QEMU 7.2.9 or later is required, to support RISC-V "Sstc" Extension. Thank you so much!

inochisa · 2024-05-04T01:58:59Z

Tested OK on QEMU 7.2.9: https://gist.github.com/lupyuen/b2520054b5ecc12a2f33909aaa46121c

@inochisa Could you update the NuttX Doc to say that QEMU 7.2.9 or later is required, to support RISC-V "Sstc" Extension. Thank you so much!

Good, In addition, I will add some comments for users want to use its own OpenSBI.

This patch can boot `rv-virt/knsh64` reported in issue apache#12275 with qemu 6.2 target in Ubuntu 22.04. Signed-off-by: Yanfeng Liu <[email protected]>

yf13 · 2024-05-04T03:27:56Z

@lupyuen and @inochisa, I sent patch #12279 to disable SSTC as I am using stock QEMU (ver 6.2 with SBI v0.9) on Ubuntu 22.04 most of the time, I believe this is easier for people to start with.

However, there is remaining issue even if SSTC is off, the "usleep 5" command doesn't return back to nsh, this indicts that the rv-virt timer related setup logic might be incomplete when SSTC is off?

inochisa · 2024-05-04T04:28:57Z

@lupyuen and @inochisa, I sent patch #12279 to disable SSTC as I am using stock QEMU (ver 6.2 with SBI v0.9) on Ubuntu 22.04 most of the time, I believe this is easier for people to start with.

I do not think it is a good idea. QEMU for risc-v is always active developed. If we stick on old version, we will lost most of the new features. In addition, 7.2.9 is not too new for most distribution.

However, there is remaining issue even if SSTC is off, the "usleep 5" command doesn't return back to nsh, this indicts that the rv-virt timer related setup logic might be incomplete when SSTC is off?

This does worth exploring. If SSTC is off, the timer should go the old path. I do not change any old code when adding SSTC, so it is more like another problem

lupyuen · 2024-05-04T04:39:42Z

@inochisa Is it possible that this is causing the Timer Issue with SSTC Disabled? #12178 (comment)

Could you help to do a Regression Test based on your patch: If SSTC is Disabled, will usleep work? Thanks

inochisa · 2024-05-04T04:41:37Z

@inochisa Is it possible that this is causing the Timer Issue with SSTC Disabled? #12178 (comment)

Could you help to do a Regression Test based on your patch: If SSTC is Disabled, will usleep work? Thanks

This is weird, I have tested on my machine. I can confirm rv-virt:knsh64 works on QEMU 9.0.0 without SSTC.

PS: with the #12178 patch.

lupyuen · 2024-05-04T04:46:04Z

@inochisa Do you mean that if ARCH_RV_EXT_SSTC=N, then usleep works OK on rv-virt:knsh64?

If so, I will test ARCH_RV_EXT_SSTC=N with usleep on my old version of QEMU. Thanks

inochisa · 2024-05-04T04:46:51Z

@inochisa Do you mean that if ARCH_RV_EXT_SSTC=N, then usleep works OK on rv-virt:knsh64?

Yes.

If so, I will test ARCH_RV_EXT_SSTC=N with usleep on my old version of QEMU. Thanks

This worth a try. Thanks.

lupyuen · 2024-05-04T05:02:14Z

@inochisa Sorry ARCH_RV_EXT_SSTC=N failed to build with rv-virt:knsh64. Did I miss something?

https://gist.github.com/lupyuen/749f8235603781c29a351c066cd2fcf9

$ git clone https://github.com/inochisa/nuttx --branch qemu-s-mode
$ git clone https://github.com/apache/nuttx-apps apps
$ cd nuttx && ./tools/configure.sh rv-virt:knsh64
$ make menuconfig
## Disable ARCH_RV_EXT_SSTC
$ grep SSTC .config

$ make

./chip/qemu_rv_start.c:54:4: error: #error "Target requires kernel in S-mode, enable CONFIG_ARCH_USE_S_MODE"
   54 | #  error "Target requires kernel in S-mode, enable CONFIG_ARCH_USE_S_MODE"
      |    ^~~~~
ERROR: riscv64-unknown-elf-gcc failed: 1
       command: riscv64-unknown-elf-gcc -MT ./qemu_rv_start.o  -M '-fno-common' '-Wall' '-Wstrict-prototypes' '-Wshadow' '-Wundef' '-Wno-attributes' '-Wno-unknown-pragmas' '-Wno-psabi' '-Os' '-fno-strict-aliasing' '-fomit-frame-pointer' '-ffunction-sections' '-fdata-sections' '-nostdlib' '-g' '-mcmodel=medany' '-march=rv64imafdc' '-mabi=lp64d' '-isystem' '/private/tmp/qemu-s-mode/nuttx/include' '-D__NuttX__' '-D__KERNEL__' '-pipe' '-I' '/private/tmp/qemu-s-mode/nuttx/arch/risc-v/src/chip' '-I' '/private/tmp/qemu-s-mode/nuttx/arch/risc-v/src/common' '-I' '/private/tmp/qemu-s-mode/nuttx/sched' ./chip/qemu_rv_start.c
make[2]: *** [/private/tmp/qemu-s-mode/nuttx/tools/Config.mk:230: qemu_rv_start.ddc] Error 1
make[1]: *** [Makefile:233: .depend] Error 2
make: *** [tools/Unix.mk:620: pass2dep] Error 2

inochisa · 2024-05-04T05:06:17Z

@inochisa Sorry ARCH_RV_EXT_SSTC=N failed to build with rv-virt:knsh64. Did I miss something?

https://gist.github.com/lupyuen/749f8235603781c29a351c066cd2fcf9

$ git clone https://github.com/inochisa/nuttx --branch qemu-s-mode
$ git clone https://github.com/apache/nuttx-apps apps
$ cd nuttx && ./tools/configure.sh rv-virt:knsh64
$ make menuconfig
## Disable ARCH_RV_EXT_SSTC
$ grep SSTC .config

$ make

./chip/qemu_rv_start.c:54:4: error: #error "Target requires kernel in S-mode, enable CONFIG_ARCH_USE_S_MODE"
   54 | #  error "Target requires kernel in S-mode, enable CONFIG_ARCH_USE_S_MODE"
      |    ^~~~~
ERROR: riscv64-unknown-elf-gcc failed: 1
       command: riscv64-unknown-elf-gcc -MT ./qemu_rv_start.o  -M '-fno-common' '-Wall' '-Wstrict-prototypes' '-Wshadow' '-Wundef' '-Wno-attributes' '-Wno-unknown-pragmas' '-Wno-psabi' '-Os' '-fno-strict-aliasing' '-fomit-frame-pointer' '-ffunction-sections' '-fdata-sections' '-nostdlib' '-g' '-mcmodel=medany' '-march=rv64imafdc' '-mabi=lp64d' '-isystem' '/private/tmp/qemu-s-mode/nuttx/include' '-D__NuttX__' '-D__KERNEL__' '-pipe' '-I' '/private/tmp/qemu-s-mode/nuttx/arch/risc-v/src/chip' '-I' '/private/tmp/qemu-s-mode/nuttx/arch/risc-v/src/common' '-I' '/private/tmp/qemu-s-mode/nuttx/sched' ./chip/qemu_rv_start.c
make[2]: *** [/private/tmp/qemu-s-mode/nuttx/tools/Config.mk:230: qemu_rv_start.ddc] Error 1
make[1]: *** [Makefile:233: .depend] Error 2
make: *** [tools/Unix.mk:620: pass2dep] Error 2

I think you disable CONFIG_ARCH_USE_S_MODE by mistake....

grep should give

$ cat .config | grep SSTC
# CONFIG_ARCH_RV_EXT_SSTC is not set

lupyuen · 2024-05-04T05:35:27Z

@inochisa Sorry I goofed.

@yf13 usleep works OK with ARCH_RV_EXT_SSTC=N, rv-virt:knsh64, QEMU 7.0.0 and OpenSBI v1.0.

Could you update to OpenSBI v1.0? I have a hunch that OpenSBI v0.9 might be buggy, we probably shouldn't use it? Thanks!

https://gist.github.com/lupyuen/da195386a4873c572aa26961e4c81d79

→ qemu-system-riscv64 --version
QEMU emulator version 7.0.0
Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

→ qemu-system-riscv64 -nographic -semihosting -M virt,aclint=on -cpu rv64 -kernel nuttx
OpenSBI v1.0
NuttShell (NSH)
nsh> usleep 5
nsh> ostest
...
ostest_main: Exiting with status -1
nsh>

yf13 · 2024-05-04T07:14:24Z

@lupyuen and @inochisa thanks for all the information!

Here I tried opensbi 1.3-1ubuntu0.22.04.2 got via apt instalal opensbi with QEMU 6.2 on Ubuntu 22.04, usleep works now with patch #12279. I don't have other opensbi versions at hand to try but it looks like QEMU 6.2 is fine now.

So it is good that we finally can support the Ubuntu stock QEMU v6.2 this way: qemu-system-riscv64 -M virt,aclint=on -semihosting -kernel nuttx -bios /usr/lib/riscv64-linux-gnu/opensbi/generic/fw_dynamic.bin

lupyuen · 2024-05-04T07:17:39Z

Thanks @yf13 ! Moving ahead, we should mandate OpenSBI v1.0 or later. Version 0.9 sounds kinda sus :-)

yf13 · 2024-05-04T07:42:20Z

@lupyuen and @inochisa, I checked with QEMU 6.2 and OpenSBI v1.0-3ubuntu1 here, it works well. So saying v1.0 in docs should be fine.

This patch can boot `rv-virt/knsh64` and `rv-virt/knsh32` reported in issue apache#12275 with qemu 6.2 target and OpenSBI v1.0 firmware on Ubuntu 22.04 host. Signed-off-by: Yanfeng Liu <[email protected]>

This patch can boot `rv-virt/knsh64` and `rv-virt/knsh32` reported in issue #12275 with qemu 6.2 target and OpenSBI v1.0 firmware on Ubuntu 22.04 host. Signed-off-by: Yanfeng Liu <[email protected]>

xiaoxiang781216 · 2024-05-04T14:54:56Z

@yf13 could you update opensbi to the workable verion:
https://github.com/apache/nuttx/blob/master/arch/risc-v/src/opensbi/Make.defs

yf13 changed the title ~~kernel build can't boot with QEMU on Ubuntu~~ rv-virt kernel builds are brokken May 2, 2024

yf13 changed the title ~~rv-virt kernel builds are brokken~~ rv-virt kernel builds are brokken May 2, 2024

yf13 changed the title ~~rv-virt kernel builds are brokken~~ Regression of rv-virt kernel builds May 2, 2024

yf13 changed the title ~~Regression of rv-virt kernel builds~~ regression of rv-virt kernel builds May 2, 2024

yf13 added a commit to yf13/nuttx that referenced this issue May 4, 2024

riscv/rv-virt: fix knsh64 booting issue

8aef204

This fixes `rv-virt/knsh64` booting issue apache#12275. Signed-off-by: Yanfeng Liu <[email protected]>

yf13 mentioned this issue May 4, 2024

risc-v/qemu: fix kernel build issue 12275 #12279

Merged

inochisa mentioned this issue May 4, 2024

Documentation/rv-virt: add required minimal QEMU version information #12280

Merged

yf13 added a commit to yf13/nuttx that referenced this issue May 4, 2024

riscv/rv-virt: fix knsh64 booting issue

ea9b9c2

This patch can boot `rv-virt/knsh64` reported in issue apache#12275 with qemu 6.2 target in Ubuntu 22.04. Signed-off-by: Yanfeng Liu <[email protected]>

yf13 closed this as completed May 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

regression of `rv-virt` kernel builds #12275

regression of `rv-virt` kernel builds #12275

yf13 commented May 2, 2024 •

edited

Loading

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

lupyuen commented May 4, 2024

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

inochisa commented May 4, 2024 •

edited

Loading

lupyuen commented May 4, 2024

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

yf13 commented May 4, 2024 •

edited

Loading

inochisa commented May 4, 2024

lupyuen commented May 4, 2024

inochisa commented May 4, 2024 •

edited

Loading

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

lupyuen commented May 4, 2024

yf13 commented May 4, 2024

lupyuen commented May 4, 2024

yf13 commented May 4, 2024

xiaoxiang781216 commented May 4, 2024

regression of rv-virt kernel builds #12275

regression of rv-virt kernel builds #12275

Comments

yf13 commented May 2, 2024 • edited Loading

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

lupyuen commented May 4, 2024

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

inochisa commented May 4, 2024 • edited Loading

lupyuen commented May 4, 2024

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

yf13 commented May 4, 2024 • edited Loading

inochisa commented May 4, 2024

lupyuen commented May 4, 2024

inochisa commented May 4, 2024 • edited Loading

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

lupyuen commented May 4, 2024

inochisa commented May 4, 2024

lupyuen commented May 4, 2024

yf13 commented May 4, 2024

lupyuen commented May 4, 2024

yf13 commented May 4, 2024

xiaoxiang781216 commented May 4, 2024

regression of `rv-virt` kernel builds #12275

regression of `rv-virt` kernel builds #12275

yf13 commented May 2, 2024 •

edited

Loading

inochisa commented May 4, 2024 •

edited

Loading

yf13 commented May 4, 2024 •

edited

Loading

inochisa commented May 4, 2024 •

edited

Loading