feat(testing): add basic section-based test declaration support #65

mystor · 2020-02-07T03:38:58Z

This is an outline of a potential section-based approach for declaring tests which supports running tests from across the dependency graph with a single test runner.

May supercede #59

Example Output:

 [INFO] run tests
 [INFO] | test, test.name="basic_alloc", test.module="mycelium_kernel"
 [INFO] | |- mycelium_kernel: vec=[], vec.addr=0x8
 [INFO] | |- mycelium_kernel: vec=[5], vec.addr=0x3fc058
 [INFO] | |- mycelium_kernel: vec=[5, 10], vec.addr=0x3fc048
 [INFO] | test, test.name="wasm_hello_world", test.module="mycelium_kernel"
[TRACE] | | invoke_index, index=0, args=RuntimeArgs([I32(1), I32(0), I32(1), I32(20)])
 [INFO] | | | fd_write, fd=1, iovs=0, iovs_len=1, nwritten=20
 [INFO] | | | |- mycelium_kernel::wasm::wasi: fd_write, buf_slice=[104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100, 10], buf_str=Ok("hello world\n")
 [INFO] | test, test.name="it_works", test.module="mycelium_util::testing"
 [INFO] | |- mycelium_util::testing: I'm running in a test!
 [WARN] |- mycelium_kernel: 3 passed | 0 failed

mystor · 2020-02-07T03:45:55Z

util/src/testing.rs

+ // Introduce an anonymous const to avoid name conflicts. The `#[used]`
+ // will prevent the symbol from being dropped, and `link_section` will
+ // make it visible.
+ const _: () = {


Unfortunately, we can't put this behind a #[cfg(test)], as that cfg flag won't be visible in dependencies. We may need to use RUSTFLAGS="--cfg mycelium_test" (or something like that) to turn on test code for the whole dependency tree.

that makes sense, i would really like to avoid linking these into non-test builds eventually. this is fine for now...

hawkw

this is awesome! i left notes on a few things but it will be very cool to get this merged.

hawkw · 2020-02-07T18:11:09Z

util/src/testing.rs

+/// Get a list of test objects.
+pub fn all_tests() -> &'static [Test] {
+ unsafe {
+ // FIXME: These should probably be `&raw const __start_*`.


is &raw new proposed syntax? i would love a link to this as I've never seen it before!

Yup! It's a way to do field indexing without requiring the object you're taking a reference to to be valid memory. (rfc: rust-lang/rfcs#2582, tracking issue: rust-lang/rust#64490)

hawkw · 2020-02-07T18:21:34Z

util/src/testing.rs

+use core::slice;
+
+/// Internal definition type used for a test.
+pub struct Test {


nit/tioli: since this has all pub fields, should there maybe be a private _p: () or something to stop it from being constructed from anywhere?

It has to be constructed from within the macro, which is why it has all-pub fields, so I can't add a _p: () or the decl_test! macro won't work.

It's technically harmless to construct it anywhere you like, it won't do anything unless it's in the correct section.

ah, nvm, i missed that! seems like it could have a const-fn ctor but that might just be boilerplate?

Yeah, I think that might just end up being boilerplate. I'm inclined to just leave it as-is.

hawkw · 2020-02-07T18:22:09Z

util/src/testing.rs

+use core::mem;
+use core::slice;
+
+/// Internal definition type used for a test.


nit: the test runner in the kernel consumes this, so i am not sure if i would call it "internal" :P

hawkw · 2020-02-07T18:23:09Z

util/src/testing.rs

+ // Introduce an anonymous const to avoid name conflicts. The `#[used]`
+ // will prevent the symbol from being dropped, and `link_section` will
+ // make it visible.
+ const _: () = {


that makes sense, i would really like to avoid linking these into non-test builds eventually. this is fine for now...

util/src/testing.rs

src/lib.rs

hawkw · 2020-02-07T18:28:18Z

src/arch/x86_64.rs

+pub(crate) fn qemu_exit(exit_code: QemuExitCode) {
+ let code = exit_code as u32;
+ unsafe {
+ asm!("out 0xf4, eax" :: "{eax}"(code) :: "intel","volatile");
+ }
+}


can we stick a

#[cfg(test)] qemu_exit(QemuExitCode::Failed);

at the end of arch::x86_64::oops, before the loop? that way, if a test panics/faults, we can report failure rather than hanging. ideally, we would be able to track which test failed, so that panics can be a nice way to fail tests rather than presenting less information, but it would be nice if panics/faults didn't make the CI build just hang

Yes, I was intending to do that, but forgot to do it before pushing. I'll add that.

hawkw · 2020-02-07T18:44:27Z

.github/workflows/rust.yml

+ - name: install qemu
+ run: sudo apt-get update && sudo apt-get install qemu


it would be nice if we could figure out a way to use the github actions cache thingy so we don't have to install qemu over and over...not a blocket though!

I didn't even know that was a thing, so I'm not sure I can help with that :-P
If you have some documentation for how the cache thing works to point me to, let me know.

we can figure this out later!

hawkw · 2020-02-07T18:45:40Z

.github/workflows/rust.yml

+ command: dev-env
+ - name: install qemu
+ run: sudo apt-get update && sudo apt-get install qemu
+ - name: run tests


would be good to have a step that runs any "regular" (i.e. not in qemu) rust tests in crates as well. i think we define some now as of #64, so it would be good for them to run on CI as well.

if this was a separate job, it could run in parallel with the qemu tests.

Yeah, we should probably do that. We'll probably want to make sure to exclude the root mycelium_kernel crate, though.

we might have to just cargo test -p $FOO for every other crate...which is fine, it's just a little flaky since it needs to be manually updated as we add more crates...

I ended up just using more #[cfg(...)] attributes to stub out a dummy fn main() for the mycelium_kernel crate when we're running on the host platform.

hawkw · 2020-02-08T00:09:11Z

clippy seems broken for some reason and i don't know why? but otherwise, this is awesome

hawkw

shipit!

feat(testing): add basic section-based test declaration support

28f00e1

mystor requested a review from hawkw February 7, 2020 03:43

mystor commented Feb 7, 2020

View reviewed changes

hawkw reviewed Feb 7, 2020

View reviewed changes

mystor added 4 commits February 7, 2020 16:28

run qemu_exit in oops handler when running tests

55cc7a7

Get #[test]-style tests passing

80f1d4a

try running host tests in CI

490cecb

Small tidying of doc comments in testing module

cb034ab

mystor added 3 commits February 7, 2020 20:32

Add clippy as an install requirement for clippy check

6c68bf2

rustfmt

a8df89b

more clippy action tweaks

03ef7cd

hawkw approved these changes Feb 8, 2020

View reviewed changes

mystor merged commit 7cf668d into master Feb 8, 2020

mystor deleted the nika/section-testing branch February 8, 2020 15:22

mystor mentioned this pull request Feb 8, 2020

feat(testing): in-qemu tests #59

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(testing): add basic section-based test declaration support #65

feat(testing): add basic section-based test declaration support #65

mystor commented Feb 7, 2020

mystor Feb 7, 2020

hawkw Feb 7, 2020

hawkw left a comment

hawkw Feb 7, 2020

mystor Feb 7, 2020

hawkw Feb 7, 2020

mystor Feb 7, 2020

hawkw Feb 7, 2020

mystor Feb 7, 2020

hawkw Feb 7, 2020

hawkw Feb 7, 2020

hawkw Feb 7, 2020

mystor Feb 7, 2020

hawkw Feb 7, 2020

mystor Feb 7, 2020

hawkw Feb 7, 2020

hawkw Feb 7, 2020

mystor Feb 7, 2020

hawkw Feb 7, 2020

mystor Feb 7, 2020

hawkw commented Feb 8, 2020

hawkw left a comment

		- name: install qemu
		run: sudo apt-get update && sudo apt-get install qemu

feat(testing): add basic section-based test declaration support #65

feat(testing): add basic section-based test declaration support #65

Conversation

mystor commented Feb 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hawkw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hawkw commented Feb 8, 2020

hawkw left a comment

Choose a reason for hiding this comment