Skip to content

Commit

Permalink
Documentation: Add Sanitizer section to RunningTests
Browse files Browse the repository at this point in the history
Also, add link to RunningTests BuildInstructions, and clean up stale
commands in RunningTests to align to current build strategies.
  • Loading branch information
ADKaster authored and linusg committed May 16, 2021
1 parent ac1d87b commit cfdd231
Show file tree
Hide file tree
Showing 2 changed files with 62 additions and 20 deletions.
5 changes: 5 additions & 0 deletions Documentation/BuildInstructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,11 @@ For the changes to take effect, SerenityOS needs to be recompiled and the disk i

To add a package from the ports collection to Serenity, for example curl, go into `Ports/curl/` and run `./package.sh`. The sourcecode for the package will be downloaded and the package will be built. After that, rebuild the disk image. The next time you start Serenity, `curl` will be available.

## Tests

For information on running host and target tests, see [Running Tests](RunningTests.md). The documentation there explains the difference between host tests run with Lagom and
target tests run on SerenityOS. It also contains useful information for debugging CI test failures.

## Customize disk image

To add, modify or remove files of the disk image's file system, e.g. to change the default keyboard layout, you can create a shell script with the name `sync-local.sh` in the project root, with content like this:
Expand Down
77 changes: 57 additions & 20 deletions Documentation/RunningTests.md
Original file line number Diff line number Diff line change
@@ -1,52 +1,89 @@
## Running SerenityOS Tests
# Running SerenityOS Tests

There are two classes of tests built during a Serenity build: host tests and target tests. Host tests run on the build
machine, and use [Lagom](../Meta/Lagom/ReadMe.md) to build Serenity userspace libraries for the host platform. Target
tests run on the Serenity machine, either emulated or bare metal.
There are two classes of tests built during a SerenityOS build: host tests and target tests. Host tests run on the build
machine, and use [Lagom](../Meta/Lagom/ReadMe.md) to build SerenityOS userspace libraries for the host platform. Target
tests run on the SerenityOS machine, either emulated or bare metal.

### Running Host Tests
## Running Host Tests

There are two ways to build host tests: from a full build, or from a Lagom-only build. The only difference is the CMake
command used to initialize the build directory.

For a full build, pass `-DBUILD_LAGOM=ON` to the CMake command.

```sh
mkdir Build
cd Build
cmake .. -GNinja -DBUILD_LAGOM=ON
mkdir -p Build/lagom
cd Build/lagom
cmake ../.. -GNinja -DBUILD_LAGOM=ON
```

For a Lagom-only build, pass the Lagom directory to CMake.
For a Lagom-only build, pass the Lagom directory to CMake. The `BUILD_LAGOM` CMake option is still required.

```sh
mkdir BuildLagom
cd BuildLagom
cmake ../Meta/Lagom -GNinja
cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON
```

In both cases, the tests can be run via ninja after doing a build:
In both cases, the tests can be run via ninja after doing a build. Note that `test-js` requires the `SERENITY_SOURCE_DIR` environment variable to be set
to the root of the serenity source tree when running on a non-SerenityOS host.

```sh
# /path/to/serenity repository
export SERENITY_SOURCE_DIR=${PWD}/..
ninja && ninja test
```

### Running Target Tests
To see the stdout/stderr output of failing tests, the reccommended way is to set the environment variable [`CTEST_OUTPUT_ON_FAILURE`](https://cmake.org/cmake/help/latest/manual/ctest.1.html#options) to 1.

Tests built for the Serenity target get installed either into `/usr/Tests` or `/bin`. `/usr/Tests` is preferred, but
```sh
CTEST_OUTPUT_ON_FAILURE=1 ninja test

# or, using ctest directly...
ctest --output-on-failure
```

### Running with Sanitizers

CI runs host tests with Address Sanitizer and Undefined Sanitizer instrumentation enabled. These tools catch many
classes of common C++ errors, including memory leaks, out of bounds access to stack and heap allocations, and
signed integer overflow. For more info on the sanitizers, check out the Address Sanitizer [wiki page](https://github.com/google/sanitizers/wiki),
or the Undefined Sanitizer [documentation](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) from clang.

Note that a sanitizer build will take significantly longer than a non-santizer build, and will mess with caches in tools such as `ccache`.
The sanitizers can be enabled with the `-DENABLE_FOO_SANITIZER` set of flags. Sanitizers are only supported for Lagom tests, as SerenityOS support
for gcc's `libsanitizer` is not yet implemented.

```sh
mkdir BuildLagom
cd BuildLagom
cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON -DENABLE_ADDRESS_SANITIZER=ON -DENABLE_UNDEFINED_SANITIZER=ON
ninja
CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test
```

To ensure that Undefined Sanitizer errors fail the test, the `halt_on_error` flag should be set to 1 in the environment variable `UBSAN_OPTIONS`.

```sh
UBSAN_OPTIONS=halt_on_error=1 CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test
```

## Running Target Tests

Tests built for the SerenityOS target get installed either into `/usr/Tests` or `/bin`. `/usr/Tests` is preferred, but
some system tests are installed into `/bin` for historical reasons.

The easiest way to run all of the known tests in the system is to use the `run-tests-and-shutdown.sh` script that gets
installed into `/home/anon/tests`. When running in CI, the environment variable `$DO_SHUTDOWN_AFTER_TESTS` is set, which
will run `shutdown -n` after running all the tests.

For completeness, a basic on-target test run will need the Serenity image built and run via QEMU.
For completeness, a basic on-target test run will need the SerenityOS image built and run via QEMU.

```sh
mkdir Build
cd Build
cmake .. -GNinja
ninja && ninja install && ninja image && ninja run
mkdir Build/i686
cd Build/i686
cmake ../.. -GNinja
ninja install && ninja image && ninja run
```

In the initial terminal, one can easily run the test runner script:
Expand Down Expand Up @@ -77,8 +114,8 @@ BootModes=self-test
the serial debug output to `./debug.log` so that both stdout of the tests and the dbgln from the kernel/tests can be
captured.

To run with CI's TestRunner system server entry, Serenity needs booted in self-test mode. Running the following shell
lines will boot Serenity in self-test mode, run tests, and exit.
To run with CI's TestRunner system server entry, SerenityOS needs booted in self-test mode. Running the following shell
lines will boot SerenityOS in self-test mode, run tests, and exit.

```sh
export SERENITY_RUN=ci
Expand Down

0 comments on commit cfdd231

Please sign in to comment.