From cfdd231a4d6036e27da584bf633bd60ff215193e Mon Sep 17 00:00:00 2001 From: Andrew Kaster Date: Sat, 15 May 2021 20:13:31 -0600 Subject: [PATCH] Documentation: Add Sanitizer section to RunningTests Also, add link to RunningTests BuildInstructions, and clean up stale commands in RunningTests to align to current build strategies. --- Documentation/BuildInstructions.md | 5 ++ Documentation/RunningTests.md | 77 ++++++++++++++++++++++-------- 2 files changed, 62 insertions(+), 20 deletions(-) diff --git a/Documentation/BuildInstructions.md b/Documentation/BuildInstructions.md index 8a47ed331fe41c..8eb398607aa2bb 100644 --- a/Documentation/BuildInstructions.md +++ b/Documentation/BuildInstructions.md @@ -267,6 +267,11 @@ For the changes to take effect, SerenityOS needs to be recompiled and the disk i To add a package from the ports collection to Serenity, for example curl, go into `Ports/curl/` and run `./package.sh`. The sourcecode for the package will be downloaded and the package will be built. After that, rebuild the disk image. The next time you start Serenity, `curl` will be available. +## Tests + +For information on running host and target tests, see [Running Tests](RunningTests.md). The documentation there explains the difference between host tests run with Lagom and +target tests run on SerenityOS. It also contains useful information for debugging CI test failures. + ## Customize disk image To add, modify or remove files of the disk image's file system, e.g. to change the default keyboard layout, you can create a shell script with the name `sync-local.sh` in the project root, with content like this: diff --git a/Documentation/RunningTests.md b/Documentation/RunningTests.md index 5da60041278a8f..d922d6dfe4bd2e 100644 --- a/Documentation/RunningTests.md +++ b/Documentation/RunningTests.md @@ -1,10 +1,10 @@ -## Running SerenityOS Tests +# Running SerenityOS Tests -There are two classes of tests built during a Serenity build: host tests and target tests. Host tests run on the build -machine, and use [Lagom](../Meta/Lagom/ReadMe.md) to build Serenity userspace libraries for the host platform. Target -tests run on the Serenity machine, either emulated or bare metal. +There are two classes of tests built during a SerenityOS build: host tests and target tests. Host tests run on the build +machine, and use [Lagom](../Meta/Lagom/ReadMe.md) to build SerenityOS userspace libraries for the host platform. Target +tests run on the SerenityOS machine, either emulated or bare metal. -### Running Host Tests +## Running Host Tests There are two ways to build host tests: from a full build, or from a Lagom-only build. The only difference is the CMake command used to initialize the build directory. @@ -12,41 +12,78 @@ command used to initialize the build directory. For a full build, pass `-DBUILD_LAGOM=ON` to the CMake command. ```sh -mkdir Build -cd Build -cmake .. -GNinja -DBUILD_LAGOM=ON +mkdir -p Build/lagom +cd Build/lagom +cmake ../.. -GNinja -DBUILD_LAGOM=ON ``` -For a Lagom-only build, pass the Lagom directory to CMake. +For a Lagom-only build, pass the Lagom directory to CMake. The `BUILD_LAGOM` CMake option is still required. ```sh mkdir BuildLagom cd BuildLagom -cmake ../Meta/Lagom -GNinja +cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON ``` -In both cases, the tests can be run via ninja after doing a build: +In both cases, the tests can be run via ninja after doing a build. Note that `test-js` requires the `SERENITY_SOURCE_DIR` environment variable to be set +to the root of the serenity source tree when running on a non-SerenityOS host. ```sh +# /path/to/serenity repository +export SERENITY_SOURCE_DIR=${PWD}/.. ninja && ninja test ``` -### Running Target Tests +To see the stdout/stderr output of failing tests, the reccommended way is to set the environment variable [`CTEST_OUTPUT_ON_FAILURE`](https://cmake.org/cmake/help/latest/manual/ctest.1.html#options) to 1. -Tests built for the Serenity target get installed either into `/usr/Tests` or `/bin`. `/usr/Tests` is preferred, but +```sh +CTEST_OUTPUT_ON_FAILURE=1 ninja test + +# or, using ctest directly... +ctest --output-on-failure +``` + +### Running with Sanitizers + +CI runs host tests with Address Sanitizer and Undefined Sanitizer instrumentation enabled. These tools catch many +classes of common C++ errors, including memory leaks, out of bounds access to stack and heap allocations, and +signed integer overflow. For more info on the sanitizers, check out the Address Sanitizer [wiki page](https://github.com/google/sanitizers/wiki), +or the Undefined Sanitizer [documentation](https://clang.llvm.org/docs/UndefinedBehaviorSanitizer.html) from clang. + +Note that a sanitizer build will take significantly longer than a non-santizer build, and will mess with caches in tools such as `ccache`. +The sanitizers can be enabled with the `-DENABLE_FOO_SANITIZER` set of flags. Sanitizers are only supported for Lagom tests, as SerenityOS support +for gcc's `libsanitizer` is not yet implemented. + +```sh +mkdir BuildLagom +cd BuildLagom +cmake ../Meta/Lagom -GNinja -DBUILD_LAGOM=ON -DENABLE_ADDRESS_SANITIZER=ON -DENABLE_UNDEFINED_SANITIZER=ON +ninja +CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test +``` + +To ensure that Undefined Sanitizer errors fail the test, the `halt_on_error` flag should be set to 1 in the environment variable `UBSAN_OPTIONS`. + +```sh +UBSAN_OPTIONS=halt_on_error=1 CTEST_OUTPUT_ON_FAILURE=1 SERENITY_SOURCE_DIR=${PWD}/.. ninja test +``` + +## Running Target Tests + +Tests built for the SerenityOS target get installed either into `/usr/Tests` or `/bin`. `/usr/Tests` is preferred, but some system tests are installed into `/bin` for historical reasons. The easiest way to run all of the known tests in the system is to use the `run-tests-and-shutdown.sh` script that gets installed into `/home/anon/tests`. When running in CI, the environment variable `$DO_SHUTDOWN_AFTER_TESTS` is set, which will run `shutdown -n` after running all the tests. -For completeness, a basic on-target test run will need the Serenity image built and run via QEMU. +For completeness, a basic on-target test run will need the SerenityOS image built and run via QEMU. ```sh -mkdir Build -cd Build -cmake .. -GNinja -ninja && ninja install && ninja image && ninja run +mkdir Build/i686 +cd Build/i686 +cmake ../.. -GNinja +ninja install && ninja image && ninja run ``` In the initial terminal, one can easily run the test runner script: @@ -77,8 +114,8 @@ BootModes=self-test the serial debug output to `./debug.log` so that both stdout of the tests and the dbgln from the kernel/tests can be captured. -To run with CI's TestRunner system server entry, Serenity needs booted in self-test mode. Running the following shell -lines will boot Serenity in self-test mode, run tests, and exit. +To run with CI's TestRunner system server entry, SerenityOS needs booted in self-test mode. Running the following shell +lines will boot SerenityOS in self-test mode, run tests, and exit. ```sh export SERENITY_RUN=ci