CMake: tests: Support skipping unsupported test with reason #14902

LDong-Arm · 2021-07-09T11:20:43Z

Summary of changes

Add a new argument TEST_SKIPPED to mbed_greentea_add_test() to indicate a test is skipped and give a reason (e.g. the Mbed target and/or configuration does not provide what the test requires).

The skip reason of a test is printed when running tests with ctest, and the test is marked as "Skipped" in the test report.

As a showcase, use this mechanism to skip the PSA Attestation test if baremetal is used or PSA and Experimental API are unavailable. Also update the ticker test (whose CTest support was added in #14892) to skip if microsecond ticker is unavailable.

Impact of changes

Migration actions required

Documentation

In-line documentation for mbed_greentea_add_test() updated in tools/cmake/mbed_greentea.cmake.

Pull request type

[] Patch update (Bug fix / Target update / Docs update / Test update / Refactor)
[x] Feature update (New feature / Functionality change / New API)
[] Major update (Breaking change E.g. Return code change / API behaviour change)

Test results

[] No Tests required for this change (E.g docs only update)
[] Covered by existing mbed-os tests (Greentea or Unittest)
[x] Tests / results supplied as part of this PR

Got expected behaviors when trying to build the PSA Attestation test in platform/FEATURE_EXPERIMENTAL_API/FEATURE_PSA/TARGET_MBED_PSA_SRV/TESTS/attestation/test,

with/without -DMBED_TEST_BAREMETAL=ON
with/without --app-config <path-to>/mbed-os/TESTS/configs/experimental.json

Reviewers

@ARMmbed/mbed-os-core

tools/cmake/mbed_greentea.cmake

...ATURE_EXPERIMENTAL_API/FEATURE_PSA/TARGET_MBED_PSA_SRV/TESTS/attestation/test/CMakeLists.txt

tools/cmake/mbed_greentea.cmake

ciarmcom · 2021-07-09T11:30:15Z

@LDong-Arm, thank you for your changes.
@ARMmbed/mbed-os-security @ARMmbed/mbed-os-core @ARMmbed/mbed-os-maintainers please review.

mergify · 2021-07-09T16:07:45Z

This PR cannot be merged due to conflicts. Please rebase to resolve them.

rwalton-arm

Looks pretty good to me. Just one comment about the log level of the skipped message.

tools/cmake/mbed_greentea.cmake

LDong-Arm · 2021-07-14T15:19:04Z

@rwalton-arm Thanks for the review.
To maintainers: Please keep CI on hold until #14892 is merged, and I'll make a further change with the PR rebased on that.

tools/cmake/mbed_greentea.cmake

Pull request has been modified.

LDong-Arm · 2021-08-11T10:53:23Z

Rebased, as #14892 is merged

Following approach defined in ARMmbed#14892 and ARMmbed#14902, events greentea tests are ported to enable running with CTest.

Patater

LGTM

Patater · 2021-08-11T13:05:33Z

CI started

mbed-ci · 2021-08-11T13:42:59Z

Jenkins CI Test : ✔️ SUCCESS

Build Number: 1 | 🔒 Jenkins CI Job | 🌐 Logs & Artifacts

CLICK for Detailed Summary

jobs	Status
jenkins-ci/mbed-os-ci_unittests	✔️
jenkins-ci/mbed-os-ci_build-cloud-example-ARM	✔️
jenkins-ci/mbed-os-ci_cmake-cloud-example-ARM	✔️
jenkins-ci/mbed-os-ci_build-cloud-example-GCC_ARM	✔️
jenkins-ci/mbed-os-ci_cmake-cloud-example-GCC_ARM	✔️
jenkins-ci/mbed-os-ci_build-greentea-ARM	✔️
jenkins-ci/mbed-os-ci_build-greentea-GCC_ARM	✔️
jenkins-ci/mbed-os-ci_cmake-example-GCC_ARM	✔️
jenkins-ci/mbed-os-ci_cmake-example-ARM	✔️
jenkins-ci/mbed-os-ci_build-example-ARM	✔️
jenkins-ci/mbed-os-ci_build-example-GCC_ARM	✔️
jenkins-ci/mbed-os-ci_greentea-test	✔️
jenkins-ci/mbed-os-ci_tfm-integration	✔️

Following approach defined in ARMmbed#14892 and ARMmbed#14902, events greentea tests are ported to enable running with CTest.

Following the approach defined in ARMmbed#14892 and ARMmbed#14902, events greentea tests are ported to enable running with CTest. events/equeue requires MBED_EXTENDED_TESTS. events/queue and events/timing both require DEVICE_USTICKER.

Following the approach defined in ARMmbed#14892 and ARMmbed#14902, events greentea tests are ported to enable running with CTest. events/queue and events/timing both require DEVICE_USTICKER.

LDong-Arm commented Jul 9, 2021

View reviewed changes

tools/cmake/mbed_greentea.cmake Outdated Show resolved Hide resolved

LDong-Arm commented Jul 9, 2021

View reviewed changes

...ATURE_EXPERIMENTAL_API/FEATURE_PSA/TARGET_MBED_PSA_SRV/TESTS/attestation/test/CMakeLists.txt Outdated Show resolved Hide resolved

tools/cmake/mbed_greentea.cmake Show resolved Hide resolved

ciarmcom added the release-type: feature label Jul 9, 2021

ciarmcom requested review from a team July 9, 2021 11:30

ciarmcom added needs: review component: security component: tools component: core labels Jul 9, 2021

mergify bot added needs: work and removed needs: review labels Jul 9, 2021

LDong-Arm requested a review from rwalton-arm July 12, 2021 09:20

rwalton-arm reviewed Jul 14, 2021

View reviewed changes

tools/cmake/mbed_greentea.cmake Outdated Show resolved Hide resolved

LDong-Arm force-pushed the greentea_skip branch from 93d3b01 to 094d24c Compare July 14, 2021 14:21

rwalton-arm previously approved these changes Jul 14, 2021

View reviewed changes

mergify bot added needs: CI and removed needs: work labels Jul 15, 2021

Patater self-requested a review July 15, 2021 10:26

Patater previously requested changes Jul 15, 2021

View reviewed changes

tools/cmake/mbed_greentea.cmake Outdated Show resolved Hide resolved

mergify bot added needs: work and removed needs: CI labels Jul 15, 2021

LDong-Arm force-pushed the greentea_skip branch from 094d24c to ee2e0b7 Compare July 15, 2021 10:49

Patater added the do not merge label Jul 15, 2021

Patater previously approved these changes Jul 15, 2021

View reviewed changes

LDong-Arm requested a review from a team August 11, 2021 10:52

hazzlim added a commit to hazzlim/mbed-os that referenced this pull request Aug 11, 2021

CMake: greentea: Port events tests to CTest

e147d50

Following approach defined in ARMmbed#14892 and ARMmbed#14902, events greentea tests are ported to enable running with CTest.

Patater added needs: review and removed needs: work labels Aug 11, 2021

Patater approved these changes Aug 11, 2021

View reviewed changes

mergify bot added needs: CI and removed needs: review labels Aug 11, 2021

Patater merged commit e588f80 into ARMmbed:master Aug 11, 2021

mergify bot added ready for merge and removed needs: CI ready for merge labels Aug 11, 2021

hazzlim added a commit to hazzlim/mbed-os that referenced this pull request Aug 11, 2021

CMake: greentea: Port events tests to CTest

eb8658b

Following approach defined in ARMmbed#14892 and ARMmbed#14902, events greentea tests are ported to enable running with CTest.

0xc0170 mentioned this pull request Aug 12, 2021

Add CMakeLists for USB greentea tests #14937

Closed

mbedmain added release-version: 6.14.0 Release-pending labels Aug 18, 2021

mbedmain removed release-type: feature Release-pending labels Aug 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMake: tests: Support skipping unsupported test with reason #14902

CMake: tests: Support skipping unsupported test with reason #14902

LDong-Arm commented Jul 9, 2021 •

edited

Loading

ciarmcom commented Jul 9, 2021

mergify bot commented Jul 9, 2021

rwalton-arm left a comment

LDong-Arm commented Jul 14, 2021

LDong-Arm commented Aug 11, 2021

Patater left a comment

Patater commented Aug 11, 2021

mbed-ci commented Aug 11, 2021

CMake: tests: Support skipping unsupported test with reason #14902

CMake: tests: Support skipping unsupported test with reason #14902

Conversation

LDong-Arm commented Jul 9, 2021 • edited Loading

Summary of changes

Impact of changes

Migration actions required

Documentation

Pull request type

Test results

Reviewers

ciarmcom commented Jul 9, 2021

mergify bot commented Jul 9, 2021

rwalton-arm left a comment

Choose a reason for hiding this comment

LDong-Arm commented Jul 14, 2021

LDong-Arm commented Aug 11, 2021

Patater left a comment

Choose a reason for hiding this comment

Patater commented Aug 11, 2021

mbed-ci commented Aug 11, 2021

Jenkins CI Test : ✔️ SUCCESS

Build Number: 1 | 🔒 Jenkins CI Job | 🌐 Logs & Artifacts

LDong-Arm commented Jul 9, 2021 •

edited

Loading