Skip to content

Commit

Permalink
Docs: move CI python testing args into CI section (#34264)
Browse files Browse the repository at this point in the history
  • Loading branch information
cecille committed Jul 10, 2024
1 parent 8ebe19d commit c6af674
Showing 1 changed file with 62 additions and 52 deletions.
114 changes: 62 additions & 52 deletions docs/testing/python.md
Original file line number Diff line number Diff line change
Expand Up @@ -85,58 +85,8 @@ The default_matter_test_main() function is used to run the test on the command
line. These two lines should appear verbatim at the bottom of every python test
file.

## Defining the test arguments

Below is the format:

```
# test-runner-runs: <run_identifier>
# test-runner-run/<run_identifier>/app: ${TYPE_OF_APP}
# test-runner-run/<run_identifier>/factoryreset: <True|False>
# test-runner-run/<run_identifier>/quiet: <True|False>
# test-runner-run/<run_identifier>/app-args: <app_arguments>
# test-runner-run/<run_identifier>/script-args: <script_arguments>
```

### Description of Parameters

- test-runner-runs: Specifies the identifier for the run. This can be any
unique identifier.

- Example: run1

- test-runner-run/<run_identifier>/app: Indicates the application to be used
in the test. Different app types as needed could be referenced from section
[name: Generate an argument environment file ] of the file
[.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml)

- Example: \${TYPE_OF_APP}

- test-runner-run/<run_identifier>/factoryreset: Determines whether a factory
reset should be performed before the test.

- Example: True

- test-runner-run/<run_identifier>/quiet: Sets the verbosity level of the test
run. When set to True, the test run will be quieter.

- Example: True

- test-runner-run/<run_identifier>/app-args: Specifies the arguments to be
passed to the application during the test.

- Example: --discriminator 1234 --KVS kvs1 --trace-to
json:\${TRACE_APP}.json

- test-runner-run/<run_identifier>/script-args: Specifies the arguments to be
passed to the test script.
- Example: --storage-path admin_storage.json --commissioning-method
on-network --discriminator 1234 --passcode 20202021 --trace-to
json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto

This structured format ensures that all necessary configurations are clearly
defined and easily understood, allowing for consistent and reliable test
execution.
The structured comments above the class definition are used to set up the CI for
the tests. Please see [Running tests in CI](#running-tests-in-ci).

## Cluster Codegen

Expand Down Expand Up @@ -610,3 +560,63 @@ example DUT on the host and includes factory reset support
- if there are things in your test that will fail on CI (ex. test vendor
checks), gate them on the PICS_SDK_CI_ONLY
- is_ci = self.check_pics('PICS_SDK_CI_ONLY')

The CI test runner uses a structured environment setup that can be declared
using structured comments at the top of the test file. To use this structured
format, use the --load-from-env flag with the run_python_tests.py runner.

Ex:
`scripts/run_in_python_env.sh out/venv './scripts/tests/run_python_test.py --load-from-env /tmp/test_env.yaml --script src/python_testing/TC_ICDM_2_1.py'`

## Defining the CI test arguments

Below is the format:

```
# test-runner-runs: <run_identifier>
# test-runner-run/<run_identifier>/app: ${TYPE_OF_APP}
# test-runner-run/<run_identifier>/factoryreset: <True|False>
# test-runner-run/<run_identifier>/quiet: <True|False>
# test-runner-run/<run_identifier>/app-args: <app_arguments>
# test-runner-run/<run_identifier>/script-args: <script_arguments>
```

### Description of Parameters

- test-runner-runs: Specifies the identifier for the run. This can be any
unique identifier.

- Example: run1

- test-runner-run/<run_identifier>/app: Indicates the application to be used
in the test. Different app types as needed could be referenced from section
[name: Generate an argument environment file ] of the file
[.github/workflows/tests.yaml](https://github.com/project-chip/connectedhomeip/blob/master/.github/workflows/tests.yaml)

- Example: \${TYPE_OF_APP}

- test-runner-run/<run_identifier>/factoryreset: Determines whether a factory
reset should be performed before the test.

- Example: True

- test-runner-run/<run_identifier>/quiet: Sets the verbosity level of the test
run. When set to True, the test run will be quieter.

- Example: True

- test-runner-run/<run_identifier>/app-args: Specifies the arguments to be
passed to the application during the test.

- Example: --discriminator 1234 --KVS kvs1 --trace-to
json:\${TRACE_APP}.json

- test-runner-run/<run_identifier>/script-args: Specifies the arguments to be
passed to the test script.
- Example: --storage-path admin_storage.json --commissioning-method
on-network --discriminator 1234 --passcode 20202021 --trace-to
json:${TRACE_TEST_JSON}.json --trace-to perfetto:${TRACE_TEST_PERFETTO}.perfetto

This structured format ensures that all necessary configurations are clearly
defined and easily understood, allowing for consistent and reliable test
execution.

0 comments on commit c6af674

Please sign in to comment.