Update target/variant/modifier logic in build_examples.py (#23054)

* Starting to add some build target logic and unit tests

* Restyle

* make basic unit tests pass

* More unit tests

* Restyle

* Update prefix logic: use the builder prefix as well

* Prepare to move build targets for linux

* More conversion tests

* more conversions

* Converted more targets

* Converted more targets

* Updted more targets

* Simplify target part

* Restyle

* All applications are converted

* Remove old target code - should not be needed anymore

* Restyle

* Building at least chip-tool works. No introspection though

* Fix typo in m5 builds

* A bit easier to parse logging

* More updates

* Add ability to output all variants of some target

* Restyle

* Add the ability to print all possible targets

* Add esp32 qemu support

* Restyle

* Fix typo

* Simplify target printout, make nrf tests work

* Restyle

* Remove some usages of target glob

* Replace glob in build scripts

* Restyle

* convert some globs into instructions, fix some target compile bugs

* make unit tests pass

* more unit tests

* Restyle

* Remove the default of "all" in build targets since that does not mean anything anymore

* Rename arm64-clang to arm64 in targets for cloudbuild smoke test

* For arm64, clang is a modifier that is required

* Remove some whitespace.Tested that targets are ok

* Fix ameba

* Fix tizen

* Fix infineon

* Fix qpg

* Fix infineon, cc13x2x7 and update android a bit

* Fix linux standalone

* Fix efr32

* Fix esp32

* Remove glob option from tasks.json

* Fix platform naming for targets - fixes darwin target naming

* Fix k32w targets - no more release

* Adjust arm64 cross compile target names

* Fix boufallolab app target

* All-clusters full thread device does not link (insufficient ram/flash) on cc13x2x7. Updating to minimal thread device only

* Update k32w example builds: nologs is mandatory for low-power builds

* Fix output dir for boufallolab: output should be all lowercase

* Add back the test/extra_tests variant to host builds

* Fix qpg size reports

* Update unit tests for available outpust now that test variant was added

* Fix expected paths for cc13x2x7 builds

* Fix typo in infineon build

* Fix BL702 path for output - duplicate boufallolab

* Allow rpc modifier for BL706

* Fix unit tests

* Move nltestrunner to esp32 test runner and include in cloudbuild

* Undo duplicate build target addition

* Fix typo: esp32 -> efr32

* Split out fake platform: only tests are really supported there

.github/workflows/examples-ameba.yaml

@@ -47,4 +47,12 @@ jobs:
  - name: Checkout submodules
  run: scripts/checkout_submodules.py --shallow --platform ameba
  - name: Build example Ameba All Clusters App
- run: scripts/run_in_build_env.sh "./scripts/build/build_examples.py --target-glob 'ameba-*' build"
+ run: |
+ ./scripts/run_in_build_env.sh \
+ "./scripts/build/build_examples.py \
+ --target ameba-amebad-all-clusters \
+ --target ameba-amebad-all-clusters-minimal \
+ --target ameba-amebad-light \
+ --target ameba-amebad-pigweed \
+ build \
+ "

.github/workflows/examples-bouffalolab.yaml

@@ -56,17 +56,17 @@ jobs:
  timeout-minutes: 30
  run: |
  ./scripts/run_in_build_env.sh \
- "./scripts/build/build_examples.py --target bouffalolab-BL706-IoT-DVK-light build"
+ "./scripts/build/build_examples.py --target bouffalolab-bl706-iot-dvk-light build"
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py bl702 bl702 lighting-app \
- out/bouffalolab-BL706-IoT-DVK-light/chip-bl702-lighting-example.out /tmp/bloat_reports/
+ out/bouffalolab-bl706-iot-dvk-light/chip-bl702-lighting-example.out /tmp/bloat_reports/
 
  - name: Build example BL702 Lighting App with RPCs
  timeout-minutes: 30
  run: |
  ./scripts/run_in_build_env.sh \
- "./scripts/build/build_examples.py --target bouffalolab-BL706-IoT-DVK-light-rpc build"
+ "./scripts/build/build_examples.py --target bouffalolab-bl706-iot-dvk-light-rpc build"
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py bl702 bl702+rpc lighting-app \
- out/bouffalolab-BL706-IoT-DVK-light-rpc/chip-bl702-lighting-example.out /tmp/bloat_reports/
+ out/bouffalolab-bl706-iot-dvk-light-rpc/chip-bl702-lighting-example.out /tmp/bloat_reports/
 
  - name: Uploading Size Reports
  uses: actions/upload-artifact@v2

.github/workflows/examples-cc13x2x7_26x2x7.yaml

@@ -68,7 +68,15 @@ jobs:
  timeout-minutes: 60
  run: |
  scripts/run_in_build_env.sh "\
- ./scripts/build/build_examples.py --target-glob cc13x2x7_26x2x7* build \
+ ./scripts/build/build_examples.py \
+ --target cc13x2x7_26x2x7-all-clusters-mtd \
+ --target cc13x2x7_26x2x7-all-clusters-minimal \
+ --target cc13x2x7_26x2x7-lock-ftd \
+ --target cc13x2x7_26x2x7-lock-mtd \
+ --target cc13x2x7_26x2x7-pump \
+ --target cc13x2x7_26x2x7-pump-controller \
+ --target cc13x2x7_26x2x7-shell \
+ build \
  --copy-artifacts-to out/artifacts \
  "
  - name: Get lock FTD size stats
@@ -104,7 +112,7 @@ jobs:
  run: |
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
  cc13x2_26x2 LP_CC2652R7 all-clusters-app \
- out/artifacts/cc13x2x7_26x2x7-all-clusters/chip-LP_CC2652R7-all-clusters-example.out \
+ out/artifacts/cc13x2x7_26x2x7-all-clusters-mtd/chip-LP_CC2652R7-all-clusters-example.out \
  /tmp/bloat_reports/
  - name: Get All Clusters Minimal App size stats
  timeout-minutes: 5

.github/workflows/examples-efr32.yaml

@@ -77,7 +77,12 @@ jobs:
  ./scripts/run_in_build_env.sh \
  "./scripts/build/build_examples.py \
  --enable-flashbundle \
- --target-glob '*-brd4187c-{lock,light,switch,window-covering,unit-test}' \
+ --target efr32-brd4187c-window-covering \
+ --target efr32-brd4187c-switch \
+ --target efr32-brd4187c-unit-test \
+ --target efr32-brd4187c-light \
+ --target efr32-brd4187c-light-rpc \
+ --target efr32-brd4187c-lock-rpc \
  build \
  --copy-artifacts-to out/artifacts \
  "

.github/workflows/examples-esp32.yaml

@@ -70,7 +70,9 @@ jobs:
  ./scripts/run_in_build_env.sh \
  "./scripts/build/build_examples.py \
  --enable-flashbundle \
- --target-glob '*-m5stack-{all-clusters,all-clusters-minimal,all-clusters-rpc-ipv6only}' \
+ --target esp32-m5stack-all-clusters \
+ --target esp32-m5stack-all-clusters-minimal \
+ --target esp32-m5stack-all-clusters-rpc-ipv6only \
  build \
  --copy-artifacts-to out/artifacts \
  "

.github/workflows/examples-infineon.yaml

@@ -126,7 +126,9 @@ jobs:
  run: |
  ./scripts/run_in_build_env.sh \
  "./scripts/build/build_examples.py \
- --target-glob 'cyw30739-cyw930739m2evb_01-{light,lock,ota-requestor-no-progress-logging}' \
+ --target cyw30739-cyw930739m2evb_01-light \
+ --target cyw30739-cyw930739m2evb_01-lock \
+ --target cyw30739-cyw930739m2evb_01-ota-requestor-no-progress-logging \
  build \
  --copy-artifacts-to out/artifacts \
  "

.github/workflows/examples-k32w.yaml

@@ -71,11 +71,11 @@ jobs:
  run: |
  scripts/run_in_build_env.sh "\
  ./scripts/build/build_examples.py \
- --target k32w-light-ota-se \
- --target k32w-light-release-no-ota \
- --target k32w-lock-low-power-release \
- --target k32w-contact-low-power-release \
- --target k32w-shell-release \
+ --target k32w-light \
+ --target k32w-light-no-ota \
+ --target k32w-lock-low-power-nologs \
+ --target k32w-contact-low-power-nologs \
+ --target k32w-shell \
  build \
  --copy-artifacts-to out/artifacts \
  "
@@ -84,21 +84,21 @@ jobs:
  run: |
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
  k32w k32w0+release light \
- out/artifacts/k32w-light-release-no-ota/chip-k32w0x-light-example \
+ out/artifacts/k32w-light-no-ota/chip-k32w0x-light-example \
  /tmp/bloat_reports/
  - name: Get lock size stats
  timeout-minutes: 5
  run: |
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
  k32w k32w0+release lock \
- out/artifacts/k32w-lock-low-power-release/chip-k32w0x-lock-example \
+ out/artifacts/k32w-lock-low-power-nologs/chip-k32w0x-lock-example \
  /tmp/bloat_reports/
  - name: Get contact size stats
  timeout-minutes: 5
  run: |
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
  k32w k32w0+release contact \
- out/artifacts/k32w-contact-low-power-release/chip-k32w0x-contact-example \
+ out/artifacts/k32w-contact-low-power-nologs/chip-k32w0x-contact-example \
  /tmp/bloat_reports/
  - name: Uploading Size Reports
  uses: actions/upload-artifact@v2

.github/workflows/examples-linux-arm.yaml

@@ -69,26 +69,26 @@ jobs:
  run: |
  ./scripts/run_in_build_env.sh \
  "./scripts/build/build_examples.py \
- --target linux-arm64-clang-all-clusters \
- --target linux-arm64-clang-chip-tool-ipv6only \
- --target linux-arm64-clang-lock \
- --target linux-arm64-clang-minmdns \
- --target linux-arm64-clang-thermostat-no-ble \
+ --target linux-arm64-all-clusters-clang \
+ --target linux-arm64-chip-tool-ipv6only-clang \
+ --target linux-arm64-lock-clang \
+ --target linux-arm64-minmdns-clang \
+ --target linux-arm64-thermostat-no-ble-clang \
  build \
  "
  - name: Bloat report - chip-tool
  timeout-minutes: 5
  run: |
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
  linux arm64 chip-tool-ipv6only \
- out/linux-arm64-clang-chip-tool-ipv6only/chip-tool \
+ out/linux-arm64-chip-tool-ipv6only-clang/chip-tool \
  /tmp/bloat_reports/
  - name: Bloat report - thermostat
  timeout-minutes: 5
  run: |
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
  linux arm64 thermostat-no-ble \
- out/linux-arm64-clang-thermostat-no-ble/thermostat-app \
+ out/linux-arm64-thermostat-no-ble-clang/thermostat-app \
  /tmp/bloat_reports/
  - name: Uploading Size Reports
  uses: actions/upload-artifact@v2

.github/workflows/examples-linux-standalone.yaml

@@ -71,7 +71,11 @@ jobs:
  timeout-minutes: 10
  run: |
  ./scripts/run_in_build_env.sh \
- "./scripts/build/build_examples.py --no-log-timestamps --target-glob '*-chip-cert' build"
+ "./scripts/build/build_examples.py --no-log-timestamps \
+ --target linux-x64-chip-cert \
+ --target linux-arm64-chip-cert \
+ build \
+ "
  - name: Build minmdns example with platform dns
  timeout-minutes: 10
  run: |

.github/workflows/examples-qpg.yaml

@@ -72,7 +72,10 @@ jobs:
  ./scripts/run_in_build_env.sh \
  "./scripts/build/build_examples.py \
  --enable-flashbundle \
- --target-glob 'qpg-{lock,light,shell,persistent-storage}' \
+ --target qpg-qpg6105-lock \
+ --target qpg-qpg6105-light \
+ --target qpg-qpg6105-shell \
+ --target qpg-qpg6105-persistent-storage \
  build \
  --copy-artifacts-to out/artifacts \
  "
@@ -85,11 +88,11 @@ jobs:
  run: |
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
  qpg qpg6105+debug lighting-app \
- out/qpg-light/chip-qpg6105-lighting-example.out \
+ out/qpg-qpg6105-light/chip-qpg6105-lighting-example.out \
  /tmp/bloat_reports/
  .environment/pigweed-venv/bin/python3 scripts/tools/memory/gh_sizes.py \
  qpg qpg6105+debug lock-app \
- out/qpg-lock/chip-qpg6105-lock-example.out \
+ out/qpg-qpg6105-lock/chip-qpg6105-lock-example.out \
  /tmp/bloat_reports/
 
  - name: Uploading Size Reports

.github/workflows/examples-tizen.yaml

@@ -54,7 +54,11 @@ jobs:
  ./scripts/run_in_build_env.sh \
  "./scripts/build/build_examples.py \
  --enable-flashbundle \
- --target-glob 'tizen-*' \
+ --target tizen-arm-all-clusters \
+ --target tizen-arm-all-clusters-minimal-no-wifi \
+ --target chip-tool \
+ --target light \
+ --target light-no-ble-no-wifi \
  build \
  --copy-artifacts-to out/artifacts \
  "

.github/workflows/full-android.yaml

@@ -111,4 +111,4 @@ jobs:
  # - name: Build Android Studio build (arm64 only)
  # run: |
  # ./scripts/run_in_build_env.sh \
- # "./scripts/build/build_examples.py --no-log-timestamps --target-glob 'android-androidstudio-arm64-chip-tool' build"
+ # "./scripts/build/build_examples.py --no-log-timestamps --target android-androidstudio-arm64-chip-tool  build"

.restyled.yaml

@@ -75,6 +75,7 @@ exclude:
  - "src/darwin/Framework/CHIP/zap-generated/MTRBaseClusters.mm" # https://github.com/project-chip/connectedhomeip/issues/20236
  - "src/darwin/Framework/CHIP/zap-generated/MTRBaseClusters.h" # https://github.com/project-chip/connectedhomeip/issues/20236
  - "examples/platform/nxp/k32w/k32w0/scripts/demo_generated_certs/**/*"
+ - "integrations/cloudbuild/*.yaml" # uglier long command line content
 
 
 changed_paths:

.vscode/tasks.json

@@ -13,18 +13,6 @@
  "base": "$gcc"
  }
  },
- {
- "label": "Build examples (glob)",
- "type": "shell",
- "command": "scripts/run_in_build_env.sh",
- "args": [
- "./scripts/build/build_examples.py --target-glob \"${input:exampleGlob}\" build"
- ],
- "group": "build",
- "problemMatcher": {
- "base": "$gcc"
- }
- },
  {
  "label": "Main Build",
  "type": "shell",

integrations/cloudbuild/build-all.yaml

@@ -27,8 +27,124 @@ steps:
  args:
  - >-
  ./scripts/build/build_examples.py --enable-flashbundle
- --target-glob '*' --skip-target-glob
- '{imx-*,tizen-*,*-androidstudio-*,*-tests*,*-chip-test}' build
+ --target ameba-amebad-light
+ --target ameba-amebad-pigweed
+ --target android-arm-chip-tool
+ --target android-arm64-chip-tool
+ --target android-arm64-tv-casting-app
+ --target android-arm64-tv-server
+ --target android-x64-chip-tool
+ --target bl602-light
+ --target bouffalolab-BL706-IoT-DVK-light-rpc
+ --target cc13x2x7_26x2x7-lock-ftd
+ --target cc13x2x7_26x2x7-lock-mtd
+ --target cc13x2x7_26x2x7-shell
+ --target cyw30739-cyw930739m2evb_01-ota-requestor-no-progress-logging
+ --target efr32-brd4161a-light-rpc
+ --target efr32-brd4161a-lock
+ --target efr32-brd4161a-switch
+ --target efr32-brd4161a-window-covering
+ --target efr32-brd4187c-window-covering
+ --target esp32-c3devkit-all-clusters
+ --target esp32-c3devkit-all-clusters-minimal
+ --target esp32-devkitc-all-clusters-ipv6only
+ --target esp32-devkitc-all-clusters-minimal-ipv6only
+ --target esp32-devkitc-bridge
+ --target esp32-devkitc-light-rpc
+ --target esp32-devkitc-lock
+ --target esp32-devkitc-ota-requestor-rpc
+ --target esp32-devkitc-shell
+ --target esp32-m5stack-all-clusters
+ --target esp32-m5stack-all-clusters-ipv6only
+ --target esp32-m5stack-all-clusters-minimal
+ --target esp32-m5stack-all-clusters-minimal-ipv6only
+ --target esp32-m5stack-all-clusters-minimal-rpc
+ --target esp32-m5stack-all-clusters-minimal-rpc-ipv6only
+ --target esp32-m5stack-ota-requestor
+ --target genio-lighting-app
+ --target imx-all-clusters-app
+ --target imx-chip-tool
+ --target imx-thermostat
+ --target infineon-psoc6-all-clusters
+ --target infineon-psoc6-all-clusters-minimal
+ --target infineon-psoc6-light
+ --target infineon-psoc6-lock
+ --target k32w-contact
+ --target k32w-light-no-ota
+ --target k32w-lock
+ --target k32w-shell
+ --target linux-arm64-all-clusters-clang
+ --target linux-arm64-all-clusters-app-nodeps
+ --target linux-arm64-all-clusters-app-nodeps-ipv6only
+ --target linux-arm64-all-clusters-ipv6only-clang
+ --target linux-arm64-all-clusters-minimal-clang
+ --target linux-arm64-all-clusters-minimal-ipv6only-clang
+ --target linux-arm64-bridge-ipv6only-clang
+ --target linux-arm64-chip-tool-clang
+ --target linux-arm64-chip-tool-ipv6only-clang
+ --target linux-arm64-chip-tool-nodeps
+ --target linux-arm64-chip-tool-nodeps-ipv6only
+ --target linux-arm64-dynamic-bridge-ipv6only-clang
+ --target linux-arm64-light-clang
+ --target linux-arm64-light-ipv6only-clang
+ --target linux-arm64-light-rpc-clang
+ --target linux-arm64-light-rpc-ipv6only-clang
+ --target linux-arm64-lock-clang
+ --target linux-arm64-lock-ipv6only-clang
+ --target linux-arm64-minmdns-clang
+ --target linux-arm64-ota-provider-nodeps-ipv6only
+ --target linux-arm64-ota-requestor-nodeps-ipv6only
+ --target linux-arm64-shell-ipv6only-clang
+ --target linux-arm64-thermostat-ipv6only-clang
+ --target linux-arm64-tv-app-ipv6only-clang
+ --target linux-arm64-tv-casting-app-ipv6only-clang
+ --target linux-x64-address-resolve-tool
+ --target linux-x64-all-clusters
+ --target linux-x64-all-clusters-nodeps
+ --target linux-x64-all-clusters-nodeps-ipv6only
+ --target linux-x64-all-clusters-coverage
+ --target linux-x64-all-clusters-ipv6only
+ --target linux-x64-all-clusters-minimal
+ --target linux-x64-all-clusters-minimal-ipv6only
+ --target linux-x64-bridge-ipv6only
+ --target linux-x64-chip-cert
+ --target linux-x64-chip-tool
+ --target linux-x64-chip-tool-coverage
+ --target linux-x64-chip-tool-nodeps-ipv6only
+ --target linux-x64-dynamic-bridge-ipv6only
+ --target linux-x64-efr32-test-runner
+ --target linux-x64-light-rpc-ipv6only
+ --target linux-x64-lock-ipv6only
+ --target linux-x64-minmdns
+ --target linux-x64-nl-test-runner
+ --target linux-x64-ota-provider
+ --target linux-x64-ota-provider-nodeps-ipv6only
+ --target linux-x64-ota-requestor-nodeps-ipv6only
+ --target linux-x64-rpc-console
+ --target linux-x64-shell-ipv6only
+ --target linux-x64-thermostat-ipv6only
+ --target linux-x64-tv-app-ipv6only
+ --target linux-x64-tv-casting-app-ipv6only
+ --target mbed-CY8CPROTO_062_4343W-light-release
+ --target mbed-CY8CPROTO_062_4343W-lock-release
+ --target mbed-CY8CPROTO_062_4343W-pigweed-release
+ --target mw320-all-clusters-app
+ --target nrf-nrf52840dk-all-clusters
+ --target nrf-nrf52840dk-all-clusters-minimal
+ --target nrf-nrf52840dk-light-rpc
+ --target nrf-nrf52840dk-lock
+ --target nrf-nrf52840dk-pump
+ --target nrf-nrf52840dk-pump-controller
+ --target nrf-nrf52840dk-shell
+ --target nrf-nrf52840dongle-light
+ --target nrf-nrf5340dk-all-clusters
+ --target nrf-nrf5340dk-all-clusters-minimal
+ --target nrf-nrf5340dk-shell
+ --target qpg-qpg6105-light
+ --target qpg-qpg6105-shell
+ --target telink-tlsr9518adk80d-light
+ --target tizen-arm-light
+ build
  --create-archives /workspace/artifacts/
  id: CompileAll
  waitFor: