Warning: The following content is outdated

A partner may generate test summaries.

In the partner section of collection_rdf_template.yaml we need to specify your partner_id (e.g. 'ilastik') and your workflow which creates and deploys your test summaries.

Dedicated partner test workflows are triggered by the collection CI trigger_partner_wf.yaml for any newly added resource version. For this to work the partner repository needs to invite @bioimageiobot as collaborator and the partner repository needs to implement a GitHub Actions workflow which is callable from another workflow and accepts a pending_matrix input:

on:
  workflow_dispatch:
    inputs:
      pending_matrix:
        description: 'json encoded gh matrix of pending validations for new or updated resources (default: test on all resources)'
        required: true
        default: '{"include": [{"resource_id": "**", "version_id": "**"}]}'
        type: string

If the partner's test itself changes due to a new release, etc, or another test is added, it is the partner's (CI's) responsibility to trigger this workflow on the appropriate resources (or all).

The workflow should generate test summaries for the resource versions given by pending_matrix. The RDFs of any resource version can be found in the gh-pages branch of the collection-bioimage-io repository as https://github.com/bioimage-io/collection-bioimage-io/tree/gh-pages/rdfs/<resource_id>/<version_id>/rdf.yaml

Test summaries are mappings with the following keys saved as .yaml files:

bioimageio_spec_version: # str
bioimageio_core_version: # str
error: # Union[None, str, Dict[str, Any]]
name: # str
nested_errors: # Optional[Dict[str, dict]]
source_name: # str
status: # Union[Literal["passed", "failed"], str]
traceback: # Optional[List[str]]
warnings: # dict

The current keys are defined here.

The partner's GitHub Actions workflow should generate such test summaries and deploy them as configured in https://github.com/bioimage-io/collection-bioimage-io/blob/main/collection_rdf_template.yaml :config:partner:<partner entry>:test_summaries as individual yaml files test_summary*.yaml in the subfolder <resource_id>/<version_id> (example).

The collection-bioimage-io CI combines all partner and the bioimageio core test summaries and deploys this test summary to https://github.com/bioimage-io/collection-bioimage-io/blob/gh-pages/rdfs/<resource_id>/<version_id>.test_summary.yaml, e.g. here from which the bioimage.io website renders them.

• repository with ilastik's test ci: https://github.com/ilastik/bioimage-io-resources
• this github actions workflow is called by the collection CI: test_bioimageio_resources.yaml
• deployed test summaries: https://github.com/ilastik/bioimage-io-resources/tree/gh-pages/test_summaries
• config of this process in collection repo: collection_rdf_template.yaml#L63-L68

• gh workflow called by collection CI: test_bioimageio_resources.yaml
• config in collection repo: collection_rdf_template.yaml#L41-L50
• Specific material from deepImageJ's model repository to run the CI:
  • A drafted GitHub actions workflow to download the installation of Fiji/ImageJ and call the deepImageJ macro that processes the test image and generates an output for each of the models in the repo. The workflow will test that all the models produce the corresponding output tiff file.
  • The workflow calls deepImageJ in headless mode