Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create a compare ORT command #7072

Closed
sschuberth opened this issue May 31, 2023 · 0 comments
Closed

Create a compare ORT command #7072

sschuberth opened this issue May 31, 2023 · 0 comments
Assignees
Labels
cli About the Command Line Interface new feature Issues that are considered to be new features

Comments

@sschuberth
Copy link
Member

In several scenarios, one may be interested in whether two ORT (pipeline) runs yield the "same" result (modulo some varying fields like timestamps etc.). Use cases include:

  1. Verify that a new ORT version does not unexpectedly generate different results when scanning a "gold standard" project in a fixed revision (and with a fixed configuration) again; i.e. use it as a check in end-to-end / integration testing before taking a new ORT version into use.
  2. Verify that a configuration change for a fixed ORT version does not cause any difference in (unexpected areas of) the result.
  3. For a fixed ORT version and configuration, see the difference in results between two different revisions of the same project.

The idea is to start with a trivial implementation for use-case 1. that just tells whether two ORT results are the same or not, without actually "interpreting" any differences on a semantic level that pays attention to the structure of the data model.

@sschuberth sschuberth self-assigned this May 31, 2023
@sschuberth sschuberth added new feature Issues that are considered to be new features cli About the Command Line Interface labels May 31, 2023
sschuberth added a commit that referenced this issue Jun 7, 2023
Resolves #7072.

Signed-off-by: Sebastian Schuberth <[email protected]>
sschuberth added a commit that referenced this issue Jun 7, 2023
Resolves #7072.

Signed-off-by: Sebastian Schuberth <[email protected]>
sschuberth added a commit that referenced this issue Jun 8, 2023
Resolves #7072.

Signed-off-by: Sebastian Schuberth <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cli About the Command Line Interface new feature Issues that are considered to be new features
Projects
None yet
Development

No branches or pull requests

1 participant