Sub test execution and output updates #79

karlnaden · 2022-12-27T15:08:06Z

Summary

Add extensions to support subtest execution from within a TestScript
Improve the feedback mechanism for profile validation errors specifically, but also executions in general

New behavior

Output stuff

validate profile id success and failure messages include validation errors and warnings
skip logic fixed to match TestScript spec
TestReports exported to run-specific folder
test inputs included in outputs

Subtest stuff

reports no longer indexed by script name, just a list, to allow the same script to be executed multiple times
moved all dynamic details outside of the TestScript (e.g., dynamic variables) to support easier copying of runnables
added extension indicating subtest execution

Code changes

Generally minor tweaks to support cloning of runnables such that they can be executed within the context of another test

Testing guidance

output updates: bundle exec bin/testscript_engine execute --config config-multipleMCODE.yml
expected results: not all will succeed
subtest: bundle exec bin/testscript_engine execute --config config-mCODESubtest.yml
expected results (failures expected with internal FHIR path, should succeed with external FHIRPath support)

   SUMMARY OF EXECUTION RESULTS: 
      Execution of [UsMCODECancerPatientReadProfile] with inputs 'targetResourceIdCancerPatient=cancer-patient-jenny-m' failed with score: [75.0].
      Execution of [UsMcodeProfileSearchFromPatientSubTest] with inputs 'cancerPatientId=cancer-patient-jenny-m' failed with score: [60.0].

subtest each: bundle exec ruby bin/testscript_engine execute --config spec/examples/subtest/config-subtest.yml
expected results (these failures are expected):

   SUMMARY OF EXECUTION RESULTS: 
      Execution of [SubtestToExecuteExample] with inputs 'checkValue=Chalmers' passed.
      Execution of [SubtestToExecuteExample] with inputs 'checkValue=Chalmers' passed.
      Execution of [SubtestToExecuteExample] with inputs 'checkValue=Chalmers' passed.

      Execution of [SubtestToExecuteExample] with inputs 'checkValue=Windsor' failed with score: [0.0].
      Execution of [SubtestToExecuteExample] with inputs 'checkValue=Windsor' failed with score: [0.0].
      Execution of [SubtestExecutionExample] with inputs 'directExecuteInput=Chalmers' failed with score: [67.0].

move display to write_reports add testscript.name to testreport reference

remove TestScript variable extension add subtest config file

* add fhirpath_service * Initial commit * Add testscript for internal fhirpath testing * Add external validator test case * Add mCODE testscripts * Minor change * Handle only equals operator * Handle only String value based on the FHIR spec * check for multiple values in assertion logic * Update README.md Co-authored-by: Jack Fraser <[email protected]> Co-authored-by: Jaehoon Lee <[email protected]> Co-authored-by: Jaehoon Lee <[email protected]> Co-authored-by: Karl Naden <[email protected]>

move display to write_reports add testscript.name to testreport reference

remove TestScript variable extension add subtest config file

…t-engine into SubTestExecution

karlnaden · 2022-12-27T21:42:03Z

bundle exec bin/testscript_engine execute --config config-mCODESubtest.yml should now execute subtests. Had to adjust to get to work with HAPI FHIRPath. Should result in

   SUMMARY OF EXECUTION RESULTS: 
      Execution of [UsMCODEComorbiditiesReadProfile] with inputs 'targetResourceIdComorbidities=cancer-related-comorbidities-jenny-m' passed.

      Execution of [UsMCODECancerPatientReadProfile] with inputs 'targetResourceIdCancerPatient=cancer-patient-jenny-m' failed with score: [75.0].
      Execution of [UsMcodeProfileSearchFromPatientSubTest] with inputs 'cancerPatientId=cancer-patient-jenny-m' failed with score: [71.0].

jhlee-mitre · 2022-12-28T04:17:16Z

lib/testscript_engine/message_handler.rb

@@ -103,7 +107,7 @@ def make_runnables
 end

 def run(*args)
- print_out messages(:begin_runnable_execution, script.id)
+ print_out messages(:begin_runnable_execution, script.name)


I agree with that use of name instead of is good idea.

jhlee-mitre · 2022-12-28T04:34:14Z

lib/testscript_engine/testscript_runnable.rb

 self.get_fixture_block = block
 self.options = options
 self.client_util = FHIR::Client.new('')
 self.available_profiles = available_profiles
+ self.engine = engine


I think by having engine object part of runnable initialization has pros and cons and we may make it a bit clearer or simpler as to what information the engine handles vs the runnable handles. But that is out of the scope in the last hour of this project. For now I'm okay to this change if you'd prefer that way.

agreed - this was a quick solution that could be handled better in another way. Main need is to pass the report back to the engine, which could be done in another way.

jhlee-mitre · 2022-12-28T04:35:18Z

lib/testscript_engine/testscript_runnable.rb

@@ -77,16 +79,16 @@ def preprocess

 def setup
 return info(:no_setup) unless script.setup
- handle_actions(script.setup.action, true)
+ handle_actions(script.setup.action, :setup)


This is a good way to detail the configuration of fail on error in different phases of testing

jhlee-mitre

No concerns and approved

karlnaden added 10 commits December 23, 2022 15:46

update skip logic to match TestScript docs

b5f507a

external validator profile validation feedback

e2c1d46

internal validator return validate profile details

b1ede00

output TestReports for each execution

b3c5af5

fixed testreport unit tests

a96ccae

include dynamic variable values in summary and TestReport

603535f

make_one_runnable method

d7b69c8

avoid report clashes

8e6abd2

move display to write_reports add testscript.name to testreport reference

basic subtest execution with variable binding

360a3cb

harmonize after merge

9ee391c

remove TestScript variable extension add subtest config file

karlnaden requested a review from jhlee-mitre December 27, 2022 15:08

karlnaden mentioned this pull request Dec 27, 2022

Validate profile id feedback #78

Closed

karlnaden and others added 16 commits December 27, 2022 14:18

subtesteach execution extension

2c6a905

update skip logic to match TestScript docs

25d55b3

external validator profile validation feedback

731849f

internal validator return validate profile details

07bd7b9

output TestReports for each execution

74f1a09

fixed testreport unit tests

cb07caa

include dynamic variable values in summary and TestReport

c21ebad

make_one_runnable method

079ed99

avoid report clashes

0200cea

move display to write_reports add testscript.name to testreport reference

basic subtest execution with variable binding

e1f2143

harmonize after merge

cab6a62

remove TestScript variable extension add subtest config file

subtesteach execution extension

d7ca8ab

Merge branch 'SubTestExecution' of github.com:fhir-crucible/testscrip…

c5c9865

…t-engine into SubTestExecution

remove duplicate documentation

0f7e6ee

working mCODE subtest

f2abe06

simplified testscript

926e9dd

jhlee-mitre reviewed Dec 28, 2022

View reviewed changes

jhlee-mitre approved these changes Dec 28, 2022

View reviewed changes

karlnaden added 6 commits December 28, 2022 00:17

generated search and read mCODE test

dd6e1cd

fixed search and read test

28c5fc0

extension definition updates, IG link

acf983d

ig pointer update

bc73053

test fix

50ef850

interpretation guidance and additional TestReport flags

2e8d623

karlnaden merged commit becf09f into main Dec 29, 2022

karlnaden deleted the SubTestExecution branch December 29, 2022 20:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sub test execution and output updates #79

Sub test execution and output updates #79

karlnaden commented Dec 27, 2022 •

edited

Loading

karlnaden commented Dec 27, 2022

jhlee-mitre Dec 28, 2022

jhlee-mitre Dec 28, 2022

karlnaden Dec 28, 2022

jhlee-mitre Dec 28, 2022

jhlee-mitre left a comment

Sub test execution and output updates #79

Sub test execution and output updates #79

Conversation

karlnaden commented Dec 27, 2022 • edited Loading

Summary

New behavior

Code changes

Testing guidance

karlnaden commented Dec 27, 2022

jhlee-mitre Dec 28, 2022

Choose a reason for hiding this comment

jhlee-mitre Dec 28, 2022

Choose a reason for hiding this comment

karlnaden Dec 28, 2022

Choose a reason for hiding this comment

jhlee-mitre Dec 28, 2022

Choose a reason for hiding this comment

jhlee-mitre left a comment

Choose a reason for hiding this comment

karlnaden commented Dec 27, 2022 •

edited

Loading