-
Notifications
You must be signed in to change notification settings - Fork 176
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(test-data-generation): add ability to generate invalid load statements #15153
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
f04219c
to
33c703f
Compare
54d32bc
to
963a9d3
Compare
sfoster1
approved these changes
May 13, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, nice
caila-marashaj
approved these changes
May 14, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
very cool!
Carlos-fernandez
pushed a commit
that referenced
this pull request
May 20, 2024
…ements (#15153) # Overview This PR aims to add the ability to call `protocol_context.load_*` methods in a way that generates invalid protocols. There is also a whole lot of refactoring that went into this PR to enable these invalid calls. I removed as much as I could to go into a different PR. ## Features 2 main features are exposed: `explict_loads` and `allow_overlapping_loads`. ## `explicit_loads` An explicit load is a `protocol_context.load_*` statement specified outside of the internal load statement generation logic from a Deck Configuration. These are stored as a dictionary which takes the slot for the key and the load for the value. Also note, that explicit loads have access to column 4 on staging areas ```python explict_loads = { "a3": <a_heater_shaker_load_statement>, "c4": <a_trash_bin_load_statement>, } ``` ### `allow_overlapping_loads` When defining a protocol, it is invalid to say, "Load the Heater-Shaker in slot C1 and also load a Trash Bin into slot C1`. The `allow_overlapping_loads` switch, allows a test writer to override this default behavior and specify the loading of multiple things into the same slot. ### How to use these new features There are 2 main ways that you can use `explict_loads` and `allow_overlapping_loads` together: `explict_loads` defined and `allow_overlapping_loads` set to False and `explict_loads` defined and `allow_overlapping_loads` set to True. When you have `allow_overlapping_loads` set to False, if you already have a fixture defined in the slot that the explicit load is acting on, the explicit load **REPLACES** the existing fixture. When you have `allow_overlapping_loads` set to True, if you already have a fixture defined in the slot that the explicit load is acting on, the explicit load is **APPENDED AFTER** the existing fixture. ### Usage - See [python_protocol_generation_strategies.py](https://github.com/Opentrons/opentrons/blob/39a6849787081c26669c13951abb6cce82b96293/test-data-generation/src/test_data_generation/python_protocol_generation/strategy/python_protocol_generation_strategies.py ) # Changelog ## Logic The implementation of explict_loads exists mainly in, ast_helpers.py and load_phase.py. `ast_helpers.py` now has 1-to-1 instantation methods for each of the utilized `protocol_content.load_*` methods. `load_phase.py` - To remove all isinstance logic, all the helper methods now return lists of assignment statements. - Add logic for either overriding or appending ## Refactors and supporting content - Rename `PossibleSlotContents` to `DeckConfigurationFixtures` - now that explicit loads define things in slots, but don't use this object, rename it to make it clearer - Also change the list-generating properties to classmethods, shouldn't have to have an object instantiated to get access to those lists - Combine all the different datashapes and constants into singular files. I was fighting so many circular imports - For constants, switch nearly everything over to `typing.Literal` instead of `enum.Enum`- cleans up calling logic - Define ProtocolConfiguration dataclass to hold all these different settings instead of passing 5 arguments to PythonProtocolGenerator - Note that all the changes in python_protocol_generator don't change any logic. Just usages of classes - Refactor test/utils.py into protocol_analysis_validator.py. Parsing the output of failed tests sucked, and was making writing them a nightmare. This fixes that. - All pytest tests now use ProtocolConfiguration objects. No more DeckConfiguration object test cases. - Add `pytest-xdist`, these tests are CPU bound so running them in parallel speeds up overall execution. Something to keep in mind as we move forward # Test Plan - [x] Manually validate various generated protocols - [x] Manually validate analysis response from the various protocols - [x] Run ~600 tests per strategy and determine if any failures are due to my generation code or are actual failures. **Resolved all the issues, they were all me** - [x] Run new protocols against analysis in app to check for any weirdness # Review requests - Please take a look at [the load statement determination logic](https://github.com/Opentrons/opentrons/pull/15153/files#diff-2835a2dd65821aa83efd7e004726e6b12ea60eea84e33a86ace6b1a6a74989edR165-R210) - The logic in [python_protocol_generation_strategies.py](https://github.com/Opentrons/opentrons/blob/161783ffd765d3e5069c60651de1ece3c330a5d0/test-data-generation/src/test_data_generation/python_protocol_generation/strategy/python_protocol_generation_strategies.py) is quite dense. Any suggestions? # Risk assessment Low
Carlos-fernandez
pushed a commit
that referenced
this pull request
Jun 3, 2024
…ements (#15153) # Overview This PR aims to add the ability to call `protocol_context.load_*` methods in a way that generates invalid protocols. There is also a whole lot of refactoring that went into this PR to enable these invalid calls. I removed as much as I could to go into a different PR. ## Features 2 main features are exposed: `explict_loads` and `allow_overlapping_loads`. ## `explicit_loads` An explicit load is a `protocol_context.load_*` statement specified outside of the internal load statement generation logic from a Deck Configuration. These are stored as a dictionary which takes the slot for the key and the load for the value. Also note, that explicit loads have access to column 4 on staging areas ```python explict_loads = { "a3": <a_heater_shaker_load_statement>, "c4": <a_trash_bin_load_statement>, } ``` ### `allow_overlapping_loads` When defining a protocol, it is invalid to say, "Load the Heater-Shaker in slot C1 and also load a Trash Bin into slot C1`. The `allow_overlapping_loads` switch, allows a test writer to override this default behavior and specify the loading of multiple things into the same slot. ### How to use these new features There are 2 main ways that you can use `explict_loads` and `allow_overlapping_loads` together: `explict_loads` defined and `allow_overlapping_loads` set to False and `explict_loads` defined and `allow_overlapping_loads` set to True. When you have `allow_overlapping_loads` set to False, if you already have a fixture defined in the slot that the explicit load is acting on, the explicit load **REPLACES** the existing fixture. When you have `allow_overlapping_loads` set to True, if you already have a fixture defined in the slot that the explicit load is acting on, the explicit load is **APPENDED AFTER** the existing fixture. ### Usage - See [python_protocol_generation_strategies.py](https://github.com/Opentrons/opentrons/blob/39a6849787081c26669c13951abb6cce82b96293/test-data-generation/src/test_data_generation/python_protocol_generation/strategy/python_protocol_generation_strategies.py ) # Changelog ## Logic The implementation of explict_loads exists mainly in, ast_helpers.py and load_phase.py. `ast_helpers.py` now has 1-to-1 instantation methods for each of the utilized `protocol_content.load_*` methods. `load_phase.py` - To remove all isinstance logic, all the helper methods now return lists of assignment statements. - Add logic for either overriding or appending ## Refactors and supporting content - Rename `PossibleSlotContents` to `DeckConfigurationFixtures` - now that explicit loads define things in slots, but don't use this object, rename it to make it clearer - Also change the list-generating properties to classmethods, shouldn't have to have an object instantiated to get access to those lists - Combine all the different datashapes and constants into singular files. I was fighting so many circular imports - For constants, switch nearly everything over to `typing.Literal` instead of `enum.Enum`- cleans up calling logic - Define ProtocolConfiguration dataclass to hold all these different settings instead of passing 5 arguments to PythonProtocolGenerator - Note that all the changes in python_protocol_generator don't change any logic. Just usages of classes - Refactor test/utils.py into protocol_analysis_validator.py. Parsing the output of failed tests sucked, and was making writing them a nightmare. This fixes that. - All pytest tests now use ProtocolConfiguration objects. No more DeckConfiguration object test cases. - Add `pytest-xdist`, these tests are CPU bound so running them in parallel speeds up overall execution. Something to keep in mind as we move forward # Test Plan - [x] Manually validate various generated protocols - [x] Manually validate analysis response from the various protocols - [x] Run ~600 tests per strategy and determine if any failures are due to my generation code or are actual failures. **Resolved all the issues, they were all me** - [x] Run new protocols against analysis in app to check for any weirdness # Review requests - Please take a look at [the load statement determination logic](https://github.com/Opentrons/opentrons/pull/15153/files#diff-2835a2dd65821aa83efd7e004726e6b12ea60eea84e33a86ace6b1a6a74989edR165-R210) - The logic in [python_protocol_generation_strategies.py](https://github.com/Opentrons/opentrons/blob/161783ffd765d3e5069c60651de1ece3c330a5d0/test-data-generation/src/test_data_generation/python_protocol_generation/strategy/python_protocol_generation_strategies.py) is quite dense. Any suggestions? # Risk assessment Low
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Overview
This PR aims to add the ability to call
protocol_context.load_*
methods in a way that generates invalid protocols. There is also a whole lot of refactoring that went into this PR to enable these invalid calls. I removed as much as I could to go into a different PR.Features
2 main features are exposed:
explict_loads
andallow_overlapping_loads
.explicit_loads
An explicit load is a
protocol_context.load_*
statement specified outside of the internal load statement generation logic from a Deck Configuration. These are stored as a dictionarywhich takes the slot for the key and the load for the value. Also note, that explicit loads have access to column 4 on staging areas
allow_overlapping_loads
When defining a protocol, it is invalid to say, "Load the Heater-Shaker in slot C1 and also load a Trash Bin into slot C1
. The
allow_overlapping_loads` switch, allows a test writer to override this default behavior and specify the loading of multiple things into the same slot.How to use these new features
There are 2 main ways that you can use
explict_loads
andallow_overlapping_loads
together:explict_loads
defined andallow_overlapping_loads
set to False andexplict_loads
defined andallow_overlapping_loads
set to True.When you have
allow_overlapping_loads
set to False, if you already have a fixture defined in the slot that the explicit load is acting on, the explicit load REPLACES the existing fixture.When you have
allow_overlapping_loads
set to True, if you already have a fixture defined in the slot that the explicit load is acting on, the explicit load is APPENDED AFTER the existing fixture.Usage
Changelog
Logic
The implementation of explict_loads exists mainly in, ast_helpers.py and load_phase.py.
ast_helpers.py
now has 1-to-1 instantation methods for each of the utilizedprotocol_content.load_*
methods.load_phase.py
Refactors and supporting content
PossibleSlotContents
toDeckConfigurationFixtures
- now that explicit loads define things in slots, but don't use this object, rename it to make it clearertyping.Literal
instead ofenum.Enum
- cleans up calling logicpytest-xdist
, these tests are CPU bound so running them in parallel speeds up overall execution. Something to keep in mind as we move forwardTest Plan
Review requests
Risk assessment
Low