Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(test-data-generation): add ability to generate invalid load statements #15153

Merged
merged 20 commits into from
May 14, 2024

Conversation

DerekMaggio
Copy link
Contributor

@DerekMaggio DerekMaggio commented May 9, 2024

Overview

This PR aims to add the ability to call protocol_context.load_* methods in a way that generates invalid protocols. There is also a whole lot of refactoring that went into this PR to enable these invalid calls. I removed as much as I could to go into a different PR.

Features

2 main features are exposed: explict_loads and allow_overlapping_loads.

explicit_loads

An explicit load is a protocol_context.load_* statement specified outside of the internal load statement generation logic from a Deck Configuration. These are stored as a dictionary
which takes the slot for the key and the load for the value. Also note, that explicit loads have access to column 4 on staging areas

explict_loads = {
    "a3": <a_heater_shaker_load_statement>,
    "c4": <a_trash_bin_load_statement>,
}

allow_overlapping_loads

When defining a protocol, it is invalid to say, "Load the Heater-Shaker in slot C1 and also load a Trash Bin into slot C1. The allow_overlapping_loads` switch, allows a test writer to override this default behavior and specify the loading of multiple things into the same slot.

How to use these new features

There are 2 main ways that you can use explict_loads and allow_overlapping_loads together: explict_loads defined and allow_overlapping_loads set to False and explict_loads defined and allow_overlapping_loads set to True.

When you have allow_overlapping_loads set to False, if you already have a fixture defined in the slot that the explicit load is acting on, the explicit load REPLACES the existing fixture.

When you have allow_overlapping_loads set to True, if you already have a fixture defined in the slot that the explicit load is acting on, the explicit load is APPENDED AFTER the existing fixture.

Usage

Changelog

Logic

The implementation of explict_loads exists mainly in, ast_helpers.py and load_phase.py.

ast_helpers.py now has 1-to-1 instantation methods for each of the utilized protocol_content.load_* methods.

load_phase.py

  • To remove all isinstance logic, all the helper methods now return lists of assignment statements.
  • Add logic for either overriding or appending

Refactors and supporting content

  • Rename PossibleSlotContents to DeckConfigurationFixtures - now that explicit loads define things in slots, but don't use this object, rename it to make it clearer
    • Also change the list-generating properties to classmethods, shouldn't have to have an object instantiated to get access to those lists
  • Combine all the different datashapes and constants into singular files. I was fighting so many circular imports
  • For constants, switch nearly everything over to typing.Literal instead of enum.Enum- cleans up calling logic
  • Define ProtocolConfiguration dataclass to hold all these different settings instead of passing 5 arguments to PythonProtocolGenerator
  • Note that all the changes in python_protocol_generator don't change any logic. Just usages of classes
  • Refactor test/utils.py into protocol_analysis_validator.py. Parsing the output of failed tests sucked, and was making writing them a nightmare. This fixes that.
  • All pytest tests now use ProtocolConfiguration objects. No more DeckConfiguration object test cases.
  • Add pytest-xdist, these tests are CPU bound so running them in parallel speeds up overall execution. Something to keep in mind as we move forward

Test Plan

  • Manually validate various generated protocols
  • Manually validate analysis response from the various protocols
  • Run ~600 tests per strategy and determine if any failures are due to my generation code or are actual failures. Resolved all the issues, they were all me
  • Run new protocols against analysis in app to check for any weirdness

Review requests

Risk assessment

Low

@DerekMaggio DerekMaggio changed the base branch from add-running-analysis-to-test-data-generation to edge May 9, 2024 20:34
@DerekMaggio DerekMaggio changed the base branch from edge to add-running-analysis-to-test-data-generation May 9, 2024 20:35
@DerekMaggio DerekMaggio force-pushed the parameterize-protocol-generation branch 2 times, most recently from f04219c to 33c703f Compare May 10, 2024 16:42
@DerekMaggio DerekMaggio changed the base branch from add-running-analysis-to-test-data-generation to edge May 10, 2024 20:47
@DerekMaggio DerekMaggio changed the title feat(test-data-generation): parameterize protocol generation (WIP) feat(test-data-generation): parameterize protocol generation May 10, 2024
@DerekMaggio DerekMaggio marked this pull request as ready for review May 13, 2024 15:20
@DerekMaggio DerekMaggio requested review from a team as code owners May 13, 2024 15:20
@DerekMaggio DerekMaggio changed the title feat(test-data-generation): parameterize protocol generation feat(test-data-generation): add ability to generate invalid load statements May 13, 2024
@DerekMaggio DerekMaggio force-pushed the parameterize-protocol-generation branch from 54d32bc to 963a9d3 Compare May 13, 2024 15:38
@DerekMaggio DerekMaggio requested review from CaseyBatten and a team May 13, 2024 15:40
Copy link
Member

@sfoster1 sfoster1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, nice

Copy link
Contributor

@caila-marashaj caila-marashaj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

very cool!

@DerekMaggio DerekMaggio merged commit 79f25a9 into edge May 14, 2024
28 checks passed
Carlos-fernandez pushed a commit that referenced this pull request May 20, 2024
…ements (#15153)

# Overview

This PR aims to add the ability to call `protocol_context.load_*`
methods in a way that generates invalid protocols. There is also a whole
lot of refactoring that went into this PR to enable these invalid calls.
I removed as much as I could to go into a different PR.

## Features 

2 main features are exposed: `explict_loads` and
`allow_overlapping_loads`.

## `explicit_loads`

An explicit load is a `protocol_context.load_*` statement specified
outside of the internal load statement generation logic from a Deck
Configuration. These are stored as a dictionary
which takes the slot for the key and the load for the value. Also note,
that explicit loads have access to column 4 on staging areas

```python
explict_loads = {
    "a3": <a_heater_shaker_load_statement>,
    "c4": <a_trash_bin_load_statement>,
}
```

### `allow_overlapping_loads` 

When defining a protocol, it is invalid to say, "Load the Heater-Shaker
in slot C1 and also load a Trash Bin into slot C1`. The
`allow_overlapping_loads` switch, allows a test writer to override this
default behavior and specify the loading of multiple things into the
same slot.

### How to use these new features

There are 2 main ways that you can use `explict_loads` and
`allow_overlapping_loads` together: `explict_loads` defined and
`allow_overlapping_loads` set to False and `explict_loads` defined and
`allow_overlapping_loads` set to True.

When you have `allow_overlapping_loads` set to False, if you already
have a fixture defined in the slot that the explicit load is acting on,
the explicit load **REPLACES** the existing fixture.

When you have `allow_overlapping_loads` set to True, if you already have
a fixture defined in the slot that the explicit load is acting on, the
explicit load is **APPENDED AFTER** the existing fixture.

### Usage

- See
[python_protocol_generation_strategies.py](https://github.com/Opentrons/opentrons/blob/39a6849787081c26669c13951abb6cce82b96293/test-data-generation/src/test_data_generation/python_protocol_generation/strategy/python_protocol_generation_strategies.py
)

# Changelog

## Logic

The implementation of explict_loads exists mainly in, ast_helpers.py and
load_phase.py.

`ast_helpers.py` now has 1-to-1 instantation methods for each of the
utilized `protocol_content.load_*` methods.

`load_phase.py` 
- To remove all isinstance logic, all the helper methods now return
lists of assignment statements.
- Add logic for either overriding or appending

## Refactors and supporting content

- Rename `PossibleSlotContents` to `DeckConfigurationFixtures` - now
that explicit loads define things in slots, but don't use this object,
rename it to make it clearer
- Also change the list-generating properties to classmethods, shouldn't
have to have an object instantiated to get access to those lists
- Combine all the different datashapes and constants into singular
files. I was fighting so many circular imports
- For constants, switch nearly everything over to `typing.Literal`
instead of `enum.Enum`- cleans up calling logic
- Define ProtocolConfiguration dataclass to hold all these different
settings instead of passing 5 arguments to PythonProtocolGenerator
- Note that all the changes in python_protocol_generator don't change
any logic. Just usages of classes
- Refactor test/utils.py into protocol_analysis_validator.py. Parsing
the output of failed tests sucked, and was making writing them a
nightmare. This fixes that.
- All pytest tests now use ProtocolConfiguration objects. No more
DeckConfiguration object test cases.
- Add `pytest-xdist`, these tests are CPU bound so running them in
parallel speeds up overall execution. Something to keep in mind as we
move forward

# Test Plan

- [x] Manually validate various generated protocols
- [x] Manually validate analysis response from the various protocols
- [x] Run ~600 tests per strategy and determine if any failures are due
to my generation code or are actual failures. **Resolved all the issues,
they were all me**
- [x] Run new protocols against analysis in app to check for any
weirdness

# Review requests

- Please take a look at [the load statement determination
logic](https://github.com/Opentrons/opentrons/pull/15153/files#diff-2835a2dd65821aa83efd7e004726e6b12ea60eea84e33a86ace6b1a6a74989edR165-R210)
- The logic in
[python_protocol_generation_strategies.py](https://github.com/Opentrons/opentrons/blob/161783ffd765d3e5069c60651de1ece3c330a5d0/test-data-generation/src/test_data_generation/python_protocol_generation/strategy/python_protocol_generation_strategies.py)
is quite dense. Any suggestions?

# Risk assessment

Low
Carlos-fernandez pushed a commit that referenced this pull request Jun 3, 2024
…ements (#15153)

# Overview

This PR aims to add the ability to call `protocol_context.load_*`
methods in a way that generates invalid protocols. There is also a whole
lot of refactoring that went into this PR to enable these invalid calls.
I removed as much as I could to go into a different PR.

## Features 

2 main features are exposed: `explict_loads` and
`allow_overlapping_loads`.

## `explicit_loads`

An explicit load is a `protocol_context.load_*` statement specified
outside of the internal load statement generation logic from a Deck
Configuration. These are stored as a dictionary
which takes the slot for the key and the load for the value. Also note,
that explicit loads have access to column 4 on staging areas

```python
explict_loads = {
    "a3": <a_heater_shaker_load_statement>,
    "c4": <a_trash_bin_load_statement>,
}
```

### `allow_overlapping_loads` 

When defining a protocol, it is invalid to say, "Load the Heater-Shaker
in slot C1 and also load a Trash Bin into slot C1`. The
`allow_overlapping_loads` switch, allows a test writer to override this
default behavior and specify the loading of multiple things into the
same slot.

### How to use these new features

There are 2 main ways that you can use `explict_loads` and
`allow_overlapping_loads` together: `explict_loads` defined and
`allow_overlapping_loads` set to False and `explict_loads` defined and
`allow_overlapping_loads` set to True.

When you have `allow_overlapping_loads` set to False, if you already
have a fixture defined in the slot that the explicit load is acting on,
the explicit load **REPLACES** the existing fixture.

When you have `allow_overlapping_loads` set to True, if you already have
a fixture defined in the slot that the explicit load is acting on, the
explicit load is **APPENDED AFTER** the existing fixture.

### Usage

- See
[python_protocol_generation_strategies.py](https://github.com/Opentrons/opentrons/blob/39a6849787081c26669c13951abb6cce82b96293/test-data-generation/src/test_data_generation/python_protocol_generation/strategy/python_protocol_generation_strategies.py
)

# Changelog

## Logic

The implementation of explict_loads exists mainly in, ast_helpers.py and
load_phase.py.

`ast_helpers.py` now has 1-to-1 instantation methods for each of the
utilized `protocol_content.load_*` methods.

`load_phase.py` 
- To remove all isinstance logic, all the helper methods now return
lists of assignment statements.
- Add logic for either overriding or appending

## Refactors and supporting content

- Rename `PossibleSlotContents` to `DeckConfigurationFixtures` - now
that explicit loads define things in slots, but don't use this object,
rename it to make it clearer
- Also change the list-generating properties to classmethods, shouldn't
have to have an object instantiated to get access to those lists
- Combine all the different datashapes and constants into singular
files. I was fighting so many circular imports
- For constants, switch nearly everything over to `typing.Literal`
instead of `enum.Enum`- cleans up calling logic
- Define ProtocolConfiguration dataclass to hold all these different
settings instead of passing 5 arguments to PythonProtocolGenerator
- Note that all the changes in python_protocol_generator don't change
any logic. Just usages of classes
- Refactor test/utils.py into protocol_analysis_validator.py. Parsing
the output of failed tests sucked, and was making writing them a
nightmare. This fixes that.
- All pytest tests now use ProtocolConfiguration objects. No more
DeckConfiguration object test cases.
- Add `pytest-xdist`, these tests are CPU bound so running them in
parallel speeds up overall execution. Something to keep in mind as we
move forward

# Test Plan

- [x] Manually validate various generated protocols
- [x] Manually validate analysis response from the various protocols
- [x] Run ~600 tests per strategy and determine if any failures are due
to my generation code or are actual failures. **Resolved all the issues,
they were all me**
- [x] Run new protocols against analysis in app to check for any
weirdness

# Review requests

- Please take a look at [the load statement determination
logic](https://github.com/Opentrons/opentrons/pull/15153/files#diff-2835a2dd65821aa83efd7e004726e6b12ea60eea84e33a86ace6b1a6a74989edR165-R210)
- The logic in
[python_protocol_generation_strategies.py](https://github.com/Opentrons/opentrons/blob/161783ffd765d3e5069c60651de1ece3c330a5d0/test-data-generation/src/test_data_generation/python_protocol_generation/strategy/python_protocol_generation_strategies.py)
is quite dense. Any suggestions?

# Risk assessment

Low
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants