feature(api, robot-server): Allow fixit commands to recover from an error #14908

TamarZanzouri · 2024-04-15T20:25:18Z

Overview

closes https://opentrons.atlassian.net/browse/EXEC-285.
Allow FIXIT command intent when waiting for recovery and allow resume from recovery.

Test Plan

Tested on dev server works as expected!

set error recovery ff to true OT_API_FF_enableErrorRecoveryExperiments=1
upload the following protocol - it should fail:

from opentrons import protocol_api

requirements = {"robotType": "Flex", "apiLevel": "2.17"}


def run(protocol: protocol_api.ProtocolContext) -> None:
    protocol.load_trash_bin("A3")

    tip_rack = protocol.load_labware("opentrons_flex_96_tiprack_50ul", "C2")
    reservoir = protocol.load_labware(
        "opentrons_96_wellplate_200ul_pcr_full_skirt", "C3"
    )
    pipette = protocol.load_instrument(
        "flex_1channel_50", mount="left", tip_racks=[tip_rack]
    )

    for source, dest in zip(reservoir.columns()[0], reservoir.columns()[1]):
        pipette.pick_up_tip()
        pipette.move_to(source.top())
        pipette.move_to(dest.top())
        pipette.drop_tip()

GET /run and make sure it is in AWAITING_RECOVERY mode.
POST a few commands with intent=FIXIT
make sure commands are executed.
when done, issue a resumeFromRecovery action. # need to check if this is true

Review requests

Changes make sense? did I miss anything?

Risk assessment

Medium. changing add_command in pe and create_command in router.

… merge fixes

sfoster1

Looks very good structurally, but I think we want a couple defensive programming enhancements and we should enforce that failed command ids only exist for fixits

api/src/opentrons/protocol_engine/commands/hash_command_params.py

api/src/opentrons/protocol_engine/protocol_engine.py

sfoster1 · 2024-04-17T20:43:21Z

robot-server/robot_server/runs/router/commands_router.py

@@ -209,7 +223,7 @@ async def create_run_command(
 command_create = request_body.data.copy(update={"intent": command_intent})

 try:
- command = protocol_engine.add_command(command_create)
+ command = protocol_engine.add_command(command_create, failed_command_id)

 except pe_errors.SetupCommandNotAllowedError as e:
 raise CommandNotAllowed.from_exc(e).as_error(status.HTTP_409_CONFLICT)


should add explicit handling and 400 errors if this was a non-fixit command that specified a failed command id

SyntaxColoring

Looks good so far!

api/src/opentrons/protocol_engine/state/commands.py

api/tests/opentrons/protocol_engine/state/test_command_history.py

robot-server/robot_server/runs/router/commands_router.py

dont pick setup when in waiting for recovery

codecov · 2024-04-19T18:13:35Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.31%. Comparing base (8f50b08) to head (364c42d).
Report is 94 commits behind head on edge.

❗ Current head 364c42d differs from pull request most recent head 775bb8a. Consider uploading reports for the commit 775bb8a to get more accurate results

Additional details and impacted files

@@             Coverage Diff             @@
##             edge   #14908       +/-   ##
===========================================
+ Coverage   67.50%   81.31%   +13.80%     
===========================================
  Files        2521      118     -2403     
  Lines       72090     4029    -68061     
  Branches     9311        0     -9311     
===========================================
- Hits        48666     3276    -45390     
+ Misses      21228      753    -20475     
+ Partials     2196        0     -2196

Flag	Coverage Δ
api	`?`
app	`?`
components	`?`
g-code-testing	`92.43% <ø> (ø)`
hardware	`?`
hardware-testing	`?`
labware-library	`?`
notify-server	`?`
ot3-gravimetric-test	`?`
protocol-designer	`?`
react-api-client	`?`
robot-server	`?`
shared-data	`76.11% <ø> (+0.79%)`	⬆️
step-generation	`?`
system-server	`?`
update-server	`?`
usb-bridge	`?`

Flags with carried forward coverage won't be shown. Click here to find out more.

see 2410 files with indirect coverage changes

sfoster1

Looks really good except for the one sneaky little print!

robot-server/robot_server/runs/router/commands_router.py

api/tests/opentrons/protocol_engine/test_protocol_engine.py

sfoster1

Looks good to me!

SyntaxColoring

A few small things. And I think CI tests are failing, maybe because of the raise in pickUpTip. Otherwise, looks great, thanks!

shared-data/fixture/schemas/1.json

api/src/opentrons/protocol_engine/commands/command.py

api/src/opentrons/protocol_engine/commands/pick_up_tip.py

SyntaxColoring

LGTM if CI passes, thank you!

* edge: (194 commits) fix(app): clone run with RTPs from HistoricalProtocolRun (#14959) fix(api): Filter out `air_gap()` calls as higher-order commands (#14985) fix(app): fix infinitely re-rendering/never rendering firmware success toasts (#14981) feat(api): add option to ignore different tip presence states (#14980) feat(opentrons-ai-client) add input textbox to container (#14968) fix(app): add robotSerialNumber to proceedToRun event (#14976) fix(api): remove homing patch fix for right mount when a 96-channel is attached (#14975) feat(api-client,app,react-api-client): upload splash logo from desktop app (#14941) fix(robot-server): notify /runs when a non-current run is deleted (#14974) feature(api, robot-server): Allow fixit commands to recover from an error (#14908) feat(hardware-testing): enable multi sensor processing in liquid probe (#14883) fix(app): prevent "run again" banner from rendering after navigating away from the current run (#14973) refactor(components): refactor roundtab stories (#14956) refactor(protocol-designer): assign module slot in createFileWizard instead of modal (#14951) fix(app, api-client): fix choose protocol slideout issue (#14949) refactor(protocol-designer): tip position modal max values round down (#14972) feat(app): add tiprack selection step to quick transfer flow (#14950) ci(shared-data): install dependencies in workflow (#14958) fix(components): fix icon stories (#14969) feat(opentrons-ai-client): introduce react-markdown to chat display component (#14965) ...

…rror (#14908)

TamarZanzouri added 19 commits April 4, 2024 14:10

added schema definition for fixtures

59710a2

initial commit WIP

755abd3

WIP added fixit commands to action validation

8fcbcee

WIP commands_history now containing fixit queue and executed commands

eb2caed

added logic for checking failed command and adding to fixit queue

3d1f734

fixed circular dependency and added test

b1bde23

added tests

942f8f0

wip next command to execute

04bb500

get next queued prioritized fixit

d28fe8d

WIP e2e test for fixit recovery

e8d2fc0

WIP resume from recovery

e059aef

merge

0840846

WIP pr feedback - move failed_command_id to base command ad route and…

9517147

… merge fixes

removed tests that are not in prod

9dfa58a

WIP failed_command_id from router to BaseCommand

a20fdda

fixed fixit in hash, renaming to protocol hash WIP

5e631ec

has protocol and command queue action set to None

f7511c0

moved setup and fixit hashing logic to add_command

e8436d6

removed raise and fix failing test

a66a8b9

sfoster1 requested changes Apr 17, 2024

View reviewed changes

fixed rehearsal

239fad5

SyntaxColoring reviewed Apr 18, 2024

View reviewed changes

TamarZanzouri added 6 commits April 18, 2024 15:04

added assert for failed command id and tests

0090dd6

bad request when giving failed command and non fixit

ca25777

dont access private methods from tests WIP

746b3e0

dont pick setup when in waiting for recovery

do not access private methods from test

86e9651

changed all tests to not access private methods

2527555

raise from add_command and catch in router

c736f81

raise 400 with bad data

764bf9d

sfoster1 requested changes Apr 19, 2024

View reviewed changes

robot-server/robot_server/runs/router/commands_router.py Outdated Show resolved Hide resolved

robot-server/robot_server/runs/router/commands_router.py Outdated Show resolved Hide resolved

TamarZanzouri commented Apr 19, 2024

View reviewed changes

api/tests/opentrons/protocol_engine/test_protocol_engine.py Outdated Show resolved Hide resolved

TamarZanzouri added 2 commits April 19, 2024 16:49

Update robot-server/robot_server/runs/router/commands_router.py

9d464a6

Update api/tests/opentrons/protocol_engine/test_protocol_engine.py

b0249a7

TamarZanzouri requested a review from sfoster1 April 19, 2024 20:50

Merge branch 'edge' into EXEC-285-add-fixit-commands

2a494ec

sfoster1 approved these changes Apr 19, 2024

View reviewed changes

TamarZanzouri changed the title ~~Exec 285 add fixit commands~~ feature(api, robot-server): Allow fixit commands to recover from an error Apr 19, 2024

TamarZanzouri and others added 6 commits April 19, 2024 17:08

linting

4234564

linting

2120bee

looks like the editor from the 80s wins again

0b3229b

lint-js

e6ff173

router name typo

5877e38

changed return error type

5da6b3a

SyntaxColoring reviewed Apr 22, 2024

View reviewed changes

shared-data/fixture/schemas/1.json Outdated Show resolved Hide resolved

api/src/opentrons/protocol_engine/commands/command.py Outdated Show resolved Hide resolved

api/src/opentrons/protocol_engine/commands/pick_up_tip.py Outdated Show resolved Hide resolved

TamarZanzouri and others added 4 commits April 22, 2024 12:59

reverted raise for testing

bd9d061

rename to failedCommandId

364c42d

Delete shared-data/fixture/schemas/1.json

635d00a

fixed failing tests

7bbc78b

SyntaxColoring approved these changes Apr 22, 2024

View reviewed changes

TamarZanzouri added 3 commits April 22, 2024 14:35

lint

b2ed3cf

format-js

057f974

removed tavern test for fixit

775bb8a

TamarZanzouri merged commit 2d57126 into edge Apr 22, 2024
41 checks passed

TamarZanzouri deleted the EXEC-285-add-fixit-commands branch April 22, 2024 19:44

shlokamin restored the EXEC-285-add-fixit-commands branch April 25, 2024 23:08

shlokamin deleted the EXEC-285-add-fixit-commands branch April 25, 2024 23:10

TamarZanzouri mentioned this pull request Apr 26, 2024

fix(api): disallow hashing we using a fixit command #15026

Merged

Carlos-fernandez pushed a commit that referenced this pull request May 20, 2024

feature(api, robot-server): Allow fixit commands to recover from an e…

eb21f4e

…rror (#14908)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature(api, robot-server): Allow fixit commands to recover from an error #14908

feature(api, robot-server): Allow fixit commands to recover from an error #14908

TamarZanzouri commented Apr 15, 2024 •

edited

Loading

sfoster1 left a comment

sfoster1 Apr 17, 2024

SyntaxColoring left a comment

codecov bot commented Apr 19, 2024 •

edited

Loading

sfoster1 left a comment

sfoster1 left a comment

SyntaxColoring left a comment

SyntaxColoring left a comment

feature(api, robot-server): Allow fixit commands to recover from an error #14908

feature(api, robot-server): Allow fixit commands to recover from an error #14908

Conversation

TamarZanzouri commented Apr 15, 2024 • edited Loading

Overview

Test Plan

Review requests

Risk assessment

sfoster1 left a comment

Choose a reason for hiding this comment

sfoster1 Apr 17, 2024

Choose a reason for hiding this comment

SyntaxColoring left a comment

Choose a reason for hiding this comment

codecov bot commented Apr 19, 2024 • edited Loading

Codecov Report

sfoster1 left a comment

Choose a reason for hiding this comment

sfoster1 left a comment

Choose a reason for hiding this comment

SyntaxColoring left a comment

Choose a reason for hiding this comment

SyntaxColoring left a comment

Choose a reason for hiding this comment

TamarZanzouri commented Apr 15, 2024 •

edited

Loading

codecov bot commented Apr 19, 2024 •

edited

Loading