-
Notifications
You must be signed in to change notification settings - Fork 177
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
question: how to safely handle a keyboard interrupt in a protocol #11360
Comments
Are you able to post your protocol, or some minimal reproduction that shows how and where exactly you are using a |
Hi Mike, |
You may email me directly at mike at opentrons dot com. However, a full protocol may be difficult for me to read or reason with. A small reproduction protocol would be much more helpful, if possible. It would also be helpful simply to see how you are constructing your
|
Hi Mike, import opentrons.execute, from opentrons import protocol_api
protocol = opentrons.execute.get_protocol_api(opentrons.protocol_api.MAX_SUPPORTED_VERSION)
def run(protocol: protocol_api.ProtocolContext):
####_(protocol defined within the run function)_
try:
run(protocol)
protocol.home()
except KeyboardInterrupt:
print(" user stopped program!")
protocol.home()
for i in protocol.loaded_instruments.values():
if i.has_tip:
i.drop_tip()
except Exception as e:
traceback.print_exc()
protocol.home()
for i in protocol.loaded_instruments.values():
if i.has_tip:
i.drop_tip() So to answer your questions:
Best, |
Thanks for the snippet, that is helpful! I definitely think this is an unsafe construct. The fact that it worked for a while just means you got luck with the timings of your ctrl-c presses. It's very important that the hardware is told to
Without the halt, this exception handler as written may start interleaving requests to the hardware layer (which is running asynchronously in another thread), causing unexpected movements like the ones you observed. I'm going to need to look into how to best do this in Jupyter / command line, but it will likely be something along the lines of "move recovery to its own script / process so that everything can settle after a |
Hi Mike, |
@laura-wyzer it's going to take me until Monday to really start testing this out, but in the mean time, do you know what happens to your protocol if you remove the |
Hi Mike, the protocol does halt if you remove the try/except block and press ctrl+c. |
Hi Mike, |
Hi, I don't know if its useful, but I wanted to pitch in and say that we are seeing a similar issue on our end that matches the behavior @mcous is mentioning. It seems like whenever a set of instructions is halted mid action (pick up tip, drop tip) and then an action is immediately queued, the robot will finish whatever it was doing after the next queued action. This behavior persists across separate jupyter notebook blocks. If I interrupt the interpreter and immediately queue the 'recover and reset' block, the 'recover and reset' block will run and then the last behavior (ex 'pick up tip') that was interrupted will complete (ex 'home the pipette above where the tips were picked up'). All this pointing to the issue that whatever is handling robot actions on the back end is vulnerable to race conditions. |
The existing protocol execution system for Python protocols is pretty fundamentally susceptible to race conditions if you're trying to reach in and cancel a protocol run from the same place you are triggering protocol commands. In other words, SSH and Jupyter. We've been on a multi-year journey of rearchitecting this system, and we're making progress! JSON protocols have been moved to the new system, but Python protocols are still a ways out. The Opentrons App, however, does not suffer these problems, because it communicates with the robot over an HTTP API. It uses this API to upload a protocol file and kick off a run. Starting, pausing, and stopping the run can all be accomplished with subsequent HTTP requests, and since they come in externally, they can safely and gracefully shut down the run. If you are able to use HTTP instead of Jupyter or SSH, I think you might have a better experience. Is this something that your workflow would be amenable to? For example, you could write a Python script that runs from your own computer to upload the protocol, run it, and wait for it to complete. In that script, you could wire a I can add more details to this thread if you're interested! |
Hi ! Please can you add more details to this thread ? |
Hi! The way I ended up solving this for my purposes was not using keyboard interrupt; it was simpler for us to simply kill the process on the robot and then home the robot. Here's the code we run:
When we want to cancel something, we send the above code to the robot and execute it by running this code on the computer we have connected to the OT2:
This slows down our cancel by about 10 seconds. If you store the cancel script on the robot itself and execute the cancel directly through the command line, it's pretty much instantaneous. |
Thank you ! |
Overview
A program we wrote with the Opentrons API uses a try/except block so that the program can stop, home the robot, and drop tips when the user hits Ctrl+C. This has been working for weeks, then all of the sudden had unexpected behavior when we interrupted the program as it was picking up a tip. After homing the robot, the pipette rapidly traversed the deck and jammed into the loaded labware, cause the tip to bend. The error below was produced:
We understand that interrupting the program as it was picking up a tip could cause an error (even though we don't know how that would happen), but are unsure why the pipette flew across the deck?
Steps to reproduce
note: do not do this with the labware on the deck since this broke our pipette tip
Current behavior
Expected behavior
Upon any keyboard interruption at any point in the protocol, the robot should follow the defined steps in our try/except block (homing the robot, dropping the tips, ending the program)
Operating system
Windows
System and robot setup or anything else?
The text was updated successfully, but these errors were encountered: