Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

commented out call to run_store.has() #10340

Merged
merged 2 commits into from
May 18, 2022
Merged

commented out call to run_store.has() #10340

merged 2 commits into from
May 18, 2022

Conversation

TamarZanzouri
Copy link
Contributor

@TamarZanzouri TamarZanzouri commented May 18, 2022

Overview

commented out call to run_store.has(run_id). this causes a delay in the POST commands end point

Quick solution for #10333.

Changelog

  • Commands router - commented out call to run_store.has()

Risk assessment

Very low. This call is being done from a depends method that is only being used by the current end point

@TamarZanzouri TamarZanzouri requested a review from a team as a code owner May 18, 2022 23:11
Copy link

@laviera laviera left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Revert database read causing long waits on database reads

@SyntaxColoring
Copy link
Contributor

For background, here's a copy-paste of the Slack message from @mcous that proposed this workaround:

The hang is happening during database access from RunStore.has(run_id), which POST .../commands hits to check the validity of a given run ID (and 404 if it's no good). Sometimes, this call takes an really long time!

Notes:

  • Freeze time is inconsistent, but can be in the neighborhood of ~30 sec
  • It sort of seems like this is some sort of "waiting to acquire a lock" problem
  • It's not out of the realm of possibility that it's an IO problem, given our hardware

Things I want to try out:

  • Tweak the has query to only pull the id column from the run table rather than all fields
    • I don't necessarily expect this to do anything, but might help if it is an IO problem

Situation assemssment:

  • A blocker to beta release, due to how badly this could mess up the jogging experience in LPC
  • The 404 from the POST .../commands endpoint is non-essential for the app. We could remove it to unblock the beta with no detrimental effects
  • Whether or not a run ID exists is also easy (enough) to cache in memory

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants