Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core][state] Task events backend - Port state API [5/n] #31278

Merged
merged 8 commits into from
Dec 23, 2022
Merged

[core][state] Task events backend - Port state API [5/n] #31278

merged 8 commits into from
Dec 23, 2022

Conversation

rickyyx
Copy link
Contributor

@rickyyx rickyyx commented Dec 21, 2022 •
edited
Loading

Signed-off-by: rickyyx [email protected]

Why are these changes needed?

Previous PRs:

This PR:

With the change,

  • list_tasks now will return tasks with attempt number as an additional column.
  • get_task might return multiple task attempt entries if there are retries.

There is also some plumbing in the test and in core (esp in the test logic) given the changes. Major changes in the PR are:

  • Add limit support to GcsTaskManager
  • Change the state aggregator to get tasks from GCS.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@rickyyx
init
96a5ed7 
Signed-off-by: rickyyx <[email protected]>
@rickyyx rickyyx changed the title init [core][state] Task backend - Port state API [6/n] Dec 22, 2022
@rickyyx
test attempts
fa8e5c8 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
revert changes for status events
dd72b2b 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
test
65b24c5 
Signed-off-by: rickyyx <[email protected]>
@rickyyx
lint
9cd1553 
Signed-off-by: rickyyx <[email protected]>
@rickyyx rickyyx marked this pull request as ready for review December 22, 2022 20:20
@ericl
Copy link
Contributor

ericl commented Dec 22, 2022

Could we split out the unrelated changes to TaskManager? These are high-risk changes, so we should review them independently of the status API.

@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Dec 22, 2022
@rickyyx rickyyx removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Dec 22, 2022
ericl
ericl approved these changes Dec 22, 2022
View reviewed changes
dashboard/state_aggregator.py Outdated Show resolved Hide resolved
@ericl ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Dec 22, 2022
@rickyyx
removed logging
7268489 
Signed-off-by: rickyyx <[email protected]>
@rickyyx rickyyx removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Dec 22, 2022
@ericl ericl merged commit eb096f7 into ray-project:master Dec 23, 2022
@rickyyx rickyyx changed the title [core][state] Task backend - Port state API [6/n] [core][state] Task backend - Port state API [5/n] Dec 23, 2022
@rickyyx rickyyx mentioned this pull request Dec 23, 2022
Merged
7 tasks
@rickyyx rickyyx changed the title [core][state] Task backend - Port state API [5/n] [core][state] Task events backend - Port state API [5/n] Dec 23, 2022
AmeerHajAli pushed a commit that referenced this pull request Jan 12, 2023
@rickyyx @AmeerHajAli
[core][state] Task backend - Port state API [6/n] (#31278)
ffb6711 
**Previous PRs:**
 - #30829: 
 - #30953: 
 - #30867: 
 - #30979: 
 - #30934
 - #31207
**This PR:** 

With the change, 
- `list_tasks` now will return tasks with attempt number as an additional column. 
- `get_task` might return multiple task attempt entries if there are retries. 


There is also some plumbing in the test and in core (esp  in the test logic) given the changes. Major changes in the PR are: 
- Add limit support to `GcsTaskManager`
- Change the state aggregator to get tasks from GCS.
tamohannes pushed a commit to ju2ez/ray that referenced this pull request Jan 16, 2023
@rickyyx @tamohannes
[core][state] Task backend - Port state API [6/n] (ray-project#31278)
1df4812 
**Previous PRs:**
 - ray-project#30829: 
 - ray-project#30953: 
 - ray-project#30867: 
 - ray-project#30979: 
 - ray-project#30934
 - ray-project#31207
**This PR:** 

With the change, 
- `list_tasks` now will return tasks with attempt number as an additional column. 
- `get_task` might return multiple task attempt entries if there are retries. 


There is also some plumbing in the test and in core (esp  in the test logic) given the changes. Major changes in the PR are: 
- Add limit support to `GcsTaskManager`
- Change the state aggregator to get tasks from GCS.

Signed-off-by: tmynn <[email protected]>
tamohannes pushed a commit to ju2ez/ray that referenced this pull request Jan 25, 2023
@rickyyx @tamohannes
[core][state] Task backend - Port state API [6/n] (ray-project#31278)
bee9bec 
**Previous PRs:**
 - ray-project#30829: 
 - ray-project#30953: 
 - ray-project#30867: 
 - ray-project#30979: 
 - ray-project#30934
 - ray-project#31207
**This PR:** 

With the change, 
- `list_tasks` now will return tasks with attempt number as an additional column. 
- `get_task` might return multiple task attempt entries if there are retries. 


There is also some plumbing in the test and in core (esp  in the test logic) given the changes. Major changes in the PR are: 
- Add limit support to `GcsTaskManager`
- Change the state aggregator to get tasks from GCS.

Signed-off-by: tmynn <[email protected]>
tamohannes pushed a commit to ju2ez/ray that referenced this pull request Jan 25, 2023
@rickyyx @tamohannes
[core][state] Task backend - Port state API [6/n] (ray-project#31278)
0e1eee1 
**Previous PRs:**
 - ray-project#30829: 
 - ray-project#30953: 
 - ray-project#30867: 
 - ray-project#30979: 
 - ray-project#30934
 - ray-project#31207
**This PR:** 

With the change, 
- `list_tasks` now will return tasks with attempt number as an additional column. 
- `get_task` might return multiple task attempt entries if there are retries. 


There is also some plumbing in the test and in core (esp  in the test logic) given the changes. Major changes in the PR are: 
- Add limit support to `GcsTaskManager`
- Change the state aggregator to get tasks from GCS.

Signed-off-by: tmynn <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ericl ericl ericl approved these changes

@wuisawesome wuisawesome Awaiting requested review from wuisawesome

@AmeerHajAli AmeerHajAli Awaiting requested review from AmeerHajAli

@robertnishihara robertnishihara Awaiting requested review from robertnishihara

@pcmoritz pcmoritz Awaiting requested review from pcmoritz pcmoritz is a code owner

@raulchen raulchen Awaiting requested review from raulchen raulchen is a code owner

@fishbone fishbone Awaiting requested review from fishbone

@scv119 scv119 Awaiting requested review from scv119

Assignees

@ericl ericl

@rkooo567 rkooo567

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@rickyyx @ericl @rkooo567