Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Core] When the GCS is restarted for HA, it overwrites the previous session directory #38796

Closed
rkooo567 opened this issue Aug 23, 2023 · 5 comments
Closed

[Core] When the GCS is restarted for HA, it overwrites the previous session directory #38796

rkooo567 opened this issue Aug 23, 2023 · 5 comments
Assignees
@vitsai
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release

Comments

@rkooo567
Copy link
Contributor

What happened + What you expected to happen

When the GCS is restarted for HA, it overwrites the previous session directory.

It means when you create a new driver after GCS restarts, it will start using a new session_dir.

This could impact correctness (and usability, for example, dashboard logs will lose previous logs) because there are some logic that we may depend on the session_dir. it is also a blocker to merge #37644

Versions / Dependencies

master

Reproduction script

N/A

Issue Severity

None
@rkooo567 rkooo567 added bug Something that is supposed to be working; but isn't release-blocker P0 Issue that blocks the release P0 Issues that should be fixed in short order triage Needs triage (eg: priority, bug/not-bug, and owning component) core Issues that should be addressed in Ray Core labels Aug 23, 2023
@rkooo567 rkooo567 removed the triage Needs triage (eg: priority, bug/not-bug, and owning component) label Aug 23, 2023
@rkooo567 rkooo567 mentioned this issue Aug 23, 2023
Merged
8 tasks
@vitsai vitsai self-assigned this Aug 28, 2023
@rkooo567
Copy link
Contributor Author

What's the progress of this task? There's one issue that's blocked by this, so I'd like to understand the timeline

@fishbone fishbone removed their assignment Aug 28, 2023
@fishbone
Copy link
Contributor

@rkooo567 sorry I forgot this item :( I'll sync with @vitsai today to get a clear timeline.

@vitsai
Copy link
Contributor

vitsai commented Aug 29, 2023

We just synced; this should be done by Wednesday.

@vitsai vitsai mentioned this issue Aug 29, 2023
Closed
8 tasks
@rkooo567
Copy link
Contributor Author

rkooo567 commented Aug 29, 2023 •
edited
Loading

Thank you! This is very important task to support multi cloud!

@vitsai
Copy link
Contributor

vitsai commented Aug 30, 2023

Mis-scoped this, need 1 more day to merge.

This was referenced Aug 31, 2023
Closed
Merged
@rkooo567 rkooo567 closed this as completed Sep 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vitsai vitsai

Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P0 Issues that should be fixed in short order release-blocker P0 Issue that blocks the release
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@vitsai @rkooo567 @fishbone