[data] [streaming] Enable actor fault tolerance by default for streaming map operator #33906
Conversation
Signed-off-by: Eric Liang <[email protected]>
python/ray/data/_internal/execution/operators/actor_pool_map_operator.py
Outdated
Show resolved
Hide resolved
| # up to three retries per task. The user can customize this in map_batches via | ||
| # extra kwargs (e.g., map_batches(..., max_restarts=0) to disable). | ||
| if "max_restarts" not in ray_remote_args: | ||
| ray_remote_args["max_restarts"] = -1 |
There was a problem hiding this comment.
is there any reason why we want to restart actor infinitely? What if user code has bug, and causes the infinite retry and stuck?
There was a problem hiding this comment.
My thoughts here were that for a long running job, there could be an indefinite number of node failures. Hence, the number of actor restarts should be unbounded. If there is a bug, then the task retry limit will eventually raise the error.
There was a problem hiding this comment.
Shall we update the documentation?
Maybe we can do separately, as I checked the docs it looks we don't have an explanation about Dataset FT yet.
python/ray/data/_internal/execution/operators/actor_pool_map_operator.py
Outdated
Show resolved
Hide resolved
Signed-off-by: Eric Liang <[email protected]>
| # up to three retries per task. The user can customize this in map_batches via | ||
| # extra kwargs (e.g., map_batches(..., max_restarts=0) to disable). | ||
| if "max_restarts" not in ray_remote_args: | ||
| ray_remote_args["max_restarts"] = -1 |
There was a problem hiding this comment.
Shall we update the documentation?
Maybe we can do separately, as I checked the docs it looks we don't have an explanation about Dataset FT yet.
…ing map operator (ray-project#33906) Signed-off-by: elliottower <[email protected]>
…ing map operator (ray-project#33906) Signed-off-by: Jack He <[email protected]>
Why are these changes needed?
Enable actor restarts (infinite) and also task retries (up to 5 by default).