add worker num paused tasks api

[sofianhnaide]

Why are these changes needed?

This is added to give the debugger another API to signal paused workloads without reliably depending on Ray having a valid task Id, this is needed for cases where manual threads are created by the user.

This work also solidify the debugger_port reading and writing to be thread safe.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[rkooo567]

Generally lgtm

Actually, I'd like to discuss a little bit more about num_paused_tasks

Is it currently used to be displayed for vs code debugger terminal? If not, I feel like it is better removing this.

1. Dashboard doesn't display worker state, and we don't have any plan to expose it in a while, so it won't be that useful
2. it will simplify a lot of code in this PR

[rkooo567]

this will deadlock if this API is called within the same thread as the callback thread (thread_io_service_).

Can you add a check to the beginning of the function that the current thread is not same as thread_io_service_?

[sofianhnaide]

done.

[sofianhnaide]

Generally lgtm

Actually, I'd like to discuss a little bit more about num_paused_tasks

Is it currently used to be displayed for vs code debugger terminal? If not, I feel like it is better removing this.

1. Dashboard doesn't display worker state, and we don't have any plan to expose it in a while, so it won't be that useful

2. it will simplify a lot of code in this PR

What we need here is a flag to signal a process being paused in the debugger. That flag can not be the port since you might have a port and be either paused or not.

I am using num_paused_tasks counter essentially as a flag, but then I can capture threading paused and unpaused safely.

The current behavior is all threads are unpaused on connect so technically an unpaused call should trump all paused but I would rather not build with that assumption giving that it won't technically simplify things for me.

If you have any suggestions to simply that I am happy to consider.

[rickyyx]

Code lgtm!

So is this orthogonal to what Sang's been working on with the threaded task id thing?

[sofianhnaide]

I have also renamed the newely added property to num_paused_threads to be explicit and not mix it with tasks