CoroutineScheduler rework

[qwwdfsad]

Fixes #840
Fixes #1046
Fixes #1286

[elizarov]

Does not seem that we need @Volatile on state:

kotlinx.coroutines/kotlinx-coroutines-core/jvm/src/scheduling/CoroutineScheduler.kt

Line 577 in dd30af0

@Volatile

It is only read in other threads after Thread.join and in toString()

[elizarov]

Looks good. All the feedback I can give is mostly cosmetic.

[elizarov]

Note, a bug was introduced somewhere in shutdown sequence. It can hang on shutdown now: https://teamcity.jetbrains.com/buildConfiguration/KotlinTools_KotlinxCoroutines_NightlyStressWindows/2652563

org.junit.runners.model.TestTimedOutException: test timed out after 60000 milliseconds
	at java.lang.Object.wait(Native Method)
	at java.lang.Thread.join(Thread.java:1260)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.shutdown(CoroutineScheduler.kt:346)
	at kotlinx.coroutines.scheduling.ExperimentalCoroutineDispatcher.shutdown$kotlinx_coroutines_core(Dispatcher.kt:125)
	at kotlinx.coroutines.TestBase.shutdownPoolsAfterTest(TestBase.kt:173)
	at kotlinx.coroutines.TestBase.onCompletion(TestBase.kt:150)
	at kotlinx.coroutines.debug.DebugTestBase.tearDown(DebugTestBase.kt:29)

[qwwdfsad]

I was avoiding adding long-scanning (dd30af0) because it could have masked all the bugs on the rendezvous of parking/unparking. Ironically, it has exposed existing problems :) It seems like it is impossible to ensure strict liveness in loads like BlockingCoroutineDispatcherMixedStealingStressTest.. I can't say this is a "bug", but rather something we should be aware of before merge.
I will elaborate on details later, for now just will fix an execution that leads to hanging

• 1 core, 2 threads (one with CPU token), both ends scanning queues before pushing themselves to the stack.
• External submitter submits CPU task to the global queue, can't find anyone to unpark, can't create a new thread, bails out
• Thread without CPU token pushes itself to the stack
• Thread with CPU token pushes itself to the stack
• Thread with CPU token finds external CPU tasks and starts executing it
• Thread without token parks
• External submitter submits blocking tasks, tries to compensate workers, but can't, pops the stack, founds the thread with CPU token (that is executing CPU task right now), successfully "unparks" it, bails out
• CPU task never completes (due to the nature of the test, it can't complete before blocking task), blocking task is kept in the queue, blocking worker is parked forever

[elizarov]

@qwwdfsad
....

• External submitter submits blocking tasks, tries to compensate workers, but can't, pops the stack, founds the thread with CPU token (that is executing CPU task right now), successfully "unparks" it, bails out

Hm. If the thread is executing CPU task right now, then it should not be unparked by tryUnpark as worker.tryForbidTermination() should return false if the thread was claimed before, because terminationState.value = TERMINATION_ALLOWED assignment is only done in the actual worker.park() method after CPU task finished executing.

Combining both terminationState and parkingState to a single state-machine that is reset after parking should be able to fix this problem.

[elizarov]

Looks way better now.