[ci] run ci state machine with shorter data length (#44783)

Currently the CI state machine uses a very long historical data to make judgment (100 tests). As we're moving things to run periodically, adjust the setting to work with a shorter history. I also adjust flaky percentage to 7% (so that it's 2 red per 30 run, which is 6.6%, is not punished, but 3 red per 30 run is)

Signed-off-by: can <[email protected]>

release/ray_release/test_automation/ci_state_machine.py

@@ -6,9 +6,9 @@
 from ray_release.test import Test, TestState
 
 
-CONTINUOUS_FAILURE_TO_FLAKY = 10 # Number of continuous failures before flaky
-CONTINUOUS_PASSING_TO_PASSING = 30 # Number of continuous passing before passing
-FLAKY_PERCENTAGE_THRESHOLD = 5 # Percentage threshold to be considered as flaky
+CONTINUOUS_FAILURE_TO_FLAKY = 3 # Number of continuous failures before flaky
+CONTINUOUS_PASSING_TO_PASSING = 10 # Number of continuous passing before passing
+FLAKY_PERCENTAGE_THRESHOLD = 7 # Percentage threshold to be considered as flaky
 FAILING_TO_FLAKY_MESSAGE = (
  "This test is now considered as flaky because it has been "
  "failing on postmerge for too long. Flaky tests do not run on premerge."
@@ -25,7 +25,7 @@
 class CITestStateMachine(TestStateMachine):
  def __init__(self, test: Test, dry_run: bool = False) -> None:
  # Need long enough test history to detect flaky tests
- super().__init__(test, dry_run=dry_run, history_length=100)
+ super().__init__(test, dry_run=dry_run, history_length=30)
 
  def _move_hook(self, from_state: TestState, to_state: TestState) -> None:
  change = (from_state, to_state)