DatafeedJobsIT#testDatafeedTimingStats_DatafeedRecreated fails

[javanna]

Build scan: https://gradle-enterprise.elastic.co/s/weggtbiofm7zw/console-log?task=:x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner

Repro line: ./gradlew ':x-pack:plugin:ml:qa:native-multi-node-tests:integTestRunner' --tests "org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.testDatafeedTimingStats_DatafeedRecreated" -Dtests.seed=862B982F15848C7 -Dtests.security.manager=true -Dtests.locale=ro-RO -Dtests.timezone=Africa/Mogadishu -Druntime.java=8 -Dtests.fips.enabled=true

Reproduces locally?: no

Applicable branches: any

Failure history:
This test has been failing every once in a while in the last months, hard to say when it started. Not sure why an issue was not logged, there seems to be #44335 but it's a very old one.

Failure excerpt:

 
  | org.elasticsearch.xpack.ml.integration.DatafeedJobsIT > testDatafeedTimingStats_DatafeedRecreated FAILED |  
  | java.lang.AssertionError: |  
  | Expected: a value greater than <0L> |  
  | but: <0L> was equal to <0L> |  
  | at __randomizedtesting.SeedInfo.seed([862B982F15848C7:CF35BA87409F0E03]:0) |  
  | at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18) |  
  | at org.junit.Assert.assertThat(Assert.java:956) |  
  | at org.junit.Assert.assertThat(Assert.java:923) |  
  | at org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.assertDatafeedStats(DatafeedJobsIT.java:272) |  
  | at org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.lambda$testDatafeedTimingStats_DatafeedRecreated$5(DatafeedJobsIT.java:178) |  
  | at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:951) |  
  | at org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.lambda$testDatafeedTimingStats_DatafeedRecreated$6(DatafeedJobsIT.java:175) |  
  | at org.elasticsearch.xpack.ml.integration.DatafeedJobsIT.testDatafeedTimingStats_DatafeedRecreated(DatafeedJobsIT.java:185)

[elasticmachine]

Pinging @elastic/ml-core (:ml)

[droberts195]

@przemekwitek please could you take a look at this one?

I see at the end of the test it calls openAndRunJob twice:

elasticsearch/x-pack/plugin/ml/qa/native-multi-node-tests/src/javaRestTest/java/org/elasticsearch/xpack/ml/integration/DatafeedJobsIT.java

Lines 181 to 182 in 0bc307f

	openAndRunJob.run();
	openAndRunJob.run();

The comment // Datafeed processed numDocs documents so search_count must be greater than 0. in that lambda is only true for the first run of the datafeed. The datafeed gets deleted and recreated but the data counts are from the job which doesn't get deleted and recreated. So second time through it's expected that the datafeed finds no documents. I can see that it ought to do a search to determine that there are no extra documents, but maybe there's uncertainty about how quickly it will do this if it already appears to be up-to-date (if the results of the first invocation are within the last query_delay seconds)?

[przemekwitek]

I'm taking a look at this issue now...

[droberts195]

Assigning medium-risk due to loss of test coverage from muting.

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)