Retry transient shard failures in search

[jimczi]

Today shard search requests are executed on each replica upon success. If all replicas fail for a shard, we consider the shard as failed and move on with the other shards.
Users can choose whether they accept partial results or not by setting allow_partial_search_results , however they have no choice but to replay the query if they want the full results (assuming that the shard failures were transient).
I am opening this discuss whether we could apply some exponential backoff to retry transient shard failures in search requests.
Failures such as:

• Rejected executed exception.
• Shard unavailable exception

could be retried with a configurable exponential backoff. This would be useful for search requests that run in the background (with _async_search) and that can afford waiting for a shard recovery.

This issue is also loosely related to #37867 since low-priority search requests could be configured to retry automatically.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[elasticmachine]

Pinging @elastic/es-distributed (:Distributed/Distributed)

[jimczi]

We discussed this in Fix-it Thursday and agreed on two possible improvements:

• We shouldn't retry non-transient failures.
• We could add a configurable delay to wait for non-assigned shards before raising an error.

These improvements are not linked so I'll open a new issue for the latter so that it can be handled separately.
Shard failures are difficult to diagnose, for instance there is no way to determine if a circuit breaker exception is due to the current shard request or because the node is overloaded.
We didn't reach a conclusion for this specific failure but we agreed that we should categorize each error type in order to determine if they should be retried or not.

[jimczi]

I opened #56236 to handle non-assigned shards in search request. This issue is now geared towards classifying shard failures that shouldn't be retried automatically.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)