Search requests should prefer nodes within the same availability zone

[jimczi]

In 7.x we deprecated search routing based on allocation awareness and decided to always use adaptive replica selection starting in 8. However this change may be an issue for some users due to cross-zone traffic costs. One way to solve this issue could be to add an opt-in option in 8 so that users can still use allocation awareness in search, but that doesn't work in all cases since search routing picks nodes that have at least one awareness attribute value in common.

So instead, we've decided to start a new discussion on how coordinating nodes could prefer nodes in the same "availability-zone". The first question we'd like to answer is whether this should be a hard requirement or should we allow cross-zone traffic if shards within the same availability zone are not ready to accept search requests ?

Depending on the answer we might also want to reduce the cross-zone traffic by splitting the request into multiple "availability-zone" search requests, each one being responsible to retrieve and merge the results coming from a single availability zone. That's something we already do for cross-cluster search in order to reduce the number of connections to the remote cluster but this is also just one possible optimization.

So the gist of this issue is to propose a better way to avoid cross-zone traffic in search that is not only based on user's conventions (based on random tags in the allocation awareness attributes).

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[AyWa]

The first question we'd like to answer is whether this should be a hard requirement or should we allow cross-zone traffic if shards within the same availability zone are not ready to accept search requests ?

I can not talk for all the user of ES, but this my current use case. (read heavy cluster with huge spike of req between day and night -> so in fact we have a lot of shards replica during spike).

We will always prefer having a shard doing cross-zone traffic than failing the request (if the node in the AZ are all busy etc). However we just prefer for cost optimization to stay in the AZ.

The reason behind that is:

• we do not control the client -> Sometimes client might be unbalanced between AZ
• we do not control the ES cluster autoscaling 100% -> sometimes an AZ scale more than an other one

I hope the information are usefull ~

[andrassy]

Wondering what the status of this is now that 8.0 is GA? Is there a new mechanism to replace the previous default behaviour, or is the old behaviour still just deprecated rather than removed? 🙏

[dorony]

Hi @jimczi, is there any update on this issue?

[tailnode]

also want this feature, our cross zone cost is high.

[tinder-igorsokolov]

the same here - after migration to the ECK/ES8 we observed eye-popping increase in the network cost.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)