ES|QL Search: allow WHERE with text predicates

[ioanatia]

When WHERE is used in the search context of the SEARCH command it can now receive the expressions using MATCH.

POST _query
{
  "query": """
  SEARCH search-movies [
    | WHERE MATCH(Title, "hobbit") AND imdbRating > 8.0
  ]
  | KEEP Title, _score
  """
}

This will apply a boolean query with a filter clause - the returned results will have scores equal to zero, since no ranking is happening.

The grammar has been simplified, so for the moment WHERE and RANK allow the same type of expressions when used in the SEARCH context.

[ioanatia]

this is an interesting behaviour that we can follow up on separately.
I would expect that we just have a bool query with two filter clauses:

 var expectedLuceneQuery =  new BoolQueryBuilder().filter(new MatchQueryBuilder("first_name", "Meg")).filter(new MatchQueryBuilder("last_name", "Ryan"))

instead we are using must clauses which comes from how the PushFiltersToSource forms the query, specifically here:

elasticsearch/x-pack/plugin/esql/src/main/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizer.java

Line 222 in 90047f9

Query queryDSL = TRANSLATOR_HANDLER.asQuery(Predicates.combineAnd(pushable));

We can see that this behaviour happens with FROM too, take this test query:

elasticsearch/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java

Lines 236 to 240 in dabd9b6

	var plan = plannerOptimizer.plan("""
	from test
	| where salary > 1000
	| stats c = count(salary)
	""", IS_SV_STATS);

the filters are pushed as must clauses:

elasticsearch/x-pack/plugin/esql/src/test/java/org/elasticsearch/xpack/esql/optimizer/LocalPhysicalPlanOptimizerTests.java

Line 255 in dabd9b6

var expected = QueryBuilders.boolQuery().must(range).must(exists);

I think maybe they should be pushed as filter instead since must clauses add to the score which for filters we don't care about?

EDIT: maybe this is okay since we wrap everything in a constant score anyway for the FROM command and Lucene optimizations should kick in?

[ChrisHegarty]

That's a good question. It works ok, by I also suspect that filter would be more appropriate here.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[ioanatia]

@elasticsearchmachine test this please

[ChrisHegarty]

@elasticmachine update branch

[ChrisHegarty]

@elasticmachine update branch

[ChrisHegarty]

LGTM