unsupported_operation_exception on collapse + docvalue_fields

[dgilmanAIDENTIFIED]

Elasticsearch Version

8.8.0

Installed Plugins

No response

Java Version

bundled

OS Version

Linux 5.19.0-1022-aws #23~22.04.1-Ubuntu SMP Fri Mar 17 15:38:24 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Problem Description

ES returns unsupported_operation_exception: null instead of a descriptive error message for a minimal query with collapse and docvalue_fields

{
  "took" : 196,
  "timed_out" : false,
  "_shards" : {
    "total" : 64,
    "successful" : 63,
    "skipped" : 0,
    "failed" : 1,
    "failures" : [
      {
        "shard" : 0,
        "index" : "idx",
        "node" : "rcZ4pURhRNSeSL349CCG-Q",
        "reason" : {
          "type" : "unsupported_operation_exception",
          "reason" : "unsupported_operation_exception: null"
        }
      }
    ]
  },
  "hits" : {
    "total" : {
      "value" : 10000,
      "relation" : "gte"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Steps to Reproduce

{
 "collapse": {"field": "col1"},
  "docvalue_fields": [
  "col2",
  "col3"
 ]
}

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[nemphys]

Any news regarding this issue?
Still there in 8.12.2, seems impossible to use docvalue_fields when using collapse without getting this error:

Caused by: java.lang.UnsupportedOperationException
at java.base/java.util.AbstractCollection.add(AbstractCollection.java:253)
at [email protected]/org.elasticsearch.search.fetch.FetchContext.docValuesContext(FetchContext.java:179)
at [email protected]/org.elasticsearch.search.fetch.subphase.FetchDocValuesPhase.getProcessor(FetchDocValuesPhase.java:33)
at [email protected]/org.elasticsearch.search.fetch.FetchPhase.getProcessors(FetchPhase.java:183)
at [email protected]/org.elasticsearch.search.fetch.FetchPhase.buildSearchHits(FetchPhase.java:108)
at [email protected]/org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:78)
at [email protected]/org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:717)
at [email protected]/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:688)
at [email protected]/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$2(SearchService.java:545)
at [email protected]/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:73)
at [email protected]/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:70)
at [email protected]/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:95)
at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
... 6 more

[nemphys]

After further debugging this, I discovered that the exception is only thrown when the collapse field is not included in the docvalue_fields; as soon as it is added to the docvalue_fields list, the error goes away :-)

[benwtrent]

wow thanks @nemphys ! What a weird and tricky bug...

We should at a minimum throw an error early indicating that if you are specifying fields to include, you must also include the collapse field.

[nemphys]

Or maybe silently include it by default when it is not there (?)

[benwtrent]

Finally had some cycles to dig more into this. Found the silly bug. Should be fixed soon :)

Thank you @nemphys for the detailed debugging & @dgilmanAIDENTIFIED for reporting!!!