Alias information in field caps API

[costin]

Field caps returns information only concrete indices which, when using the API against indices makes it difficult to interpret the response.

For example, take an alias alias_one which points to index index_a and index_b and alias_two which points to index_c.
A field caps on alias* will properly match alias_one and alias_two but the response will refer to the backing indices only, namely index_a, index_b and index_c. Not only the indices name do not match the pattern which is surprising (I asked for alias* which do I get index_a) but also one can't tell the relationship between aliases and indices and thus what fields belong to what alias.
The workaround is to ask the aliases for all matching indices - not a deal breaker but it is an extra call and further more, the API is not CCS-compatible.

It would be helpful to have the alias information in the response, as we do for indices.

[elasticmachine]

Pinging @elastic/es-search (:Search/Search)

[javanna]

I can see how this behaviour can be surprising, but it aligns with how search, get and other API work: we have never guaranteed that once aliases are requested, the response won't refer to the corresponding concrete indices. It's always been the other way around: the response will refer to the concrete indices that hold the data. With that said I would close this with no further action. Please feel free to comment or reopen if you think otherwise.

[costin]

Note this has security implications see #75810 (and its related issues).
Essentially as it stands, there's information loss both in terms of mapping and security access.
To clarify the issue is about obtaining metadata about how the resolution happens in a matter that is CCS compatible.

[javanna]

The security implications are not local to the field caps API and I believe the entire alias support in Elasticsearch is subject to it. For this, I believe the issue to follow is #29874 .

While the issue around field_caps can be worked around with an additional call, the PIT issue you linked above is slightly worse because you get a PIT id that can't be authorized against the aliases, which you must provide to the API to get more results.

To clarify, I can see how this is problematic but I don't see it being resolved by simply adapting the field_caps output alone. It's a structural problem around aliases and how they are handled by an Elasticsearch API and how they are authorized.

I did not get the link to CCS in this conversation, could I ask you to expand on the CCS related issue you mentioned?

[costin]

To me the above is another side-effect of how aliases are treated in ES - they are considered 'symlinks' (redirects) when in fact they act as a view on top of one or multiple indices, both from a content perspective (filter with query) and/or security (alias with different permissions than the target index).
The lack of information about them in field_caps is another side-effect of that.
That is, I don't think adding this information in field_caps helps the underlying aliasing issue however it does bubble it up and makes the metadata accessible (potentially by reconstructing it).

QL relies on field_caps to get metadata about indices and aliases which is used also for CCS. However since there's no aliasing information in field_caps for remote clusters QL cannot get this information and treats everything as an index (table) which exacerbates the security problem in case it exists:

• the user checks the target, we report it as a table/index not a view/alias due to CCS
• the user gives all the permissions to alias, things keep on failing - tries to diagnose which tends to be more complicated due to CCS typically being a remote cluster running in a different environment with a different set of rules
• the user reports the problem and learns about the extra permissions which might be or not an option

Overall not a pleasant experience. Aliasing information would give proper insight into the actual mapping and differentiate between concrete vs lookalike indices.
It will not solve the underlying problem however it will not hide it under the rug either.

[javanna]

Thanks for all the context, I am re-opening for discussion with the team.

I think what you were referring to for the CCS scenario is that the additional call to resolve aliases that can be made in the local cluster can't be sent to the remote cluster as that API does not support cross-cluster calls. So while you are able to work-around the problem in the local cluster, when CCS comes into play that is no longer possible.

One more question: indices appear globally as well as per field. Do you envision an additional entry for each field that returns the potentially matching aliases, as well as an aliases array at the top-level?

[costin]

Thanks.
No real preference regarding the response format - per field aliasing is needed, the global one could be inferred however it would useful to avoid having to recompute the mapping from the fields. Potentially enable/disable the behavior through dedicated a dedicated param.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[javanna]

This has been open for quite a while with no activity. While it is something that may be interesting to solve, it does not align with our current priorities. For now I'm going to close this as something we aren't planning on implementing. We can re-open it later if needed.