[Query Insghts] Memory budgeting for Query Insights services

[ansjcy]

Is your feature request related to a problem? Please describe

As we add more query insights services like top n queries by different metrics types and top queries shapes, we are introducing higher memory usage to store all the metrics. We should provide the ability for the user to set memory budget for query insights features to avoid over-using memory on each node.

Describe the solution you'd like

We can provide separate, but generic configuration endpoints to set limits on memory usages for diffrent query inisght services. For example, for top n queries, we can provide something like:

 search.insights.top_queries.latency.max_memory_usage: "5mb",

to set the maximum memory that the top n queries by latency service can use to 5mb.

Related component

Search:Query Insights

Describe alternatives you've considered

We can also consider providing only one configuration endpoint to set limit on resource usage for all the query insights components. It would make the configuration easier but will also take away the customization for different query insights features. If one feature is using too much memory it would starve other query insights services.

Additional context

Please refer to the meta issue: #11522

[peternied]

[Triage - attendees 1 2 3 4 5 6 7 8]
@ansjcy Thanks for creating this issue. Please work with maintainers in the area to see that this approach will not introduce too much overhead for cluster administration.