Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Transforms should handle bucket script #1205

Open
nicerloop opened this issue Jul 11, 2024 · 1 comment
Open

[FEATURE] Transforms should handle bucket script #1205

nicerloop opened this issue Jul 11, 2024 · 1 comment
Labels
enhancement New request

Comments

@nicerloop
Copy link

nicerloop commented Jul 11, 2024

Is your feature request related to a problem?
I want to have some computation done inside OpenSearch in a continuous manner, using transforms.
My use-case is very similar to the one described foe ElasticSearch in this blog post: https://xeraa.net/blog/2021_elasticsearch-transforms-duration-status-updates/
The query without transform produces the expected result, but I want this result updated and stored inside OpenSearch each minute.
When trying to declare the transform with a bucket script aggregation, I get an error message corresponding to #671

What solution would you like?
OpenSearch Index Transforms handle bucket scripts aggregations https://opensearch.org/docs/latest/aggregations/pipeline-agg/#bucket_script-bucket_selector
I can then port the use-case in https://xeraa.net/blog/2021_elasticsearch-transforms-duration-status-updates/ to OpenSearch

What alternatives have you considered?
We can externally query OpenSearch each minute and update the target index with the retrieved result. This requires an external component, and superfluous network exchanges, data extraction from OpenSearch, and data storage in OpenSearch, all realized without any support for optimizing the data scope to handle. It is a crude brute-force workaround that will put down our OpenSearch storage.

Do you have any additional context?
If necessary guidance is provided, considering I can code in Java, I could envision doing the work, if it not too broad for a single person.

@dblock
Copy link
Member

dblock commented Jul 29, 2024

[Catch All Triage - 1, 2, 3, 4]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New request
Projects
None yet
Development

No branches or pull requests

2 participants