Requesting feedback: Transforms

Hi,

I am a contributor for ODFE IndexManagement. We recently published a RFC detailing our plans to launch transforms as part of IndexManagement - link.

Transforms will enable one to pivot selected existing data into a summarized view based on the selected data features. We are hoping this feature will help users to view their data in new ways for further analysis and make it easy for further processing as it is will be in a more digestible format.

We are planning to implement transform feature as a background running job that users can schedule. As a scheduled job users can run the job multiple times based off a schedule or just once.

The reason for this post is to gather potential user feedback and ensure we are releasing it with features that would be most helpful for users from start - if you have any use case for transforms or have any questions regarding this we want to hear from you.

Please feel free to comment here or in the issue linked in RFC.

2 Likes

Hi Ravi, @thalurur

Thanks for this feature, it will be very helpful for lifecycle management of rollover index. We’re indexing large amount of data (logs) and old indices should have only aggregated data using transform.

When do you plan to release Transform ?

Thanks

Hi Julien -

We have shipped index rollups. Take a look here and see if it meets your needs: Index Rollups - Open Distro for Elasticsearch Documentation

Would love to get your feedback. The idea behind transform is very similar to rollup but it is intended to be more flexible to allow you to pre-compute your data or pivot it - much like you would in a data frame. With rollups - there may be less of a need for transforms but would love to get folks feedback.

Hi @carlmead,

Yes, but rollup has too much limitation for our use case. Here my feedback on rollup : Rollup index is not easily user friendly

That’s why we are waiting for Transform feature !

Thanks,
Julien

Awesome - thanks Julien - will pass it on.