Requesting feedback: Transforms

thalurur · January 27, 2021, 11:35pm

Hi,

I am a contributor for ODFE IndexManagement. We recently published a RFC detailing our plans to launch transforms as part of IndexManagement - link.

Transforms will enable one to pivot selected existing data into a summarized view based on the selected data features. We are hoping this feature will help users to view their data in new ways for further analysis and make it easy for further processing as it is will be in a more digestible format.

We are planning to implement transform feature as a background running job that users can schedule. As a scheduled job users can run the job multiple times based off a schedule or just once.

The reason for this post is to gather potential user feedback and ensure we are releasing it with features that would be most helpful for users from start - if you have any use case for transforms or have any questions regarding this we want to hear from you.

Please feel free to comment here or in the issue linked in RFC.

searchymcsearchface · January 28, 2021, 4:18pm

Julien · February 11, 2021, 10:27am

Hi Ravi, @thalurur

Thanks for this feature, it will be very helpful for lifecycle management of rollover index. We’re indexing large amount of data (logs) and old indices should have only aggregated data using transform.

When do you plan to release Transform ?

Thanks

carlmead · February 15, 2021, 7:33pm

Hi Julien -

We have shipped index rollups. Take a look here and see if it meets your needs: Index Rollups - Open Distro for Elasticsearch Documentation

Would love to get your feedback. The idea behind transform is very similar to rollup but it is intended to be more flexible to allow you to pre-compute your data or pivot it - much like you would in a data frame. With rollups - there may be less of a need for transforms but would love to get folks feedback.

github.com

opendistro-for-elasticsearch/index-management/blob/main/docs/transform-rfc.md

# Transform RFC
The purpose of this request for comments (RFC) is to introduce our plans to enhance Index Management with transform functionality to collect feedback and discuss our plans with the community. This RFC is meant to cover the high level functionality of transform and doesn't go into the design details/architecture/implementation details.

## Problem Statement:
Elasticsearch provides the ability to perform aggregations, help to summarize and retrieve insights from the data. However in some cases, the amount of data that needs to processed can overwhelm the hardware or might not be possible because of hardware constraints and can involve building complex query expressions. Transforms help in these cases by transforming and summarizing the data to a different index in the background which can be used later for further analysis.

An example would be if there are millions of documents of user request data ingested currently and want to analyze certain subset of users based on their request time or request frequency with which they use the system. Though it is possible to do this using raw data, but its inefficient. Instead using transform a new index can be created setting user as the group, computing all the required metrics and associating with the user. This makes it much simpler to use this data later on for further analysis, and since this will be done in background it will be run in a controlled fashion.

## Proposed Solution:
### Transforming data:
We are planning to introduce a set of APIs and new ISM actions that allow a user to configure and manage transform jobs. We plan to launch the features in a phased manner.

For first phase we are planning to make the APIs available to create, delete, preview, start, stop and stats APIs for transform jobs. These transform jobs can be scheduled to be run at a requested time or immediately. We initially are planning to support single occurrence transform jobs. 

As part of the transform job you will be able to transform data based a pivot, a set of features and calculate the metrics for each of these features. We are going to have a set of commonly used aggregations and groupings to begin with and expand these with time. There will also be ability to filter the source data used for transform job if needed.

### Accessing transformed data:
Once your data is transformed into the target index you will be able to query it like any other index. The documents in the index are stored in the format described in the transform job.

## Providing feedback:

This file has been truncated. show original

Julien · February 17, 2021, 1:11pm

Hi @carlmead,

Yes, but rollup has too much limitation for our use case. Here my feedback on rollup : Rollup index is not easily user friendly

That’s why we are waiting for Transform feature !

Thanks,
Julien

carlmead · February 17, 2021, 4:41pm

Awesome - thanks Julien - will pass it on.