-
Notifications
You must be signed in to change notification settings - Fork 1.4k
Description
We're looking to enhance Elsa's data processing capabilities. To do this, we're drawing ideas from Azure Data Factory. Here's a breakdown of the new concepts we're considering for integration:
-
Data Set: This would be a reference to a specific set of data located somewhere. It's like a pointer telling Elsa where to find the data it needs.
-
Connected Service: Think of this as a "connection string". It's a way for Elsa to connect to various services to read and write data. This is crucial for moving data in and out of Elsa.
-
Activity Connection to Data Sets: We plan to enable each activity in Elsa to be linked to a source Data Set and a target Data Set (also known as a sink Data Set). When an activity runs, it would be able to pull data from its source Data Set and send it to its target Data Set. This process could involve data mapping and transformation, depending on what the activity is designed to do.
-
Background Execution and Data Processing: Activities would be able to run in the background, even on different nodes within a cluster, which is something Elsa already supports. They can handle data in chunks, sending processed data to subsequent activities or creating new data sets.
-
Data Set vs Variables and Storage Driver: A significant upgrade in Elsa would be the way it handles data sets compared to its current Variables and Storage Driver concept. The idea is to store Data Sets externally on the Linked Service, providing more flexibility and advanced functionality.
-
User Experience Enhancements: From a user interface perspective, users will still have the ability to use outputs from activities. But in addition to capturing outputs via Variables, users would also have the option to direct these outputs to a chosen Data Set.
In summary, these enhancements are aimed at making Elsa more capable in handling data, especially in terms of processing, storing, and transferring data between different activities and services.
Metadata
Metadata
Assignees
Labels
Type
Projects
Status