In 302, the system may obtain data from the source data store. For example, the system may obtain (or retrieve, receive, etc.) data from a data file provided by a source data store. For example, a vendor may provide a data file from a vendor database. In addition, in some implementations, the system (or source component) may directly retrieve such data from a system associated with a vendor (e.g., via a direct vendor database query).

In 303, the system may initiate an execution of the set of components to perform one or more sequences of operations to transform the data. In one implementation, the sequence of operations (or workflow or processing sequence) may be determined based on the transformation graph. In one implementation, the execution of the set of components may be performed as an execution pipeline (or data pipeline). For example, a pipeline may include a sequence of operations (or commands, program runs, tasks, threads, procedures, etc.) conceptually executed in parallel in which the output (or output stream) from the execution of one component may be automatically provided as the input (or input stream) to the next component of the sequence. The one or more sequences of operations may include executing one or more components as further described in FIG. 4.

In 304, the system may provide the transformed data to the target data store. For example, the system may initiate the uploading (e.g., load) of one or more files (or output files) created from executing the transformation process to the target data store. In one implementation, a target component may initiate such an uploading. For example, the target component may update a target database.

FIG. 4 is a process flow diagram illustrating an example of a process for executing a set of components according to one or more implementations of the disclosure. As described, one or more components may be executed as part of an execution pipeline initiated by the system. For example, data that is output from a particular component may be provided as input into a component that is next in the sequence. For instance, the components may perform operations on the columns of the data as part of a transformation process.

In 401, the system may execute a source component (e.g., source component 220) to read the obtained data and convert the obtained data to a data structure used for the transformation process. In one implementation, the data structure may include a 2-dimensioal labeled data structure, a table data structure, a DataFrame, or any other format that may be used, for example, for manipulating panel data. The source component may perform a read operation of various types of files. As described, the components may be implemented using the Python programming language and related libraries. Accordingly, such components may be configured as further described below.

For example, a configuration for a source component may include:

″config″: {

″file_path_name″: “input file″,

″schema″:“schema file ″,

″file_type″:“DL, json, FW″,

″delimiter″:“[,|,t]″,

″contain_column_names″:″Y″,

}

For example, as shown, the source component may read an input file including a JSON file type with a specified delimiter value.

In 402, the system may execute one or more transformation components (e.g., transformation components 230) to transform the data to a format for updating the target data store. The transformation components may perform various operations as further described in FIG. 5.

In 403, the system may execute a target component to write the transformed data to an output file. For example, the target component may initiate uploading the output file to a target data store as part of a backup process.

For example, a configuration for a target component may include:

″config″: {

″file_path_name″: “output file path″,

″target_filetype″:“dl, json″,

″delimiter″:“[,|,t]″

}

For example, as shown, the target component may create an output file including a JSON file type with a specified delimiter value.

FIG. 5 is a process flow diagram illustrating an example of various transformation operations that may be performed by a set of transformation components according to one or more implementations of the disclosure. The system (e.g., data management system 110 or host system 120) may execute (or initiate the execution of) the components as described. It should be noted that the numbering of operations 501-511 is provided to merely identify and distinguish between operations and that execution of the components shown in operations 501-511 may be performed in any order as part of the transformation process.

In 501, the system (e.g., data management system 110 or host system 120) may execute (or initiate the execution of) a filter component to filter data based on an expression. For example, if the filter expression evaluates to true, the data will be passed-through, and if the filter expression evaluates to false, the data will be removed from further processing or will be redirected.

For example, a configuration for a filter component may include:

″filterCondition″: {

″selectCond″: “filter expression″,

″deSelectCond″: ″Y/N″

}

In 502, the system may execute a reformat component to change the structure of data records or remap certain data records. For example, the reformat operation may perform field-level data transformations or mappings for a given input.

For example, a configuration for a reformat component may include:

″transformation″: {

″route0″: {

“1″: “output_field = (input_field == ‘condition′,

‘True′, strip(input_field))“,

“2”: “output_field_2 = input_field”

},

″route1″: {

″1″: ″custom_column2 = star_rating“

}

},

″pass_through″: {

″route0″: ″N″,

″route1″: ″Y″

},

″exclude_columns″:{

″route0″ : [“input_field1″],

″route1″ : [“input_field2″]

},

″router″:{

″route0″ : “input_field == ‘R’″,

″route1″ : “input_field != ‘R’″

},

″is_empty_fail″: ″Y″

In 503, the system may execute a lookup component to retrieve specified data (e.g., data records) from a source data store. For example, the lookup operation may include inputting a primary input file and/or a lookup input file as the input (or input stream) and output specified columns from the lookup input file.

For example, a configuration for a lookup component may include:

″config″: {

″lookup_column″:“col_1, col_2, ..., col_n“,

″lookup_keys_column″:“col_1, col_2, ..., col_n“,

″source_column″:“col_1, col_2, ..., col_n“

}

In 504, the system may execute a replicate component to create a specified number of duplicate copies of specified data. For example, the replicate operation may output multiple copies (e.g., duplicates, replications, etc.) of the input.

For example, a configuration for a replicate component may include:

″config″: {

″start″: “starting number″,

″step″: “increment value″,

″column_name″: “column_name″

}

In 505, the system may execute a sort component to sort data records based on specified criteria. For example, the sort operation may order data based on a key value.

For example, a configuration for a sort component may include:

″config″: {

″sort_key″: {

″1″:“column_name,ascending=False″ ,

“2″:“column_name,ascending=True“,

“3″:“column_name″

},

″na_position″:″first/Last″

}

In 506, the system may execute a groupby (or aggregate) component to group data records. For example, data records having the same key value may be grouped. The component may output a summary of a particular function of the grouping. For example, the functions may include providing a value for a minimum, maximum, mean, sum, count, first, and last.

For example, a configuration for a groupby component may include:

“Sorted_input”: Y/N

“Input_select” : filter expression

“Key”: key1,key2,...

“Aggregation”:{

“1”:”output_column1 = input_column.func( )”

.

.

}

“Keep”: all/ list of column_names

“Output_select”: filter expression

In 507, the system may execute a join component to perform a join operation for data records from two inputs. For example, the join operation may perform an SQL-style join operation. The join may include an inner, outer, left, right, or cross join. For example, the join component may have three outputs which may include: insertions, which contain records that exist in input 1 and not in input 2; deletions, which contains records that exist in input 2 and not in input 1; and matches, which contain records that are the same in both inputs.

For example, a configuration for a join component may include:

″join_type″: (″inner″, “outer”, “right”, “left”, or “cross”)

″on″: {

″left_on″: (“col_1, col_2, ..., col_n“),

″right_on″: (“col_1 col_2, ..., col_n“)

},

″output_unmatched_records″: (″y″ or “n”),

″transformation″: {

“1”:output_column= input_column

},

″pass_through″: ”(“in0” or “in1”)”,

″exclude_columns″: ”(column_name)”

“is_empty_fail”: (“y” or “n”)

In 508, the system may execute a union component to combine two or more inputs into a single output.

For example, a configuration for a union component may include:

″config″: {

},

“class″: ″Union″,

″inputs″: [

“in-1″,“in-2″,...

],

″name″: ″union-1″,

″node_id″: “x”,

″outputs″: [

“on-1″

]

In 509, the system may execute a sequence component to generate a sequence of numbers. For example, a sequence operation may generate a sequence of consecutive numbers.

For example, a configuration for a sequence component may include:

″config″: {

″start″: “starting number″,

″step″:“increment value″,

″column_name″:“column_name″

}

In 510, the system may execute a deduplication component to remove duplicate data records. For example, the deduplication operation may identify duplicate records and may remove the first, last, or all duplicate records. The deduplication operation may also collect removed duplicate records through a deselect output.

For example, a configuration for a deduplication component may include:

″config″: {

″key″:″ col_1, col_2, ..., col_n ″,

″keep_order″: ″first/last/False″,

″deselect_port″: ″Y“/”N”

}

In 511, the system may execute a phase component to create dependencies for controlling the data flow of operations. For example, the phase operation may create dependencies between multiple components and may provide an option to select one or more inputs for further processing.

For example, a configuration for a phase component may include:

″config″: {

″select_input″: [“in-1”,”in-2”]

},

“classs″: ″phase″,

″inputs”: [

“in-1″,”in-2”,...,in-n

],

″name″: ″phase-1″,

″node_id″: ″3 ″,

″outputs″: [

“on-1“, ”on-2”

]

It should be noted that the above configuration examples are provided using the Python programming language and related libraries (e.g., pandas). However, implementations using other programming languages to provide substantially equivalent functionality is also contemplated. As described, an execution of a sequence of the components may be based on a transformation graph.

FIG. 6 is a diagram illustrating an example transformation graph 600 of a set of components (e.g., components 118) representing one or more sequences of operations to perform a data transformation according to one or more implementations of the disclosure.

As shown, the transformation graph 600 may represent a set of components (e.g., as nodes of the graph) that represent a sequence of operations (e.g., as represented by the directional vertices). For example, as described, the components may perform operations on the columns of the data as part of a transformation process.

As shown in this example, the source data may correspond to data that is related to data at various time points, which in this example includes data associated with a current day, and data associated with a previous day. In this example, three source components (A1, B1, and C1) read data from three sources. Sequence A may commence by executing a source component A1 that reads data from a file (e.g., current day file) and converts the data into a particular data structure (e.g., DataFrame object). The converted data may be provided to a filter component A2 that filters the data. The data that is filtered out may be written to a reject file by target component A3. The remaining data may be used by a lookup component B3. In addition, the lookup component B3 may obtain data from a reformat component B2. As shown, sequence B may commence by executing source component B1 that reads data from a file. As described above, source component B1 may convert the data into a particular data structure (e.g., DataFrame object), which is then reformatted by reformat component B2. Accordingly, lookup component B3 may use reformatted data from B2 and the filtered data from A2 to lookup data in one or more columns of an input file. This data may then be provided to the reformat component B4 to extract a set of required fields. The extracted data may then be provided to a sort component B5 that performs a sort based on the keys of the data records. Next, the deduplication component B6 may remove the duplicate records. For example, the duplicate records may be determined based on the keys for each record. Next, the join component C4 may perform a join operation. As shown, source component C1 may read a file (e.g., a previous day file) and convert the data as described above. The converted previous day data may then be provided to reformat component C2. The reformatted data may then be provided to replicate component C3 which create two copies of the data. For example, as shown in this example, one copy is provided to groupby component D1, and the other copy to join component C4. For example, an inner join between the current day data that was deduplicated by component B6 and the previous day data replicated by component C3 may be performed. Accordingly, the joined current day data and previous day data may be provided to reformat component C5. Next, this reformatted data may be provided to the target component T1 that writes an update load ready file. In addition, this reformatted data may be provided to a union component C6 that combines the current day updated records with new records provided by the join component C4. Accordingly, sequence component C7 may generate sequence numbers and provide such information to another join component D2 which joins the previous day data with the data combined in C6. This joined data may then be provided to target component T2 that writes the data to an insert load ready file.

As shown in this example, two outputs may be created (T1 and T2). For example, the system may include a master list of all accounts and details. Accordingly, the system may receive a data file daily with account updates or changes to a subset of the accounts. The system may process these accounts (e.g., perform a transformation process as described herein) and create a data file to account for only these changes (e.g., T2). However, the master list may not be synchronized with such changes for the current day, an accordingly, the master list may also need to be updated. Accordingly, the system may create an additional file (e.g., T1) to account for such changes that is used to synchronize the master list with the actual data.

Accordingly, the sequences of operations in this example obtain data from multiple data sources, transform the data, and write the data to output files for updating a target data store.

FIG. 7 is a process flow diagram illustrating an example of a method for performing a data transformation using a set of components according to one or more implementations of the disclosure. Process 700 may use processing logic, which may include software, hardware, or a combination thereof. For example, process 700 may be performed by a system including one or more components described in operating environment 100 (e.g., data management system 110, host system 120, etc.).

In 701, the system (e.g., data management system 110) may obtain a transformation graph of a set of components. In one implementation, the system may create a transformation graph of the set of components. For example, the system may identify one or more source data stores that provide data to be extracted by the system, and based on the source data stores, determine a set of transformation components (or operations) that are required to load the data onto a specified target data store. As described, each component may be executable as an independent service and may perform a predefined operation. The transformation graph may represent one or more sequences of operations to transform data from one or more source data stores (e.g., source data store 130) to a format for updating the target data store (e.g., target data store 140).

The system may then initiate an execution of a set of components based on the transformation graph. The execution of the set of components may include executing one or more source components (e.g., source component