In one embodiment, the transaction log may record each version of the table, the data files associated with a respective version of the data table, information pertaining to the type of transactions that were performed on the data table, the order in which the transactions were performed (e.g., transaction sequence number, a timestamp of the transaction), and an indication of data files that were subject to the transaction, and the like. In some embodiments, the transaction log may include change data for a transaction that also records the changes for data written into a data table with respect to the previous version of the data table. The change data may be at a relatively high level of granularity and may indicate the specific changes to individual records with an indication of whether the record was inserted, deleted, or updated due to the corresponding transaction.

FIG. 3 is a block diagram of an architecture of a control layer 106, in accordance with an embodiment. In one embodiment, the control layer 106 includes an interface module 325, a transaction module 330, a query processing module 335, a cluster management module 340, and an access control module 350. The control layer 106 also includes a data notebook store 360. The modules 325, 330, 335, and 340 may be structured for execution by a computer system, e.g., 900 having some or all of the components as described in FIG. 9, such that the computer system 900 operates in a specified manner as per the described functionality.

The interface module 325 provides an interface and/or a workspace environment where users of client devices 116 (e.g., users associated with tenants) can access resources of the data processing service 102. For example, the user may retrieve information from data tables associated with a tenant, submit data processing requests such as query requests on the data tables, through the interface provided by the interface module 325. The interface provided by the interface module 325 may include notebooks, libraries, experiments, queries submitted by the user. In one embodiment, a user may access the workspace via a user interface (UI), a command line interface (CLI), or through an application programming interface (API) provided by the workspace module 325.

For example, a notebook associated with a workspace environment is a web-based interface to a document that includes runnable code, visualizations, and explanatory text. A user may submit data processing requests on data tables in the form of one or more notebook jobs. The user provides code for executing the one or more jobs and indications such as the desired time for execution, number of cluster worker nodes for the jobs, cluster configurations, a notebook version, input parameters, authentication information, output storage locations, or any other type of indications for executing the jobs. The user may also view or obtain results of executing the jobs via the workspace.

The workspace module 328 deploys workspaces within the data processing service 102. A workspace as defined herein may refer to a deployment in the cloud that functions as an environment for users of the workspace to access assets. An account of the data processing service 102 represents a single entity that can include multiple workspaces. In one embodiment, an account associated with the data processing service 102 may be associated with one workspace. In another embodiment, an account may be associated with multiple workspaces. A workspace organizes objects, such as notebooks, libraries, dashboards, and experiments into folders. A workspace also provides users access to data objects, such as tables or views or functions, and computational resources such as cluster computing systems.

In one embodiment, a user or a group of users may be assigned to work in a workspace. The users assigned to a workspace may have varying degrees of access permissions to assets of the workspace. For example, an administrator of the data processing service 102 may configure access permissions such that users assigned to a respective workspace are able to access all of the assets of the workspace. As another example, users associated with different subgroups may have different levels of access, for example users associated with a first subgroup may be granted access to all data objects while users associated with a second subgroup are granted access to only a select subset of data objects.

The transaction module 330 receives requests to perform one or more transaction operations from users of client devices 116. As described in conjunction in FIG. 2, a request to perform a transaction operation may represent one or more requested changes to a data table. For example, the transaction may be to insert new records into an existing data table, replace existing records in the data table, delete records in the data table. As another example, the transaction may be to rearrange or reorganize the records or the data files of a data table to, for example, improve the speed of operations, such as queries, on the data table. For example, when a particular version of a data table has a significant number of data files composing the data table, some operations may be relatively inefficient. Thus, a transaction operation may be a compaction operation that combines the records included in one or more data files into a single data file.

The query processing module 335 receives and processes queries that access data stored by the data storage system 110. The query processing module 335 may reside in the control layer 106. The queries processed by the query processing module 335 are referred to herein as database queries. The database queries are specified using a declarative database query language such as the SQL. The query processing module 335 compiles a database query specified using the declarative database query language to generate executable code that is executed. The query processing module 335 may encounter runtime errors during execution of a database query and returns information describing the runtime error including an origin of the runtime error representing a position of the runtime error in the database query. In one embodiment, the query processing module 335 provides one or more queries to appropriate clusters of the data layer 108, and receives responses to the queries from clusters in which the queries are executed.

The unity catalog module 345 is a fine-grained governance solution for managing assets within the data processing service 102. It helps simplify security and governance by providing a central place to administer and audit data access. In one embodiment, the unity catalog module 345 maintains a metastore for a respective account. A metastore is a top-level container of objects for the account. The metastore may store data objects and the permissions that govern access to the objects. A metastore for an account can be assigned to one or more workspaces associated with the account. In one embodiment, the unity catalog module 345 organizes data as a three-level namespace, a catalogue is the first layer, a schema (also called a database) is the second layer, and tables and views are the third layer.

In one embodiment, the unity catalog module 345 enables read and write of data to data stored in cloud storage of the data storage system 110 on behalf of users associated with an account and/or workspace. In one instance, the unity catalog module 345 manages storage credentials and external locations. A storage credential represents an authentication and authorization mechanism for accessing data stored on the data storage system 110. Each storage credential may be subject to access-control policies that control which users and groups can access the credential. An external location is an object that combines a cloud storage path (e.g., storage path in the data storage system 110) with a storage credential that authorizes access to the cloud storage path. Each storage location is subject to access-control policies that control which users and groups can access the storage credential. Therefore, if a user does not have access to a storage credential in the unity catalog module 345, the unity catalog module 345 does not attempt to authenticate to the data storage system 110.

In one embodiment, the unity catalog module 345 allows users to share assets of a workspace and/or account with users of other accounts and/or workspaces. For example, users of Company A can configure certain tables owned by Company A that are stored in the data storage system 110 to be shared with users of Company B. Each organization may be associated with separate accounts on the data processing service 102. Specifically, a provider entity can share access to one or more tables of the provider with one or more recipient entities.

Responsive to receiving a request from a provider to share one or more tables (or other data objects), the unity catalog module 345 creates a share in the metastore of the provider. A share is a securable object registered in the metastore for a provider. A share contains tables and notebook files from the provider metastore that the provider would like to share with a recipient. A recipient object is an object that associates an organization with a credential or secure sharing identifier allowing that organization to access one or more shares of the provider. In one embodiment, a provider can define multiple recipients for a given metastore. The unity catalog module 345 in turn may create a provider object in the metastore of the recipient that stores information on the provider and the tables that the provider has shared with the recipient. In this manner, a user associated with a provider entity can securely share tables of the provider entity that are stored in a dedicated cloud storage location in the data storage system 110 with users of a recipient entity by configuring shared access in the metastore.

The access control module 350 receives access control policy specifications and automates and synchronizes the implementation of the access control policy specified in the access control policy specifications across multiple data platforms supported by the data processing service 102. The access control specification may specify access control to a dataset for a user using conditions based on attributes of the dataset. For example, the user may have access to data that has an attribute within a particular range or the attribute having a specific value. For example, the user may have access to data where an attribute representing a state has a value within a specified set of states such as California, Washington, and Oregon. The access control policy may specify that the user has access to certain subset of data after masking the data. For example, the user may have access to data that is tagged as PII (personally identifiable information) provided the data is masked using a masking technique specified in the access control policy.

FIG. 4 is a block diagram of an architecture of a cluster computing system 402 of the data layer 108, in accordance with an embodiment. In some embodiments, the cluster computing system 402 of the data layer 108 includes driver node 450 and worker pool including multiple executor nodes. The nodes may be structured for execution by a computer system, e.g., 900 having some or all of the components as described in FIG. 9, such that the computer system 900 operates in a specified manner as per the described functionality.

The driver node 450 receives one or more jobs for execution, divides a job into job stages, and provides job stages to executor nodes, receives job stage results from the executor nodes of the worker pool, and assembles job stage results into complete job results, and the like. In one embodiment, the driver node receives a request to execute one or more queries from the query processing module 335. The driver node 450 may compile a database query and generate an execution plan. The driver node 450 distributes the query information including the generated code to the executor nodes. The executor nodes execute the query based on the received information.

The worker pool can include any appropriate number of executor nodes (e.g., 4 executor nodes, 12 executor nodes, 256 executor nodes). Each executor node in the worker pool includes one or more execution engines (not shown) for executing one or more tasks of a job stage. In one embodiment, an execution engine performs single-threaded task execution in which a task is processed using a single thread of the CPU. The executor node distributes one or more tasks for a job stage to the one or more execution engines and provides the results of the execution to the driver node 410. According to an embodiment, an executor node executes the generated code for the database query for a particular subset of data that is processed by the database query. The executor nodes execute the query based on the received information from the driver node 450.

FIG. 5 is a block diagram of an architecture of a driver node 450, in accordance with an embodiment. In one instance, the driver node 450 includes a query parser 510, a query rewrite module 520, a logical plan generation module 530, and a physical plan generation module 540. The modules and nodes may be structured for execution by a computer system, e.g., 900 having some or all of the components as described in FIG. 9, such that the computer system 900 operates in a specified manner as per the described functionality.

The query parser 510 receives a database query for processing and parses the database query. The database query is specified using a declarative database query language such as SQL. The query parser 510 parses the database query to identify various tokens of the database query and build a data structure representation of the database query. The data structure representation identifies various components of the database query, for example, any SELECT expressions that are returned by the database query, tables that are input to the query, a conditional clause of the database query, a group by clause, and so on. According to an embodiment, the data structure representation of the database query is a graph model based on the database query.

The query rewrite module 520 performs transformations of the database query, for example, to improve the execution of the query. The improvement may be in terms of execution time, memory utilization, or other resource utilization. A database query may process one or more tables that store a significant number of records that are processed by the database query. Since the declarative database query language does not specify the procedure for determining the result of the database query, there are various possible procedures for executing the database query.

The query rewrite module 520 may transform the query to change the order of processing of certain steps, for example, by changing the order in which tables are joined, by changing the order in which certain operations such as filtering of records of a table is performed in relation to other operations. The query rewrite module 520 may transform the database query to cause certain temporary results to be materialized. The query rewrite module 520 may eliminate certain operations if the operations are determined to be redundant. The query rewrite module 520 may transform a database query so that certain computations such as subqueries or expressions are shared. The query rewrite module 520 may transform the database query to pushdown certain computations, for example, by changing the order in which certain predicates are applied to the computation as early as possible. The query rewrite module 520 may transform the database query to modify certain predicates to use more optimized versions of the predicates that are computationally equivalent but provide better performance.

The logical plan generation module 530 generates a logical plan for the database query. The logical plan includes representation of the various steps that need to be executed for processing the database query. According to an embodiment, the logical plan generation module 530 generates an unresolved logical plan based on the transformed query graph representation. Various relation names (or table names) and column names may not be resolved in an unresolved logical plan. The logical plan generation module 530 generates a resolved logical plan from the unresolved logical plan by resolving the relation names and column names in the unresolved logical plan. The logical plan generation module 530 further optimizes the resolved logical plan to obtain an optimized logical plan.

The physical plan generation module 540 generates a physical plan from the logical plan generated by the logical plan generation module 530. The physical plan specifies details of how the logical plan is executed by the data processing service 102. The physical plan generation module 540 may generate different physical plans for the same logical plan and evaluate each physical plan using a cost model to select the optimal physical plan for execution. The physical plan further specifies details of various operations of the logical plan. As an example, if the logical plan includes a join operator, the physical plan may specify the type of join that should be performed for implementing the join operator. For example, the physical plan may specify whether the join operator should be implemented as a hash join, merge join, or sort join, and so on. The physical plan may be specific to a database system, whereas the logical plan may be independent of database systems and may be executed on any target database system by converting to a physical plan for that target database system.

The code generator 550 generates code representing executable instructions for implementing the physical plan for executing a database query. The generated code includes a set of instructions for each operator specified in the execution plan. The generated code is specified using a programming language that may be compiled and executed.

System Architecture of Access Control Module

FIG. 6 illustrates a query plan generated from a database query and processes for executing the query, in accordance with an embodiment. The database query may process one or more datasets 640a, 640b. The query plan 610 is generated by compiling the database query. The query plan includes nodes corresponding to database operators and edges representing connections between nodes. For example, the query plan 610 includes nodes corresponding to database operators 650a, 650b, 650c, 650d, 650e and edges 625a, 625b, 625c, 625d. An edge 625 going from node 650m to 650n indicates that the output of the database operator corresponding to node 650m is provided as input to the database operator corresponding to node 650n.

The database query may be executed using a cluster of nodes, for example, an Apache Spark™ cluster. The datasets 640a, 640b may be stored in a storage system, for example, Amazon S3™ that provides an object storage service. A database operator may be a data access operator that accesses data from a dataset 640a, 640b. A database operator may be a data processing operator that receives data from a data access operator or a data processing operator. For example, the output of the data access operator 650e is provides as input to the data processing operator 650b via edge 625c. The output of the data processing operator 650d is provides as input to the data processing operator 650a via edge 625a. According to an embodiment, the data access operators are leaf nodes of the query plan, and the data processing operators are non-leaf nodes of the query plan. The node represented by the data processing operator 650a is a root node that does not provide data to any other data processing node and accordingly generates the result of the database query that is sent to the client that sent the database query for processing.

Examples of data processing operators include join operators that may perform sort join, merge join, nested loop join and so on. The data access operator may correspond to a scan operator. The data processing operators may include select operator that selects one of more columns of a dataset. The data processing operator may compute an expression, for example, for evaluating a condition or for determining a value of a result. A data processing operator may execute a function that may be a built-in function supported by the database or a user defined function that is provided by the user and may be implemented using a programming language that is distinct from the database query language.

A node may represent an access control filter, for example, access control filter that is inserted by the query processing module 335 in the query plan to implement the access control policy. The data of the dataset 640a is accessed by the data access operator 650d and provided to the access control filter 650c via the edge 625d. A subset of the data of the dataset 640a that was accessed by the data access operator 650d is provided as input to the data processing operator 650b via edge 625b after being filtered by the access control filter 650c.

As shown in FIG. 6, the query plan is divided into two sets of database operators, a set 630 of data access operators that include the access control filters also and a set 620 of data processing operators. The data processing service 102 executes the query plan using two types of processes, executor processes 660 and sidecar processes 670. The query plan may be divided across multiple executor nodes as shown in FIG. 4. Each executor node may include one executor process 660 and one sidecar process 670. A sidecar process 670 may also be referred to herein as a helper process. According to an embodiment, the executor process 660 is executed in a container that includes the sidecar process as a helper process. The sidecar process 670 and the executor processes 660 communicate using interprocess communications 680, for example, pipes, message passing, sockets, and so on.

Process of Query Compilation

FIG. 7 is a flowchart of a method for synchronizing access control policies across multiple data platforms, in accordance with an embodiment. The process shown in FIG. 7 may be performed by one or more components (e.g., the query processing module 335 of the control layer 106) of a data processing system/service (e.g., the data processing service 102). Other entities may perform some or all of the steps in FIG. 7. The data processing service 102 as well as the other entities may include some or of the component of the machine (e.g., computer system) described in conjunction with FIG. 9. Embodiments may include different and/or additional steps, or perform the steps in different orders.

The data processing service 102 receives 710 an access control policy specification that specifies the access control policy. The access control policy specification may specify conditions associated with data access for users. According to an embodiment, the conditions are specified as expressions based on attributes describing the data. For example, the attribute may be a column or field of the data or the attribute may be a tag associated with the data. For example, an attribute may be a tag that specifies whether a field of the dataset includes PII. According to an embodiment, the conditions in the access control policy specification are specified using the syntax conforming to the database query language used for specifying database queries processed by the data processing service 102, for example, SQL (structured query language).

The query processing module 335 receives 720 a database query for processing. The database query may be received from a client, for example, a client application running on a client device. The client is associated with a user and the query processing module 335 processes the database query in accordance with the access control policy. Accordingly, the database query is executed to generate results such that the user is provided results only based on the subset of data that the user has access to according to the access control policy.

The query processing module 335 compiles 730 the database query to generate a query plan, for example, a query plan similar to that illustrated in FIG. 6. The query plan includes a set of database operators. According to an embodiment, the query plan is a graph made up of nodes representing database operators and edges connecting the database operators. The graph corresponding to a database query may include an edge E from a database operator O1 to a database operator O2 indicating that the output generated by the database operator O1 is provided as input to the database operator O2. The query plan may include one or more root nodes such that there are no outgoing edges connecting a root node to another database operator. A root node generates results of the database query that are sent to the client.

According to an embodiment, the query plan includes one or more data access operators and one or more data processing operators. The data access operators access data from datasets that may be stored on a storage system, for example, a file in a file system. The data access operators are leaf nodes of the query plan, and the data processing operators are no leaf nodes of the query plan that receive data processed by the data access operators.

The query processing module 335 determines 740 filter conditions based on access control policies specified in the access control policy specification. The filter conditions include instructions to determine a subset of data of a dataset, for example, a file. The subset of data represents the data of the dataset that is accessible to the user providing the database query according to the access control policy.

The query processing module 335 includes 750 access control filters in the query plan based on the filter conditions. For example, the query processing module 335 may determine that the subset of data of dataset D1 stored in file F1 that is accessible to the user is determined using a condition C1. The query processing module 335 generates an access control filter F1 based on the conditions C1. The query processing module 335 identifies the data access operator O1 of the query plan that accesses data of the dataset D1. Assume that the data access operator O1 has an edge going from the data access operator O1 to a data processing operator O2. The query processing module 335 modifies the query plan by inserting the access control filter F1 between the data access operator O1 and the data processing operator O2. Accordingly, the output of the data access operator O1 is filtered by the access control filter F1 to generate a subset S1 of data of the dataset D1. The subset S1 represents a subset of data of the dataset D1 that is accessible to the user according to the access control policy. The subset S1 is fed as input to the data processing operator O2. Similarly, the query processing module 335 may insert an access control filter for an output of every data access operator that processes a dataset identified in the access control policy specification such that the user has access to only a subset of the data of the dataset. If the access control policy does not restrict access to data of a dataset for the user, the query processing module 335 does not insert any access control filter to restrict the output of the data access operator accessing data of that dataset. Accordingly, if the access control policy does not restrict access to data of a dataset for the user, the query plan directly feeds the data output by the corresponding data access operator to one or more data processing operators.

Process of Query Execution

FIG. 8 is a flowchart of a method for synchronizing access control policies across multiple data platforms, in accordance with an embodiment. The process shown in FIG. 8 may be performed by one or more components (e.g., the executor node of the cluster computing system 402) of a data processing system/service (e.g., the data processing service 102). Other entities may perform some or all of the steps in FIG. 8. The data processing service 102 as well as the other entities may include some or of the component of the machine (e.g., computer system) described in conjunction with FIG. 9. Embodiments may include different and/or additional steps or perform the steps in different orders.

According to an embodiment, the query processing module 335 compiles the database query according to the process shown in FIG. 7 and distributes the query plan across a plurality of executor nodes. Alternatively, the query plan may be executed by a single executor node. The executor node starts 810 the executor process 660 and starts 820 a helper process, for example, sidecar process 670. The helper process communicates with the executor process using an interprocess communication mechanism 680.

According to an embodiment, the driver node 450 provides temporary credentials to the helper process. The temporary credentials allow the helper process with temporary access to one or more datasets being processed by the database query. The temporary credentials typically provide coarse grained access to the dataset, for example, access to entire data stored in a file storing the dataset. The helper process