132, pod VM(s) 138, and support VM(s) 144) includes a guest OS (e.g., guest OSs 134, 140, and 146, respectively) and one or more applications (e.g., application(s) 136, 142, and 148, respectively). The guest OS may be a standard OS and the applications may run on top of the guest OS. An application may be any software program, such as a word processing program, a virtual desktop interface (VDI), or other software program. The applications can includes containerized applications executing in pod VMs 138 and non-containerized applications executing directly on guest OSs in in native VMs 132. Support VMs 144 have specific functions within host cluster 120. For example, support VMs 144 can provide control plane functions, edge transport functions, and/or the like. As discussed in more detail below with respect to FIG. 2, a host 130 may run one or more VMs 205 running one or more ML/AI applications and one or more virtual ML/AI accelerators.

Host(s) 130 may be constructed on a server grade hardware platform 160, such as an ×86 architecture platform. Hardware platform 160 of a host 130 may include components of a computing device such as one or more central processing units (CPUs) 162, memory 164, one or more physical network interfaces (PNICs) 166, storage 168, and other components (not shown). A CPU 162 is configured to execute instructions, for example, executable instructions that perform one or more operations described herein and that may be stored in memory 164 and storage 168. PNICs 166 enable host 130 to communicate with other devices via a physical network, such as management network 115 and data network 170.

Memory 164 is hardware allowing information, such as executable instructions, configurations, and other data, to be stored and retrieved. Memory 164 may be volatile memory or non-volatile memory. Volatile or non-persistent memory is memory that needs constant power in order to prevent data from being erased, such as dynamic random access memory (DRAM).

Storage 168 represents persistent, non-volatile memory, storage devices that retains its data after having power cycled (turned off and then back on), which may be byte-addressable, such as one or more hard disks, flash memory modules, solid state disks (SSDs), magnetic disks, optical disks, or other storage devices, as well as combinations thereof. In some embodiments, hosts 130 access a shared storage using PNICs 166. In another embodiment, each host 130 contains a host bus adapter (HBA) through which input/output operations (IOs) are sent to the shared storage (e.g., over a fibre channel (FC) network). A shared storage may include one or more storage arrays, such as a storage area network (SAN), a network attached storage (NAS), or the like. In some embodiments, shared storage 168 is aggregated and provisioned as part of a virtual SAN (vSAN). As discussed in more detail below with respect to FIG. 2, a memory 164 may store one or more computational graphs.

Hypervisor 150 architecture may vary. Hypervisor 150 can be installed as system level virtualization software directly on the server hardware (often referred to as “bare metal” installation) and be conceptually interposed between the physical hardware and the guest OSs executing in the VMs. Alternatively, the virtualization software may conceptually run “on top of” a conventional host OS in the server. In some implementations, hypervisor 150 may comprise system level software as well as a “Domain 0” or “Root Partition” VM (not shown) which is a privileged machine that has access to the physical hardware resources of the host 130. In this implementation, one or more of a virtual switch, a virtual router, a virtual tunnel endpoint (VTEP), etc., along with hardware drivers, may reside in the privileged VM. One example of hypervisor 150 that may be used is a VMware ESXi™ hypervisor provided as part of the VMware vSphere® solution made commercially available from VMware, Inc. of Palo Alto, California.

Data center 102 includes a management plane and a control plane. The management plane and control plane each may be implemented as single entities (e.g., applications running on a physical or virtual compute instance), or as distributed or clustered applications or components. In alternative embodiments, a combined manager/controller application, server cluster, or distributed application, may implement both management and control functions. In the embodiment shown, network manager 108 at least in part implements the management plane and controller 106 at least in part implements the control plane

The control plane determines the logical overlay network topology and maintains information about network entities such as logical switches, logical routers, and endpoints, etc. As used herein, the term “overlay” may be used synonymously with “logical” and refers to the logical network implemented at least partially within networking environment 100. The logical topology information is translated by the control plane into network configuration data, such as forwarding table entries to populate forwarding tables at virtual switches in each host 130. A virtual switch provided by a host 130 may connect virtualized endpoints running on the same host 130 to each other as well as to virtual endpoints on other hosts. Logical networks typically use Layer 2 (L2) routing based on data link layer addresses (e.g., Medium Access Control (MAC) addresses). The network configuration data is communicated to network elements of host(s) 130.

Controller 106 generally represents a control plane that manages configuration of VMs within data center 102. Controller 106 may be one of multiple controllers executing on various hosts 130 in data center 102 that together implement the functions of the control plane in a distributed manner. Controller 1065 may be a computer program that resides and executes in a server in data center 102, external to data center 102 (e.g., in a public cloud), or, alternatively, controller 106 may run as a virtual appliance (e.g., a VM) in one of the hosts 130. Although shown as a single unit, it should be understood that controller 106 may be implemented as a distributed or clustered system. That is, controller 106 may include multiple servers or VCIs that implement controller functions. It is also possible for controller 106 and network manager 108 to be combined into a single controller/manager. Controller 106 collects and distributes information about the network from and to endpoints in the network. Controller 106 is associated with one or more virtual and/or physical CPUs (not shown). Processor(s) resources allotted or assigned to controller 106 may be unique to controller 106, or may be shared with other components of data center 102. Controller 106 communicates with hosts 130 via management network 115, such as through control plane protocols. In some embodiments, controller 106 implements a central control plane (CCP).

Network manager 108 and virtualization manager 110 generally represent components of a management plane comprising one or more computing devices responsible for receiving logical network configuration inputs, such as from a user or network administrator, defining one or more endpoints (e.g., VCIs) and the connections between the endpoints, as well as rules governing communications between various endpoints.

In some embodiments, virtualization manager 110 is a computer program that executes in a server in data center 102 (e.g., the same or a different server than the server on which network manager 108 executes), or alternatively, virtualization manager 110 runs in one of the VMs. Virtualization manager 110 is configured to carry out administrative tasks for data center 102, including managing hosts 130, managing VMs running within each host 130, provisioning VMs, transferring VMs from one host 130 to another host, transferring VMs between data centers, transferring application instances between VMs or between hosts 130, and load balancing among hosts 130 within data center 102. Virtualization manager 110 takes commands as to creation, migration, and deletion decisions of VMs and application instances on data center 102. Virtualization manager 110 also makes independent decisions on management of local VMs and application instances, such as placement of VMs and application instances between hosts 130. In some embodiments, virtualization manager 110 also includes a migration component that performs migration of VMs between hosts 130. One example of a virtualization manager 110 is the vCenter Server™ product made available from VMware, Inc. of Palo Alto, California. One example of control plane functions is vMotion® made commercially available from VMware, Inc. of Palo Alto, California.

In some embodiments, network manager 108 is a computer program that executes in a server in networking environment 100, or alternatively, network manager 108 may run in a VM (e.g., in one of hosts 130). Network manager 108 communicates with host(s) 130 via management network 115. Network manager 108 may receive network configuration input from a user or an administrator and generates desired state data that specifies how a logical network should be implemented in the physical infrastructure of data center 102. Network manager 108 is configured to receive inputs from an administrator or other entity (e.g., via a web interface or application programming interface (API)), and carry out administrative tasks for data center 102, including centralized network management and providing an aggregated system view for a user. One example of a network manager 108 is the NSX™ product made available from VMware, Inc. of Palo Alto, California.

Container orchestrator 112 provides a platform for automating deployment, scaling, and operations of application containers across host cluster(s) 120. In some embodiments, the virtualization layer of a host cluster 120 is integrated with an orchestration control plane. For example, virtualization manager 110 may deploy the container orchestrator 112. The orchestration control plane can include the container orchestrator 112 and agents 152, which may be installed by virtualization manager 110 and/or network manager 108 in hypervisor 150 to add host 130 as a managed entity. Although container orchestrator 112 is shown as a separate logical entity, container orchestrator 112 may be implemented as one or more native VM(s) 132 and/or pod VMs 138. Further, although only one container orchestrator 112 is shown, data center 102 can include more than one container orchestrator 112 in a logical cluster for redundancy and load balancing. In some systems, containers are grouped into logical units called “pods” that execute on nodes in a cluster (also referred to as “node cluster”). A node can be a physical server or a pod VM 138.

In some embodiments, the network environment 100 includes one or more hardware agnostic vAIAs. FIG. 2 is a block diagram of an example hardware agnostic vAIA running on a host 130, according to one or more embodiments. As shown, one or more VMs running on host 130, such as VM 205, VM 210, and VM 215, run an ML/AI application (application 220 on VM 205, application 225 on VM 210, and application 230 on VM 215).

In some embodiments, when the end user deploys the ML/AI application, the under user will declare the infrastructure needs of the application, such as target RAM, CPU, disk, and ML features the application requires. The ML/AI application will then be deployed on a host that has ML/AI accelerator hardware to meet the target requirements (typically there will be many hosts, possibly with different HW.) The ML application (the VM) can then migrate to other hosts (for load balancing or system maintenance purposes), provided the new host it migrates to can also meet all the requirements.

Different hardware may support a different, sometimes overlapping, subset of tensor operations and data-types. In some embodiments, capabilities levels are defined to allow ML/AI workloads to migrate between heterogeneous hardware ML/AI accelerators. For example, a first capability level may be defined for 8-bits integer arithmetic CNNs, another capability level may defined for full tensor operation with arbitrary data types. The different hardware ML/AI accelerators may support different capability levels. In some embodiments, the user selects a capability level (e.g., a minimum capability level to support the ML/AI workload) for a given ML/AI workload. Accordingly, the migration of the ML/AI workload may be based on the specified capability level.

The ML/AI applications are associated with an ML/AI framework using an ML/AI library. As shown, VM 205 uses ML/AI library 255, VM 210 uses ML/AI library 260, and VM 215 uses ML/AI library 265. The VMs may use the same ML/AI framework or different ML/AI frameworks. In some embodiments, multiple ML/AI frameworks are supported, and the ML/AI framework to be used is selected by a user.

One example of an ML framework is TensorFlow. TensorFlow is an open-source library for numerical computation using data flow graphs. TensorFlow provides a simple and powerful interface for creating complex models. TensorFlow supports a variety of deep learning architectures, including CNNs, RNNs, and more. The computation is organized as a computational graph, where each node in the graph represents an operation, and the edges represent the data that flows between the nodes. The data that flows between the nodes is represented as multidimensional arrays called tensors. Tensors are the fundamental building blocks of TensorFlow, and they can represent a variety of data types, including scalars, vectors, matrices, and higher-dimensional arrays. A tensor is an object that represents multi-dimensional arrays of numerical data. Tensors can be scalars (0-dimensional), vectors (1-dimensional), matrices (2-dimensional), or higher-dimensional arrays. In machine learning, tensors may be used to represent data, such as images, sound waves, or text, that are processed by neural networks. For example, a color image can be represented as a 3-dimensional tensor, where the first dimension represents the height of the image, the second dimension represents the width of the image, and the third dimension represents the color channels (red, green, and blue). TensorFlow also provides a high-level API that makes it easy to create and train complex machine learning models. For example, the TensorFlow API can be used to define a CNN for image classification by simply defining the layers of the network and specifying the loss function and optimization algorithm.

Another example of an ML framework is PyTorch. PyTorch is an open-source ML library that is primarily used for building deep learning models for a variety of applications such as image recognition, natural language processing, and computer vision. PyTorch uses a dynamic computational graph, which allows a model to be built on-the-fly as it runs, as opposed to a static graph used by other frameworks like TensorFlow. PyTorch uses tensor computation, which allows for efficient numerical computations. This makes PyTorch ideal for large-scale data processing, as it can handle vast amounts of data quickly and efficiently. PyTorch also offers a number of pre-built functions that make it easier for developers to work with complex data structures such as matrices and vectors.

Another example of an ML framework is ONNX-RT (Open Neural Network Exchange Runtime). ONNX-RT is an open-source software framework designed to accelerate the performance of deep learning models by optimizing their execution on various hardware platforms such as CPUs, GPUs, and FPGAs. ONNX-RT supports the ONNX (Open Neural Network Exchange) format, which is an open-source format for representing deep learning models that allows developers to easily move their models between different deep learning frameworks, such as TensorFlow and PyTorch.

The ML/AI frameworks may define several thousand tensor operations, from trivial element-wise arithmetic to complex FFTs. In some embodiments, the user may define custom tensor operations, such as through C++ or CUDA routines.

Host 130 runs one or more vAIAs 240. In some embodiments, each ML/AI application may be associated with its own vAIA. In some embodiments, a vAIA may be associated with multiple ML/AI applications. As shown in FIG. 2, vAIA 240 is shared by application 220 on VM 205, application 225 on VM 210, and application 230 on VM 215. In some embodiments, the vAIA 240 performs computations to assist (e.g., speed up) the ML/AI application 220, 225, 230. In some embodiments, the vAIA is a process running on the host 130 or on a VM running on the host 130 that is responsible or handling I/Os.

VAIA 240 is a software emulation of a physical hardware ML/AI accelerator(s) 250. VAIA 240 enables an OS (e.g., the guest OSs running on the VMs) to interact with the physical hardware ML/AI accelerator(s) 250 as if it were physically present on the system. vAIA 240 represents one or more ML/AI computation(s) in a hardware agnostic form using a computational graph 270. The ML/AI computation(s) may be done by one or more of the hardware ML/AI accelerators 250 underlying the vAIA 240.

In some embodiments, each vAIA 240 abstracts the aggregation of multiple locally attached physical hardware ML/AI accelerators. In some embodiments, each vAIA 240 abstracts away the interconnect topology and low level hardware specific communication interfaces, ensuring that lateral communication across ML/AI accelerators is direct—without guest VM involvement. In some embodiments, the existence of multiple hardware accelerators is hidden from the VM and the vAIA 240 behaves as a single accelerator, with the combined capacity and throughput of the multiple attached physical accelerators.

In some embodiments, vAIA 240 transfers data between guest memory and the host 130. A memory-mapped input/output (MMIO) mechanism may allow a driver to read from and write to registers, which control its functionality. These registers are mapped to a specific area of system memory, and the device driver can interact with them just as it would with physical hardware. A first-in, first-out (FIFO) buffer may be used to transfer data between the device and the driver. Data is written to the buffer by the driver and read from the buffer by the device, or vice versa. The size of the buffer determines how much data can be transferred at a time, and the device driver must manage the buffer to ensure that data is not lost or corrupted. Interrupt request lines (IRQs) are signals sent to the OS to indicate that it needs attention. When the device driver receives an IRQ, it responds appropriately, such as by reading or writing data, resetting the device, or performing another action.

Unlike other virtual PCIs, in vAIA 240 the state consists of tensors (N-dimensional arrays) and computational graph(s) 270 and graph execution commands operate on the state. In some embodiments, an ML/AI model for an ML/AI application (e.g., application 220, 225, or 230) is parsed, using an ML/AI framework (e.g., ML/AI library 255, 260, or 265), to generate the one or more computational graph(s) 270.

The computational graph(s) 270 are a graphical representation of a mathematical function or algorithm. Computational graph(s) 270 visualize the flow of data through a network of nodes and connections. For example, a computation graph 270 may be a directed acyclic graph whose nodes describe operations between tensors. A tensor operation can be any function between tensors. The nodes represent mathematical operations or functions, and the connections represent the flow of data between them. In some embodiments, the network of nodes are made up of two main types of nodes: input nodes and computation nodes. Input nodes are used to represent the input data that is fed into the graph, while computation nodes represent the mathematical operations that are performed on the data. The connections between the nodes represent the flow of data between them. Each connection has a weight associated with it, which determines how much influence the output of one node has on the input of the next node. The weights may be represented as arrows pointing from one node to another node. In some embodiments, the computation nodes are further divided into different types, depending on the type of mathematical operation they perform. For example, there may be nodes that perform addition, multiplication, or activation functions such as the sigmoid function or the rectified linear unit (ReLU) function.

In some embodiments, the computational graph(s) 270 are used to perform forward and backward propagation, which are the two main phases of a neural network. Forward propagation involves feeding the input data through the graph and computing the output of the final node. Backward propagation involves computing the gradients of the loss function with respect to the weights of the connections, which is used to update the weights during the training process.

FIG. 3 depicts an example computational graph 300 (e.g., an example of a computational graph 270). The example computational graph 300 describes the computation of X=A+B−C, where A, B, and C are the input tensors 302, 304, and 306, respectively. Computational node 308 is the addition operation, computational node 310 is the subtraction operation, and X is the output tensor 312. The example computational graph 300 is two layers deep, however, graphs for state-of-the-art NNs can be, for example, hundreds of layers deep.

In some embodiments, the computation graph(s) 270 are stored in memory 164 (and/or in storage 158 as) shown in FIG. 2.

In some embodiments, the ML/AI computations for the ML/AI applications 220, 225, or 230 are lowered to an intermediate form expected by the vAIA, for example, by the creation, uploading, downloading, and destruction of tensors and the creation, execution, and destruction of computational graph(s) 270.

In some embodiments, each VM includes an ML/AI framework specific plugin. As shown in FIG. 2, VM 205 includes vAIA plugin 275, VM 210 includes vAIA plugin 280, and VM 215 includes vAIA plugin 285. In some embodiments, the ML/AI plugin runs on the guest OS of the respective VM.

The ML framework, including the ML/AI library 255, 260, or 265, generates tensors based on the computation associated with the respective ML/AI application 220, 225, or 230. In some embodiments, the ML framework generates computation graph(s) 270 based on the tensors and the vAIA plugin 275, 280, or 285 are used to provide the computation graph(s) 270 to the vAIA 240. In some embodiments, the vAIA plugin 275, 280, or 285 provides the tensors to the vAIA 240 and the vAIA 240 infers computational graph(s) 270 implicitly from the tensors.

In some embodiments, host 130 includes a driver VM 290. In some embodiments, the driver VM 290 is a pod VM 138. The driver VM 290 may run a guest OS 140 (e.g., a Linux kernel) and a container engine running on the guest OS that supports one or more containers running on the pod VM 138. As shown in FIG. 2, the driver VM 290 includes vAIA backend 292, graph compiler 294, and one or more driver(s) 296. The graph compiler 294 and driver(s) 296 may be provided by a vendor of the underlying hardware ML/AI accelerators 250.

In some embodiments, vAIA backend 292 is associated with one or more locally attached physical hardware ML/AI accelerator(s) 250, as shown in FIG. 2. In some embodiments, the one or more physical hardware ML/AI accelerators 250 are attached via a PCI bus. In some embodiments, the one or more vAIA backend 292 is associated with a remote hardware ML/AI accelerator. In some aspects, a remote hardware ML/AI accelerator can be used by running a hardware abstraction layer (e.g., such as by leveraging BitFusion or Radium) inside the driver VM 290. For example, in some embodiments, the tensors or computation graph(s) 270 are sent across a network and executed by a remote ML/AI accelerator.

In some embodiments, the vAIA 240 transfers guest OS commands and data to the driver VM 290 through a paravirtualized memory sharing device. In some embodiments, vAIA 240 transfers the computation graph(s) 270 to the driver VM 290 via the vAIA backend 292. The VAIA backend 292 may translate the computational graph(s) 270 into a form that can be consumed by the graph compiler 294.

In some embodiments, graph compiler 294 takes a trained ML/AI model and compiles the model into a format that can run on different ML/AI accelerator hardware. In some embodiments, graph compiler 294 analyzes computation graph(s) 270 and performs a series of optimizations to optimize the computational graph(s) 270 for target physical hardware ML/AI accelerator(s) 250. The target hardware may be pre-configured.

One example of a graph compiler 294 is an ONNX-RT graph compiler. The ONNX-RT Graph Compiler takes a trained ML/AI model as input and transforms it into a format that can be executed on different hardware devices. The ONNX-RT graph compiler analyzes the computational graph and performs a series of optimizations to make the graph more efficient for the target hardware. The optimizations can include node fusion, where multiple nodes are combined into a single node to reduce memory usage and computation time. The graph compiler can also perform kernel selection, where the optimal mathematical operations are selected for each node based on the target hardware. Once the graph has been optimized, the ONNX-RT graph compiler generates a platform-specific runtime graph. This platform-specific runtime graph is optimized for the target hardware and includes all the necessary mathematical operations and memory allocations. The runtime graph is then executed on the hardware ML/AI accelerators 250.

Another example of a graph compiler 294 is a TVM (Tensor Virtual Machine) graph compiler. The TVM graph compiler breaks down a ML model into a series of computations that can be represented as a directed acyclic graph (DAG). Each node in the DAG represents a computation, and the edges between nodes represent the data flow between computations. The TVM graph compiler then applies a series of optimizations to this graph, including graph-level optimizations like operator fusion, as well as device-level optimizations like memory layout and data movement.

Another example of a graph compiler 294 is a MLIR (Multi-Level Intermediate Representation) graph compiler. The MLIR graph compiler is part of the TensorFlow ML framework. The MLIR graph compiler is designed to be a multi-level IR, which means that it supports multiple levels of abstraction for expressing ML models. MLIR supports high-level abstractions like TensorFlow's computation graph and, at the lower level, MLIR supports low-level abstractions like LLVM IR, which is a low-level representation of a program that can be executed efficiently on a variety of hardware architectures. The MLIR graph compiler takes a TensorFlow computation graph and compiles it down to a lower-level representation that can be executed on accelerators. The MLIR graph compiler uses a series of optimization passes to optimize the computation graph for efficiency and then generates code for the target accelerator.

In some embodiments, the driver VM 290 has direct access to the physical hardware ML/AI accelerator(s) 250. In some embodiments, the driver VM 290 runs unmodified hardware vendor user-space and kernel-space drivers 296, such as Linux drivers, to avoid the need of porting any drivers. Driver VM 290 controls the hardware ML/AI accelerator(s) 250 using the unmodified hardware drivers 296.

From the VM perspective, the vAIA 240 works like a physical hardware device generally does, but without mimicking any particular physical device. Not having to mimic any existing hardware means the vAIA can be optimized for the virtual environment. Because the VAIA 240 does not behave like any hardware previously known to the guest OS, virtual device drivers may be used in the VM.

In some embodiments, for migration of the ML/AI workload, the vAIA 240 quiesces between commands (e.g., between computational graph 270 execution or between a tensor upload/download command.) In some embodiments, mid-graph execution is performed while quiescing.

In some embodiments, vAIA 240 masks features of the underlying physical hardware ML/AI accelerator(s) 250 in order to facilitate the migration between different hardware. As an illustrative example, an ML/AI accelerator A supports tensors with 32-bits and 16-bits floating point representation, and an ML/AL accelerator B supports 16-bits and 8-bits tensors. vAIA 240 may mask out features to a common denominator (16-bits in this case) to allow the computation to transparently move across the ML/AI accelerators A and B. In addition to tensor numeric types, hardware also may diverge in supported tensor operations and maximum tensor size, which vAIA 240 can mask out down to the common denominator.

FIG. 4 depicts a block diagram of a workflow 400 for use of a vAIA on a host, according to one or more embodiments. The workflow 400 may be understood with reference to the example host 130 illustrated in FIG. 2.

As shown, workflow 400 includes, at step 402, running an ML/AI application (e.g., application 220) on a VCI (e.g., VM 205) on a host (e.g., host 130). In some embodiments, the ML/AI application is a query/response application, an image recognition application, a speech recognition application, a natural language processing application, a recommendation generation application, a fraud detection application, or any other type of application using ML/AI.