num="0034">In aspects, the model merger 204 merges select layers of different models. Some systems may consolidate layers in early sections of multi-layer models to reduce computation when executing inferences on a common set of input data (e.g., input video stream). These systems generate common stems of layers to reduce paths of processing data. In contrast, the model merger 204 merges select layers from any location within respective models to reduce the memory footprint of the models.

The refresh determiner 232 determines whether to refresh (or merge models) when a value of data drift is greater than a predetermined threshold.

The sharable layer determiner 234 determines a set of layers that are sharable among the models. In aspects, layers are sharable when a set of properties associated with the respective layers are within a predetermined threshold of similarity, including a case where values of each property of the set of properties among the respective layers match. For example, two layers are sharable when both layers have the same input data size, the same output data size, the same kernel size, and the same size of strides. In aspects, layers that are sharable between distinct models. In some other aspects, a single model may include layers that sharable. The sharable layer determiner 234 may analyze properties of layers for determining sharable layers in a descending order starting from the layer with the highest memory consumption. For example, the sharable layer determiner 234 generates a descending list of layers based on memory consumption for respective models. By consolidating sharable layers that are high in memory consumption, the disclosed technology may efficiently reduce a memory footprint of the overall models (e.g., by merging the heaviest layers first). In aspects, the sharable layer determiner 234 determines a set of sharable layers based on a combinational search on at least a part of layers in the sorted list of layers.

The model generator 236 generates a merged model from the models with the sharable layers by instantiating a single layer that corresponds to the sharable layers. In aspects, the generated model may include multiple entry points as input to the model and multiple exit points as output to the model, each pair of the entry points and the exit points representing a pair of input and output for previously distinct models (i.e., sub-models of the merged model). In aspects, the sub-models of the generated merged model may share one or more instantiations of intermediate layers.

The model trainer 238 trains the generated merged model. In aspects, the model trainer 238 uses training data 242, which is a combination of training data for the previously distinct models. In aspects, merging of the models balances trade-offs between reduced accuracy in predicting inferences and a reduction in memory consumption. The training data 242 may include a predetermined threshold of accuracy that is sufficient or acceptable across all the models that are being merged. When a generated model exhibits a degree of accuracy that surpasses the predetermined threshold, the model may be suitable for performing data analytics. In some aspects, models reserve memory-intensive layers for learning specialized tasks. Accordingly, use of the aggregate training data across the merged models attains a sufficient level of accuracy for inference execution across the merged models. In aspects, the model trainer 238 may also update weights associated with consolidated layers in the generated model. In some examples, the model trainer 238 generates the trained model by executing the respective models or an exemplar model with high-fidelity on sampled frames from a target video feed.

The model transmitter 240 transmits at least the instantiated sharable layer to the edge server 202 for updating the models that are affected by the instantiated sharable layer in the edge server 202. After transmitting the model with at least the instantiated sharable layer, the layer attribute receiver 230 continues receiving the status information associated with layers of the models.

In some aspects, including during a bootstrapping phase, the model transmitter 240 transmits unmerged models to the edge server 202. The edge server 202 performs inference execution using the models based on GPU time-sharing mechanisms. In some other aspects, the model transmitter 240 transmits the merged structure of the generated model and the updated weights to the edge server 202. The edge server 202 replaces individual models in GPU memory with the merged model during swapping operations. As the swapping operations continue over time, the merged models may completely replace the individual models. When a size of the memory of the edge server 202 is insufficient to create a single merged model that includes all layers of the individual models, the edge server 202 may time-slice the merged models on the GPU based on some scheduling mechanisms for executing instructions. In aspects the combined use of the merged models and time-slicing reduce the need for swapping models.

In aspects, the edge server 202 transmits one or more sample frames to the model merger 204 for determining when the merged models need to be refreshed. Data drift, or deviations in semantic accuracy of inference execution, may occur over time as the edge server 202 continues to operate on live video feeds. When the refresh determiner 232 determines the merged models need to be refreshed, the refresh determiner 232 uses the merged structure and weights associated with the merged structure as the initial condition to refine the layered structure of the merged models.

As will be appreciated, the various methods, devices, applications, features, etc., described with respect to FIG. 2 are not intended to limit the system 200 to being performed by the particular applications and features described. Accordingly, additional controller configurations may be used to practice the methods and systems herein and/or features and applications described may be excluded without departing from the methods and systems disclosed herein.

FIG. 3 illustrates an example of data structure for models in accordance with aspects of the present disclosure. The data structure 300 includes a merged model 301. In aspects, the merged model 301 represents a model that the model generator 236 in the model merger 204 has generated based on multiple models with sharable layers.

In the example, the merged model 301 includes three sub-models that were respectively distinct models prior to the merger: a first sub-model 302A, a second sub-model 302B, and a third sub-model 302C. The first sub-model 302A includes an input as an entry point to the model, a first layer 306A, a first shared layer 340, a third layer 310A, a first shared layer 340 (repeating), a last layer 314A, and output 316A as an exit point of the model. In aspects, prior to the merger, the model (now the first sub-model 302A) included two layers with common properties as repeating layers and thus were sharable. The first shared layer 340 in both positions in the first sub-model 302A uses a single instance of the first shared layer 340 in the memory as indicated in the shared layers 330.

The second sub-model 302B includes input 304B, a first layer 306B, a second layer 308B, the first shared layer 340, the second shared layer 342, a last layer 314B, and output 316B. The third sub-model 302C includes input 304C, a first layer 306C, the second shared layer 342, a third layer 310A, the first shared layer 340, a last layer 314C, and output 316C. The first shared layer 340 in respective positions in respective sub-models use a single instance of the first shared layer 340 in the shared layers 330. Similarly, the second shared layer 342 in the second sub-model 302B and the third sub-model 302C use a single instance of the second shared layer 342 in the shared layers 330. As illustrated in the example, the disclosed technology merges models by sharing layers with common properties regardless of positions of the sharable layers in respective sub-models. Accordingly, in this example, use of the first shared layer 340 upon the merging models may reduce memory consumption from four instantiations to one instantiation. Similarly, use of the second shared layer 342 upon merging of the models may reduce memory consumption from two instantiations to one.

FIG. 4 illustrates an example of data structure for properties of layers in models in accordance with aspects of the present disclosure. The data structure 400 includes layer properties 402 for respective layers of models (e.g., a set of the distinct three models prior to the merger to generate the merged model 301 as shown in FIG. 3). In aspects, the layer properties 402 includes model ID 404, layer ID 406, input data size 408, kernel size 410, stride 412 (in pixels), and output data size 414. For example, a model with model ID 001 includes four layers (layer IDs 001, 002, 003, 004, and 005). In particular, some of layers have property values that are in common.

As shown as emphases in FIG. 4, the four layers have common property values (i.e., the input data size of 4096, the kernel size of 512, the stride of 8, and the output data size of 1024. The four layers correspond to the second layer and the fourth layer of the first model, the third layer of the second model, and the fourth layer of the third model (e.g., as an example after the merger, refer to the first shared layer 340 of the first sub-model 302A, of the second sub-model 302B, and of the third sub-model 302C in the merged model 301 as shown FIG. 3.) Accordingly, sharable layers 420 includes the four layers as sharable layers, for which the model generator consolidates into one instantiation of a layer (e.g., the first shared layer 340 in the shared layers 330 as shown in FIG. 3).

FIG. 5 illustrates an example data structure of a layer of a model in accordance with aspects of the present disclosure. The data structure 500 includes a layer 504 with input 502 and output 508. The layer includes a weight 506. In aspects, a layer functionality includes extracting features (e.g., people, faces, automobiles, etc.) of data and identifying a predetermined shape that represents objects. In some aspects, the layer 504 is a convolutional layer that filters image based on a predetermined filtering condition and outputs filtered data. In some other aspects, the layer 504 may be a part of a recursive neural network. The layer 504 may be a part of a convolutional neural network. In aspects, the weight 506 determines how various aspects of input data of the layer influence output data of the layer.

FIG. 6A is an example of a method for merging models in accordance with aspects of the present disclosure. A general order of the operations for the method 600A is shown in FIG. 6A. Generally, the method 600A begins with start operation 602 and ends with end operation 618. The method 600 may include more or fewer steps or may arrange the order of the steps differently than those shown in FIG. 6A. The method 600A can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. Further, the method 600A can be performed by gates or circuits associated with a processor, an ASIC, an FPGA, a SOC or other hardware device. Hereinafter, the method 600A shall be explained with reference to the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, methods, etc., described in conjunction with FIGS. 1, 2, 3, 4, 5, 6B, 7, and 8A-B.

Following start operation 602, the method 600A begins with transmit operation 604, in which a set of models are transmitted to the edge server. In aspects, one or more service applications use the set of models in the edge server. The edge server includes processors and heterogeneous accelerators to execute the service applications. The processors include a CPU and the heterogeneous accelerators include GPU and NIC and other accelerators based on ASIC, FPGA, and NPU, as examples.

Sort operation 606 sorts layers of the models into a list based on memory consumption. In aspects, memory consumption by respective layers of the models may vary. By prioritizing consolidating layers with higher levels of memory consumption, the disclosed technology may reduce the memory footprint of the models more substantially than consolidating layers with lower levels of memory consumptions. In aspects, memory consumption of layers in a model exhibit a power-law distribution, where few layers contribute most of the model's memory usage while the larger number of remaining layers incur little overhead.

Based on memory consumption, select operation 608 selects a layer of a model to determine sharable layers. In aspects, the select operation 608 selects the layer with the highest level of memory consumption. The select operation 608 may select models used for the same or similar data analytics of data. For example, models used for processing video stream data from video cameras capturing scenes from the same or similar locations (e.g., video footage of a location from varying angles of views) or similar types of locations (e.g., scenes of hallways inside buildings, scenes of busy streets with automobile traffic). In aspects, data models used for analyzing similar scenes for the same or similar purposes may have sets of layers with similar properties.

Determine operation 610 determines sharable layers of the models. In aspects, the determine operation 610 determines layers as sharable based on values of properties of the layers. The properties of the layers may include input size, output size, kernel size, and stride size (e.g., the layer properties 402 as shown in FIG. 4). In aspects, the determine operation 610 determines layers of the models as sharable when values of respective properties among the layers match. Additionally or alternatively, the determine operation 610 determines sharable layers when the values of respective properties among the layers are within a predetermined threshold of similarity.

Generate operation 612 generates a model by merging the models. The merger of the model includes consolidating instances of sharable layers (e.g., the first shared layer 340 as shown in FIG. 3). In aspects, the generate operation 612 consolidates distinct instances of repeating layers of a model into one instantiation. In some other aspects, the generate operation 612 consolidates distinct instances of sharable layers across models into one instantiation.

Train operation 614 trains the merged model including the consolidated instantiation of sharable layers. In aspects, the train operation 614 trains the merged model for multiple epochs and/or until the trained layer attains a level accuracy that is higher than a predetermined threshold.

In aspects, the method 600A simultaneously merges and trains layers across models in an incremental fashion. The method 600A selects the first layer from the sorted list and sharing the first layer across models in the workload in which the first layer appears in the edge server. For merging, weights of a randomly selected model are used for initialization. Each time, a layer is merged, the train operation 614 trains the layer until the merged model satisfies a predetermined threshold of accuracy for each of the individual models. Additionally or alternatively, the train operation 614 ends training when a predetermined time has lapsed (e.g., a predetermined number of epochs of training). The pick-merge-train operation continues until it no longer is practical to merge any more layers without a reduction in accuracy of the merged model. Additionally or alternatively, the pick-merge-train operation ends when the cloud resources allocated to the training by the pipeline operator or an intelligent controller (e.g., the RAN intelligent controller 152 (RIC) in the cloud datacenter 150 as shown in FIG. 1).

Transmit operation 616 transmits the merged model to the edge server. In aspects, the transmit operation 616 cause the edge server to swap one or more models with the merged models. The method 600A ends with the end operation 618.

As should be appreciated, operations 602-618 are described for purposes of illustrating the present methods and systems and are not intended to limit the disclosure to a particular sequence of steps, e.g., steps may be performed in different order, additional steps may be performed, and disclosed steps may be excluded without departing from the present disclosure.

FIG. 6B is an example of a method for iteratively refreshing the models in accordance with aspects of the present disclosure. A general order of the operations for the method 600B is shown in FIG. 6B. Generally, the method 600B begins with start operation 650 and ends with end operation 660. The method 600B may include more or fewer steps or may arrange the order of the steps differently than those shown in FIG. 6B. The method 600B can be executed as a set of computer-executable instructions executed by a computer system and encoded or stored on a computer readable medium. Further, the method 600B can be performed by gates or circuits associated with a processor, an ASIC, an FPGA, a SOC or other hardware device. Hereinafter, the method 600B shall be explained with reference to the systems, components, devices, modules, software, data structures, data characteristic representations, signaling diagrams, methods, etc., described in conjunction with FIGS. 1, 2, 3, 4, 5, 6A, 7, and 8A-B.

Following start operation 650, the method 600 begins with a set of operations as indicated by “A,” which corresponds to a set of operations from the transmit operation 604 to the transmit operation 616 as shown in FIG. 6A. Following the transmit operation 616 that transmits the merged model, receive operation 652 receives information associated with data drift as detected by output of the merged model.

Determine operation 654 determines whether to refresh the merged model. In aspects, the determine operation 654 determines to refresh when a degree of data drift output from one or more sub-models in the merged model exceed a predetermined threshold.

Decision operation 656 decides whether to proceed to detach operation 658 when the determine operation 654 determines the merged model should be refreshed. In aspects, the detach operation 658 splits a model exhibiting the data drift from the merged model. In some other aspects, the detach operation 658 detaches one or more consolidated layers into independent layers among sub-models of the merged model. The detach operation 658 includes retraining of both the detached models. In some other aspects, the detach operation 658 includes retraining of the merged model with the one or more consolidated layers being detached within the merged model.

When the determine operation 654 determines not to refresh the merged model, the decision operation 656 proceeds without performing the detach operation 658 to the end operation 664. The method 600B ends with the end operation 660.

As should be appreciated, operations 650-660 are described for purposes of illustrating the present methods and systems and are not intended to limit the disclosure to a particular sequence of steps, e.g., steps may be performed in different order, additional steps may be performed, and disclosed steps may be excluded without departing from the present disclosure.

FIG. 7 is a block diagram illustrating physical components (e.g., hardware) of a computing device 700 with which aspects of the disclosure may be practiced. The computing device components described below may be suitable for the computing devices described above. In a basic configuration, the computing device 700 may include at least one processing unit 702 and a system memory 704. Depending on the configuration and type of computing device, the system memory 704 may comprise, but is not limited to, volatile storage (e.g., random access memory), non-volatile storage (e.g., read-only memory), flash memory, or any combination of such memories. The system memory 704 may include an operating system 705 and one or more program tools 706 suitable for performing the various aspects disclosed herein such. The operating system 705, for example, may be suitable for controlling the operation of the computing device 700. Furthermore, aspects of the disclosure may be practiced in conjunction with a graphics library, other operating systems, or any other application program and is not limited to any particular application or system. This basic configuration is illustrated in FIG. 7 by those components within a dashed line 708. The computing device 700 may have additional features or functionality. For example, the computing device 700 may also include additional data storage devices (removable and/or non-removable) such as, for example, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIG. 7 by a removable storage device 709 and a non-removable storage device 710.

As stated above, a number of program tools and data files may be stored in the system memory 704. While executing on the at least one processing unit 702, the program tools 706 (e.g., an application 720) may perform processes including, but not limited to, the aspects, as described herein. The application 720 includes a layer status receiver 722, a sharable layer determiner 724, a model generator 726, and a model trainer 728, as described in more detail with regard to FIG. 2. Other program tools that may be used in accordance with aspects of the present disclosure may include electronic mail and contacts applications, word processing applications, spreadsheet applications, database applications, slide presentation applications, drawing or computer-aided application programs, etc.

Furthermore, aspects of the disclosure may be practiced in an electrical circuit comprising discrete electronic elements, packaged or integrated electronic chips containing logic gates, a circuit utilizing a microprocessor, or on a single chip containing electronic elements or microprocessors. For example, aspects of the disclosure may be practiced via a system-on-a-chip (SOC) where each or many of the components illustrated in FIG. 7 may be integrated onto a single integrated circuit. Such an SOC device may include one or more processing units, graphics units, communications units, system virtualization units, and various application functionality all of which are integrated (or “burned”) onto the chip substrate as a single integrated circuit. When operating via an SOC, the functionality, described herein, with respect to the capability of client to switch protocols may be operated via application-specific logic integrated with other components of the computing device 700 on the single integrated circuit (chip). Aspects of the disclosure may also be practiced using other technologies capable of performing logical operations such as, for example, AND, OR, and NOT, including but not limited to mechanical, optical, fluidic, and quantum technologies. In addition, aspects of the disclosure may be practiced within a general-purpose computer or in any other circuits or systems.

The computing device 700 may also have one or more input device(s) 712, such as a keyboard, a mouse, a pen, a sound or voice input device, a touch or swipe input device, etc. The output device(s) 714 such as a display, speakers, a printer, etc. may also be included. The aforementioned devices are examples and others may be used. The computing device 700 may include one or more communication connections 716 allowing communications with other computing devices 750. Examples of the communication connections 716 include, but are not limited to, radio frequency (RF) transmitter, receiver, and/or transceiver circuitry; universal serial bus (USB), parallel, and/or serial ports.

The term computer readable media as used herein may include computer storage media. Computer storage media may include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, or program tools. The system memory 704, the removable storage device 709, and the non-removable storage device 710 are all computer storage media examples (e.g., memory storage). Computer storage media may include RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other article of manufacture which can be used to store information and which can be accessed by the computing device 700. Any such computer storage media may be part of the computing device 700. Computer storage media does not include a carrier wave or other propagated or modulated data signal.

Communication media may be embodied by computer readable instructions, data structures, program tools, or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term “modulated data signal” may describe a signal that has one or more characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), infrared, and other wireless media.

FIGS. 8A and 8B illustrate a computing device or mobile computing device 800, for example, a mobile telephone, a smart phone, wearable computer (such as a smart watch), a tablet computer, a laptop computer, and the like, with which aspects of the disclosure may be practiced. In some aspects, the client utilized by a user (e.g., as an operator of servers in the on-premises edge in FIG. 1) may be a mobile computing device. With reference to FIG. 8A, one aspect of a mobile computing device 800 for implementing the aspects is illustrated. In a basic configuration, the mobile computing device 800 is a handheld computer having both input elements and output elements. The mobile computing device 800 typically includes a display 805 and one or more input buttons 810 that allow the user to enter information into the mobile computing device 800. The display 805 of the mobile computing device 800 may also function as an input device (e.g., a touch screen display). If included as an optional input element, a side input element 815 allows further user input. The side input element 815 may be a rotary switch, a button, or any other type of manual input element. In alternative aspects, mobile computing device 800 may incorporate more or less input elements. For example, the display 805 may not be a touch screen in some aspects. In yet another alternative aspect, the mobile computing device 800 is a portable phone system, such as a cellular phone. The mobile computing device 800 may also include an optional keypad 835. Optional keypad 835 may be a physical keypad or a “soft” keypad generated on the touch screen display. In various aspects, the output elements include the display 805 for showing a graphical user interface (GUI), a visual indicator 820 (e.g., a light emitting diode), and/or an audio transducer 825 (e.g., a speaker). In some aspects, the mobile computing device 800 incorporates a vibration transducer for providing the user with tactile feedback. In yet another aspect, the mobile computing device 800 incorporates input and/or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., a HDMI port) for sending signals to or receiving signals from an external device.

FIG. 8B is a block diagram illustrating the architecture of one aspect of computing device, a server (e.g., the edge servers 116 and the servers 134, and other servers as shown in FIG. 1), a mobile computing device, etc. That is, the mobile computing device 800 can incorporate a system 802 (e.g., a system architecture) to implement some aspects. The system 802 can implemented as a “smart phone” capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, and media clients/players). In some aspects, the system 802 is integrated as a computing device, such as an integrated digital assistant (PDA) and wireless phone.