122, and/or any combination thereof.

FIG. 3 illustrates an example system 300 that can be used to generate replacement frames for a media stream and to playback media content to a user. System 300 can include any number of processor-based devices, including but not limited to one or more cameras and/or media systems, such as media system 104, discussed above. As illustrated, system 300 includes cameras 302, 304, various media systems, including various playback devices (e.g., displays or TVs) 304, 306, and audio playback device 308. It is understood that a greater (or fewer) number of devices can be used, and that processing tasks, e.g., to generate replacement frames, may be performed by a single device, distributed amongst two or more devices in system 300, and/or performed by one or more remote systems (such as servers 126).

The devices of system 300 can be configured to communicate with various remote systems, such as content server 120 and/or system server 126, via one or more computer networks such as the Internet. In operation, the devices of system 300 can be used to receive and playback media content of different types, including video streams for entertainment content (e.g., movies, televisions shows, etc.), and/or to facilitate monitoring and surveillance of the surrounding environment. For example, cameras 302, 304 may be configured to monitor (record) events occurring within field of view 303, for example to provide a homeowner (user) with notifications or other alerts when specific events are detected. By way of example, a user of system 300 may be able to play recorded surveillance footage of events occurring within field-of-view 303 on any of playback devices 304, 306, 308.

As discussed above, playback of media content (e.g., surveillance footage, movies, TV shows, etc.) may be interrupted due to device and/or network malfunctions, such as increased latency or jitter resulting in the loss of video frames in a media stream. The resulting discontinuities can degrade the user's playback experience and it would therefore be helpful to generate replacement frames to fill these discontinuities. Aspects of the disclosed technology provide solutions for generating replacement frames that can be used to in place of the dropped frames, to provide a more continuous streaming experience to the user. In some implementations, the replacement frames can depict events that are indistinguishable from those events that would have been depicted in the dropped frames.

In some approaches, replacement frames can be created using a generative machine-learning approach, such as by using a Generative Adversarial Network (GAN), Variational Auto Encoders (VAE), and/or Diffusion-based models etc. It is understood that other types of generative ML models may be used, without departing from the scope of the disclosed technology.

Media Playback Examples

By way of example, the devices of system 300 may be used to playback streaming media content, for example, that is received from a remote content server, such as content server 120. If device or network latency errors occur, the resulting packet loss may prevent all frames of a media stream from reaching the playback device, such as TV 304. In such cases, portions of the stream may be provided to a generative ML model and used to construct replacement data for the media stream, including replacement video frames and/or audio data, so that the stream discontinuities are unnoticed by the user. The replacement frames and/or audio data can be reconstructed from any information available about the media stream, including but not limited to metadata describing the stream, including but not limited to information about the content origin, title, type, episode, genre, score, and/or video and/or audio data collected by additional or other devices, for example, from different vantage points. and the like. Replacement frame construction can also be based on the content of the frames and audio information preceding or following the discontinuity or temporal location of the dropped frames.

In some approaches, discontinuities may be identified, for example, based on frame number metadata indicating an ordering of frames for a particular media stream. In such instances, gaps in the received frames, as identified from missing or incomplete frame numbering, can signify how many replacement frames need to be generated-or conversely, a length of replacement frame content needed to fill the discontinuity. In other approaches, discontinuities may be automatically identified through analysis of the content of one or more received frames, such as using one or more ML models trained for discontinuity detection, and/or using a machine vision approach.

Security System Examples In some implementations, one or more replacement frames may be generated to fill discontinuities in video streams originating from one or more devices in system 300, such as camera 302 and/or 304, which may be used for security monitoring purposes. In such instances, frame numbering metadata may be unavailable for use in identifying media stream discontinuities, and other types of metadata may be more salient. For example, cameras 304 and 304 can both be configured to identify and log events observed in their respective field-of-view, and in some instances to cross-reference or corroborate events jointly observed in overlapping field-of-view 303. Over time, each device 304, 304 may train a camera-specific ML model tuned to identify observed events at the premise of system 300. Such camera specific models can be used to perform event detection, identify discontinuities in video streams (e.g., due to network latency), and to generate replacement frames to fill identified continuities. Replacement frame generation can be based on event metadata, as well as frames captured before and/or after an identified discontinuity, video and/or audio data collected by additional or other devices, for example, from different vantage points. For example, if camera 302 observes an approach of visitor 305, but device/network issues result in the loss of subsequent frames, then event detection metadata (e.g., a metadata tag indicating “package delivery”) may be used to generate one or more replacement frames to fill the discontinuity, for example, by showing visitor 305 leaving a package and walking away.

In some instances, data collected by one camera may be used to train and improve the generation of replacement frames for another device. For example, if camera 304 records an event, e.g., a package delivery by visitor 305, but the event is not entirely captured by camera 302, then the replacement frames generated to fill the discontinuity in the video stream from camera 302 may be based on one or more frames collected by camera 304. Further details regarding the generation of replacement frames, including the training of ML models for frame generation are discussed with respect to FIGS. 4-6, below.

FIG. 4 is a diagram illustrating an example system 400 that can be used to generate replacement frames. System 400 can be configured to receive different types of information about a given video (media) stream, including but not limited to video data 402, that includes image frames of the video stream, audio data 404 that can include audio information corresponding with video data 402, and metadata 406, that can include any information about the video data 402 and/or audio data 404. For example, metadata 406 may include event information for one or more events identified in video data 402, media content information (e.g., title, episode, frame numbering, etc.), and/or other types of information about the video stream. Video and/or audio data collected by additional or other devices, for example, from different vantage points can also be used as additional signals to generate replacement frames.

All or a portion of video data 402, audio data 404 and/or metadata 406 can then be used to identify one or more discontinuities in the media stream (block 408). For media streams that include entertainment content (e.g., movies, TV shows) frame index (or frame numbering) information may be available (in metadata 406) and used to identify discontinuities, e.g., by identifying which frames have not been received. In some instances, frame numbering information may also be available for other types of video streams, such as those coming from a security system (e.g., system 300) as discussed in relation to FIG. 3.

In other aspects, discontinuity identification 408 may be based on an analysis of video data 402 and/or audio data 404, for example, by using ML or computer-vision based approaches to determine where, and how many, frames have been dropped. In such cases, data for a media stream (including one or more of video data 402, audio data 404 and/or metadata 406) can be provided to an ML model that is trained to identify discontinuities. In other approaches, machine vision techniques may be used to identify object discontinuities, such as when a person represented in one frame jumps to an improbable location in the subsequent frame, suggesting that one or more intervening frames may be missing, i.e., a discontinuity. Audio data 404 may also be used to identify discontinuities, such as when there are interruptions or unexpected breaks. Identified discontinuities may be referenced using a frame number and/or a time stamp indicating an insertion point in the media stream/content where replacement frames are to be inserted.

Identified discontinuities (block 408) can be passed, along with video data 402, audio data 404 and/or metadata 406 to a generative ML model (block 410) and used to generate replacement frames (block 412). The replacement frames can then be added to the media content based on the temporal and/or numerical reference for a corresponding discontinuity. The replacement frames can therefore be used to fill the discontinuity and provide a completed media stream for playback by the user (block).

In some instances, creation of replacement frames (block 412) and or playback of generated replacement frames (414) may be restricted due to user privacy policies or settings. For example, users may have the ability to opt-out of having content generated that includes their likeness, including visual or audible reconstructions of how the user may look or sound. In some implementations such restrictions may be applied to devices owned or controlled by the user, and in other implementations the restrictions may apply more globally, such as to other devices that are not necessarily owned and/or controlled by the user.

FIG. 5 is a diagram illustrating an example system 500 that can be used to train a generative ML model for replacement frame generation. Training of an ML model for use in generating replacement frames can be performed on a set of training data that includes known media content 502, such as media streams for which a complete set of video frame data and audio data exists. In some implementations portions of video and/or audio data may be removed from a media stream (block 504) and then the resulting media stream, which contains one or more discontinuities, can be provided to a generative ML model (block 506). The generative ML model can the produce replacement frames (block 508) based on the received content, and the replacement frames can be compared to the removed frames (block 510) to determine an accuracy of the generative ML model. That is, a loss function for feedback/training to the generative ML model can be based on a difference of the removed frames (known) and the replacement frames that are produced by the ML model (predicted).

In some examples, the generative ML model may produce replacement frames (block 508) with the addition of other types of information, including but not limited to metadata describing the stream, audio information about the stream, and the like. It is understood that the generative ML model may also be configured to generate other types of data, such as audio data, that may be missing, using a similar training process. In some implementations, such as security system scenarios, model training may be performed with the benefit of other types of information, such as video and/or audio data collected by additional or other devices, for example, from different vantage points. Further details regarding ML training are described in further detail with respect to FIG. 6, below.

FIG. 6 is a diagram illustrating an example system 600 that can be used to train a generative ML model (e.g., generative ML model 608) for replacement frame generation using camera-specific ML models. Camera-specific models can be models trained or optimized for use with a specific device (camera), for example, that is deployed in a static or semi-static location. Camera-specific models can be trained (or optimized) for specific device specifications (e.g., camera resolution, frame capture rate, and/or image adjustment parameters, etc.) and/or characteristics and/or for features of a particular environment in which they are deployed, such as light levels, and field-of-view, etc. In some instances, camera-specific models can include ML models trained to perform object detection, including the recognition of inanimate objects (e.g., cars, packages, etc.), animate objects (e.g., pets, people) or specifically pre-identified behaviors (e.g., package delivery, visits by a neighbor or family member, arrival of caretakers or service providers, etc.).

Training of a given camera-specific model can begin with the acquisition of audio and/or video data for a given environment, such as by one or more of cameras 302, 304 in system 300, discussed above. Video and/or audio feeds for a specific device can be analyzed over time, for example to identify and store patterns of observed objects, events, and/or behaviors. For example, package delivery events can follow similar a similar pattern, with predictable sounds and/or observed objects (e.g., delivery personnel, packages, etc.) across multiple video frames, and lasting for predictable time durations. Over time, camera-specific models can become attuned to an associated environment and highly accurate at event identification as well as replacement frame generation the respective environment.

For newly received video streams, discontinuities can be detected (block 606), and passed to a generative ML model (block 608). In operation, the generative ML model 608 can use historic information about patterns (objects, events, behaviors) observed by a corresponding camera device, as well as information about identified discontinuities 606 to generate one or more replacement frames (block 610). By way of example, the replacement frames 610 may be used to represent animate and/or inanimate objects, as well as behaviors by the represented objects, based on the learned context in which the device (camera) is deployed. In some instances, user feedback may be used to determine if replacement frame outputs are accurate or acceptable to the user (e.g., a homeowner or operator of a security system), the user feedback can be used to further train/tune the generative ML model (block 612). As such, camera-specific models can improve accuracy of event identification, discontinuity identification and replacement frame generation over time, and with continued user feedback.

FIG. 7 is a diagram illustrating steps of a process 700 for generating replacement frames.

In step 710, process 700 includes receiving a set of vide frames. As discussed above, the video frames may be received as part of a video stream, such as during the receipt of media content from a remote content server (e.g., server 120). The received video frames may also be received from an image capture device, such as a camera that is deployed from monitoring in a home or business setting.

In step 720, process 700 includes identifying a discontinuity in the set of video frames. Discontinuities can result from dropped or corrupted frames in the set of received video frames. In some approaches, discontinuities can be identified based on video frame metadata, such as frame numbers. For example, non-consecutive frame numbering can indicate frames that have been lost/dropped, and used to determine a length of generated replacement content that is needed.

In step 730, process 700 includes generating one or more replacement frames associated with the discontinuity based on at least one video frame selected from among the set of video frames.

In step 740, process 700 includes providing one or more replacement frames to a user. As discussed above, user feedback may be used to further train a generative ML model, e.g., to improve the accuracy of generated frame content.

FIG. 8 is a diagram illustrating an example of a neural network architecture 800 that can be used to implement some or all of the neural networks described herein. The neural network architecture 800 can include an input layer 820 can be configured to receive and process data to generate one or more outputs. The neural network architecture 800 also includes hidden layers 822a, 822b, through 822n. The hidden layers 822a, 822b, through 822n include “n” number of hidden layers, where “n” is an integer greater than or equal to one. The number of hidden layers can be made to include as many layers as needed for the given application. The neural network architecture 800 further includes an output layer 821 that provides an output resulting from the processing performed by the hidden layers 822a, 822b, through 822n.

The neural network architecture 800 is a multi-layer neural network of interconnected nodes. Each node can represent a piece of information. Information associated with the nodes is shared among the different layers and each layer retains information as information is processed. In some cases, the neural network architecture 800 can include a feed-forward network, in which case there are no feedback connections where outputs of the network are fed back into itself. In some cases, the neural network architecture 800 can include a recurrent neural network, which can have loops that allow information to be carried across nodes while reading in input.

Information can be exchanged between nodes through node-to-node interconnections between the various layers. Nodes of the input layer 820 can activate a set of nodes in the first hidden layer 822a. For example, as shown, each of the input nodes of the input layer 820 is connected to each of the nodes of the first hidden layer 822a. The nodes of the first hidden layer 822a can transform the information of each input node by applying activation functions to the input node information. The information derived from the transformation can then be passed to and can activate the nodes of the next hidden layer 822b, which can perform their own designated functions. Example functions include convolutional, up-sampling, data transformation, and/or any other suitable functions. The output of the hidden layer 822b can then activate nodes of the next hidden layer, and so on. The output of the last hidden layer 822n can activate one or more nodes of the output layer 821, at which an output is provided. In some cases, while nodes in the neural network architecture 800 are shown as having multiple output lines, a node can have a single output and all lines shown as being output from a node represent the same output value.

In some cases, each node or interconnection between nodes can have a weight that is a set of parameters derived from the training of the neural network architecture 800. Once the neural network architecture 800 is trained, it can be referred to as a trained neural network, which can be used to generate one or more outputs. For example, an interconnection between nodes can represent a piece of information learned about the interconnected nodes. The interconnection can have a tunable numeric weight that can be tuned (e.g., based on a training dataset), allowing the neural network architecture 800 to be adaptive to inputs and able to learn as more and more data is processed.

The neural network architecture 800 is pre-trained to process the features from the data in the input layer 820 using the different hidden layers 822a, 822b, through 822n in order to provide the output through the output layer 821.

In some cases, the neural network architecture 800 can adjust the weights of the nodes using a training process called backpropagation. A backpropagation process can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter/weight update is performed for one training iteration. The process can be repeated for a certain number of iterations for each set of training data until the neural network architecture 800 is trained well enough so that the weights of the layers are accurately tuned.

To perform training, a loss function can be used to analyze an error in the output. Any suitable loss function definition can be used, such as a Cross-Entropy loss. Another example of a loss function includes the mean squared error (MSE), defined as E_total=Σ(1/2(target−output){circumflex over ( )}2). The loss can be set to be equal to the value of E_total.

The loss (or error) will be high for the initial training data since the actual values will be much different than the predicted output. The goal of training is to minimize the amount of loss so that the predicted output is the same as the training output. The neural network architecture 800 can perform a backward pass by determining which inputs (weights) most contributed to the loss of the network, and can adjust the weights so that the loss decreases and is eventually minimized.

The neural network architecture 800 can include any suitable deep network. One example includes a Convolutional Neural Network (CNN), which includes an input layer and an output layer, with multiple hidden layers between the input and out layers. The hidden layers of a CNN include a series of convolutional, nonlinear, pooling (for downsampling), and fully connected layers. The neural network architecture 800 can include any other deep network other than a CNN, such as an autoencoder, Deep Belief Nets (DBNs), Recurrent Neural Networks (RNNs), among others.

As understood by those of skill in the art, machine-learning based techniques can vary depending on the desired implementation. For example, machine-learning schemes can utilize one or more of the following, alone or in combination: hidden Markov models; RNNs; CNNs; deep learning; Bayesian symbolic methods; Generative Adversarial Networks (GANs); support vector machines; image registration methods; and applicable rule-based systems. Where regression algorithms are used, they may include but are not limited to: a Stochastic Gradient Descent Regressor, a Passive Aggressive Regressor, etc.

Machine learning classification models can also be based on clustering algorithms (e.g., a Mini-batch K-means clustering algorithm), a recommendation algorithm (e.g., a Minwise Hashing algorithm, or Euclidean Locality-Sensitive Hashing (LSH) algorithm), and/or an anomaly detection algorithm, such as a local outlier factor. Additionally, machine-learning models can employ a dimensionality reduction approach, such as, one or more of: a Mini-batch Dictionary Learning algorithm, an incremental Principal Component Analysis (PCA) algorithm, a Latent Dirichlet Allocation algorithm, and/or a Mini-batch K-means algorithm, etc.

Example Computer System

Various aspects and examples may be implemented, for example, using one or more well-known computer systems, such as computer system 900 shown in FIG. 9. For example, the media device 106 may be implemented using combinations or sub-combinations of computer system 900. Also or alternatively, one or more computer systems 900 may be used, for example, to implement any of the aspects and examples discussed herein, as well as combinations and sub-combinations thereof.

Computer system 900 may include one or more processors (also called central processing units, or CPUs), such as a processor 904. Processor 904 may be connected to a communication infrastructure or bus 906.

Computer system 900 may also include user input/output device(s) 903, such as monitors, keyboards, pointing devices, etc., which may communicate with communication infrastructure 906 through user input/output interface(s) 902.

One or more of processors 904 may be a graphics processing unit (GPU). In some examples, a GPU may be a processor that is a specialized electronic circuit designed to process mathematically intensive applications. The GPU may have a parallel structure that is efficient for parallel processing of large blocks of data, such as mathematically intensive data common to computer graphics applications, images, videos, etc.

Computer system 900 may also include a main or primary memory 908, such as random access memory (RAM). Main memory 908 may include one or more levels of cache. Main memory 908 may have stored therein control logic (e.g., computer software) and/or data.

Computer system 900 may also include one or more secondary storage devices or memory 910. Secondary memory 910 may include, for example, a hard disk drive 912 and/or a removable storage device or drive 914. Removable storage drive 914 may be a floppy disk drive, a magnetic tape drive, a compact disk drive, an optical storage device, tape backup device, and/or any other storage device/drive.

Removable storage drive 914 may interact with a removable storage unit 918. Removable storage unit 918 may include a computer usable or readable storage device having stored thereon computer software (control logic) and/or data. Removable storage unit 918 may be a floppy disk, magnetic tape, compact disk, DVD, optical storage disk, and/any other computer data storage device. Removable storage drive 914 may read from and/or write to removable storage unit 918.

Secondary memory 910 may include other means, devices, components, instrumentalities or other approaches for allowing computer programs and/or other instructions and/or data to be accessed by computer system 900. Such means, devices, components, instrumentalities or other approaches may include, for example, a removable storage unit 922 and an interface 920. Examples of the removable storage unit 922 and the interface 920 may include a program cartridge and cartridge interface (such as that found in video game devices), a removable memory chip (such as an EPROM or PROM) and associated socket, a memory stick and USB or other port, a memory card and associated memory card slot, and/or any other removable storage unit and associated interface.

Computer system 900 may include a communication or network interface 924. Communication interface 924 may enable computer system 900 to communicate and interact with any combination of external devices, external networks, external entities, etc. (individually and collectively referenced by reference number 928). For example, communication interface 924 may allow computer system xx00 to communicate with external or remote devices 928 over communications path 926, which may be wired and/or wireless (or a combination thereof), and which may include any combination of LANs, WANs, the Internet, etc. Control logic and/or data may be transmitted to and from computer system 900 via communication path 926.

Computer system 900 may also be any of a personal digital assistant (PDA), desktop workstation, laptop or notebook computer, netbook, tablet, smart phone, smart watch or other wearable, appliance, part of the Internet-of-Things, and/or embedded system, to name a few non-limiting examples, or any combination thereof.

Computer system 900 may be a client or server, accessing or hosting any applications and/or data through any delivery paradigm, including but not limited to remote or distributed cloud computing solutions; local or on-premises software (“on-premise” cloud-based solutions); “as a service” models (e.g., content as a service (CaaS), digital content as a service (DCaaS), software as a service (Saas), managed software as a service (MSaaS), platform as a service (PaaS), desktop as a service (DaaS), framework as a service (FaaS), backend as a service (BaaS), mobile backend as a service (MBaaS), infrastructure as a service (IaaS), etc.); and/or a hybrid model including any combination of the foregoing examples or other services or delivery paradigms.

Any applicable data structures, file formats, and schemas in computer system 900 may be derived from standards including but not limited to JavaScript Object Notation (JSON), Extensible Markup Language (XML), Yet Another Markup Language (YAML), Extensible Hypertext Markup Language (XHTML), Wireless Markup Language (WML), MessagePack, XML User Interface Language (XUL), or any other functionally similar representations alone or in combination. Alternatively, proprietary data structures, formats or schemas may be used, either exclusively or in combination with known or open standards.

In some examples, a tangible, non-transitory apparatus or article of manufacture comprising a tangible, non-transitory computer useable or readable medium having control logic (software) stored thereon may also be referred to herein as a computer program product or program storage device. This includes, but is not limited to, computer system 900, main memory 908, secondary memory 910, and removable storage units 918 and 922, as well as tangible articles of manufacture embodying any combination of the foregoing. Such control logic, when executed by one or more data processing devices (such as computer system