134 to learn broad elements including, image or speech recognition, general sentence structure, common phrases, vocabulary, natural language structure, and other elements. In some implementations, this first foundational model is trained using self-supervision, or unsupervised training on such datasets.

In some implementations, the second portion of training, including fine-tuning, includes unsupervised, supervised, reinforced, or any other type of training. In some implementations, this second portion of training includes some elements of supervision, including learning techniques incorporating human or machine-generated feedback, undergoing training according to a set of guidelines, or training on a previously labeled set of data, etc. In a non-limiting example associated with reinforcement learning, the outputs of AI model 134 while training can be ranked by a user, according to a variety of factors, including accuracy, helpfulness, veracity, acceptability, or any other metric useful in the fine-tuning portion of training. In this manner, AI model 134 can learn to favor these and any other factors relevant to users when generating a response.

In some implementations, AI model 134 includes one or more pre-trained models, or fine-tuned models. In a non-limiting example, in some implementations, the goal of the “fine-tuning” can be accomplished with a second, or third, or any number of additional models. For example, the outputs of the pre-trained model can be input into a second AI model that has been trained in a similar manner as the “fine-tuned” portion of training above. In such a way, two more AI models can accomplish work similar to one model that has been pre-trained, and then fine-tuned.

Client devices 140A-n can be personal computers (PCs), laptops, notebook computers, mobile phones, smartphones, tablet computers, digital assistants, network-connected televisions (e.g., smart TVs), conference room hardware (e.g., cameras, microphones, speakers, etc.), or any other computing devices. The computer system of FIG. 6 can be an example of a client device. In various embodiments, client devices 140A-n can also be referred to as “user devices.” Client devices 140A-n can run an operating system (OS) that manages hardware and software of the client devices. Client devices 140A-n can further include a web browser, application, or other software for displaying virtual conference user interfaces and interacting with servers 120-130. Client devices 140A-n can be used by users such as virtual conference participants. In general, and as described herein, functions described in embodiments as being performed by a virtual conferencing platform and/or server devices 120-130 can also or alternatively be performed on client devices 140A-n in other embodiments. In addition, the functionality attributed to a particular component can be performed by different or multiple components operating together.

Client device 140A can include capture sensor 142, which can be one or more cameras, one or more microphones, other types of sensors, or combinations thereof. Capture sensor 142 can be used for capturing video and audio streams of one or more virtual conference participants. Client device 140A can further include stream modification service 132 for modifying a media stream(s) of capture sensor 142 to include semantic metadata as described with reference to server 130. Stream modification service 132 can use on-device machine learning models or other techniques for identifying participants in the media stream and generating corresponding semantic metadata. In an embodiment, stream modification service 132 can generate semantic metadata on a regular basis, such as on a frame-by-frame basis or every n frames. In an embodiment, stream modification service 132 can generate semantic metadata once, such as at the beginning of a stream. In an embodiment, stream modification service 132 can generate semantic metadata in response to one or more events, such as in response to an update request from a server or client device, in response to a participant entering or leaving the field of view of a media stream, in response to a participant starting or stopping speaking, or similar.

Client device 140A can present GUI 146, which can be a GUI of a virtual conferencing application, a web browser, or similar. GUI 146 can be used for displaying video streams of one or more virtual conference participants. GUI 146 can include a layout for displaying video streams such as a grid layout, a row layout, a column layout, a presenter layout, or similar. Client device 140A can further include visual representation modification service 148, for modifying a visual representation of a media stream based on semantic metadata embedded in the media stream. Visual representation modification service 148 can translate, rotate, crop, scale, split, or otherwise transform or modify visual representations of media streams based on the semantic metadata. Visual representation modification service 148 can perform modifications on a regular basis (e.g., a frame-by-frame basis), when updated semantic metadata is received, when a participant requests an update, or when similar cues are received. Example visual representation modifications are further described with reference to FIGS. 3A-B.

In various embodiments, client devices can include subsets of the components depicted with reference to client device 140A. For example, client device 140B can be a conference room camera and can thus include capture sensor 152 and stream modification service 132. In another example, client device 140n can be a conference room TV and can thus include GUI 156 and visual representation modification service 158.

FIG. 2 is a block diagram of an example encoder-decoder architecture 200 for a virtual conferencing platform that modifies visual representations of media streams using embedded semantic metadata, in accordance with an embodiment. Encoder-decoder architecture 200 (also referred to as “system” or “codec” herein) can include encoder 210 and decoder 220 operating on video frames 202, semantic metadata 204, and media stream 206. In various embodiments, system 200 can include more or fewer components in different configurations than those depicted in FIG. 2. For example, system 100 can include additional processing for audio data.

Encoder 210 can include custom data processing service 212 for adding custom data processing operations in the encoding of video frames 202 into media stream 206. Custom data processing service 212 can be configured or otherwise programmed to embed semantic metadata 204 in media stream 206 along with video frames 202. For example, semantic metadata can be attached to a respective video frame and transferred over a network together with the video frame to provide synchronized delivery of the metadata and video frame. In an embodiment, semantic metadata 204 can be embedded into RTP (RFC-3550) payloads of video frames 202 that constitute media stream 206. In an embodiment, encoder 210 can be included in stream modification service 132 of server 130 and/or client devices 140A-n of FIG. 1.

Similarly, decoder 220 can include custom data processing service 222 for adding custom data processing operations in the decoding of video frames 202 from media stream 206. Custom data processing service 222 can be configured or otherwise programmed to extract semantic metadata 204 from media stream 206 along with video frames 202. In an embodiment, decoder 220 can be included in stream modification service 132 of server 130 and/or visual representation modification services 148-158 of FIG. 1. For example, stream modification service 132 of server 130 can receive a media stream from client device 140B, decode the media stream using decoder 220, add metadata (or additional metadata) using encoder 210, and provide the media stream to client device 140A.

In an embodiment, custom data processing services 212-222 correspond to an application programming interface (API), such as the WebRTC Insertable Streams API. For example, custom data processing services 212-222 can be provided by a web browser and configured or otherwise programmed to embed or extract semantic metadata from a WebRTC MediaStreamTrack using the WebRTC Insertable Streams API.

FIGS. 3A-B illustrate example graphical user interfaces 300A-B presenting media stream visual representations before and after modification of the visual representations, in accordance with an embodiment. FIGS. 3A-B depict GUI frame 310 before modification. GUI frame 310 includes media stream visual representation 320, which can correspond to a media stream including embedded semantic metadata. GUI frame 310 can be one of multiple GUI frames (e.g., in a grid) each including a respective media stream visual representation. Media stream visual representation 320 includes video conference participants 322A-B, which can each correspond to semantic metadata representing their respective positions, sizes, bounding boxes, etc. within the video frame.

In an embodiment, as depicted in FIG. 3A, a visual representation of a media stream can be modified by splitting it into two or more visual representations occupying the same space in the graphical user interface. For example, media stream visual representation 320 can be split into media stream visual representations 330A-B, which occupy the same GUI frame 310. GUI frame 310 can include one or more dividers (e.g., divider 312) to separate the visual representations. As described herein, media stream visual representations 330A-B can be derived from media stream visual representation 320 by splitting, scaling, translating, cropping, rotating, or otherwise transforming or modifying video frames of the media stream to generate video sub-frames associated with media stream visual representations 330A-B.

In an embodiment, as depicted in FIG. 3B, a visual representation of a media stream can be modified by splitting it into two or more visual representations each occupying separate spaces in the graphical user interface. For example, media stream visual representation 320 can be split into media stream visual representations 350A-B, which each occupy respective GUI frames 340A-B that are equivalent or analogous to GUI frame 310. GUI frames 340A-B can be positioned in a row, column, grid, or other layout along with other GUI frames for other media streams.

In various embodiments, combinations of modifications and presentations of media stream visual representations depicted in FIGS. 3A-B can be used. For example, media stream visual representations 350A-B can be derived from media stream visual representations 330A-B in a multi-step splitting process of an embodiment. In various embodiments, other types of modifications and presentations of media stream visual representations not depicted in FIGS. 3A-B can be used.

FIG. 4 is a sequence diagram of an example interaction 400 between client devices 140A-B and servers 120-130 for modifying visual representations of media streams in virtual conferencing platforms using embedded semantic metadata, in accordance with an embodiment. In some embodiments, operations depicted in FIG. 4 could occur in a different order or be performed by different components than depicted. Various embodiments can include additional operations or components not depicted in FIG. 4 or a subset of operations or components depicted in FIG. 4. The operations depicted in FIG. 4 can correspond to different communication sessions or different timing intervals. For example, some operations can proceed in immediate succession or can be part of a single communication session, while other operations can be spread out over time or can be part of different communication sessions.

At operation 402, client device 140B obtains video frames from one or more capture sensors, such as capture sensor 152 of FIG. 1. At operation 404, client device 140B identifies semantic information in the obtained video frames, e.g., by using an AI model or other technique provided by stream modification service 132. At operation 406, client device 140B embeds the semantic information as metadata in a media stream, e.g., by using stream modification service 132 and/or encoder 210.

At operation 408, client device 140B provides the media stream with embedded semantic metadata to media streaming service 122 of server 120. In an embodiment, server 120 provides the media stream to client device 140A at operation 409 without additional server-side processing of the embedded metadata (e.g., operations proceed at operation 420). In an embodiment, server 120 provides the media stream to stream modification service 132 of server 130 at operation 410 for additional metadata processing.

At operation 412, server 130 identifies semantic information in the obtained video frames, e.g., by using an AI model or other technique provided by stream modification service 132. At operation 414, server 130 embeds the semantic information as additional metadata in the media stream, e.g., by using stream modification service 132 and/or encoder 210. Prior to embedding the semantic information as metadata in the media stream, server 130 can first decode the media stream using decoder 220 and re-encode the media stream after adding the additional metadata. In an embodiment, server 130 can additionally or alternatively modify the semantic metadata embedded at operation 406 by client device 140B. Prior to modifying the metadata, server 130 can first decode the media stream using decoder 220 and re-encode the media stream after modifying the metadata.

At operation 416, server 130 provides the media stream with additional embedded semantic metadata to server 120. At operation 418, server 120 provides the media stream to client device 140A.

At operation 420, client device 140A extracts the semantic metadata from the media stream, e.g., by using visual representation modification service 148 and/or decoder 220. At operation 422, client device 140A modifies video frames of the media stream using, e.g., visual representation modification service 148. Modifications can include cropping, translating, rotating, etc. as previously described. Modifications can be based on the metadata and/or characteristics of client device 140A, such as screen size, active layout, number of media streams to be presented, etc. At operation 424, client device 140A presents video frames of the media stream, e.g., using visual representation techniques described with reference to FIG. 3A-B.

In an embodiment, operations 402-424 can be repeated on a periodic basis, such as frame-by-frame or at a regular multi-frame interval. In an embodiment, operations 402-424 can occur once at the beginning of a stream. In an embodiment, operations 402-424 can occur in response to a trigger, such as a user interaction on one of client devices 140A-B, a participant entering or leaving the field of view of a capture device, a participant starting or stopping speaking, or similar.

FIGS. 5A-B are a flow diagram of an example method 500 for modifying visual representations of media streams in virtual conferencing platforms using embedded semantic metadata, in accordance with an embodiment. Method 500 can be performed by processing logic that can comprise hardware (e.g., circuitry, dedicated logic, etc.), computer-readable instructions such as software or firmware (e.g., run on a general-purpose computing system or a dedicated machine), or a combination thereof. For instance, an example system can include a memory and a processing device coupled to the memory device to perform operations comprising the blocks of method 500. Method 500 can also be associated with a set of instructions stored on a non-transitory computer-readable medium (e.g., magnetic or optical disk, etc.). The instructions, when executed by a processing device, can cause the processing device to perform operations comprising the blocks of method 500. In at least one embodiment, method 500 is performed by one or more of servers 120-130 or client devices 140A-n of FIG. 1, or components thereof. In at least one embodiment, method 500 is performed by computing system 600 of FIG. 6. In some embodiments, blocks depicted in FIG. 5 could be performed simultaneously or in a different order than depicted. Various embodiments can include additional blocks not depicted in FIGS. 5A-B or a subset of blocks depicted in FIGS. 5A-B. For example, blocks depicted with a dashed outline (e.g., blocks 512-518) can be absent in an embodiment.

At block 502, processing logic presents a virtual meeting user interface (UI) of a virtual meeting during a virtual meeting between a plurality of participants, wherein the virtual meeting UI comprises a plurality of regions each corresponding to a video stream provided by one of a plurality of client devices of the plurality of participants, the plurality of regions comprising a first region corresponding to one or more first video streams provided to a first client device of the plurality of client devices. A region of the virtual meeting UI can correspond to one or more of GUI frames 310 or 340A-B of FIG. 3, and the video streams can be associated with the corresponding media stream representations. The plurality of participants can include participants 322A-B. The client device can correspond to one of client devices 140A-n of FIG. 1.

At block 504, the processing logic receives, at the first client device, the one or more first video streams each comprising respective semantic metadata, wherein respective semantic metadata of a first video stream of the one or more first video streams indicates a spatial location in the first video stream of a first participant of the plurality of participants. The semantic metadata can be metadata 204 of FIG. 2. As previously described, the metadata can indicate various characteristics of video conference participants, such as spatial location in the video frame, spatial size in the video stream, bounding box, direction of motion, whether a participant is speaking, etc.

At block 506, the processing logic identifies one or more content presentation layout characteristics of the first client device. In an embodiment, the one or more content presentation layout characteristics of the first client device comprises at least one of: a screen size of the first client device, an aspect ratio of the first client device, a layout grid size of the first client device, or a video stream count.

At block 508, the processing logic causes a visual representation of the first video stream to be modified in the first region based at least on the spatial location in the first video stream of the first participant and the one or more content presentation layout characteristics of the first client device. In an embodiment, causing the visual representation of the first video stream to be modified in the first region comprises cropping, translating, rotating, scaling, splitting, or otherwise transforming the visual representation.

At block 510, the processing logic presents the virtual meeting UI comprising the first region with the modified visual representation of the first video stream on the first client device (e.g., as depicted in FIGS. 3A-B).

In an embodiment, the respective semantic metadata of the first video stream further indicates a second spatial location in the first video stream of a second participant of the plurality of participants. Causing the visual representation of the first video stream to be modified in the first region can comprise splitting a frame of the first video stream into a first video sub-frame corresponding to the first participant and a second video sub-frame corresponding to the second participant. Presenting the virtual meeting UI comprising the first region with the modified visual representation of the first video stream on the first client device can comprise presenting the first video sub-frame and the second video sub-frame on the first client device in the first region (e.g., as depicted in FIG. 3A).

At block 512, the processing logic obtains, at the first client device, a second video stream from a video sensor of the first client device. The video sensor can be capture sensor 142 or 152, for example.

At block 514, the processing logic identifies a second spatial location in the second video stream of a second participant of the plurality of participants. In an embodiment, identifying the second spatial location in the second video stream of the second participant comprises providing the second video stream as input to an artificial intelligence (AI) model trained to identify video conference participants and respective spatial locations in video streams, and obtaining an output of the AI model comprising the second spatial location. The AI model can be AI model 134, for example.

At block 516, the processing logic modifies the second video stream to comprise second semantic metadata, wherein the second semantic metadata indicates the second spatial location in the second video stream of the second participant.