106 can be trained and/or retrained to recognize references to the previously unrecognized content in the audio data of the virtual meeting and then generate a textual representation of the previously unrecognized content.

For example, training data can be generated based on the previously unrecognized content, where one or more possible pronunciations (e.g., from pronunciation generation subsystem 114) can be used as the input data and the previously unrecognized content can be used as the target output data. In some embodiments, the one or more possible pronunciations are converted to a latent space (e.g., embedding space, vector space, etc.) of the machine learning model and the converted representations of the possible pronunciations are used as the training input data.

In some embodiments, speech recognition subsystem 104 has an associated lexicon and/or vocabulary of words that can be recognized by speech recognition subsystem 104. To modify speech recognition subsystem 104 to recognize the previously unrecognized content, the previously unrecognized content can be added to the lexicon and/or vocabulary.

Virtual meeting system 102 can search (e.g., parse, read, analyze, scan, etc.) one or more content items associated with a virtual meeting to find previously unrecognized content to be used to modify speech recognition subsystem 104. The searched content items can include names of participants of the virtual meeting (e.g., from a meeting invitation, provided by the participant as they join the meeting, etc.) and/or documents associated with the virtual meeting (e.g., documents attached to a meeting invitation).

In some embodiments, the content items associated with the virtual meeting can include text extracted from content shared during the virtual meeting. For example, a participant can share a presentation during the virtual meeting (e.g., can share their device screen where they have a presentation open during a video conference meeting). Virtual meeting system 102 can extract text from the images shared by the participant (e.g., frames of the presentation, frames of the participant's device screen, etc.), for example, using optical character recognition (OCR) software (e.g., via optical character recognition subsystem 108).

In some embodiments, the content items associated with the virtual meeting can include text exchanged between one or more participants of the virtual meeting (e.g., in-meeting chat messages).

In some embodiments, the content items associated with the virtual meeting can include text extracted from one or more documents associated with an organization of a participant of the virtual meeting. For example, during a team virtual meeting (e.g., audio call, video call, etc.), a speech recognition system can be used that has previously been modified to recognize and correctly transcribe speech audio related to content found in documents of the organization (e.g., company-specific jargon). The modified speech recognition system can be used during multiple virtual meetings of the organization to increase the quality of all text generated from speech audio related to those meetings.

As mentioned above, optical character recognition subsystem 108 can extract text from images shared during a virtual meeting and add the extracted text to the content items virtual meeting system 102 searches for previously unrecognized content. In some embodiments, optical character recognition subsystem 108 can include one or more machine learning models trained to convert images to text. In some embodiments, optical character recognition subsystem 108 can include other software configured to recognize text in an image and extract that text from the image. For example, optical character recognition subsystem 108 can be configured to extract text from images of handwritten notes (e.g., on a whiteboard, on paper, etc.), digital copies of documents (e.g., a scanned image of a physical document), screenshots from a device of a user, and/or the like.

Unrecognized content detection subsystem 110 can be configured to evaluate the content items found by virtual meeting system 102 and select one or more content items that have not previously been recognized by speech recognition subsystem 104 (e.g., previously unrecognized content). In some embodiments, unrecognized content detection subsystem 110 compares the text associated with a content item against the lexicon and/or vocabulary of speech recognition subsystem 104 to determine if speech recognition subsystem 104 can recognize speech audio corresponding to the text. In some embodiments, unrecognized content detection subsystem 110 provides the text associated with a content item to one or more machine learning models to determine if speech recognition subsystem 104 can recognize speech audio corresponding to the text. In some embodiments, unrecognized content detection subsystem 110 can evaluate each content item found by virtual meeting system 102 and determine whether it should be included in the set of previously unrecognized content or not.

In some embodiments, unrecognized content detection subsystem 110 can add metadata to a previously unrecognized content item to aid in speech recognition of the previously unrecognized content item. For example, unrecognized content detection subsystem 110 can include information related to how the previously unrecognized content item was used in the content (e.g., part of speech, etc.). Speech recognition subsystem 104 can use the metadata during text generation to provide a more accurate textual representation of the audio. For example, if the context where a previously unrecognized content item was used suggests that the previously unrecognized content item is a noun (e.g., a product name, etc.), speech recognition subsystem 104 can avoid generating text associated with the previously unrecognized content item when a verb, adjective, adverb, etc. is expected in the generated text.

Pronunciation generation subsystem 114 can receive the previously unrecognized content from unrecognized content detection subsystem 110 and generate one or more possible pronunciations for the content. In some embodiments, pronunciation generation subsystem 114 can split the previously unrecognized content into one or more graphemes (e.g., orthographic symbols), and then combine corresponding phonemes (e.g., units of the sound system of a language) to generate one or more possible pronunciations for the previously unrecognized content. In some embodiments, pronunciation generation subsystem 114 can include one or more machine learning models trained to perform grapheme-to-phoneme (G2P) conversion tasks.

The possible pronunciations generated by pronunciation generation subsystem 114 can be used to modify speech recognition subsystem 104. For example, as described above, the possible pronunciations can be used to create training data that can be used to train a new machine learning model or to retrain an existing machine learning model.

Virtual meeting system 102 can be connected to datastore 118 via network 116. Datastore 118 can be a persistent storage that is capable of storing content items, machine learning models, previously unrecognized content, possible pronunciations, training data, and/or the like. Datastore 118 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage-based disks, tapes or hard drives, network attached storage (NAS), storage area network (SAN), and so forth. In some embodiments, datastore 118 may be a network-attached file server. In some embodiments, datastore 118 may be some other type of persistent storage such as an object-oriented database, a relational database, and so forth. In some embodiments, datastore 118 may be hosted on or may be a component of virtual meeting system 102. In some embodiments, datastore 118 may be provided by a third-party service such as a cloud platform provider.

System 100 can include one or more virtual meeting participant devices (e.g., virtual meeting participant device 120, virtual meeting participant device 122, etc.). Each virtual meeting participant device can correspond to a user and can enable the user to connect to and interface with virtual meeting system 102. Each virtual meeting participant device can include an audiovisual component that can generate audio and video data to be streamed to the virtual meeting system 102. The audiovisual component can include a device (e.g., a microphone) to capture an audio signal representing speech of a user and generate audio data (e.g., an audio file or audio stream) based on the captured audio signal. The audiovisual component can include another device (e.g., a speaker) to output audio data to a user associated with a virtual meeting participant device. In some implementations, the audiovisual component includes an image capture device (e.g., a camera) to capture images and generate video data (e.g., a video stream) of the captured images.

In some embodiments, at least one of the virtual meeting participant devices is associated with a physical conference or meeting room and can include or be coupled to a media system that can include one or more display devices, one or more speakers, and one or more cameras. The display device can be, for example, a smart display or a non-smart display (e.g., a display that is not itself configured to connect to the network). Users that are physically present in the room can use the media system rather than their own devices (e.g., one or more of devices 120 and 122) to participate in the virtual meeting, which can include other remote users. For example, the users in the room that participate in the virtual meeting can control the display device to show a slide presentation or watch slide presentations of other participants. Sound and/or camera control can similarly be performed. The virtual meeting participant devices associated with a physical conference or meeting room can generate audio and video data to be streamed to the virtual meeting system 102 (e.g., using one or more microphones, speakers, and cameras).

In some embodiments, virtual meeting participant device 120 and/or virtual meeting participant device 122 can include one or more processing devices, volatile and non-volatile memory, data storage, one or more input/output peripherals such as network interfaces. FIG. 6 illustrates an example architecture of computing devices. In some embodiments, virtual meeting participant device 120 and/or virtual meeting participant device 122 may be singular devices such as smartphones, tablets, laptops, desktops, workstations, edge devices, embedded devices, servers, network appliances, security appliances, etc. In some embodiments, virtual meeting participant device 120 and/or virtual meeting participant device 122 comprise multiple devices of similar or varying architecture such as computing clusters, data centers, co-located servers, enterprise networks, geographically disparate devices connected via virtual private networks (VPNs), etc. In some embodiments, virtual meeting participant device 120 and/or virtual meeting participant device 122 comprise hardware devices such as those just described, virtual resources such as virtual machines (VMs) and containerized applications, or a combination of hardware and virtual resources.

In some embodiments, virtual meeting participant device 120 and/or virtual meeting participant device 122 can share local content (e.g., content local to the respective device) during a virtual meeting facilitated by virtual meeting system 102. In some embodiments, virtual meeting system 102 can extract text from the shared content and use the extracted text to modify speech recognition subsystem 104 to perform contextual speech recognition for the virtual meeting.

In implementations of the disclosure, a “user” can be represented as a single individual. However, other implementations of the disclosure encompass a “user” being an entity controlled by a set of users or an organization and/or an automated source such as a system or a platform. In situations in which the systems discussed here collect personal information about users, or can make use of personal information, the users can be provided with an opportunity to control whether virtual meeting system 102, speech recognition subsystem 104, optical character recognition subsystem 108, and/or unrecognized content detection subsystem 110 collect user information (e.g., information about a user’s social network, social actions or activities, profession, a user’s preferences, or a user’s current location), or to control whether and/or how to receive content from virtual meeting system 102 and/or speech recognition subsystem 104 that can be more relevant to the user. In addition, certain data can be treated in one or more ways before it is stored or used, so that personally identifiable information is removed. For example, a user’s identity can be treated so that no personally identifiable information can be determined for the user, or a user’s geographic location can be generalized where location information is obtained (such as to a city, ZIP code, or state level), so that a particular location of a user cannot be determined. Thus, the user can have control over how information is collected about the user and used by virtual meeting system 102, speech recognition subsystem 104, optical character recognition subsystem 108, and/or unrecognized content detection subsystem 110.

FIG. 2 depicts an example virtual meeting 202, in accordance with at least one embodiment. Virtual meeting 202 can include image content 204, one or more virtual meeting participants (e.g., virtual meeting participant 206, virtual meeting participant 208, etc.), context-enhanced live captions 210, and virtual meeting chat 212. In some embodiments, virtual meeting 202 can be facilitated by virtual meeting system 102 of FIG. 1.

One or more participants (e.g., virtual meeting participant 206 and virtual meeting participant 208) can communicate within virtual meeting 202. The communication can be audio, video, text (e.g., virtual meeting chat 212), and/or a combination thereof. For example, a video recorded by a camera of virtual meeting participant 206 can be included in virtual meeting 202 so virtual meeting participant 208 can see virtual meeting participant 206, and vice versa.

Context-enhanced live captions 210 can be generated during virtual meeting 202 and can be made visible to the virtual meeting participants. In some embodiments, context-enhanced live captions 210 can be generated by speech recognition subsystem 104 of FIG. 1. Context-enhanced live captions 210 can include a textual representation of the speech audio of virtual meeting 202. The speech recognition system (e.g., speech recognition subsystem 104 of FIG. 1) generating context-enhanced live captions 210 can be modified before and/or during virtual meeting 202 based on content associated with virtual meeting 202 to improve the accuracy of context-enhanced live captions 210. In some embodiments, the speech recognition system can be modified after virtual meeting 202 to improve the accuracy of a transcript and/or AI-generated summary generated based on audio data of virtual meeting 202.

In some embodiments, one or more participants can share content from their device within virtual meeting 202. Image content 204 can include the content shared by a participant during the virtual meeting. For example, virtual meeting participant 206 can share content from their device, and a graphic representation of that content can be shared in virtual meeting 202 with the other participants. In some embodiments, text can be extracted from image content 204, and the text can be used to provide additional context to (e.g., to modify) the speech recognition system (e.g., speech recognition subsystem 104 of FIG. 1) that is being used to generate context-enhanced live captions 210.

In some embodiments, one or more participants can communicate using text within virtual meeting 202 via virtual meeting chat 212. The text exchanged between participants can be added to the content items that are evaluated to determine whether they are previously unrecognized content items. For example, participants may send text messages via virtual meeting chat 212 that include the name of a company product. The name of the company product can be identified as “previously unrecognized content” and can be used to modify the speech recognition system. Then, when the name of the company product is spoken by a participant in virtual meeting 202, the speech recognition system can accurately generate text based on the spoken name of the company product.

In some embodiments, if a participant shares a document via virtual meeting chat 212 (e.g., by sending a link to a shared document, by attaching a file to the text message, etc.), the virtual meeting system can parse the contents of the shared document to find previously unrecognized content that can be used to modify the speech recognition system.

In some embodiments, the speech recognition system used to generate context-enhanced live captions 210 of virtual meeting 202 can have been modified during a previous virtual meeting. For example, a speech recognition system can be modified based on context of a first virtual meeting (e.g., a first virtual meeting within an organization). The modified speech recognition system can be used during a second virtual meeting (e.g., a second virtual meeting within the same organization). The modified speech recognition system can be further modified during the second virtual meeting. The speech recognition system can detect and correctly transcribe (e.g., convert from audio to text) content items that were first detected as previously unrecognized content items in relation to the first virtual meeting.

FIG. 3 depicts a flow diagram of an example method for contextual speech recognition of virtual meetings, in accordance with at least one embodiment. FIG. 4 depicts a flow diagram for modifying a speech recognition system used for contextual speech recognition of virtual meetings, in accordance with at least one embodiment. FIG. 5 depicts a flow diagram for modifying a speech recognition system used for contextual speech recognition of virtual meetings, in accordance with at least one embodiment. Method 300, method 400, and/or method 500 can be performed by processing logic that can include hardware (circuitry, dedicated logic, etc.), software (e.g., instructions run on a processing device), or a combination thereof. In at least one implementation, some or all of the operations of method 300, method 400, and/or method 500 can be performed by one or more components of system 100 for contextual speech recognition of virtual meetings of FIG. 1. For example, in some embodiments, method 300 can be performed by virtual meeting system 102 of FIG. 1. In some embodiments, method 400 and/or method 500 can be performed by machine learning model training engine 112 and/or pronunciation generation subsystem 114 of FIG. 1.

For simplicity of explanation, the methods of this disclosure are depicted and described as a series of acts. However, acts in accordance with this disclosure can occur in various orders and/or concurrently, and with other acts not presented and described herein. Furthermore, not all illustrated acts may be required to implement the methods in accordance with the disclosed subject matter. In addition, those skilled in the art will understand and appreciate that the methods could alternatively be represented as a series of interrelated states e.g., via a state diagram. Additionally, it should be appreciated that the methods disclosed in this specification are capable of being stored on an article of manufacture to facilitate transporting and transferring such methods to computing devices. The term “article of manufacture,” as used herein, is intended to encompass a computer program accessible from any computer-readable device or storage media.

Referring to FIG. 3, at block 302, processing logic can receive audio data of a virtual meeting. The virtual meeting can be an audio-only meeting, a video meeting, or the like. At block 304, processing logic can identify, within a plurality of content items related to the virtual meeting, content not previously recognized by a speech recognition system designated to convert the audio data of the virtual meeting into text. The plurality of content items related to the virtual meeting can include text extracted from one or more sources. For example, processing logic can search (e.g., parse, read, analyze, scan, etc.) one or more content items associated with a virtual meeting. The searched content items can include names of participants of the virtual meeting (e.g., from a meeting invitation, provided by the participant as they join the meeting, etc.) and/or documents associated with the virtual meeting (e.g., documents attached to a meeting invitation).

In some embodiments, the plurality of content items related to the virtual meeting can include text extracted from content shared during the virtual meeting. For example, a participant can share a presentation during the virtual meeting (e.g., can share their device screen where they have a presentation open during a video conference meeting). Processing logic can extract text from the images shared by the participant (e.g., frames of the presentation, frames of the participant's device screen, etc.) for example, using optical character recognition (OCR) software. The extracted text can be added to the plurality of content items related to the virtual meeting.

In some embodiments, the OCR software can include one or more machine learning models trained to convert images to text. In some embodiments, the OCR software can include other software configured to recognize text in an image and extract that text from the image. For example, the OCR software can be configured to extract text from images of handwritten notes (e.g., on a whiteboard, on paper, etc.), digital copies of documents (e.g., a scanned image of a physical document), screenshots from a device of a user, and/or the like.

In some embodiments, the plurality of content items related to the virtual meeting can include text exchanged between one or more participants of the virtual meeting (e.g., in-meeting chat messages).

In some embodiments, the plurality of content items related to the virtual meeting can include text extracted from one or more documents associated with an organization of a participant of the virtual meeting. For example, processing logic can search (e.g., parse, read, analyze, scan, etc.) a collection of documents of an organization to find previously unrecognized content (e.g., company-specific jargon) that may be discussed during a team virtual meeting.

In some embodiments, to identify content not previously recognized by the speech recognition system designated to convert the audio data of the virtual meeting into text, text of one or more content items of the plurality of content items can be compared against a lexicon and/or vocabulary of the speech recognition system to determine if the speech recognition system can recognize speech audio corresponding to the text. In some embodiments, the text associated with a content item can be provided to one or more machine learning models to determine if the speech recognition system can recognize speech audio corresponding to the text.

If the text of the content item is not in the lexicon and/or vocabulary of the speech recognition system and/or if the one or more machine learning models determines that the speech recognition system cannot recognize speech audio corresponding to the text, the content item and/or the text of the content item can be identified as previously unrecognized content.

At block 306, processing logic can cause the speech recognition system to be modified based on the previously unrecognized content. In some embodiments, causing the speech recognition system to be modified based on the previously unrecognized content includes generating one or more possible pronunciations for the previously unrecognized content and adding the possible pronunciations to a lexicon of the speech recognition system. In some embodiments, the unrecognized content can be added to the lexicon of the speech recognition system. In some embodiments, causing the speech recognition to be modified based on the previously unrecognized content includes providing a representation of the previously unrecognized content as input to a machine learning model of the speech recognition system. In some embodiments, causing the speech recognition system to be modified based on the previously unrecognized content includes performing one or more steps of method 400 and/or method 500, as discussed below.

At block 308, processing logic can cause the audio data of the virtual meeting to be converted into the text using the modified speech recognition system, wherein the text comprises at least part of the previously unrecognized content. In some embodiments, the text can be live captions visible during the virtual meeting, a transcription of the virtual meeting, or a summary of the virtual meeting generated using one or more machine learning models.

Referring to FIG. 4, at block 402, to cause the speech recognition system to be modified based on the previously unrecognized content, processing logic can generate one or more possible pronunciations for the previously unrecognized content. At block 404, processing logic can generate training data comprising the one or more possible pronunciations as inputs and the previously unrecognized content as target output. At block 406, processing logic can retrain a first machine learning model of the one or more machine learning models using the generated training data. For example, machine learning model training engine 112 of FIG. 1 can use the training data based on the previously unrecognized content to retrain at least one of machine learning model(s) 106 of speech recognition subsystem 104 of FIG. 1.

Referring to FIG. 5, at block 502, to cause the speech recognition system to be modified based on the previously unrecognized content, processing logic can generate one or more possible pronunciations for the previously unrecognized content. At block 504, processing logic can generate training data comprising the one or more possible pronunciations as inputs and the previously unrecognized content as target output. At block 506, processing logic can train a new machine learning model using the generated training data to recognize the previously unrecognized content. In some embodiments, the new machine learning model is trained to recognize a representation of the previously unrecognized content. For example, the machine learning model may be trained to recognize a representation of a pronunciation of the previously unrecognized content. In some embodiments, the representation includes a vector-space (e.g., a latent-space, an embedding space) representation of a possible pronunciation of the previously unrecognized content. At block 508, processing logic can add the new machine learning model to the one or more machine learning models of the speech recognition system. For example, machine learning model training engine 112 of FIG. 1 can use the training data based on the previously unrecognized content to train a new machine learning model, which can be added to machine learning model(s) 106 of speech recognition subsystem 104 of FIG. 1.

FIG. 6 is a block diagram illustrating an exemplary computer system, in accordance with at least one embodiment of the present disclosure. The computer system 600 can correspond to virtual meeting system 102, described with respect to FIG. 1. The computer system 600 can also correspond to virtual meeting participant device 120 and/or virtual meeting participant device 122, described with respect to FIG. 1. Computer system 600 can operate in the capacity of a server or an endpoint machine in an endpoint-server network environment, or as a peer machine in a peer-to-peer (or distributed) network environment. The machine can be a television, a personal computer (PC), a tablet PC, a set-top box (STB), a Personal Digital Assistant (PDA), a cellular telephone, a web appliance, a server, a network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while only a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein.

The example computer system 600 includes a processing device (processor) 602, a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), double data rate (DDR SDRAM), or DRAM (RDRAM), etc.), a static memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), and a data storage device 616, which communicate with each other via a bus 628.

Processor (processing device) 602 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like, and may include processing logic