114 may include instructions that, when executed by a processor(s), cause the processor(s) to, without limitation, receive input data representative of user inputs to the one or more input device(s) 112, transmit the input data to the server(s) 104, in response receive a video stream from the client-game interface manager 124 using the communication interface 132A, and cause display of the video on the display 110. In other words, the application 114 may operate as a facilitator for enabling streaming of video, which may include the frame(s) 140 of gameplay or other content associated with the application on the client device(s) 102.

In one or more embodiments, the client device(s) 102 may use the application 114 to display the gameplay video or other video content on the display 110. In examples where the display data is received by the client device(s) 102, the system 100 may be part of a game streaming system, such as the content streaming system 800 of FIG. 8, described in more detail below.

The display 110 may include any type of display capable of displaying the video (e.g., a light-emitting diode display (LED), an organic LED display (OLED), a liquid crystal display (LCD), an active matrix OLED display (AMOLED), a quantum dot display (QDD), a plasma display, an LED/LCD display, and/or another type of display). In some examples, the display 110 may include more than one display (e.g., a dual-monitor display for computer gaming, a first display for configuring a game and a virtual reality display for playing the game, etc.). In some examples, the display is a touch-screen display, such as a touch-screen of a smart phone, tablet computer, laptop computer, or the like, where the touch-screen includes at least one of the input device(s) 112 of the client device(s) 102.

The input device(s) 112 may include any type of devices that are capable of providing user inputs to the game. The input device(s) may include a keyboard, a mouse, a touch-screen display, a controller(s), a remote(s), a headset (e.g., sensors of a virtual reality headset), and/or other types of input devices.

The communication interfaces such as the communication interface 132A and the communication interface 132B (referred to collectively or individually herein as “communication interface(s) 132”) may include one or more components and features for communicating across one or more networks, such as the network(s) 108. The communication interface(s) 132 may be configured to communicate via any number of the network(s) 108, described herein. For example, to communicate in the system 100 of FIG. 1, the client device(s) 102 may use an Ethernet or Wi-Fi connection through a router to access the Internet in order to communicate with the server(s) 104, and/or with other client device(s) 102.

Now referring to FIG. 2, FIG. 2 is a flow diagram showing an event identification process 200, in accordance with at least some embodiments of the present disclosure. The event identification process 200 may be implemented on the system 100 of FIG. 1. For example, the event identifier 118 may use the process 200 to identify one or more events. The event identification process 200 may be associated with one or more frames 202 of gameplay data, which may correspond to the frame(s) 140 of FIG. 1. The frame(s) 202 may include a display or presentation of one or more text elements, such as a text element 204A, a text element 204B, and a text element 204C. The text elements may include the text analyzed by the event identifier 118 using the parser 116 to detect events and may be displayed or presented in textual form, as shown, or using different representations. For example, one or more text elements may be displayed in a GUI and/or frame generated by the game instance 130 (as shown). However, one or more text elements may additionally or alternatively be included in one or more additional GUIs, files, or data structures, such as a log file related to the gameplay (e.g., generated by the game instance 130), without necessarily being displayed or represented in-game. Text elements may include any text information generated by or using a game instance. For example text elements may include natural language strings, sentences, numerals, timestamps, variable names, values and/or other text corresponding to an internal representation of game state, etc.

Text information corresponding to one or more of the text elements may be transmitted in one or more messages 208 (e.g., from the communication interface 132B of the game instance 130 to the communication interface 132A of the client device 102). A message 208 may contain text information and any of a number of other parameters, fields, or values. For example, a message 208 may include a parameter or field named “event” associated with a text element and expressed as a natural language string(s) 210, as shown. Although the string 210 is in natural language form, in other examples, the string may not be in natural language form. FIG. 2 also shows an “api” field which may include an identifier of the API associated with the message 208, a “ts” field which may include a timestamp(s) of the message 208 and/or the string(s) 210, a “pid” field which may include a game, program and/or process identifier associated with the game instance 130.

The natural language string 210 may be transmitted as part of the one or more messages 208 to a component configured to map one or more parts of the natural language string 210 to one or more event templates. For example, the one or more messages 208 may be received by the client device 102 and mapped to an event template by the event identifier 118 using the parser 116. The natural language string 210 may include at least a portion of game text 212, presented and/or generated by the game instance 130 during the gameplay of a game (e.g., one or more timestamped strings of text). One or more portions of the game text 212 may correspond to one or more conditions being satisfied in a game session. For example, the game text 212 includes the text “[20:25] Player eliminated a boss named Don using the Hammer,” which may be descriptive of the one or more conditions (e.g., player eliminated, by a boss, etc.). In one or more embodiments, the conditions may be evaluated by game code and relate to gameplay and/or in-game events (e.g., victory, points scored, milestones, eliminations, item acquisition, etc.). For example, the game instance 130 may generate a message, log entry (e.g., corresponding to the text element 204A), and/or notification (e.g., corresponding to the text element 204B or the text element 204C) when a player of the game wins a match, scores points, or when any of a number of actions, interactions, or combinations thereof occur during gameplay. In some examples, the notification may be displayed to one or more players in the game during gameplay (e.g., upon corresponding conditions being satisfied).

The event identifier 118 may use the parser 116 to map the game text 212 to one or more event templates. In one or more embodiments, the game text 212 is mapped to an event template based at least on matching particular words or phrases associated with an event template to the game text 212. In some examples, the game text 212 may be mapped to a template using natural language processing such as named entity recognition (“NER”), regular expression analysis, or any of a number of other NLP techniques. An event template may be associated with a particular game, a particular version of a game, a series of games, or a genre or type of games. In some examples, a generic or base event template may be used for all games if a more specific template is not available for a particular game or is otherwise not suitable for use. An event template may be located on one or more client device(s) 102 or may be located on the data store(s) 134 with associated configurations 136 (e.g., corresponding to configurations files described herein) and retrieved through the network(s) 108. In one or more embodiments, a value of the “pid” field may be used to determine which event template(s) are available for mapping the game text 212 associated with the string 210.

The event identifier 118 may use the parser 116 with an event template to analyze the game text 212 received in the message(s) 208. For example, the parser 116 may select and use an event template comprising or associated with the indicators “Player eliminated,” “named,” and “using the” to parse the game text 212. By way of example and not limitation, the event template may take the form “Player was eliminated [by a (n)<Param1:<player|npc|boss> [named Param2:<name>]>]][with a (n)<Param3:weapon|item|custom>].” As can be seen, the event template includes data that indicates various terms or phrases used to map text to the event template, as well as parameter fields and data that indicates or identifies potential values for those parameter fields. As another example, the parser 116 may match the phrases “Player's side scored” and “points” from the game text 212 to an event template having those same or similar phrases (or data that otherwise indicates or identifies those terms or phrases). Here the event template may take the form “Player's side scored [a (n)<Param1:name>][for <Param2:<amount> points]”. However, event templates described herein may take a variety of potential forms. In one or more embodiments, more than one event template may be mapped to a same string or portion of the game text 212. In one or more embodiments, each event template may correspond to one or more events.

As indicated above, the event template(s) may include or indicate one or more parameter fields and/or values thereof associated with a potential event. For example, the text, “Player eliminated a boss named Don using the Hammer,” of the game text 212 may be mapped to an event template having the parameter fields: Param1, Param2, and Param3. The event identifier 118 may use the parser 116 to extract values 216 of parameter fields from the game text 212 using the corresponding event template. As an example, the parser 116 may determine that Param1 of the event template can be mapped to the descriptor, “boss,” Param2 can be mapped to the name “Don,” and that Param3 can be mapped to the item, “Hammer.” As another example, the text, “Blue Team Flag Captured,” of the game text 212 may be mapped to an event template comprising, Param1 and Param2. The event template may be selected based at least on the event identifier 118 matching the phrase “Team Flag” to the event template. Using the event template, Param1 may be mapped to team signifier “Blue,” while Param2 may be mapped to the flag action, “Captured.”

In one or more embodiments, the event identifier 118 may attempt to map one or more of the parameter fields of an event template to the text based at least on at least some of the text being mapped to the event template (e.g., using the indicators). Also, as with the indicators, the values of the parameter fields need not be explicit in or exactly match the definition of an event template, allowing for a wider variety of terms, phrases, values, and sentence structures to be mapped to corresponding events. In one or more embodiments, event templates may identify categories or types of terms, phrases, values, and sentence structures used for mapping, such as and without limitation parts of speech identifiers. Also in one or more embodiments, text in an event template may be mapped to the game text 212 based at least on determining the game text 212 includes one or more synonyms or variations to the text rather than exact matching.

In one or more embodiments, the event identifier 118 may identify an event based at least on mapping a string(s) of text to an event template(s) associated with the event. In one or more embodiment, in order for an event template to be used to identify an event, it must both be mapped to one or more strings of the game text 212, and have at least one of (e.g., all) of its parameter fields mapped to the one or more strings. For example, “Player eliminated a boss named Don using the Hammer,” may be mapped to an event template. If the event identifier 108 determines it is unable to map one or more of the parameter fields of the event template to the text (e.g., Param1), the event identifier 108 may not use the event template to identify a corresponding event. In other examples, one or more event templates may allow for partial element mappings, optional element mappings, and/or variations on element mappings.

In one or more embodiments, each event template may correspond to one or more events or portions thereof. For example, if the event identifier 118 is able to successfully map “Player eliminated a boss named Don using the Hammer,” to the event template (e.g., including the parameter fields), the event identifier 118 may identify an event corresponding to the event template. Additionally or alternatively, one or more events may correspond to and/or be identified using multiple event templates. For example, the event identifier 118 may identify an event based at least on mapping one or more portions of the game text 212 to one or more event templates (e.g., each obtained over multiple messages, in multiple strings of text, and/or corresponding to multiple timestamps). In one or more embodiments, the event identifier 118 generates data representing and/or indicating an identified or detected event(s), which may be referred to as event data.

Event data for an event may comprise, by way of example and not limitation, data representing, indicating, or otherwise associated with the extracted parameter(s) 216, one or more portions of the game text 212, timestamps, an event identifier, one or more event template identifiers used to identify the event, and/or any other information that may be associated with an event. For example, the event data may include game information, game session identifiers, player information, or any of a number of other information. The generated event data may be organized and stored in one or more event files 220 and/or other a data structures. For example, the data structure(s) may be associated with a highlight generation system 300, in accordance with at least some embodiments of the present disclosure. In one or more embodiments, event data for one or more events may be stored in XML, JSON, txt, or any other file or storage format.

In one or more embodiments, the event identifier 118 assigns and/or associates a timestamp with one or more of the events, which may be included in event data. In one or more embodiments, the event identifier 118 uses the parser 116 to extract data corresponding to the timestamp(s) from the game text 212 and/or one or more other portions of the message(s) 208, such as the “ts” field. For example, the game text 212 is shown as including timestamps generated by the game instance 130, which may be used as or otherwise correspond to associated event timestamps. In one or more embodiments, a timestamp of an event may correspond to an order or time in which a message 208 used to identify an event is received or sent (e.g., e.g., based at least on a system time of a system, device, or component that received or sent the message 208).

Now referring to FIG. 3, FIG. 3 is an example system diagram of a highlight generation system 300, in accordance with at least some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Further, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be carried out by hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory.

The highlight generation system 300 may include, among other things, the client device(s) 102, the server(s) 104, and the network(s) 108. The highlight generation system 300 (and the components and/or features thereof) may be implemented using one or more computing devices, such as the computing device 900 of FIG. 9, described in more detail below.

The client device(s) 102 may include the display 110, the application(s) 114, an events plugin 310, the highlight generator 120, one or more other plugins 314, and one or more data store(s) 320. The server(s) 104 may include the game instance 130, the client interface 126, a game API 302, a video capturer 304, and/or one or more data store(s) 324.

The application(s) 114 (e.g., a client application of the server(s) 104) may operate as a facilitator for enabling gameplay of the game instance 130. For example, the application(s) 114 may, without limitation, display a game stream received via the network(s) 108 from the client interface 126. In some examples, as described herein, the application(s) 114 may include some of the features or functionality of the highlight generator 120, such as those related to generating highlights, recordings, or snapshots, for programming recording settings, etc. In addition, in some examples, the application(s) 114 may cause generation of the highlights, storage of the highlights, analysis of the game stream for determining when highlights should be generated, sharing of the highlights, etc. In some embodiments, the client device(s) 102 may include any number of application(s) 114 where features and functionality of the highlight generation system 300 are distributed.

In one or more embodiments, the client device(s) 102 includes the events plugin 310. The events plugin 310 may be configured to identify and extract event information from one or more messages associated with a game session. For example, the events plugin may include the parser 116 and/or the event identifier 118 of FIG. 1. The events plugin 310 may receive from the application 114 a message comprising at least one string of text (e.g., the message 208), transmitted over the network(s) 108, from then client interface 126 of the server(s) 104. In some embodiments, the events plugin 310 may additionally comprise the game data analyzer 122 of FIG. 1. In one or more embodiments at least some of the features or functionality of the events plugin 310 may be incorporated into the application(s) 114.

The other plugin(s) 314, may refer to any of a number of other plugins on the client device(s) 102. In one or more embodiments, the other plugins 314 may be used in place of, or in combination with, the events plugin 310. In one or more embodiments, the other plugins 314 may be configured to perform methods of detecting game events that are alternative to the NLP methods performed by the events plugin 310. For example, the events plugin 310 may be used to perform NLP to detect events in a game session, while the other plugin(s) 314 may simultaneously, or in parallel, perform OCR, computer vision, object detections, and/or non-NLP based techniques to detect events within that same game session. In one or more embodiments, the event identifier 118 may use the other plugin(s) 314 to detect and/or identify events. For example, the event identifier 118 may supplement NLP (e.g., and/or non-computer vision) event identification with non-NLP (e.g., and/or computer vision) methods and/or identify one or more events without NLP or without using the parser 116. In one or more embodiments the game data analyzer 112 may employ computer vision functionality to identify significant events from events identified by the event identifier 118.

While the events plugin 310 and the other plugins 314 are shown as being on the client device 102, in one or more embodiments, at least some of the features or functionality of one or more of those components may be implemented on the one or more servers 104 (e.g., the event identifier 118, the parser 116, the game data analyzer 112, computer vision event identification, etc.). Similarly, at least some of the features or functionality of the highlight generator 120 may be implemented on the server(s) 104.

In one or more embodiments, the server(s) 104 may render the game instance 130 of a game, and communicate the game stream of the game to the application(s) 114 over the client interface 126. The game API 302 may receive API calls from the game instance 130. In some examples, the game API 302 may facilitate communication between the game instance 130 and the client interface 126. In some embodiments, the game API may be included as part of a game interface, such as the game interface 128 of FIG. 1. The client interface 126 may interpret, translate, and/or pass API calls made by the game instance 130 to the game API 302 and provide corresponding messages to the application 114, such as the message 208. In one or more embodiments, the client interface 126 may use a real time streaming protocol (RTSP) to facilitate one or more of the communication between the application(s) 114 and the client interface 126 (e.g., the message(s) 208, a game stream, a highlight stream, etc.).

The video capturer 304 may comprise one or more components for capturing game video data. In at least one embodiment, the video capturer 304 may include at least some of the features or functionality of the render capture component 814 of FIG. 8. In one or more embodiments, the video capturer 304 may interpret requests to capture highlights of the game instance 130 on the server(s) 104. In one or more embodiments, a request from one or more client device(s) 102 may be received by the video capturer 304, instructing the video capturer 304 to record video data associated with a game highlight. For example, using the network(s) 108, the highlight generator 120 may request one or more frames of video data corresponding to a highlight occurring in a game session of the game instance 130. In one or more embodiments, a highlight may be based at least on one or more events identified using the event identifier 118 (e.g., timestamps and event parameters thereof). The video capturer 304, may identify and retrieve the one or more frames of video data from one or more data Store(s) 324. In one or more embodiments, the video capturer 304 may record data in the data store(s) 324 using the buffer 336 to record data, prior to receiving a request associated with generating a highlight. In one or more embodiments, the events plugin 310 may include at least some of the features and functionality of the video capturer 304. In such embodiments, the events plugin 310 may use the data store(s) 320 and/or a buffer 322 similar to how the video capturer 304 uses the data store(s) 324 and/or the buffer 336.

Now referring to FIG. 4, FIG. 4 is an example screenshot from a graphical user interface 400 for generating, displaying, and/or recording game highlights, in accordance with at least some embodiments of the present disclosure. The graphical user interface 400 may include a highlight selection control region 402, a highlight label 404, a highlight replay region 410, a highlight timeline 420, and/or a highlight information display region 430. In some examples, the graphical user interface 400 may include controls for generating and recording gameplay highlight video data.

The highlight selection control region 402 may comprise an indication and associated controls associated with any of a number of highlights, such as highlights 408A, 408B, 408C, and 408E. The highlight selection control region 402 may display indications of highlights corresponding to one or more games and/or game sessions. In some examples, the highlight selection control region 402 may provide controls for the selection of one or more highlights to be displayed in other portions of the graphical user interface 400 such as the highlight replay region 410.

The highlight replay region 410 may display a highlight replay 412 that was generated from one or more events of a game session using the highlight generator 120. The highlight may be displayed in the highlight replay region 410 as an image, a video, or any of a number of other representations. The highlight replay region 410 may also comprise controls and selection elements for manipulating a highlight in the highlight replay region 410. For example, the controls may allow a user to start/pause the playback of a highlight video clip or change the playback speed or resolution of a highlight video.

The highlight timeline 420 may comprise an indication or graphic elements that represent events identified by the event identifier 118 and/or the game data analyzer 112 in a game session. The highlight timeline 420 may include highlight or event indications, such as a highlight indication 422 corresponding to the highlight active in the highlight replay region 410. In some examples, the highlight timeline 420 may comprise controls for selecting and displaying other highlights using the corresponding highlight indications.

The highlight information display region 430 may include graphic and textual elements that are associated with game highlights and/or corresponding events. For example, the highlight information display region 430 may provide elements that include information associated with a game session. For example, game scores or outcomes from a particular game session may be displayed. In one or more embodiments, the highlight information may correspond to one or more events, event types, parameter fields and/or values associated with or extracted from text by the event identifier 118. Similarly, the highlight label 404 and/or other information displayed in the graphical user interface 400 may correspond to one or more events, event types, parameter fields and/or values associated with or extracted from text by the event identifier 118. As an example, “scored” may be associated with an event type of an event used to generate the highlight 408C.

Now referring to FIG. 5, each block of method 500, and other methods described herein, comprise a computing process that may be performed using any combination of hardware, firmware, and/or software. For instance, various functions may be carried out by a processor executing instructions stored in memory. The methods may also be embodied as computer-usable instructions stored on computer storage media. The methods may be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. In addition, the methods are described, by way of example, with respect to the system 100 of FIG. 1. However, the methods may additionally or alternatively be executed by any one system, or any combination of systems, including, but not limited to, those described herein.

FIG. 5 is a flow diagram showing an example method for determining an event from at least one string of text in natural language-processed form, in accordance with at least some embodiments of the present disclosure. The method 500, at block B502, includes receiving a message comprising at least one string of text in natural language form. For example, the client device 102 may receive, via an API, the at least one message 208 comprising the natural language string(s) 210 in natural language form, wherein the natural language string(s) 210 may have been generated by the game instance 130 based at least on one or more conditions being satisfied in a game session, the one or more conditions corresponding to gameplay of the game.

The method 500, at block B504, includes determining an event corresponding to gameplay based at least on identifying one or more indicators of the event in the at least one string of text. For example, the event identifier 118 may determine an event corresponding to the gameplay based at least on identifying one or more indicators of the event in the at least one string of text using natural language processing.

The method 500, at block B506, includes transmitting data that causes generation of one or more highlights of a game session based at least on the event. For example, the event identifier 118 may transmit event data to the highlight generator 120 and/or the game data analyzer 122 to cause generation of one or more highlights of the game session based at least on the event.

FIG. 6 is a flow diagram showing an example method 600 for determining an event from at least one string of text using an event template, in accordance with at least some embodiments of the present disclosure. The method 600, at block B602, includes receiving at least one message comprising at least one string of text generated by a game based at least on one or more condition being satisfied in a game session. For example, the client device 102 may receive at least one message comprising at least one string of text generated by the game instance 130 of a game based at least on one or more conditions being satisfied in a game session, the one or more conditions corresponding to gameplay of the game.

The method 600, at block B604, includes mapping the at least one string of text to an event template. For example, the event identifier 118 may use the parser 116 to map the at least one string of text to an event template that defines one or more parameters of an event corresponding to the gameplay.

The method 600, at block B606, includes extracting one or more parameters from the string of text using the event template. For example, the event identifier 118 may extract the one or more parameters from the string of text using the event template.

The method 600, at block B608, includes transmitting data that causes a display in a user interface that is based at least on the one or more parameters of the event. For example, the highlight generator 120 may transmit data that causes a display in the graphical user interface 400 that is based at least on the one or more parameters of the event.

FIG. 7 is a flow diagram showing an example method 700 for determining an event based at least on identifying one or more indicators in at least one string of text, in accordance with at least some embodiments of the present disclosure. The method 700, at block B702, includes receiving at least one message comprising at least one string of text generated by an application. For example, the client device 102 may receive at least one message comprising at least one string of text, the at least one string of text generated by an application (e.g., the game instance 130) based at least on one or more conditions being satisfied in an application session, the one or more conditions corresponding to a state of the application.

The method 700, at block B704, includes determining an event corresponding to a state of the application based at least on identifying one or more indicators of the event in the at least one string of text. For example, the event identifier 118 may determine an event corresponding to the state of the application based at least on identifying one or more indicators of the event in the at least one string of text using natural language processing.

The method 700, at block B706, includes transmitting data that causes a display in a user interface that is based at least on the event. For example, the highlight generator 120 may transmit data that causes a display in the graphical user interface 400 that is based at least on the one or more parameters of the event.

Example Content Streaming System

Now referring to FIG. 8, FIG. 8 is an example system diagram for a content streaming system 800, in accordance with at least some embodiments of the present disclosure. FIG. 8 includes application server(s) 802 (which may include similar components, features, and/or functionality to the example computing device 900 of FIG. 9), client device(s) 804 (which may include similar components, features, and/or functionality to the example computing device 900 of FIG. 9), and network(s) 806 (which may be similar to the network(s) described herein). In some embodiments of the present disclosure, the system 800 may be implemented. The application session may correspond to a game streaming application (e.g., NVIDIA GeFORCE NOW), a remote desktop application, a simulation application (e.g., autonomous or semi-autonomous vehicle simulation), computer aided design (CAD) applications, virtual reality (VR) and/or augmented reality (AR) streaming applications, deep learning applications, and/or other application types.

In the system 800, for an application session, the client device(s) 804 may only receive input data in response to inputs to the input device(s), transmit the input data to the application server(s) 802, receive encoded display data from the application server(s) 802, and display the display data on the display 824. As such, the more computationally intense computing and processing is offloaded to the application server(s) 802 (e.g., rendering—in particular ray or path tracing—for graphical output of the application session is executed by the GPU(s) of the game server(s) 802). In other words, the application session is streamed to the client device(s) 804 from the application server(s) 802, thereby reducing the requirements of the client device(s) 804 for graphics processing and rendering. In one or more embodiments, video provided by the client interface 126 may be of the application session (e.g., a game session), which may be streamed to one or more viewers and/or a player of the game. For example, the player may receive a stream of the game in a gaming application that presents the stream, whereas a viewer may receive the stream in a video player application that presents the game (e.g., within a web browser). In various examples, the application server(s) 802 may comprise one or more of the server(s) 104 or one or more of the server(s) 104 may be separate from the application server(s) 802.

For example, with respect to an instantiation of an application session, a client device 804 may be displaying a frame of the application session on the display 824 based at least on receiving the display data from the application server(s) 802. The client device 804 may receive an input to one of the input device(s) and generate input data in response. The client device 804 may transmit the input data to the application server(s) 802 via the communication interface 820 and over the network(s) 806 (e.g., the Internet), and the application server(s) 802 may receive the input data via the communication interface 818. The CPU(s) may receive the input data, process the input data, and transmit data to the GPU(s) that causes the GPU(s) to generate a rendering of the application session. For example, the input data may be representative of a movement of a character of the user in a game session of a game application, firing a weapon, reloading, passing a ball, turning a vehicle, etc. The rendering component 812 may render the application session (e.g., representative of the result of the input data) and the render capture component 814 may capture the rendering of the application session as display data (e.g., as image data capturing the rendered frame of the application session). The rendering of the application session may include ray or path-traced lighting and/or shadow effects, computed using one or more parallel processing units—such as GPUs, which may further employ the use of one or more dedicated hardware accelerators or processing cores to perform ray or path-tracing techniques—of the application server(s) 802. In some embodiments, one or more virtual machines (VMs)—e.g., including one or more virtual components, such as vGPUs, vCPUs, etc.—may be used by the application server(s) 802 to support the application sessions. The encoder 816 may then encode the display data to generate encoded display data and the encoded display data may be transmitted to the client device 804 over the network(s) 806 via the communication interface 818. The client device 804 may receive the encoded display data via the communication interface 820 and the decoder 822 may decode the encoded display data to generate the display data. The client device 804 may then display the display data via the display 824.

Example Computing Device

FIG. 9 is a block diagram of an example computing device(s) 900 suitable for use in implementing some embodiments of the present disclosure. Computing device 900 may include an interconnect system 902 that directly or indirectly couples the following devices: memory 904, one or more central processing units (CPUs) 906, one or more graphics processing units (GPUs) 908, a communication interface 910, input/output (I/O) ports 912, input/output components 914, a power supply 916, one or more presentation components 918 (e.g., display(s)), and one or more logic units 920. In at least one embodiment, the computing device(s) 900 may comprise one or more virtual machines (VMs), and/or any of the components thereof may comprise virtual components (e.g., virtual hardware components). For non-limiting examples, one or more of the GPUs 908 may comprise one or more vGPUs, one or more of the CPUs 906 may comprise one or more vCPUs, and/or one or more of the logic units 920 may comprise one or more virtual logic units. As such, a computing device(s) 900 may include discrete components (e.g., a full GPU dedicated to the computing device 900), virtual components (e.g., a portion of a GPU dedicated to the computing device 900), or a combination thereof.

Although the various blocks of FIG. 9 are shown as connected via the interconnect system 902 with lines, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 718, such as a display device, may be considered an I/O component 914 (e.g., if the display is a touch screen). As another example, the CPUs 906 and/or GPUs 908 may include memory (e.g., the memory 904 may be representative of a storage device in addition to the memory of the GPUs 908, the CPUs 906, and/or other components). In other words, the computing device of FIG. 9 is merely illustrative. Distinction is not made between such categories as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “hand-held device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and/or other device or system types, as all are contemplated within the scope of the computing device of FIG. 9.

The interconnect system 902 may represent one or more links or busses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 902 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and/or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 906 may be directly connected to the memory 904. Further, the CPU 906 may be directly connected to the GPU 908. Where there is direct, or point-to-point connection between components, the interconnect system 902 may include a PCIe link to carry out the connection. In these examples, a PCI bus need not be included in the computing device 900.

The memory 904 may include any of a variety of computer-readable media. The computer-readable media may be any available media that may be accessed by the computing device 900. The computer-readable media may include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, the computer-readable media may comprise computer-storage media and communication media.

The computer-storage media may include both volatile and nonvolatile media and/or removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and/or other data types. For example, the memory 904 may store computer-readable instructions (e.g., that represent a program(s) and/or a program element(s), such as an operating system. Computer-storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which may be used to store the desired information and which may be accessed by computing device 900. As used herein, computer storage media does not comprise signals per se.

The computer storage media may embody computer-readable instructions, data structures, program modules, and/or other data types in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term “modulated data signal” may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, the computer storage media may include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

The CPU(s) 906 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and/or processes described herein. The CPU(s) 906 may each include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) that are capable of handling a multitude of software threads simultaneously. The CPU(s) 906 may include any type of processor, and may include different types of processors depending on the type of computing device 900 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 900, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 900 may include one or more CPUs 906 in addition to one or more microprocessors or supplementary co-processors, such as math co-processors.

In addition to or alternatively from the CPU(s) 906, the GPU(s) 908 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and/or processes described herein. One or more of the GPU(s) 908 may be an integrated GPU (e.g., with one or more of the CPU(s) 906 and/or one or more of the GPU(s) 908 may be a discrete GPU. In embodiments, one or more of the GPU(s) 908 may be a coprocessor of one or more of the CPU(s) 906. The GPU(s) 908 may be used by the computing device 900 to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPU(s) 908 may be used for General-Purpose computing on GPUs (GPGPU). The GPU(s) 908 may include hundreds or thousands of cores that are capable of handling hundreds or thousands of software threads simultaneously. The GPU(s) 908 may generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s) 906 received via a host interface). The GPU(s) 908 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory may be included as part of the memory 904. The GPU(s) 908 may include two or more GPUs operating in parallel (e.g., via a link). The link may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs through a switch (e.g., using NVSwitch). When combined together, each GPU 908 may generate pixel data or GPGPU data for different portions of an output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

In addition to or alternatively from the CPU(s) 906 and/or the GPU(s) 908, the logic unit(s) 920 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 900 to perform one or more of the methods and/or processes described herein. In embodiments, the CPU(s) 906, the GPU(s) 908, and/or the logic unit(s) 920 may discretely or jointly perform any combination of the methods, processes and/or portions thereof. One or more of the logic units 920 may be part of and/or integrated in one or more of the CPU(s) 906 and/or the GPU(s) 908 and/or one or more of the logic units 920 may be discrete components or otherwise external to the CPU(s) 906 and/or the GPU(s) 908. In embodiments, one or more of the logic units 920 may be a coprocessor of one or more of the CPU(s) 906 and/or one or more of the GPU(s) 908.

Examples of the logic unit(s) 920 include one or more processing cores and/or components thereof, such as Tensor Cores (TCs), Tensor Processing Units (TPUs), Pixel Visual Cores (PVCs), Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming Multiprocessors (SMs), Tree Traversal Units (TTUs), Artificial Intelligence Accelerators (AIAs), Deep Learning Accelerators (DLAs), Arithmetic-Logic Units (ALUs), Application-Specific Integrated Circuits (ASICs), Floating Point Units (FPUs), input/output (I/O) elements, peripheral component interconnect (PCI) or peripheral component interconnect express (PCIe) elements, and/or the like.

The communication interface 910 may include one or more receivers, transmitters, and/or transceivers that enable the computing device 900 to communicate with other computing devices via an electronic communication network, included wired and/or wireless communications. The communication interface 910 may include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating over Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and/or the Internet.

The I/O ports 912 may enable the computing device 900 to be logically coupled to other devices including the I/O components 914, the presentation component(s) 918, and/or other components, some of which may be built in to (e.g., integrated in) the computing device 900. Illustrative I/O components 914 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I/O components 914 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs may be transmitted to an appropriate network element for further processing. An NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 900. The computing device 900 may be include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations of these, for gesture detection and recognition. Additionally, the computing device 900 may include accelerometers or gyroscopes (e.g., as part of an inertia measurement unit (IMU)) that enable detection of motion. In some examples, the output of the accelerometers or gyroscopes may be used by the computing device 900 to render immersive augmented reality or virtual reality.

The power supply 916 may include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 916 may provide power to the computing device 900 to enable the components of the computing device 900 to operate.

The presentation component(s) 918 may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up-display (HUD), other display types, or a combination thereof), speakers, and/or other presentation components. The presentation component(s) 918 may receive data from other components (e.g., the GPU(s) 908, the CPU(s) 906, etc.), and output the data (e.g., as an image, video, sound, etc.).

Example Data Center

FIG. 10 illustrates an example data center 1000 that may be used in at least one embodiments of the present disclosure. The data center 1000 may include a data center infrastructure layer 1010, a framework layer 1020, a software layer 1030, and/or an application layer 1040.

As shown in FIG. 10, the data center infrastructure layer 1010 may include a resource orchestrator 1012, grouped computing resources 1014, and node computing resources (“node C.R.s”) 1016(1)-1016(N), where “N” represents any whole, positive integer. In at least one embodiment, node C.R.s 1016(1)-1016(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input/output (“NW I/O”) devices, network switches, virtual machines (“VMs”), power modules, and/or cooling modules, etc. In some embodiments, one or more node C.R.s from among node C.R.s 1016(1)-1016(N) may correspond to a server having one or more of the above-mentioned computing resources. In addition, in some embodiments, the node C.R.s