The Chroma key process further means to process digital assets together to produce an experience or video that can be captured recorded, streamed, playing back, merged, distributed, co-located or viewed on a device (or devices) to communicate broadcast news, entertainment, or visual conference calling. The Chroma key process allows users to make a subject or subjects appear to be in a location (or locations) different, or in some cases the same, location as an additional subject or subjects. Replacement content and a well-lit Subject that is recorded, captured or streamed against a monochromatic screen are separate collections of data or digital assets for processing videography, Chroma key or digital asset management.

Video files used as replacement content are commonly described as “loops.” Loops may be repeated over and over again as a background in a composited video. For example a 10 second loop may be repeated multiple times in a 10 minute video. Loops are optionally sections of larger video files. The jump from an end of a video back to the beginning of a video to form a loop is referred to as a loop point or splice point. In various embodiments, loops are not limited to video files. For example, a virtual environment may have time varying aspects (e.g., parts that move or change with time) that form a loop at a splice point. In another example, a loop may include audio data (e.g., an audio track of a virtual environment or a video) which is repeated at a splice/loop point.

A preferred loop is a loop that has been selected and/or modified to reduce or minimize discontinuity at a loop point (e.g., where the ends of a piece of content are spliced together to form the loop). A Preferred Loop includes, for example, a video clip that is curated, selected, and/or edited before, during or after the Chroma key process so that the loop point doesn't look unnatural to the human eye. In a specific example, a loop playing back as replacement content that includes people are milling in a crowd will likely include a sudden jump in peoples' position and movement at a random loop point. People's position will appear to jump as the loop starts over at the beginning of the video loop, this discontinuity is visible and a viewer can tell the replacement content of the composited video is not real or visually or audibility natural. It is desirable that a preferred loop have a smooth transition between end and start of the loop at a loop point. Systems and methods described herein are configured to identify preferred loops and splice points that minimize such discontinuity, such that a resulting composited video displays naturally. A preferred loop is a loop that is pleasing to the eye so that the loop/splice point plays back and the images are altered so they people milling appear to do so naturally with minimized discontinuities. As described elsewhere herein, a preferred loop may be generated by identifying a preferred splice point between end and start of loop video frames, and/or by editing a splice so as to merge several video frames to reduce discontinuities associated with particular objects.

FIG. 1. illustrates a System 100 for managing media assets, including replacement content and composited images, according to various embodiments of the invention. System 100 includes a Media Server 110, a Network 115, Clients 120, and an optional External Source 125. System 100 is optionally configured to support a library of replacement content and to support use of the replacement content to generate composited content including images, image sequences (video), and/or virtual/augmented environments.

Network 115 includes one or more communication network such as the Internet, a telephone/cellular network, a wide area network, a wireless network, and/or the like. Network 115 can include modems, Wi-Fi hardware, switches, Ethernet, antennas, etc. Network 115 is configured for communication between elements of System 100.

Clients 120, individually identified as Client 120A, Client 120B, etc., can include personal computers, tablet computers, mobile devices, AR/VR/XR headsets, game consoles, omni-dimensional and/or entertainment/communication devices configured to communicate via Network 115. Clients 120 typically include a display configured for viewing composited images and/or a speaker configured for listening to composited audio. Clients 120 may also include tactile user interfaces configured to provide other types of sensory information to a user. As is discussed elsewhere herein, Clients 120 may further include various types of microprocessors, logic and storage. Clients 120 optionally include a camera and/or microphone (not shown) configured to record images and/or video. Clients 120 optionally include a light source.

Media Server 110 is configured to store a library of replacement content, to received new replacement content and to provide replacement content and/or composited content to Clients 120. Media Server 110 optionally includes a plurality of geographically distributed computing devices.

As illustrated in FIG. 1, Media Server 110 comprises a Storage 130. Storage 130 includes non-transitory digital memory and is optionally configured to store replacement content, to store indexes to replacement content, and/or to store the various logic included in Media Server 110. Storage 130 is optionally distributed among multiple storage devices. As discussed elsewhere herein, the replacement content can be characterized, and thus indexed, by a wide variety of characteristics. The indexes to replacement content can be based on any of this metadata. The stored replacement content can include any of the various replacement content discussed herein. Storage 130 is optionally configured to store indexes to video stored on an external source of video, the indexes being to segments of the video stored on the external source, the segments including preferred loops of the replacement content.

For example, Storage 130 is optionally configured to store replacement content, foreground content, and/or composited content. Common examples of replacement content include 2D images, image sequences (video), virtual environments, and/or the like. In some embodiments, Storage 130 is configured to store replacement content including preferred splice points and/or anchor points. For example, a video may be stored in association with a plurality of splice points that would result in preferred video loops from the video. In a specific example, a link to a YouTube™ video may be stored in association with a plurality of splice points indicating the beginnings and endings of one or more subsets of the YouTube™ video that would result in preferred video loops. In another example, a virtual environment may be stored in association with series of anchor points (or an anchor surface) such that the virtual environment can be properly scaled and/or positioned relative to a subject in a composited virtual environment. In a specific example, a virtual environment may be associated with a scaling factor configured to scale the virtual environment such that a subject appears properly sized in a composited virtual environment including the subject. In a specific example, a virtual environment may include an anchor surface on a virtual floor. The anchor surface is configured such that the feet of a subject can be properly position on the floor in a composited environment/video.

Storage 130 may be configured to store virtual environments in rendered on non-rendered forms, the rendered forms being dependent on a point of view. Virtual environments, to be used as replacement content, may also be stored in association with alternative textures, movement mechanics (of structures or objects within the environment), and/or lighting options.

Storage 130 is optionally configured to store replacement content and/or foreground content as an omni-dimensional time varying virtual environment. For example, a 3D virtual environment having time varying content, such as a virtual environment having virtual objects moving, changing appearance, and/or otherwise changing therein. Optionally, such replacement content can include, for example:

- 1) A computer-generated time varying virtual environment that has been analyzed and indexed to identify repetitive movement therein and to determine time segments of the virtual environment that would result in preferred loops if used as replacement content. For example, a virtual environment including a virtual non-player character that moves in a predictable pattern may be indexed for preferred splice points such that the non-player character does not produce an undesirable discontinuity at a preferred loop point.
- 2) Replacement content may comprise the same virtual environment form a variety of viewpoints and/or lighting positions.
- 3) Replacement content associated with controls and/or handles configured to manipulate a virtual environment to be used as replacement content. For example, a virtual environment may be associated with a control that allows a user to rearrange furniture within a virtual room, to control movement and/or speed of objects with the virtual environment, and/or control any other aspect of the virtual environment. In some embodiments, such controls may be manipulated in real-time responsive to a subject of a composited video. For example, if the subject moves their hand is a specified manner a virtual object within the virtual environment may respond with a change in location or other characteristic (e.g., change color, change shape, make a sound, change appearance, etc.). Thus, a virtual replacement content may be responsive in real-time to acts of a subject and this responsiveness may be visible in a composited output.
- 4) Replacement content associated with tools or metadata configured to allow a user to identify changes in a time varying virtual environment that will allow for a preferred loop. Such tools may allow a user to observe which factors influence certain features of the virtual environment over time so shifting of segments or objects within an indexed time segment will increase the quality of preferred loops using the virtual environment.
- 5) Replacement content associated with a time series analysis configured to show how a digital asset variable changes over time to identify objects in order to re-create them at loop points to make viewing the asset appear more natural or make it more useful with other assets; and/or

Parts of Storage 130 are optionally included on Clients 120, External Source 125, and or some other external storage device.

As illustrated in FIG. 1, Media Sever 110 further includes Search Logic 135. Search Logic 135 is configured to search, procure and curate digital assets, such as videos or virtual environments stored in Storage 130 or digital assets provided by users or procured from the internet. Specifically, Search Logic 135 may be configured to find and identify candidate replacement content found in Storage 130 and/or locations external to Media Server 110. A purpose of Search Logic 135 is to obtain content that may have preferred video loops such that these may be added to a library thereof. For example, Search Logic 135 may be configured to search stored digital assets based on fixed or variable criteria to curate, index, edit and procure stored replacement content, and/or to index stored assets as segments of replacement content. Search Logic 135 may be configured to process and/or identify any of the types of replacement content discussed herein. Search Logic 135 may use Loop Logic 145, discussed further below, to identify and/or score preferred loops within procured content. In some embodiments, Search Logic 135 is configured for searching for preferred loops, or indexes thereto, stored in Storage 130, optionally based on search criteria such as keywords, loop score, loop length, etc.

In some embodiments, Search Logic 135 is configured to search external digital assets for replacement content based on user defined criteria to curate, index and procure stored replacement content. In some embodiments, Search Logic 135 is configured to search a library of digital assets using cookies, stored indexes, indexes from a recommendation engine, aspects of foreground content to score the likelihood of a match between foreground and replacement content, in order to procure replacement content. The searched library can include preferred loops stored in Storage 130. Search Logic 135 is optionally configured to search a library of computer-generated digital assets seeking scored indexed segments and procuring, processing and curating potential matched segments so replacement content and foreground content match for length, lighting, perspective, angles and other aspects of segments to procure desired aspects of foreground content to match with replacement content, or vice-versa. Search Logic 135 is optionally configured to search external digital assets for replacement content based on tags within a library of digital assets or segments. Search Logic 135 is optionally configured to search external digital assets for foreground content within a library of digital assets or segments.

In one embodiment, Search Logic 135 is configured to crawl the internet, retrieve discovered content (e.g., videos or time dependent virtual content) and to process the discovered content using Loop Logic 145 to identify preferred loops that would make good replacement content. In one embodiment, Search Logic 135 is configured to received content from third parties and to process the received content to identify preferred loops that would make good replacement content. For example, Search Logic 135 may receive a link to a video or data representing a time dependent virtual environment from Client 120A and process the video or virtual environment using Loop Logic 145 to identify preferred loops therein. In some embodiments, the third party is compensated based on popularity and/or use of the identified preferred loops.

Search Logic 135 is optionally configured to identify replacement content within the storage or the indexes. For example, a user (e.g., a Camera operator or user of Client 120A) may use Search Logic 135 to find replacement content using image tags, characteristics of images (e.g., length, type, format, brightness, lighting direction, color, night/day), scored loops, preferred Loops, natural loops, unnatural loops that may require correction using dynamic splicing, and/or any other content characteristic discussed herein.

Media Server 110 typically further includes Sharing Logic 140. Sharing Logic 140 is configured to distribute replacement content to third parties, e.g., Clients 120. For example, Sharing Logic 140 may include an application programming interface or user interface configured for a third party to search for replacement content. Such a search and/or recommendations may be based on any of the characteristics of replacement content discussed herein, and/or external factors such as internet search or eCommerce trends40. For example, a user may search for replacement content including a virtual environment that can be rendered from an omni-directional viewpoint and includes two virtual animals whose position within the virtual environment is time dependent. The user may further specify that the virtual environment be characterized by metadata tags of “round space station.” The user may further specify that the replacement content has a preferred loop over a designated quality score and that lasts at least 30 seconds. The user may further request that the replacement content have user controllable features, such as viewpoint and/or configurable textures. The user may further request that the replacement content be editable, for example, using a video editor or a virtual environment editor. A user may further request replacement content has a preferred loop based on time-of-day for its lighting (daytime or nighttime), interior or exterior loops that match the time of day as inside or outside. User may search for preferred loops by landscape, portrait, frames per second, resolution, level of brightness, etc., in combination with any of the above. The user can search for any criteria that can be indexed by any data point in a digital asset.

In some embodiments, Sharing Logic 140 is configured to share both replacement content and foreground content with Clients 120. The shared content may be merged, integrated, comingled, co-located, processed together, and/or used to generate composited content and/or images. Multiple replacement content and/or multiple foreground content may be used to generate composited content/images. In some embodiments Sharing Logic 140 is further configured to share the composited content/images. Distribution of any of the content discussed herein is optionally in real-time, i.e., performed as the content is generated.

Sharing Logic 140 is optionally configured to share any of the content discussed herein in a peer-to-peer network and/or with mobile devices. In some embodiments Sharing Logic 140 is configured to share a segment of video stored on an external source of video to a client, the segment including only a part of the video indexed as a preferred loop.

Sharing Logic 140 is optionally configured to distribute content by streaming data to Clients 120. The distributed content can include any of the contend discussed herein and may be previously recorded, captured, composited, edited or processed to add integrate, merge or add value to other content using the Chroma key process, either manually or automatically.

Sharing Logic 140 is logic optionally configured to share content where several computing devices (e.g., Clients 120) capture images of respective subjects in respective foreground content. The foreground content is then shared to other devices such that the different subjects appear to be in the same location using the same replacement content. For example, Client 120A and Client 120B may each be used to capture images of their respective users, Media Server 110 may be used to combine foreground content of both of the images of the respective users with a common background asset such that the respective users appear to be together in a single common location. This can be performed in real-time and provided to Clients 120 and/or other parties using Sharing Logic 140.

Sharing Logic 140 is optionally configured for file sharing or streaming of digital assets (content) where several computing devices can stream, record or capture unique replacement content, and a single subject from a single device can share, stream or send foreground content including the subject to other computing devices so that the subject will appear to be in multiple locations using the different replacement content recorded by each of the multiple devices. This can be streamed by Sharing Logic 140 in real-time.

Sharing Logic 140 is optionally configured to share replacement content and/or composited content including virtual environments generated by several computing devices. For example, a virtual environment may include a room rendered from a first viewpoint on a first External Source 125 and rendered from a second viewpoint on a second (or the same) External Source 125. These two renderings may then be used as replacement content to be combined with subjects in different composited images representing the two different viewpoints. The virtual environments, including subjects, may then be viewed from each subjects' viewpoint. In a specific example, this allows images of actual users (e.g., people) to be placed within an interactive and real-time virtual environment. Optionally, the foreground images are captured from multiple viewpoints, so that the foreground images used in the composited images match the viewpoint of the virtual environment and an orientation of each user. Thus, if the first user is standing behind the second user, the first user may see the back of the first user's head, in the composited image of the virtual environment.

Sharing Logic 140 is optionally configured to where a computing device can share scored and indexed segments that are processed and edited to be shared with other computing devices in real-time, so the Chroma key process produces matched segments such that foreground content and replacement content appear together naturally and optionally at different camera angles and points of view.

Sharing Logic 140 is logic optionally configured to where several computing devices share replacement content that can be processed as a segment of video frames that play back as replacement content and objects or frames within a segment can be cut, spliced, rotated or shifted in time as a whole or in part to appear naturally in the Chroma key process.

Sharing Logic 140 is optionally configured to share or distribute replacement content in a 3D defined environment and processed so the environment can be moved in time for digital sequencing where objects within that computer generated environment can be moved around for a different perspective by the viewer where objects within the environment can be moved to enhance narrative and realism, and appear visually more natural or realistic.

Media Server 110 typically further includes Loop Logic 145. Loop Logic 145 is configured to identify preferred loops within any of the content discussed herein and optionally to generate a score representing qualities of the identified loops. For example, Loop Logic 145 may be configured to identify splice points representing the beginning and end of a preferred loop within a video. More than one preferred loop, optionally of different lengths, may be identified within a specific video. Likewise, multiple preferred loops may be identified within a time variant virtual environment, optionally having different and/or moving viewpoints, different lengths, and/or other distinguishing characteristics. Scores representing qualities of the identified loops may be based on quality of a loop point and/or other aspects of the preferred loop (e.g., camera stability and/or lighting stability). Loop points are optionally manually suggested by a user. Scores may be generated by scoring logic and may be based on, for example, a continuity of a loop at a loop point/splice. Splices optionally having noticeable discontinuities receiving lower scores relative to splices having hard to notice or no discontinuities between the end and start of a given loop.

Loop Logic 145 optionally includes a neural network trained to identify splice points (beginning and endings of content segments) that result in preferred loops. Loop Logic 145 optionally includes comparison logic configured to compare images or virtual environments at different times within a video or a time dependent virtual environment and to identify comparisons that show minimum difference and, as such, include good candidates for splice points. Loop Logic 145 is optionally configured to identify preferred loops with respect to both audio and visual features.

In some embodiments, Loop Logic 145 is configured to consider dynamic splicing techniques in identifying and scoring preferred loops to be used as replacement content. For example, Loop Logic 145 may be configured to consider use of Dynamic Splicing Logic 185 (discussed elsewhere Loop Logic 145 is optionally configured to select splicing points within any of the content discussed here appear more pleasing to the human eye at the loop point.

Loop Logic 145 is optionally configured to source and identify objects within frames of a video or within time segments of a virtual environment. For example, objects within the final frame of a loop may be compared to objects within the first frame of a loop. The comparison may include, appearance, location, and/or orientation of the objects. Loop Logic 145 is logic optionally configured to not discriminate by time or timeline.

Described in these embodiments are Chroma key processing of digital assets, and these editing and management processes do not discriminate by time, meaning the Chroma key process might integrate both Foreground and replacement content, captured, streamed or played back live, or recorded at different times; replacement content or Foreground Content can be shifted, lengthened or shortened in time to synchronize with Foreground Content subjects that may or may not be in real-time and could be played back at different speeds, correct any audio or video latency or to match lighting between foreground and replacement content.

Media Server 110 optionally further includes Video Modification Logic 150. Video Modification Logic 150 is configured to edit content to create preferred loops, the content optionally including video, and omni-dimensional environment, and/or a time variant virtual environment. Video Modification Logic 150 is optionally configured edit digital assets used as replacement content and foreground content to create better composited content. Such editing can include tools configured to make the replacement content and foreground content have similar characteristics, such that the resulting composited image appears more realistic or natural. Specifically, the editing can include change of viewpoint, scaling, modification of light intensity, light color, light direction, time (e.g., loop length or speed), frame rate, resolution, orientation (portrait or landscape), color adjustment, and/or any other characteristics discussed herein.

In some embodiments, Video Modification Logic 150 is configured to detect imperfections in replacement content and/or foreground content, for example using a trained neural network. Such imperfections may result from or include a non-monochromatic matte, or a shadow creating a non-monochromatic pixel grouping in the matte, or any other disruption in the Chroma key process. The neural network is optionally trained using machine language, human segmentation and/or historical background and/or foreground content.

In some embodiments, Video Modification Logic 150 is configured to modify content in real-time to create better composited images. For example, Video Modification Logic 150 may automatically adjust brightness of pixels within a replacement video loop to better match brightness of pixels within foreground content. Such modification may be performed automatically or manually.

Video Modification Logic 150 optionally includes a Transcoder 180. Transcoder 180 is configured to convert content file types. For example, changing video files from a “.mov” to a “.mp4” file. Transcoder 180 is typically used to make formats of replacement content and foreground content the same and/or a desired format of the composited content. In some embodiments, Transcoder 180 is configured to convert virtual environments or rendered views thereof from a first format to a second format.

Video Modification Logic 150 optionally includes a Fader 170. Fader 170 is configured for editing any of the contents discussed herein to improve the appearance of replacement content at a loop point. Specifically, Fader 170 may be used to modify content (e.g., frames) near a loop point such that pixels are merged, cross-faded, and/or integrated with each other to create a tethering effect. Such modification may improve a score of a preferred loop and make the loop appear more natural to the human eye at the loop point.

Video Modification Logic 150 optionally includes an Editor 175. Editor 175 includes logic configured to edit replacement content, replacement content, foreground content, and/or resulting composited content. Editor 175 can be configured to be operated manually by a user and/or to operate automatically in response to a trained artificial intelligence. Editor 175 can include any of the features commonly found in video, image, and/or virtual environment editors. Further, Editor 175 can be configured to perform specific operations related to loop points. For example, Editor 175 optionally includes a scoring function (scoring logic) configured to generate a score for a loop (e.g., for a loop point/splice), a fade function configured to use control operation of Fader 170 at a loop point, functions configured to adjust characteristics of first content to be more like second content. For example, Editor 175 optionally includes automatic or user controllable functions configured to adjust characteristics of a set of frames immediately prior to a loop point to be more like a set of frames immediately following the loop point. In a specific example Editor 175 may be configured to adjust a color or brightness of pixels on either side of a loop point to better match each other. Such adjustments may be accomplished by a “match color” or a “match brightness” control. The result of such functions can be to bring the characteristics found in each set of pixels closer to each other, without having to specify particular colors or brightness. Editor 175 is optionally configured to rotate content, e.g., to convert landscape content to portrait content. Editor 175 is optionally configured to automatically or manually trim a preferred loop, or manually edit a video segment to generate a preferred loop. Editor 175 is optionally configured to trim advertisements from content. For example, if ads are played before a sourced digital asset such as a YouTube video the advertisement may be trimmed off.

Video Modification Logic 150 optionally further includes Color Key Logic 180. Color Key Logic 180 is configured for modifying content elements in the Chroma key process, such as modifying pixels or lighting to perfect imperfections in the matt or replacement content that negatively impacts the Chroma key process. For example, Color Key Logic 180 may be configured to detect defective pixels, color variation, and/or shadows on a green screen. In some embodiments Color Key Logic 180 is configured to suggest lighting changes and/or automatically make changes to lighting of background and/or foreground content.

Media Server 110 optionally further comprises Syncing Logic 155. Synching Logic 155 is configured to synchronize content such that audio, video and other data in replacement content match and integrate to play back with foreground content to appear natural upon viewing. In one example, Synching Logic 155 is configured to synchronize various audio and video received from remote locations, e.g., over Network 115. Typically, Synching Logic 155 is configured such that playback of composited content appears natural.

Media Server 110 optionally further comprises Upload Logic 160. Upload Logic 160 is configured for receiving and/or providing content relative to Media Server 110. For example, in some embodiments Upload Logic 160 is configured for users to upload content (e.g., replacement content) to Storage 130 for inclusion in a library of replacement content. In some embodiments, Upload Logic 160 is configured to provide content to members of Clients 120 for use as foreground and/or replacement content in a Chroma key process. Upload Logic 160 is optionally configured to use Search Logic 135 and/or Sharing Logic 140 as described elsewhere herein. Content received via Upload Logic 160 may automatically be modified using Video Modification Logic 150 and/or processed using Loop Logic 145. For example, a user of Client 120A may provide a video segment via Upload Logic 160 and Loop Logic 145 may then automatically identify preferred loop points (and scores thereof) withing the provided video segment. The video segment may be added to a library of replacement content responsive to the loop point scores, e.g., only loop points having significantly high scores may be added.

Media Server 110 optionally further comprises Tagging Logic 165. Tagging Logic is configured to tag digital assets including replacement content and/or foreground content. Such tags may be used by Search Logic 135, Sharing Logic 140, and/or Upload Logic 160 to facilitate identification of content. Tags may be generated manually but are typically automatically generated using a trained machine learning system and can include keywords and/or descriptive metadata. For example, in some embodiments, a user may search for replacement content including “sleeping cats” that has at least a minimum preferred loop score. Tags are optionally stored in association with the associated content and in association with other metadata such as copyright status, watermarks present, blockchain location, NFT status, file time, loop length, resolution, color profile, version, creation date, location, lighting characteristics, use history, and/or any other characteristic discussed herein. In some embodiments, Tagging Logic 165 is configured to tag different preferred loops within content using different tags. For example, a video clip may include a preferred loop of traffic at an intersection and a preferred loop of pedestrians in the same intersection. Such loops are likely to be assigned different tags. In some embodiments, Tagging Logic 165 is configured to selectively process the or more preferred loops within replacement content, while ignoring or deemphasizing parts of the replacement content that is not part of a preferred loop.

Media Server 110 optionally further comprises Dynamic Splicing Logic 185. Dynamic Splicing Logic 185 is configured to create splices between the begging and ends of a loop that include more than just switching from the frame before the loop point to the frame following the loop point. Dynamic Splicing Logic 185 is logic optionally configured to process any of the content discussed herein. Dynamic Splicing Logic 185 can be used to reduce or eliminate discontinuities in active replacement content. For example, in replacement content including traffic it may be difficult to identify high quality splice points.

Dynamic Spicing Logic 185 may be configured to use Fader 170 or Editor 175 to modify content frames to produce a more preferred loop. For example, objects may be moved within a frame using Editor 175, or a frame may be segmented into segments that are then separately assigned to different transitionary frames around a splice. In a specific example, an image of a vehicle as seen in frames immediately prior to a loop point may be segmented and only the segment including the vehicle are then added to frames immediately following the loop point, while other segments may be treated differently. Thus, frames around the loop point may include segments from different original frames to produce a more preferred loop. Objects, e.g., a moving car or person, may be carried/continued to different sides of a loop point. In such embodiments, the loop point may not occur between specific frames, but may occur across a plurality of frames, with different segments occurring in different members of the frames included in the loop point. Selection of such segments is optionally automatic or manual. For example, a specific segment may be moved or otherwise manipulated with frames near the loop point dither automatically or manually.

In another example, assuming traffic moving from right to left in replacement content. The segmentation between pixels that are included in the end of the loop and pixels that are included in the beginning of the loop may move from right to left with the vehicles. As such, a specific vehicle is not broken up in a discontinuity at the loop point. The segmented area may be bounded by a moving segment divider (linear, non-linear, and/or non-continuous), which optionally separate objects within a plurality of frames.

In various embodiments, Dynamic Splicing Logic 185 is configured to try a variety of alternative segmentation strategies in order to identify which segmentation strategies result in the best preferred loop scores. This may include identification and manipulation of objects within a video or virtual environment and movement of these objects to different places or assignment of these objects to different frames within the loop point. In some embodiments, an object may be removed from the loop point entirely. For example, a walking person may be removed from frames including in the loop point such that that the image of that person does not create a discontinuity. Removal of an object is optionally performed during rendering of a two- or three-dimensional virtual environment.

Dynamic Splicing Logic 185 is logic optionally configured to process pixels within foreground or replacement content where an imperfect pixel in a monochromatic matte is corrected to maintain monochromatic continuity to improve or enhance the Chroma key process and appear pleasing to the human eye.

Dynamic Splicing Logic 185 is optionally configured to process digital assets in a system where the left side of the frame at the loop point comes from the beginning of the loop, and the right side of the frame at the loop point comes from the end of the loop, or vice versa. Thus, pixels can be indexed and shifted so objects within a series of frames in any environment can be created, modified to fit together and played back in a visually pleasing way.

In combination, Dynamic Splicing Logic 185 and Loop Logic 145 are optionally configured to modify replacement content to be modified, edited, interlaced or processed as a grouping of video frames that are looped to play back or streamed over and over for use as replacement. Sets if individual frames can change perspective like a virtual camera, and objects within the virtual segment can be shifted and spliced like 2D video but instead, shifted and spliced by computer code so that objects may appear to be closer than they are. Objects can appear in different frames to account for the selection of different Preferred Loop markers so the replacement content can be edited to enhance a preferred loop.

Loop Logic 145 is logic optionally configured to use Dynamic Splicing Logic 185 to correct imperfections in the synchronicity of digital assets in the Chroma key process such that the resulting video appears more natural. The Chroma key process optionally captures a subject in real-time using replacement content that was recorded earlier and played back on devices in real-time. However, in this embodiment logic is configured to process and edit digital assets used as replacement content that could be captured, streamed or recorded in real-time, and the Subject as Foreground Content could have been captured earlier and recorded in a different location, then appears to be in the same location as the live replacement content.

Dynamic Splicing Logic 185 is optionally configured to control gaming engines that loop content and generate preferred loops. Replacement content may be generated by a gaming engine in real time. For example, Loop Logic 145 is optionally configured to process, edit, replace or reproduce a person's image or avatar in 3D replacement content or Foreground Content such that the replacement content and foreground content appear natural in a preferred loop. Loop Logic 145 is logic optionally configured to mimic one or more person's image or avatar in 3D or 4D replacement content (e.g., a time variant virtual environment) so the Subject appears natural within a virtual environment.

System 100 optionally further includes one or more External Source 125 of digital assets. External Source 125 can include Storage 126 configured to store digital assets and Access Logic 127 configured for accessing the stored digital assets. Access Logic 127 may include a query system and/or a recommendation system. Access Logic 127 is optionally configured to retrieve digital assets based on search criteria such as author, subscription channel, text, tags, locations, date and time, a recommendation based on search history, colors, night, day, length, or any other criteria associated with foreground or replacement content. Examples of External Source 125 can include YouTube™, Shutter Stock™ and Getty Images™. As discussed elsewhere herein, Search Logic 135 is optionally configured to retrieve digital assets from External Source 125.

Media Server 110 and/or Clients 120 optionally further include Chroma Key Logic 190. Chroma Key Logic is configured to perform Chroma key operations as described herein and may include any of the commercially available Chroma key applications available. Chroma Key Logic 190 is optionally distributed between Media Server 110 and members of Clients 120. For example, segmentation may be performed on Media Server 110 while assembly of content in to composited content may be performed on Clients 120, or vise versa. In some embodiments, Chroma Key Logic 190 is configured to generate multiple versions of composited content, each version having different replacement content and/or other characteristics. Such versions may be used for A/B testing or be directed at different audiences/markets. Chroma Key Logic 190 is optionally configured to balance audio content included in replacement content and foreground content. For example, using manual or automatic controls.

Media Server 110 further includes Processor 195. Processor 195 is a digital processor configured to execute logical operations and may include electronic or optical circuits therefor. Processor 195 is configured to execute at least Search Logic 135, Sharing Logic 140, Loop Logic 145, Video Modification Logic 150, Dynamic Splicing Logic 185, Upload Logic 160, and/or Tagging Logic 165.

FIG. 2. Illustrates methods of adding a video and/or or other types of content to a digital asset media library, according to various embodiments of the invention. Such a library may be stored in Storage 130 and managed by Media Server 110. While the steps illustrated in FIG. 2 are presented with reference to video content, these steps may be applied to any of the other types of content discussed herein, such as time variant virtual or augmented reality environments. The steps illustrated in FIG. 2 are optionally performed using the system 100 illustrated in FIG. 1.

In a Receive Video Step 210 a digital asset is received. The digital asset can include, for example, video live streamed or previously stored. The digital asset may be received from one of Clients 120 or from External Source 125. The digital asset may include video and/or 3D time variant virtual environments to be used as foreground or background replacement content to produce composited content. The digital asset may include content captured by a camera, content generated using an artificial intelligence system, content generated by hand (e.g., drawn), composted content, and/or content otherwise generated. The digital asset may have been generated using an embodiment of the systems illustrated in FIG. 1.

In an optional Approve Video Step 215, the received video is approved for storage or use based on criteria specified by a user. For example, a user may only approve video available for copyright license, video having specific image tags, or video designated as being appropriate for a specific viewer age range. In some embodiments a trained neural network is used to exclude video including explicit content, or video having a specific violence, etc. Approve Video Step 215 is optionally performed using Upload Logic 160. Unapproved content is optionally abandoned.

In an Optional Store Step 220 the received video is stored, for example, in Storage 130. Store Step 220 is optional in embodiments wherein the received video is being used in a real-time streaming mode.

In an Index Video Step 225, the received video is indexed. The indexing may include indexing the entire video, identifying loop points within the video, identification of preferred loops within the video, tagging of the entire video, tagging of individual preferred loops within the video, scoring of individual preferred loops within the video, and/or the like. For example, in some embodiments, Index Video Step 225 includes identification and scoring of a plurality of preferred loops, which are designated as replacement content, within the received video, tagging of each of the preferred loops based on (textual, visual, and/or audio) content of the loops, and generation of further metadata regarding each of the loops. The further metadata may include loop length, color and/or lighting characteristics, legal rights, and/or any of the other characteristics of replacement content discussed herein. Multiple preferred loops may be identified and scored withing a single received video. The preferred loops may overlap or be distinct. Index Video Step 225 may be performed using Loop Logic 145, Video Modification Logic 150, and/or Dynamic Splicing Logic 185. Index Video Step 225 may be partially or fully automated. Preferred loops may represent a subset of the received video. Scoring logic may be included in, for example, Editor 175, Loop Logic 145, Dynamic Splicing Logic 185, and/or Search Logic 135.

In some embodiments, Index Video Step 225 includes indexing video so that it can be accessed quickly and precisely. Portions of larger video files can be marked with start and end points (splice points) of preferred loops, each of which are optionally separately indexed. Indexing can be for streaming and/or storage for use in the Chroma key process.

In some embodiments, Index Video Step 225 includes indexing video according to an amount of screen space or storage replacement or foreground content may require when it is displayed, broadcast, captured, streamed, recorded or played back. The indexing may include a financial cost or subscription required for use of the video content. The indexing may include an amount of time a digital asset or is displayed, broadcast, captured, streamed, recorded or played back so the content creator or copyright holder may earn revenue from calculating the fractional or whole use of the digital asset as it is captured, streamed, or recorded in the Chroma key process. The indexing may be configured for efficient search and retrieval of particular preferred loops.

In various embodiments, the preferred loops are identified based on a quality of an associated loop point, identified based on a lack of discontinuities at an associated loop point, identified based on positions of objects within the received content, and/or identified based on a realistic transition at a loop point. The preferred loops may be automatically identified using a trained neural network or other artificial intelligence system. The preferred loops are optionally represented by splice points within the received content. For example, a preferred loop may be defined by a link to the received content and a pair of splice points within the received content. Alternatively, preferred loops may be stored separately from the received content from which they are derived.

In an optional Transcode Video Step 230, the indexed video may be transcoded between formats, for example, using Transcoder 180. Digital assets may require video segments be in compatible file types in order to be used together to generate composited content, or to be used more efficiently. An example would be a replacement content digital asset in a “.mov” format can be transcoded to the same file format or codec as the foreground content with an “.mp4” file type so the Chroma key process will work as desired.

FIG. 3. Illustrates methods of providing a digital asset, such as replacement content, according to various embodiments of the invention. The digital asset may be used to produce composited content, for example, in a Chroma key process. While the methods of FIG. 3 are described in relation to replacement content, they are alternatively used to provide other digital assets, such as foreground content. While the methods of FIG. 3 are described in relation to video content, they are alternatively applied to other digital assets, such as time variable virtual environments, augmented reality environments, audio, and/or the like.

In a Receive Request Step 310 a request for replacement content is received. The request typically includes search term such as keywords to be matched with a title or tags associated with the replacement content. The request may further include a desired minimum preferred loop score to filter for preferred loops having a score equal to or greater than the desired minimum. The request may specify any of the preferred loop characteristics discussed herein. For example, a request may specify a preferred loop at least 10 seconds long, including “cats” and “cars”, in evening lighting, having a cost less than $1, and having a score in the top 5% of preferred loop scores. The request is optionally received at Media Server 110 from Client 120A via Network 115.

In an Identify Step 315 one or more preferred loops are identified based on the received request and any search criteria therein. Identify Step 315 optionally includes execution of a database query on data stored within Storage 130. The identified digital assets may include stored digital assets or a scheduled live-streaming of digital assets. In some embodiments, Identify Step 315 is facilitated by a recommendation engine.

In a Retrieve Video Step 320 one or more of the identified preferred loops are retrieved. The retrieved preferred loops may be retrieved from Storage 130, External Source 125, and/or one of Clients 120 (e.g., in a peer-to-peer relationship). The retrieved preferred loops are optionally ranked according to loop point and/or preferred loop score(s). A “loop point” score representing the quality of a loop point and a “preferred loop” score also representing other characteristics of the preferred loop, e.g., camera stability.

In an optional Modify Video Step 325 the retrieved preferred loops are modified, for example, using Video Modification Logic 150. The modification(s) can include any of the types of modifications discussed with respect to Video Modification Logic 150 and/or Dynamic Splicing Logic 185. For example, Modify Video Step 325 may include: dynamic splicing of the preferred loops, adjusting brightness and/or color of the preferred loops, changing resolution of the preferred loops, changing file type of the preferred loops, and/or matching characteristics of the preferred loops to foreground content. In a specific example, lighting within a video or a time varying virtual environment may be modified to match lighting of foreground content. Such modifications are optionally performed using Video Modification Logic 150 in real-time in response to changes in lighting or other time varying characteristics of foreground content.

Modify Video Step 325 optionally includes addition or removal of a watermark. Modify Video Step 325 optionally includes editing, transcoding, fading, interlacing, and/or dynamically splicing of the preferred loop. Modify Video Step 325 optionally includes changing the length of a preferred loop. Modify Video Step 325 optionally includes blurring of content and/or retexturing of a 3D rendering.

In some embodiments, Modify Video Step 325 includes modifying lighting of content to be used as replacement content to match lighting of content to be used as foreground content. This modification may be performed automatically or manually and may including adjusting bicolor white light at 5600 k or Tungsten light 3200 k or multicolor lighting red/blue/green etc.

In some embodiments, Modify Video Step 325 includes scaling of digital assets such that replacement content and foreground content appear more natural when combined in composited content. For example, replacement content may be scaled to better match foreground content.

In some embodiments, Modify Video Step 325 includes addition (or removal) of objects or characters to (or from) digital assets. Such objects or characters may appear with a subject in resulting composited content. In various embodiments computer-generated characters include animated objects, animals, people, heroes, branded products, advertisements, and/or the like. A user may choose to edit and place these objects such that a subject appears to be interacting with the object in composited content. In a specific example, a car or other moving object may be removed from replacement content.

In various embodiments, Modify Video Step 325 includes altering replacement content in real-time responsive to actions of a subject in foreground content. For example, position of an object may be altered if the subject appears to pick up the object. Such modification is optionally performed using a neural network trained to detect movements of the subject and embodiments of Editor 175 configured to respond to the detected movements. In another example, Video Modification Logic 150 may be used to change a point of view within replacement content including a virtual environment base on movement of a subject. For example, the replacement logic may change as a subject (person) walks through a museum and audio/video descriptions are generated to describe the replacement content to match the changing location of the subject within the replacement content. In this approach a person may experience a virtual reality museum in which the subject (e.g., the person or a tour guide) are seen traveling through the museum.

In various embodiments, Modify Video Step 325 includes addition of two- or three-dimensional surface textures to foreground Content, replacement content, and/or composited content. The surface textures may include reflections, shadows, dirt, fuzz, rain, snow or intentionally low-quality video textures, or any texture used in rendering of three-dimensional environments.

In some embodiments, Modify Video Step 325 includes changing a perspective of either the foreground content or replacement content. For example, where the replacement content includes a virtual environment, the point of view from which the virtual environment may be changed using Video Modification Logic 150.

In various embodiments, Modify Video Step 325 includes localization of digital assets. For example, replacement content may be changed based on geography or language so text or objects being displayed in the replacement content will change accordingly. In a specific example, a sign displayed in the Korean language may automatically be displayed in a local language or to a language desired by a user. In some embodiments, Modify Video Step