226 takes aggregated audio signals at frame level and computes the loudness (e.g., in dBs). A frame is evaluated to be silent if the loudness level if the loudness level of the frame falls beneath a predetermined loudness threshold.

Ad-break prediction ML model 228 is configured to perform a frame-by-frame evaluation of media content 256 based on the outputs of black frame detection ML model 224 and silent frame detection ML model 226 to predict the presence of one or more candidate ad-insertion points and a respective probability score associated with each of those one or more candidate ad-insertion points to provide one or more ad-break predictions 258B for media content 256 including ad-slugs. If several consecutive frames are determined to be both black and silent, they may be determined to belong to an ad-break and the FFCB and LFCB markers may be applied at the timestamps associated with the first and last frames of the predicted ad-break.

Evaluation and probability scoring of candidate ad-insertion points may be determined by the machine-learned statistical distribution of previously human reviewer approved segments in audio and video training data. Applying statistically learned thresholds in the audio and video signals, ad-break prediction ML model 228 can identify outlier candidate ad-insertion points that may require human review and can pass those candidates to a Quality Assurance (QA) flow for feedback and future learning. Additionally, if no ad-slugs are detected using AI-based pipeline for ad-slugged content 220, media content 256 may be automatically provided to AI-based pipeline for seamless content 230 for further evaluation.

FIG. 2C shows a more detailed exemplary diagram of AI-based pipeline for seamless content 230, according to one implementation. As shown in FIG. 2C, AI-based pipeline for seamless content 230 may include component separation module 222, black frame detection ML model 224, video transition ML model 232, scene detection or collection module 234, establishing scene detection or collection module 236, audio transition ML model 238 and seamless content ad-break prediction ML model 246. Also shown in FIG. 2C are media content 256, one or more ad-break predictions 258C for seamless content, video component 223 of media content 256, and audio component 225 of media content 256.

It is noted that one or more ad-break predictions 258C for seamless content correspond in general to one or more ad-break predictions 158/258 in FIGS. 1 and 2A. Consequently, one or more ad-break predictions 258C for seamless content may share any of the characteristics attributed to one or more ad-break predictions 158/258 by the present disclosure, and vice versa. It is further noted that any features identified in FIG. 2C by reference numbers identical to that/those shown in FIG. 2A correspond respectively to those previously shown and described features. Thus, although not shown in FIG. 2A, AI-based pipeline for seamless content 230 in that previous figure may include features corresponding respectively to component separation module 222, black frame detection ML model 224, video transition ML model 232, scene detection or collection module 234, establishing scene detection or collection module 236, audio transition ML model 238, and seamless content ad-break prediction ML model 246.

As noted above, component separation module 222 is configured to receive media content 256, to separate video component 223 from audio component 225, and to output video component 223 to black frame detection ML model 224. However, in contrast to AI-based pipeline for ad-slugged content 220 shown in FIG. 2B, in AI-based pipeline for seamless content 230, component separation module 222 outputs audio component 225 of media content 256 to audio transition ML model 238.

As further noted above, black frame detection ML model 224 may utilize frame-level hashing and brightness checks to determine if a frame is both uniform in color and falls beneath a predetermined brightness threshold. It is noted that for media content 256 that does not include ad-slugs, extra signals to help detect natural breaks within the narrative flow of media content 256 are needed. AI-based pipeline for seamless content 230 is configured to use content metadata including scenes (either manually annotated, i.e., collected, or algorithmically assigned by a scene detection ML model), establishing shots (either manually annotated, i.e., collected, or algorithmically assigned by an establishing scene detection ML model), along with ML model identified video and audio transitions.

It is noted that in the context of video, a “shot” refers to a sequence of video frames that is captured from a unique camera perspective without cuts and other cinematic transitions. A “scene” refers to a sequence of shots that correspond to one another thematically, either by sharing a common background or location, or by being directed to the same event, i.e., series of actions. An establishing shot may be used to signal the beginning of a new scene. For example, where the action to be depicted in a scene are to take place in a particular venue, such as a building or outdoor location for example, an establishing shot may provide a wide angle view of the venue and its environs, prior to subsequent shots of actions within the venue. Thus the time stamp or frame number of an establishing shot may indicate a natural ad-break preceding the establishing shot, while time stamps or frame numbers between establishing shots may be unlikely to provide good candidates for ad-breaks.

Regarding video transition ML model 232, it is noted that ideal ad-breaks typically occur around direct transitions to black (hard transitions) or fade-to-black transitions (soft transitions). The first and last frames around hard transition segments are easier to detect. Soft transitions can pose a problem because their fade/dissolve sequence can make it difficult to define the beginning and end of the fade frame sequence. Video transition ML model 232 is configured to check for soft and hard transitions as a supplement to black frame detection ML model 224. For example, video transition ML model 232 may be implemented as a Contrastive Language-Image Pre-Training (CLIP) model receiving the output of black frame detection ML model 224.

Audio transition ML model 238 is configured to ensure that expected audio breaks exhibit enough discontinuity to signal a natural transition point within audio component 225. Audio transition ML model 238 in the form of an NN, for example, may make comparisons between decomposed audio signals (e.g., in the form of spectrograms) taken before and after candidate ad-insertion points and evaluate the two signals for discontinuity. Discontinuity in the audio signal before and after the candidate ad-insertion point indicates that neither speech nor music are interrupted by the candidate ad-insertion point. Audio transition ML model 238 may also check for characteristic patterns in sound levels that indicate a purposeful decrease and subsequent increase in volume around the candidate ad insertion point.

Candidate ad-insertion points are evaluated by seamless content ad-break prediction ML model 246 based on the data and metadata described above by reference to black frame detection ML model 224, video transition ML model 232, and audio transition ML model 238, and in some implementations scene detection or collection module 234 and establishing scene detection or collection module 236 as well. Scene and establishing shot information help to ensure that scenes are not interrupted by ad-breaks and that the audio and visual transitions occur at the start and end of an ad-break. Evaluation and probability scoring of candidate ad-insertion points may be determined by the machine-learned statistical distribution of previously human reviewer approved segments in audio and video training data.

FIG. 2D shows a more detailed exemplary diagram of AI-based pipeline for seamless content with chapter markers 240, according to one implementation. It is noted that like AI-based pipeline for seamless content 230, AI-based pipeline for seamless content with chapter markers 240 includes component separation module 222, black frame detection ML model 224, video transition ML model 232, scene detection or collection module 234, establishing scene detection or collection module 236, and audio transition ML model 238. However, in addition to those features, AI-based pipeline for seamless content with chapter markers 240 further includes chapter alignment ML model 238 and seamless content with chapter markers ad-break prediction ML model 248. Also shown in FIG. 2D are media content 256, one or more ad-break predictions 258D for seamless content with chapter markers, video component 223 of media content 256, and audio component 225 of media content 256.

It is further noted that one or more ad-break predictions 258D for seamless content with chapter markers correspond in general to one or more ad-break predictions 158/258 in FIGS. 1 and 2A. Consequently, one or more ad-break predictions 258D for seamless content with chapter markers may share any of the characteristics attributed to one or more ad-break predictions 158/258 by the present disclosure, and vice versa. It is also noted that any features identified in FIG. 2D by reference numbers identical to that/those shown in FIGS. 2A and 2C correspond respectively to those previously shown and described features. Thus, although not shown in FIG. 2A, AI-based pipeline for seamless content with chapter markers 240 in that previous figure may include features corresponding respectively to component separation module 222, black frame detection ML model 224, video transition ML model 232, scene detection or collection module 234, establishing scene detection or collection module 236, audio transition ML model 238, chapter alignment ML model 242, and seamless content with chapter markers ad-break prediction ML model 248.

Chapter alignment ML model 242 is configured to detect and correct errors in chapter alignment. Candidate ad-insertion points are evaluated by seamless content with chapter markers ad-break prediction ML model 248 based on the data and metadata described above by reference to black frame detection ML model 224, video transition ML model 232, scene detection or collection module 234, establishing scene detection or collection module 236, audio transition ML model 238, and also based on the location of chapter markers. As noted above, scene and establishing shot information ensure that scenes are not interrupted ad-breaks and that the audio and visual transitions occur at the start and end of an ad-break. Candidate ad-insertion points may be evaluated based on machine-learned characteristics using human reviewer approved training data.

Moreover, because chapter markers identify timestamps or frames deemed by creative authorities of media content 256 to be suitable as transition points in the narrative flow of media content 256, in use cases in which media content 256 includes chapter markers, each of the candidate ad-insertion points identified using ad-break prediction ML model 248 may be required to coincide with a chapter marker. That is to say, candidate ad-insertion points that fail to coincide with chapter markers may be prohibited.

The functionality of AI system 100 and software code 110/210 will be further described by reference to FIG. 3. FIG. 3 shows flowchart 370 presenting an exemplary method for performing AI-based ad-break prediction, according to one implementation. With respect to the method outlined in FIG. 3, it is noted that certain details and features have been left out of flowchart 370 in order not to obscure the discussion of the inventive features in the present application.

Referring now to FIG. 3 in combination with FIGS. 1, 2A, 2B, and 2C, flowchart 370 begins with: receive media content 156/256 including video component 223 and audio component 225 (action 371). As noted above, media content 156/256 may include AV content in a variety of forms, such as episodic TV content or movie content, for example. As further noted above, in addition, or alternatively, in some implementations, media content 156/256 may be or include digital representations of persons, fictional characters, locations, objects, and identifiers such as brands and logos, for example, which populate a VR, AR, or MR environment. Moreover, media content 156/256 may depict virtual worlds that can be experienced by any number of users synchronously and persistently, while providing continuity of data such as personal identity, user history, entitlements, possessions, payments, and the like. Moreover, and as also noted above, in some implementations, media content 156/256 may be or include a hybrid of traditional AV and fully immersive VR/AR/MR experiences, such as interactive video.

In some use cases, as noted above, media content 156/256 may be received from media content source 154, while in other use cases media content 156/256 may be received from user device 160 utilized by user 164. Media content 156/256 may be received, in action 371, by software code 110/210, executed by hardware processor 104 of AI system 100.

Flowchart 370 further includes: recognize whether media content 156/256 is ad-slugged media content or seamless media content (action 372). As described above, ad-slugged media content refers to media content that includes one or more pre-existing sequences of black frames free of audio and usable as ad-breaks, while seamless media content refers to media content that is entirely free of ad-slugs and other predetermined indicators of narrative transition. In some implementations. the recognition of media content as ad-slugged media content or seamless media content may be performed based on sampling or otherwise analyzing media content 156/256 for sequences of black frames. Alternatively, in some implementations, media content 156/256 may include metadata identifying media content 156/256 as ad-slugged media content or seamless media content. Action 372 may be performed by software code 110/210, executed by hardware processor 105 of AI system 100, and using content recognition module 212.

Flowchart 370 further includes: detect black video frames of media content 156/256 (action 373). As described above by reference to FIGS. 2B and 2C. black frame detection ML model 224 may be configured to utilize frame-level hashing, and brightness checks to detect black frames. That is to say, detecting black frames of media content 156/256 in action 373 may include performing a frame-level hashing of video component 223 of media content 156/256 to provide a respective hash for each video frame, and evaluating, using the respective hash, a brightness of each video frame relative to a predetermined brightness threshold.

In some implementations, the detection of black frames of media content 156/256 in action 373 may further include evaluating, using the respective hash of each video frame, a color uniformity of each video frame relative to a predetermined color uniformity threshold. It is noted that typically all black frames included in media content 156/256 will satisfy such a predetermined color uniformity threshold. In implementations in which media content 156/256 is recognized as ad-slugged content in action 372, the detection of black frames in media content 156/256 may be performed by software code 110/210. executed by hardware processor 104 of AI system 100, and using black frame detection ML model 224 of AI-based pipeline for ad-slugged content 220. Alternatively, in implementations in which media content 156/256 is recognized as seamless content in action 372, the detection of black frames in media content 156/256 may be performed by software code 110/210, executed by hardware processor 104 of AI system 100, and using black frame detection ML model 224 of AI-based pipeline for seamless content 230.

Flowchart 370 may further include: detect, when media content 156/256 is ad-slugged media content, silent video frames of the ad-slugged media content (action 374). As described above by reference to FIG. 2B, silent frame detection ML model 226 may be configured to take aggregated audio signals at frame level and to compute the loudness (e.g., in dBs). A frame is evaluated to be silent if the loudness level of the frame falls beneath a predetermined loudness threshold. Thus, detecting silent frames of media content 156/256 in action 374 may include aggregating audio signals of audio component 225 of media content on a per video frame basis to provide a respective frame-level audio signal for each video frame, and evaluating, using the respective frame-level audio signal, the loudness of each frame-level audio signal relative to the predetermined loudness threshold. Detection of silent frames in media content 156/256 may be performed by software code 110/210, executed by hardware processor 104 of AI system 100, and using silent frame detection ML model 226 of AI-based pipeline for ad-slugged content 220.

Flowchart 370 may further include: identify, using the black video frames and the silent video frames when media content 156/256 is ad-slugged media content, one or more candidate ad-insertion points for the ad-slugged media content and a respective probability score associated with each of the one or more candidate ad-insertion points, to provide one or more ad-break predictions 158/258/258B for the ad-slugged media content (action 375). As described above by reference to FIG. 2B, ad-break prediction ML model 228 is configured to perform a frame-by-frame evaluation of media content 156/256 based on the outputs of black frame detection ML model 224 and silent frame detection ML model 226 to predict the presence of one or more candidate ad-insertion points and a respective probability score associated with each of those one or more candidate ad-insertion points. If several consecutive frames are determined to be both black and silent, they may be determined to belong to an ad-break in the form of an ad-slug.

Evaluation and probability scoring of candidate ad-insertion points may be determined by the machine-learned statistical distribution of previously human reviewer approved segments in audio and video training data. Applying statistically learned thresholds in the audio and video signals, ad-break prediction ML model 228 can identify outlier candidate ad-insertion points that may require human review and can pass those candidates to a QA flow for feedback and future learning. However, when media content 156/256 is ad-slugged media content including multiple predetermined ad-slugs. ad-break predictions 158/258/258B for the ad-slugged content will typically each identify a respective one of the predetermined ad-slugs as a candidate ad-insertion point, and the respective probability scores associated with those candidate insertion points will usually be approximately the same. For example, where the probability score is expressed as a percentage, with one hundred percent signifying complete certainty and zero percent signifying complete uncertainty, the probabilities associated with predetermined ad-slugs identified by AI-based pipeline for ad-slugged content 220 can be expected to be clustered within a range of a few percent, such as up to ten percent for example.

As noted above, if no ad-slugs are detected using AI-based pipeline for ad-slugged media content 220, media content 156/256 may be automatically provided to AI-based pipeline for seamless content 230 for further evaluation. The one or more ad-break predictions provided in action 375 may be generated by software code 110/210, executed by hardware processor 104 of AI system 100, and using ad-break prediction ML model 228 of AI-based pipeline for ad-slugged content 220.

It is noted that actions 374 and 375 are performed only in use cases in which media content 156/256 is recognized as ad-slugged media content in action 372. In use cases in which media content 156/256 is recognized as being seamless media content in action 372, actions 374 and 375 may be omitted, and the method outlined by flowchart 370 may move directly from action 373 to: perform evaluations of blackness transitions between sequential video frames, and performing one or more evaluations of audio continuity across respective one or more sequences of black video frames (action 376).

Referring to FIGS. 1, 2A, 2C, and 3 in combination, the evaluations of blackness transitions between sequential video frames occurring in action 376 may be performed by software code 110/210, executed by hardware processor 104 of AI system 100, and using video transition ML model 232 of AI-based pipeline for seamless media content 230. Regarding video transition ML model 232, as noted above, ideal ad-breaks typically occur around direct transitions to black (hard transitions) or fade-to-black transitions (soft transitions). Video transition ML model