308, the at least one processor may determine alignment between one or more of the query proposals and the support videos. For example, if a first query proposal is “deadlifting” and the at least one processor determines that a first support video depicts deadlifting, then the at least one processor may determine that the first support video is aligned (e.g., common) to the first query proposal.

At a fifth block 310, the at least one processor may determine one or more of the query proposals that are most common to the support videos. For example, the at least one processor may classify each query proposal as either common among the support videos or not common among the support videos. If the query proposal is classified as a common, then the query proposal is detected as an action that is common with the query video, and/or is an action common between the query video and a support video. The at least one processor may also define each start and end times of the common action within the query video. For example, the at least one processor may determine one or more time windows of the query video that depict the common action.

In some examples, the fourth block 308 may include query-to-support (QtoS) context cross-attention and support-to-support temporal dynamic cross-attention, as described in more detail below.

FIG. 4 is a block diagram illustrating an example process flow 400 for common-action localization. The process flow 400 is illustrated as including three separate processes: a first process 402, a second process 404, and a third process 406. The first process 402 and the second process 404 may correspond to the fourth block 308 of FIG. 3. The third process 406 may correspond to the fifth block 310 of FIG. 3.

At the first process 402, a set of query proposals (X_Q) and an l^thsupport video

(

X
S
l

)

are inputs. Each query proposal is a potential action instance depicted in the untrimmed query video (V_Q). Using the query proposals and each support video, a cross-attention weight

(

A

Q
→
S

(
l
)

(from query to support)) is applied to each pair of query proposal and support video by determining a row-wise soft maximum of the cross-correlation matrix. For example, cross correlation may be performed using the following equation:

A

Q
→
S

(
l
)

=

X
Q
T

⁢

W

Q
→
S

⁢

X
S

(
l
)

Equation
⁢

1

Where W_Q→Sis a learnable matrix shared for all L support videos. The result are cross attenuated frame-level features

(

X
˜

S

(
l
)

)

for the l^thsupport video, where

X
˜

S

(
l
)

is defined as follows:

X
~

S

(
l
)

=

X
Q

⁢

A

Q
→
S

(
l
)

+

X
S

(
l
)

Equation
⁢

2

Accordingly,

X
˜

S

(
l
)

is obtained for all L support videos, individually

(

{

X
˜

S

(
l
)

}

l
=
1

L

)

.

FIG. 5 is a block diagram illustrating an example flow process for the first process 402 of FIG. 4. The block diagram includes the set of query proposals (X_Q) 502, the l^thsupport video

(

X
S
l

)

504, the learnable matrix (W_Q→S) 506, the cross-attention weight

(

A

Q
→
S

(
l
)

)

508 and the resulting cross attenuated frame-level features

(

X
˜

S

(
l
)

)

510.

In certain aspects, first process 402 of FIG. 4 may compute attention (e.g., degree of relevance a query proposal has to the support video) for the frames of one support video at a time. That is, instead of only a dominant segment of a first support video getting attention while other support videos are suppressed, all frames of all the support videos are attended. Accordingly, important frames of each support video are appropriately transformed to the context of the query.

Referring back to FIG. 4, the second process 404 may apply a one-dimensional temporal convolution on a frame-level. For example, tuple-level features (e.g., all support videos) and frame-level features (e.g., individual support video) can together convey a more robust video representation. That is, cross-attention can provide collaborative use of features in different temporal granularity. Also, for compatibility among different support videos (e.g., for commonality extraction), the temporal dynamics from all the support videos may be propagated to an individual support video comprising resulting attenuated support frame-level features

(

{

Y
˜

S

(
l
)

}

l
=
1

L

)

.

Thus,

Y
˜

S

(
l
)

is obtained for all the support videos, individually.

FIG. 6 is a block diagram illustrating an example flow process for the second process 404 of FIG. 4. The block diagram includes the cross-attenuated frame-level features for all the L support videos

(

{

X
˜

S
⁢
′

(
l
)

}

l
=
1

L

)

602, the cross attenuated frame-level features

(

X
˜

S

(
l
)

)

604, a learnable matrix (W_S′→S) 606, a cross-attention weight

(

A

S
⁢
′

→
S

(
l
)

)

608 and the resulting attenuated support frame-level features

(

Y
˜

S

(
l
)

)

610.

Referring back to FIG. 4, the third process 406 may use all the support videos from first process 402 and the second process 404 to attend all query proposals, at once. Thus, cross-attenuation is used to generate an enhanced query proposal ({tilde over (X)}_Q). FIG. 7 is a block diagram illustrating an example flow process for the third process 406 of FIG. 4. Here, the set of query proposals (X_Q) 702 and the attenuated support frame-level features

(

{

Y
˜

S

(
l
)

}

l
=
1

L

)

704 are input to a learnable matrix (W_S→Q) 706 with a cross-attention weight (A_S→Q) 708 applied, and the resulting enhanced query proposal ({tilde over (X)}_Q) 710. The enhanced query proposal ({tilde over (X)}_Q) 710 may be output to a relational classifier 408.

FIG. 8 is a block diagram illustrating an example of the relational classifier 408 illustrated in FIG. 4. The relational classifier 408 may include an auxiliary relational module 804 and an action classifier 806.

The auxiliary relational module 804 may take, as inputs, the enhanced query proposal (e.g., an ith enhanced query proposal 710 of FIG. 7 ({tilde over (X)}_Q,i)) concatenated with average of all the support video features (e.g., avg. of

Y
˜

S

(
l
)

610 of FIG. 6), and a pseudo action class indicator (e.g., 0/1). Pseudo action classes may not relate to a true action class even in training. Instead, the pseudo action class may be obtained by k-means clustering of the enhanced query proposal. For example, the mapped k-means cluster may be the pseudo action class of a query proposal. In some examples, the pseudo action class indicator may be a 1 or 0, wherein 1 is for a pseudo action class, and 0 otherwise. The auxiliary relational module 804 may output a prediction of whether the support videos and query proposal is similar or non-similar.

The action classifier 806 may take as an input an ith enhanced query proposal ({tilde over (X)}_Q,i) weighted by the distance between support videos and the proposed query. The action classifier 806 may output: (1) action or non-action probabilities, and (2) offsets (refinement values for start and end times of the query proposal, for correct localization). The offsets may be used when the action probability is higher than 0.5.