311. The midway point on the vertical line is the group vertical center. The group vertical size is calculated as the absolute value of the difference between the topmost coordinate of all of the buffers to the bottom most coordinate of all of the buffers (e.g., the distance between the two coordinates).

For the horizontal axis, the group frame generator calculates a group horizontal center as a center of the horizontal line and a group horizontal size as a length of the horizontal line in Block 313. The midway point on the horizontal line is the group horizontal center. The group horizontal size is calculated as the absolute value of the difference between the right most coordinate of all of the buffers to the left most coordinate of all of the buffers (e.g., the distance between the two coordinates).

In Block 315, the group frame generator calculates a ratio of group horizontal size to group vertical size. For example, the group frame generator divides the horizontal size by the vertical size. In Block 317, the group frame generator identifies the group orientation by comparing the ratio to a threshold. The threshold is based on the size of the output image. For example, if the size of the output image is 16:9, then the threshold may be set to 1.77, which is 16 divided by 9. Other thresholds may be used. If the ratio is greater than the threshold, then the group is a horizontal group. If the ratio is less than the threshold, then the group is a vertical group. The group may be deemed either or both a vertical and horizontal group when the ratio equals the threshold.

The group frame generator selects either the group vertical center or the group horizontal center based on group orientation in Block 321. If the group is a horizontal group, then the group horizontal center is selected. If the group is a vertical group, then the group vertical center is selected.

The group framing template is positioned using the group horizontal center or the group vertical center based on the orientation to obtain a positioned group framing template in Block 323. If the group orientation is horizontal, then the group horizontal center is aligned with the vertical line of the template. If the group is a vertical group, the group vertical center is aligned with the horizontal line. The zoom amount is adjusted such that the ends of the vertical line determined in Block 307 aligns with the top and bottom edges of the template in the case of a vertically oriented group. In the case of the horizontally oriented group, the zoom amount is adjusted such that the ends of the horizontal line determined in Block 313 aligns with the right and left edges of the template. In some cases, when the group is at the edges of the image, once the template is positioned as described above, the template is determined to include a portion not covered by the image. In such a scenario, the template is moved right, left, top, or bottom, so that the edge of the template is the edge of the image.

In Block 325, once positioned a buffer is added around the positioned group framing template to generate a group frame definition. The group frame definition then specifies the location within the image of the positioned template and the zoom level to generate the group frame.

In Block 327, the video stream is outputted using group frame definition. The group frame definition is passed to the image framing processor that adjusts the images according to the group frame definition and sends the images to the downstream device.

In Block 329, the image framing processor monitors the video stream for refreshing. Specifically, the image framing processor continues to extract images from the video stream to detect events in the video stream. For example, if the event is that a member of the group moved outside of the group frame definition, then a corresponding timer for the movement of the member is initiated responsive to the detection. The amount of the corresponding timer is set such to account for the fact that the member may immediately return and to avoid flickering of the image. If the member does not return within the elapse of the time, a reframing of the group frame definition is triggered responsive to not detecting the member in the group frame at an elapse of a threshold amount of time according to the timer.

As another example, the event may be that a group size decreases, such as the horizontal or vertical size of the group. Namely, the members of the group may get closer to each other. A corresponding timer is then triggered. If the group size does not increase, then a reframing is triggered responsive to the group size remaining decreased at an elapse of a threshold amount of time according to the timer.

As another example, the detection may be that a member of the group leaves the room. Namely, the member may be out of the area captured by the image. A corresponding timer is initiated. If the member does not return, then a reframing is triggered responsive to not detecting the member in the room at an elapse of a threshold amount of time according to the timer.

FIGS. 4-11 show an example in accordance with one or more embodiments. FIGS. 4-11 show examples of group framing of a horizontal oriented group and a vertical oriented group. Group framing is the framing in a conference room. The goal is to obtain meeting equity in a hybrid meeting. The meeting equity is for the people attending in the room by expanding the group and focusing on the group. Thus, people attending in the room are increased in size after group framing as compared to prior to group framing, such that the people attending in the room are closer to the size of the meeting participants outside of the room.

Turning to FIG. 4, in a conference room, people may be located in any location of the conference room including standing, sitting or in another position. Further, the faces of people may be directed at the camera, away from the camera, or any angle between facing the camera and away from the camera. In the example, the input images (402, 404) are different images extracted from different video streams. Both of the examples correspond to live conference calls in which people in the conference room are interacting with people outside of the conference room through conferencing devices. A camera captures the input stream of the conference room. The housing of the camera includes the video processor that both processes the video stream and performs the group framing on the video stream. Input image (402) shows people on either side of the conference room spread out horizontally while input image (404) shows people in the conference room near each other, but vertically spaced. The subsequent figures describe the analysis of the images of FIG. 4 to perform the group framing.

Turning to FIG. 5, the head detection model identifies the heads of individual people in the conference room as shown in images (502, 504). The head detection model outputs a head bounding box as shown by the solid line squares (e.g., bounding box (506)). A head buffer around the head bounding box is added as shown by dashed line boxes (e.g., buffer (508)). The head buffer is defined so as to include a person's full head. In the example, the head buffer increases the head bounding box by 50%. The increased region that includes the head bounding box with the head buffer is referred to as head buffer bounding box and is the region of the dashed line box (e.g., buffer (508)).

FIG. 6 shows a group frame template (600). The group frame template (600) includes a vertical line (i.e., Y-Line (602)) and a horizontal line (i.e., X-Line (604)), which are defined at particular positions in the template and are orthogonal to the edges of the template that the lines intersect. The size of the group frame template is based on the size of the output image. In the example, the group frame template has a 16:9 ratio of the horizontal size to the vertical size. The ratio may be based on the expected ratio for the output image. The 16:9 is only an example and other sizes may be used without departing from the scope of the technology.

The vertical line (612) and the horizontal line (614) are defined based on which portions to add to the group frame. In the example, the vertical line is down the center of the rectangle to focus the output image on the horizontal center of the group. The horizontal line is located ⅓ of the vertical length of the template from the top of the template (600). The purpose of the ⅓ rather than center is to show more of the people (e.g., torso/legs) than to show unused areas. Specifically, because the matching to the template uses the head bounding boxes which is at the top of a persons' body, if the horizontal line (604), were at the center, then more ceiling and area above participants' heads would be shown. The centering of the vertical line and the positioning of the horizontal line in the template of FIG. 6 is only an example. Other positions may be used. For example, the horizontal line may be at any position above the center.

Turning to FIG. 7, FIG. 7 shows the example of the groups of FIG. 4 with the bounding boxes and determining whether the groups are wide or narrow. As shown in FIG. 7, for both images (702, 720), a horizontal line (706, 728) is defined from the left edge of the left most buffer to the right edge of the right most buffer. Halfway along the horizontal line (706, 728) is the group horizontal center (708, 726). Similarly, a vertical line (704, 722) is defined from the top edge of the topmost buffer to the bottom edge of the bottom most buffer. Halfway along the vertical line (704, 722) is the group vertical center (710, 724).

Using the vertical and horizontal lines, a determination is made whether the group layout is wide, or the group layout is narrow. Generally, the determination as to whether the group is wide or narrow is a determination of the distribution of people in the room.

The vertical distance between the top edge of the top-most head buffer bounding box and the bottom edge of the bottom-most head buffer bounding box (e.g., length of vertical line (704, 722)) is calculated. The horizontal distance between the left edge of the left-most head buffer bounding box and the right edge of the right-most head buffer bounding box (e.g., length of horizontal line (706, 728)) is calculated. The horizontal distance is divided by the vertical distance to obtain a value. If the value is greater than a threshold, then the group layout is wide. If the value is less than a threshold, then the group layout is narrow. In FIG. 7, because the template is size 16:9, the threshold is 16/9 or around 1.77. Other thresholds may be used.

Although the edges of the head buffer bounding boxes are used, other locations may be used. For example, the edges of the head bounding boxes may be used. In another scenario, if face detection or body detection is performed, then the face bounding boxes/body bounding boxes may be used.

In image (702), the length of the horizontal line (706) is 2.7 and the length of the vertical line (704) is 0.69. 2.7 divided by 0.69 is 3.9. Because 3.9 is greater than 1.77, the group orientation is wide or horizontally oriented. In contrast, in image (720), the length of the horizontal line (728) is 1.53 and the length of the vertical line (722) is 1.16. 1.53 divided by 1.16 is 1.31. Because 1.31 is less than 1.77, the group orientation is narrow or vertically oriented.

FIG. 8 shows the example of positioning the group framing template, the group framing template is placed as shown in the two images (800, 820) to touch the outer edges of the outer most head buffer bounding boxes. As shown, the template is placed so that the opposite edges of at least two sides of the template touches the outermost edges of at least two of the head buffer bounding boxes. Thus, all of the head buffer bounding boxes are within the template. Zooming in of the input image or resizing the template proportionally may be performed.

Next, further positioning of the group framing template (808, 828) is performed as shown in images (802, 822) based on whether the group layout is wide or narrow. If the group layout is wide as shown in image (802), the horizontal line is placed as close as possible to the group vertical center (GVC). The result is the positioning (808) of the template.

If the group layout is narrow, the vertical line is placed as close as possible to the group horizontal center (GHC). The result is the positioning (828) of the template.

If the group is neither wide nor narrow, then the center may be performed as if the group were either wide or narrow. As another example, both operations may be performed. As another example, the template may be applied and the zooming may be performed.

FIG. 9 shows adding a buffer to the positioned template (906) of image (900). As shown in image (902) of FIG. 9, a buffer is added to create a group frame (908). The buffer may be equal on each side or different on each side as shown in FIG. 9. In the example, 10% of the vertical distance added as a top buffer while 30% of the vertical distance is added as a bottom buffer. 20% of the horizontal distance is added as a buffer on the left and right side. The result as shown in image (904) is the group frame in the output video stream (910).

FIG. 10 shown examples of when to reframe based on a person's movement in the video stream. The top row is the original image. The bottom row is the resulting group frame. The participants move in the conference room from left to right.

Time (1000) shows the initial image with the initial framing Δt time (1002), the participants move out of the group frame and remain in the image region captured by the camera. The reframing timer starts and after a first threshold amount of time (e.g., 8 seconds), reframing is performed as shown in the bottom row.

At time (1004), the participants move closer together. The reframing timer starts and after a second threshold amount of time (e.g., 15 seconds), reframing is performed as shown in the bottom row.

At time (1006), a participant leaves the room. The reframing timer starts and after a third threshold amount of time (e.g., 15 seconds), reframing is performed as shown in the bottom row.