The information determination unit 122 is configured to track the target object and determine the current position of the target object in the scene in a case that the target object is present. That is, in the case where it is determined that the target object is present, the target object is tracked. It may be understood that the camera is configured to acquire the scene picture in real time, and locate the two-dimensional coordinate point of the target object in the scene coordinate system in real time, so as to realize the target trajectory tracking.

By way of example, FIG. 2 is a schematic diagram illustrating trajectory tracking according to an embodiment of the present disclosure. As shown in FIG. 2, a previous position H1 (hx1, hy1) is within (0,0) to (cx, cy), cx is a maximum value of the abscissa of the scene picture in a scene coordinate system, and cy is a maximum value of the ordinate of the scene picture in the scene coordinate system. When an observer moves, a start point is H1 (hx1, hy1), an end point is H2(hx2, hy2), that is, the current position H2. At this time, a displacement hxm=hx2−hx1 is obtained, and hym=hy2−hy1. The end point here is a position point in the tracking process, and is not a start position point of the target object.

The information determination unit 122 is configured to determine a current coordinate point of the current position mapped to an interface coordinate system of a user interface according to the current position of the target object in the scene and a mapping relationship between a pre-established scene coordinate system and the interface coordinate system. Here, a storage unit in the system control module 102 stores the mapping relationship between the pre-established scene coordinate system and the interface coordinate system of the user interface. The information determination unit 122 is specifically configured to map a two-dimensional coordinate point of the current position in the scene coordinate system to the interface coordinate system according to the mapping relationship between the scene coordinate system and the interface coordinate system after determining the current position of the target object in the scene, and determine the current coordinate point mapped to the interface coordinate system.

Continuing with FIG. 2, a visual window of a screen has a maximum abscissa of xm and a maximum ordinate of ym. The information determination unit 122 is configured to determine an offset amount of movement (vxm, vym) according to the displacement hxm=hx2−hx1 and hym=hy2−hy1 of the target object, and determine the current coordinate point V(vx, vy) according to the offset amount (vxm, vym) and the previous coordinate point. Here, vxm/xm=hxm/cx, vym/ym=hym/cy.

The update information generation unit 123 is configured to generate the update information of the interface components in the user interface according to the current coordinate point and the predefined auxiliary parameter information. The current coordinate point may be understood as a reference point that enables the interface components in the user interface to move. The auxiliary parameter information is predefined parameter information used for assisting the current coordinate point to update states of the interface components, and is used for representing the association relationship between the current coordinate point and each interface component in the user interface. The details of the auxiliary parameter information may refer to the following description of a pre-processing unit, and thus is not detailed here. The update information of the interface components here may be, for example, movement information and/or scaling information of the interface components, where the movement information may include, for example, a movement trajectory or an end coordinate point, and the like; the scaling information may include, for example, a scaling ratio, a scaling reference point, and the like. The movement trajectory, the end coordinate point, the scaling ratio and the scaling reference point here are all information in the interface coordinate system.

The display module 103 is configured to display the interface components according to the update information. For example, the interface components with updated positions are displayed. The position update may be a movement and/or a zoom of the interface components within the visual window or the like.

In the above units, the real-time interaction between each interface component in the user interface and the target object is realized by constructing the association relationship between the current position of the target object and the current coordinate point of the user interface and by utilizing the auxiliary parameter information (the association relationship between the current coordinate point and each interface component in the user interface), the 3D interaction effect is simulated through the 2D display mode and the interaction experience of the user is improved.

In some embodiments, the target object includes a person. The detection unit 121 is specifically configured to identify at least part of the features of the human body and determine whether the target object is present. In particular, some features may be selected that are capable of representing a higher intelligibility of the person, such as a facial feature (including a feature of five sense organs), a head feature, a torso feature, or the like.

Optionally, the detection unit 121 is specifically configured to perform face recognition on the scene picture, and determine whether the target object is present. Specifically, the face features in the scene picture are identified, it is judged whether a picture feature meeting the face features is present or not. If the picture feature meeting the face features is present, it is determined that the target object is present, and otherwise, it is determined that the target object is not present.

The current position includes a target key point. The information determination unit 122 is specifically configured to track the target object and determine a target key point in the face according to a face feature of the target object in a case where it is determined through the face recognition that the target object is present. For example, in the case where it is determined that the target object is present, the two-dimensional coordinate point of the target object in the scene coordinate system is located in real time, thereby achieving the target trajectory tracking. Meanwhile, the target key point in the human face is determined according to the face features of the target object, such as the feature of the five sense organs, specific face bones or the like. The target key point may be, for example, a two-dimensional coordinate point of the detected five sense organs in the scene coordinate system, a two-dimensional coordinate point of a bone key point in the scene coordinate system, or the like.

The information determination unit 122 is specifically configured to determine a current coordinate point corresponding to the target key point according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system. For example, according to the mapping relationship between the scene coordinate system and the interface coordinate system, the target key point in the scene coordinate system is mapped into the interface coordinate system, and the mapped current coordinate point is determined.

The predefined auxiliary parameter information includes a predefined auxiliary locating point on the interface coordinate system and a dimensional scaling factor of each interface component. The update information generation unit 123 is configured to generate update information of the interface components according to the current coordinate point, the auxiliary locating point, and the dimensional scaling factor of each interface component.

Here, the auxiliary locating point is a two-dimensional coordinate point on the interface coordinate system, and is used for assisting in locating the position of each interface component in the user interface. For example, the auxiliary locating point A′ is used for assisting in locating the position of the interface component 31, and the auxiliary locating point B′ is used for assisting in locating the position of the interface component 31 and/or the interface component 32. The dimensional scaling factor of each interface component may be determined based at least on an initial position of each interface component in the user interface and the auxiliary locating point. The dimensional scaling factor of each interface component is used for assisting in locating the position of each interface component.

In some embodiments, the interface component includes at least one of a control, a status bar, a card bar, a background image. The control may be a software entry in a UI interface, and is usually presented in the UI interface in the form of a software icon; the status bar may be used for displaying a status of a currently opened window or software in the UI interface, for example, the status bar includes a status of a plurality of opened folders, or a status of other software applications, or may be a bar for prompting information, such as a “screenshot” bar, a “data” bar, or the like; the card bar may be a window in the UI interface for displaying picture or text information. The background image is, for example, a wallpaper in the UI interface, etc.

In some embodiments, the system control module 102 further includes a first pre-processing unit; the first pre-processing unit is configured to determine an auxiliary locating point on the interface coordinate system according to an initial coordinate point of an initial position of the tracked target object mapped to the interface coordinate system and an initial position of each interface component in the user interface. The dimensional scaling factor of each interface component is determined according to the auxiliary locating point, the initial coordinate point and the initial position of each interface component. Here, the initial position is a position of the target object in the scene coordinate system when the detection unit 121 detects the target object for the first time. A position of the initial position mapped to the interface coordinate system is the initial coordinate point. The initial position of each interface component initialized in the user interface is known.

FIG. 3a is a schematic diagram illustrating a positional relationship between auxiliary locating points and interface components on an interface coordinate system according to an embodiment of the present disclosure. As shown in FIG. 3a, coordinates of vertices of a visual window in the user interface are known, that is, A(0, 0), B(0, ym), C(xm, 0), and D(xm, ym). Here, the visual window is a screen window of the UI interface that may be viewed by the user. The initial coordinate point is V0. An initial position of the interface component 31 (e.g., the card bar) includes coordinates of four vertices denoted as M1, M2, M3, and M4, respectively; an initial position of the interface component 32 (e.g., status bar) includes coordinates of four vertices denoted as F1, F2, F3, and F4, respectively. An auxiliary locating point A′ is used for assisting in locating the interface component 31. Specifically, the auxiliary locating point A′ is an intersection point of an extension line of a line connecting the initial coordinate point V0 and the point M1 and a line segment AB. An auxiliary locating point B′ is used for assisting in locating the interface components 31 and 32, and is an intersection point of an extension line of a line connecting the initial coordinate point V0 and the point F1 and the line segment AB, and is an intersection point of an extension line of a line connecting the initial coordinate point V0 and the point M3 and the line segment AB. An auxiliary locating point C′ is used for assisting in locating the interface component 31. Specifically, the auxiliary locating point C′ is an intersection point of an extension line of a line connecting the initial coordinate point V0 and the point M2 and a line segment CD. An auxiliary locating point D′ is used for assisting in locating the interface components 31 and 32, and is an intersection point of an extension line of a line connecting the initial coordinate point V0 and the point F2 and the line segment CD, and is an intersection point of an extension line of a line connecting the initial coordinate point V0 and the point M4 and the line segment CD. At the same time, V0, F3 and B are on a same straight line, and V0, F4 and D are on a same straight line.

A dimensional scaling factor of the interface component 31 is a ratio of A′M1 to A′V0, also C′M2 to C′V0, also B′M3 to BV0, also D′M4 to D′V0, all denoted as m. Here, m=B′M3/BV0=A′M1/A′V0=C′M2/C′V0=D′M4/D′V0. A dimensional scaling factor of the interface component 32 is a ratio of B′F1 to B′V0, also D′F2 to D′V0, also BF3 to BV0, also DF4 to DV0, all denoted as f. Here, f=B′F1/B′V0=D′F2/D′V0=BF3/BV0=DF4/DV0.

The update information generation unit 123 is specifically configured to generate the update information of each interface component according to the current coordinate point, the auxiliary locating point, and the dimensional scaling factor of each interface component. By way of example, FIG. 3b is a schematic diagram illustrating a positional relationship between a current coordinate point and interface components on an interface coordinate system according to an embodiment of the present disclosure. As shown in FIG. 3b, the current coordinate point V(vx, vy), the auxiliary locating point A′(0, a), the auxiliary locating point B′(0, b), the auxiliary locating point C′(xm, a), the auxiliary locating point D′(xm, b), the dimensional scaling factor m of the interface component 31, and the dimensional scaling factor f of the interface component 32 are known, update information of the interface component 31 (that is, updated M1, M2, M3, and M4), and update information of the interface component 32 (that is, updated F1, F2, F3, and F4) are determined. The updated M1, M2, M3 and M4 are as follows:

M
⁢
1

=

[

m
×
vx

,

a
+

m
/

(

vy
-
a

)

]

M
⁢
2

=

[

xm
-

m
×

(

xm
-
vx

)

,
 

a
-

m
×

(

a
-
vy

)

]

M
⁢
3

=

[

m
×
vx

,
 

b
-

m
×

(

b
-
vy

)

]

M
⁢
4

=

[

xm
-

m
×

(

xm
-
vx

)

,
 

b
-

m
×

(

b
-
vy

)

]

The updated F1, F2, F3, and F4 are as follows:

F
⁢
1

=

[

f
×
v
⁢
x

,
 

b
-

f
×

(

b
-

v
⁢
y

)

]

F
⁢
2

=

[

xm
-

f
×

(

x
⁢
m

-

v
⁢
x

)

,
 

b
-

f
×

(

b
-

v
⁢
y

)

]

F
⁢
3

=

[

f
×
v
⁢
x

,
 

y
⁢
m

-

f
×

(

y
⁢
m

-

v
⁢
y

)

]

F
⁢
4

=

[

xm
-

f
×

(

x
⁢
m

-

v
⁢
x

)

,
 

y
⁢
m

-

f
×

(

y
⁢
m

-

v
⁢
y

)

]

Here, a, b, f, and m are constants, and may be set according to design requirements, which are not particularly limited in the embodiment of the present disclosure.

It should be noted that, at any time, A (0, 0), B (0, ym), C (xm, 0), D (xm, ym), A′ (0, a), B′ (0, b), C′ (xm, a), and D′ (xm, b) are fixed, V, M1 and A′ are located on a same straight line, V, M2 and C′ are located on a same straight line, V, M3 and B′ are located on a same straight line, V, M4 and D′ are located on a same straight line, V, F1 and B′ are located on a same straight line, V, F2 and D′ are located on a same straight line, V, F3 and B are located on a same straight line, and V, F4 and D are located on a same straight line. Therefore, the positions of the interface components are continuously updated as the current coordinate point V is continuously changed.

In the units, the interface components may move as the target object moves by utilizing the association relationship between the target object and the current coordinate point and the association relationship between the current coordinate point and each interface component, thereby realizing a dynamic interaction between the target object and each interface component.

In some embodiments, FIG. 4a is a schematic front view of a user interface according to an embodiment of the present disclosure; FIG. 4b is a schematic diagram of a layered user interface according to an embodiment of the present disclosure. As shown in FIG. 4a and FIG. 4b, the user interface includes layers using the same interface coordinate system; the user interface including the layers includes a foreground layer 401 and a background layer 403, and at least one intermediate layer 402 located between the foreground layer 401 and the background layer 403; a plurality of interface components are included, such as a status bar 41, a card bar 42, and a background image 43, as shown.

The system control module 102 further includes an interface control unit; the interface control unit is configured to control the display module 103 to display different interface components in the foreground layer, the middle layer, and the background layer. For example, as shown in FIG. 4b, the interface control unit is configured to control the display module 103 to display the status bar 41 in the foreground layer, the card bar 42 in the middle layer, and the background image 43 in the background layer. In the embodiment of the present disclosure, the user interface is layered, which creates the vertical feeling of the picture through the parallax principle, and can present a relatively real 3D interface display effect for the user.

In some embodiments, in order to further improve the user interaction experience, when the user selects the interface component in the middle layer, the user interface display system may further adjust the interface component in the middle layer to the foreground layer for display. Specifically, the currently selected interface component is zoomed into display, so as to highlight the interface component currently selected by the user.

One possible implementation is that the update information generation unit 123 is specifically configured to receive a component selection instruction for selecting an interface component, and acquire a predefined dimensional scaling factor of each interface component; replace the dimensional scaling factor of the currently selected interface component with the dimensional scaling factor of the interface component positioned in the foreground layer in the case where the currently selected interface component is not positioned in the foreground layer of the user interface; and generate the update information of each interface component according to the current coordinate point, the replaced dimensional scaling factor and the auxiliary locating point on the predefined interface coordinate system.

By way of example, FIG. 5a is a schematic diagram of interface components without being updated according to an embodiment of the present disclosure; FIG. 5b is a schematic diagram of the updated interface components of FIG. 5a. As shown in FIGS. 5a and 5b, the component selection instruction may be an instruction of one interface component located in the middle layer selected by the user, for example, when the card bar 42 is selected, the dimensional scaling factor m of the card bar 42 is known, and the dimensional scaling factor m of the card bar 42 is replaced with the dimensional scaling factor f of the status bar 41, and the update information of the card bar 42 and the update information of the status bar 41 are generated according to the current coordinate point V (vx, vy), the replaced dimensional scaling factor and the auxiliary locating point A′ (0, a), the auxiliary locating point B′ (0, B), the auxiliary locating point C′ (xm, a) and the auxiliary locating point D′ (xm, B) on the predefined interface coordinate system. The updated M1, M2, M3 and M4 are as follows:

M
⁢
1

=

[

f
×
vx

,

a
+

f
×

(

vy
-
a

)

]

M
⁢
2

=

[

xm
-

f
×

(

xm
-
vx

)

,
 

a
-

f
×

(

a
-
vy

)

]

M
⁢
3

=

[

f
×
vx

,
 

b
-

f
×

(

b
-
vy

)

]

M
⁢
4

=

[

xm
-

f
×

(

xm
-
vx

)

,
 

b
-

f
×

(

b
-
vy

)

]

The updated F1, F2, F3, and F4 are as follows:

F
⁢
1

=

[

m
×
vx

,
 

b
-

m
×

(

b
-

v
⁢
y

)

]

F
⁢
2

=

[

xm
-

m
×

(

x
⁢
m

-

v
⁢
x

)

,
 

b
-

m
×

(

b
-

v
⁢
y

)

]

F
⁢
3

=

[

m
×
vx

,
 

ym
-

m
×

(

y
⁢
m

-

v
⁢
y

)

]

F
⁢
4

=

[

xm
-

m
×

(

x
⁢
m

-

v
⁢
x

)

,
 

ym
-

m
×

(

y
⁢
m

-

v
⁢
y

)

]

Here, a, b, f, and m are constants, and may be set according to design requirements, which are not particularly limited in the embodiment of the present disclosure.

Another possible embodiment is that the update information generation unit 123 is specifically configured to receive a component selection instruction for selecting an interface component, replace a dimensional scaling factor of the currently selected interface component with a target scaling factor, and generate the update information of each interface component according to the current coordinate point, the replaced target scaling factor and the auxiliary locating point on the predefined interface coordinate system.

Here, the target scaling factor may be empirically determined data. For example, the target scaling factor is selected to be r. The component selection instruction may be an instruction of any interface component selected by the user, for example, the status bar 41 is selected, the dimensional scaling factor f of the status bar 41 is known, the dimensional scaling factor f of the status bar 41 is replaced by the target scaling factor r, and the update information of the status bar 41 is generated according to the current coordinate point V (vx, vy), the replaced dimensional scaling factor r, and the auxiliary locating points B′ (0, b) and D′ (xm, b) on the predefined interface coordinate system. The updated F1, F2, F3 and F4 are as follows:

F
⁢
1

=

[

r
×
vx

,
 

b
-

r
×

(

b
-

v
⁢
y

)

]

F
⁢
2

=

[

xm
-

r
×

(

x
⁢
m

-

v
⁢
x

)

,
 

b
-

r
×

(

b
-

v
⁢
y

)

]

F
⁢
3

=

[

r
×
vx

,
 

ym
-

r
×

(

y
⁢
m

-

v
⁢
y

)

]

F
⁢
4

=

[

xm
-

r
×

(

x
⁢
m

-

v
⁢
x

)

,
 

ym
-

r
×

(

y
⁢
m

-

v
⁢
y

)

]

Here, a, b, and r are constants, and may be set according to design requirements, which are not particularly limited in the embodiment of the present disclosure.

It should be noted that the status bar 41 obtained by using the replaced dimensional scaling factor r is zoomed into display as compared with the original status bar 41 obtained by using the non-replaced dimensional scaling factor f.

In some embodiments, the current position includes a target key point and a depth distance reference point. The information determination unit 122 is specifically configured to track the target object and determine a target key point in the face according to a face feature of the target object in a case where it is determined through the face recognition that the target object is present. A shoulder feature of the target object is identified, and a point of a middle position of two shoulders is taken as a depth distance reference point according to positions of the two shoulders in the scene coordinate system.

For example, in the case where it is determined that the target object is present, the two-dimensional coordinate point of the target object in the scene coordinate system is located in real time, thereby achieving the target trajectory tracking. Meanwhile, the target key point in the human face is determined according to the face features of the target object, such as the feature of the five sense organs, specific face bones or the like. The target key point may be, for example, a two-dimensional coordinate point of the detected five sense organs in the scene coordinate system, a two-dimensional coordinate point of a bone key point in the scene coordinate system, or the like. According to the shoulder feature, such as a shoulder bone feature, two-dimensional coordinate points of bone key points of two shoulders of the human body in the scene coordinate system are determined, and a central point of the two-dimensional coordinate points of the two shoulders (that is, a point of a middle of the two shoulders) is determined and is used as a depth distance reference point.

The update information generation unit 123 is specifically configured to determine a component scaling factor from the depth distance reference point and the target key point; and generate the update information of each interface component according to the component scaling factor, the current coordinate point, the auxiliary locating point on the predefined interface coordinate system and the dimensional scaling factor of each interface component.

FIG. 6 is a schematic diagram illustrating a comparison between two different distances from a target object to a user interface according to an embodiment of the present disclosure. As shown in FIG. 6, the target key point may select a central point 61 of two pupils of the target object, and a length z0 of the line connecting a depth distance reference point 62 and a central point 61 of two pupils of one target object may be used as an initial scaling factor z0. An initial scaling factor of the target object at the previous time is z0_1, and an initial scaling factor of the target object at the later time is z0_2. A component scaling factor z1 may be obtained by using a preset algorithm according to the initial scaling factor z0. The preset algorithm may be determined based on a mapping relationship between the scene coordinate system and the interface coordinate system and adjustment of a matching ratio parameter of some hardware devices (e.g., a camera, etc.). That is, the component scaling factor z1 may be determined by converting the initial scaling factor z0 through the mapping relationship between the scene coordinate system and the interface coordinate system and then adjusting the matching ratio parameter, where a specific matching ratio parameter is set according to an actual application scene and a selected device parameter, which is not particularly limited in the embodiment of the present disclosure.

By way of example, the component scaling factor z1_1 of the target object at the previous time, the current coordinate point V (vx, vy), the auxiliary locating point A′ (0, a), the auxiliary locating point B′ (0, b), the auxiliary locating point C′ (xm, a), the auxiliary locating point D′ (xm, b), the dimensional scaling factor m of the interface component 31, and the dimensional scaling factor f of the interface component 32 are known, the update information of the interface component 31 (that is, updated M1, M2, M3, and M4), and the update information of the interface component 32 (that is, updated F1, F2, F3, and F4) are determined. The updated M1, M2, M3 and M4 are as follows:

M
⁢
1

=

[

z
⁢

1
-

⁢
1
×
m
×
v
⁢
x

,
 

a
+

z
⁢

1
-

⁢
1
×
m
×

(

vy
-
a

)

]

M
⁢
2

=

[

xm
-

z
⁢

1
-

⁢
1
×
m
×

(

xm
-
vx

)

,
 

a
-

z
⁢

1
-

⁢
1
×
m
×

(

a
-
vy

)

]

M
⁢
3

=

[

z
⁢

1
-

⁢
1
×
m
×
vx

,
 

b
-

z
⁢

1
-

⁢
1
×
m
×

(

b
-
vy

)

]

M
⁢
4

=

[

xm
-

z
⁢

1
-

⁢
1
×
m
×

(

xm
-
vx

)

,
 

b
-

z
⁢

1
-

⁢
1
×
m
×

(

b
-
vy

)

]

The updated F1, F2, F3, and F4 are as follows:

F
⁢
1

=

[

z
⁢

1
-

⁢
1
×
f
×
vx

,
 

b
+

z
⁢

1
-

⁢
1
×
f
×

(

b
-
vy

)

]

F
⁢
2

=

[

xm
-

z
⁢

1
-

⁢
1
×
f
×

(

xm
-
vx

)

,
 

b
-

z
⁢

1
-

⁢
1
×
f
×

(

b
-
vy

)

]

F
⁢
3

=

[

z
⁢

1
-

⁢
1
×
f
×
vx

,
 

ym
-

z
⁢

1
-

⁢
1
×
f
×

(

ym
-
vy

)

]

F
⁢
4

=

[

xm
-

z
⁢

1
-

⁢
1
×
f
×

(

xm
-
vx

)

,
 

ym
-

z
⁢

1
-

⁢
1
×
f
×

(

ym
-
vy

)

]

By way of example, the component scaling factor z1_2 of the target object at the previous time, the current coordinate point V (vx, vy), the auxiliary locating point A′ (0, a), the auxiliary locating point B′ (0, b), the auxiliary locating point C′ (xm, a), the auxiliary locating point D′ (xm, b), the dimensional scaling factor m of the interface component 31, and the dimensional scaling factor f of the interface component 32 are known, the update information of the interface component 31 (that is, updated M1, M2, M3, and M4), and the update information of the interface component 32 (that is, updated F1, F2, F3, and F4) are determined. The updated M1, M2, M3 and M4 are as follows:

M
⁢
1

=

[

z
⁢

1
-

⁢
2
×
m
×
v
⁢
x

,
 

a
+

z
⁢

1
-

⁢
2
×
m
×

(

vy
-
a

)

]

M
⁢
2

=

[

xm
-

z
⁢

1
-

⁢
2
×
m
×

(

xm
-
vx

)

,
 

a
-

z
⁢

1
-

⁢
2
×
m
×

(

a
-
vy

)

]

M
⁢
3

=

[

z
⁢

1
-

⁢
2
×
m
×
vx

,
 

b
-

z
⁢

1
-

⁢
2
×
m
×

(

b
-
vy

)

]

M
⁢
4

=

[

xm
-

z
⁢

1
-

⁢
2
×
m
×

(

xm
-
vx

)

,
 

b
-

z
⁢

1
-

⁢
2
×
m
×

(

b
-
vy

)

]

The updated F1, F2, F3, and F4 are as follows:

F
⁢
1

=

[

z
⁢

1
-

⁢
2
×
f
×
vx

,
 

b
+

z
⁢

1
-

⁢
2
×
f
×

(

b
-
vy

)

]

F
⁢
2

=

[

xm
-

z
⁢

1
-

⁢
2
×
f
×

(

xm
-
vx

)

,
 

b
-

z
⁢

1
-

⁢
2
×
f
×

(

b
-
vy

)

]

F
⁢
3

=

[

z
⁢

1
-

⁢
2
×
f
×
vx

,
 

ym
-

z
⁢

1
-

⁢
2
×
f
×

(

ym
-
vy

)

]

F
⁢
4

=

[

xm
-

z
⁢

1
-

⁢
2
×
f
×

(

xm
-
vx

)

,
 

ym
-

z
⁢

1
-

⁢
2
×
f
×

(

ym
-
vy

)

]

Here, a, b, f, and m are constants, and may be set according to design requirements, which are not particularly limited in the embodiment of the present disclosure.

In the embodiment, the target key point is found by using the face features, the depth distance reference point is found by using the shoulder features, and the line connecting the target key point and the depth distance reference point is used as the reference condition for scaling the interface component, to realize scaling of the interface component, so that the relatively real 3D interaction effect is simulated in the 2D display mode, the hardware resources of the system are saved, and the compatibility of UI interface display and other application can be better realized.

In some embodiments, the information determination unit 122 is specifically configured to identify a pupil feature in the face, and take a point of a middle position of two pupils as a target key point according to positions of the two pupils in the scene coordinate system in a case where it is determined through the face recognition that the target object is present. A central point of the two pupils is selected as the target key point to position the target object more accurately. In addition, the pupil feature has a higher differentiation degree, so that the face recognition and locating are easy to perform.

In some embodiments, as shown in FIGS. 4a and 4b, the user interface includes at least one layer, including the background layer 403; the interface component includes the background image 43. The system control module 102 further includes an interface control unit configured to control the display module 103 to display the background image in the background layer.

In some embodiments, FIG. 7 is a schematic diagram illustrating a positional relationship between a background image and a vertex of a visual window on an interface coordinate system according to an embodiment of the present disclosure. As shown in FIG. 7, the figure number 71 indicates the background image. In order to enhance a sense of space, the background image 71 also changes along with the target object. In order to ensure that an edge of the background image 71 is not exposed in the visual window 72 during the movement, it is necessary to set a suitable size of the background image 71, which is necessarily larger than that of the visual window.

The system control module 102 further includes a second pre-processing unit; the second pre-processing unit is configured to determine size information of the background image 71 according to a predefined magnification factor of the background image 71 and size information of the visual window 72.

As shown in FIG. 7, the size information of the visual window 72 includes coordinate information of each vertex of the visual window 72, and coordinates of vertexes of the visual window 72 are A (0, 0), B (0, ym), C (xm, 0), and D (xm, ym). The background image 71 has the magnification factor p, which may be set according to actual design requirements, and is not particularly limited in the embodiment of the present disclosure. p=L1+L2. The size information of the background image 71 includes a width Wx and a height Wy, and a size of the background image 71 is magnified proportionally to a size of the visual window 72, satisfying Wx/xm=Wy/ym. The background image 71 has the width Wx=xm+p and the height Wy=(ym+ym/xm×p). The size information of the background image 71 further includes coordinate information of each vertex of the background image 71, and coordinates of vertexes of the background image 71 are W1, W2, W3, and W4. The coordinates of the vertexes of the background image 71 change as the current coordinate point is updated. Initially, a geometric center of the background image 71 coincides with a geometric center of the visual window 72.

In some embodiments, the target object includes a person. The detection unit 121 is specifically configured to perform a face recognition on the scene picture and determine whether the target object is present. In particular, some features may be selected that are capable of representing a higher intelligibility of the person, such as a facial feature (including a feature of five sense organs), a head feature, a torso feature, or the like. Optionally, the detection unit 121 is specifically configured to perform face recognition on the scene picture, and determine whether the target object is present. Specifically, the face features in the scene picture are identified, it is judged whether a picture feature meeting the face features is present or not. If the picture feature meeting the face features is present, it is determined that the target object is present, and otherwise, it is determined that the target object is not present.

The current position includes a target key point. The information determination unit 122 is specifically configured to track the target object and determine a target key point in the face according to a face feature of the target object in a case where it is determined through the face recognition that the target object is present. For example, in the case where it is determined that the target object is present, the two-dimensional coordinate point of the target object in the scene coordinate system is located in real time, thereby achieving the target trajectory tracking. Meanwhile, the target key point in the human face is determined according to the face features of the target object, such as the feature of the five sense organs, specific face bones or the like. The target key point may be, for example, a two-dimensional coordinate point of the detected five sense organs in the scene coordinate system, a two-dimensional coordinate point of a bone key point in the scene coordinate system, or the like.

The information determination unit 122 is specifically configured to determine a current coordinate point corresponding to the target key point according to the target key point and a mapping relationship between the scene coordinate system and the interface coordinate system. For example, according to the mapping relationship between the scene coordinate system and the interface coordinate system, the target key point in the scene coordinate system is mapped into the interface coordinate system, and the mapped current coordinate point is determined.

The predefined auxiliary parameter information further includes the size information of the visual window of the predefined user interface, the magnification factor and the size information of the predefined background image. The update information generation unit 123 is further configured to determine an auxiliary reference point at which the current coordinate point is centrosymmetric with respect to the geometric center of the visual window according to the current coordinate point and the size information of the visual window; and generate the update information of the background image according to the size information of the visual window, the magnification factor of the background image, the auxiliary reference point and the size information of the background image.

As shown in FIG. 7, the auxiliary reference point V_mirror varies with the tracked current coordinate point V(vx, vy). At any time, the auxiliary reference point V_mirror and the current coordinate point V(vx, vy) are centrosymmetric with respect to the geometric center of the visual window. A position coordinate of the auxiliary reference point V_mirror in the interface coordinate system is (xm-vx, ym-vy). At any time, the point V_mirror, a point A and a point W1 are all located on a same straight line, and the point V_mirror, a point C and a point W2 are all located on a same straight line, the point V_mirror, a point B and a point W3 are located on a same straight line, and the point V_mirror, a point D and a point W4 are all located on a same straight line. Therefore, the position of the background image is continuously updated as the current coordinate point V is continuously changed. Specifically, a coordinate of the point W1 of the background image is [−p/xm×(xm−vx), −p/xm×(ym−vy)].

In the embodiment, the size of the background image is magnified, the position of the background image is updated along with the position of the target object, so that the sense of space of the UI interface is improved, and the interactive experience of the user is improved.

The user interface display system has been described as above.

The embodiment of the present disclosure further provides a user interface display method, a main body performing the method may be a terminal device integrated with the user interface display system. The user interface display method may be applied to a large-screen terminal. The “large screen” may be understood as a display screen with a large screen size and a high resolution, such as an ultra high definition display screen with a screen size of more than 40 inches and a resolution of 8K.

FIG. 8 is a flowchart of a user interface display method according to an embodiment of the present disclosure. As shown in FIG. 8, the method includes steps S11 to S15, specifically:

S11, acquiring a scene picture, and detecting whether a target object is present in the scene picture.

It should be noted that for a specific implementation process of step S11, reference may be made to the above description of the image acquisition module 101 and the detection unit 121 in the system control module 102 in the user interface display system, and repeated descriptions are omitted.

S12, tracking the target object and determining a current position of the target object in a scene in a case where the target object is present.

It should be noted that for a specific implementation process of step S12, reference may be made to the above description of the information determination unit 122 in the system control module 102, and repeated descriptions are not repeated.

S13, determining a current coordinate point of the current position mapped to an interface coordinate system of a user interface according to the current position of the target object in the scene and a mapping relationship between a pre-established scene coordinate system and the interface coordinate system.

It should be noted that for a specific implementation process of step S13, reference may be made to the above description of the information determination unit 122 in the system control module 102, and repeated descriptions are not repeated.

S14, generating update information of interface components in the user interface according to the current coordinate point and predefined auxiliary parameter information.

It should be noted that for a specific implementation process of step S14, reference may be made to the above description of the update information generation unit 123 in the system control module 102, and repeated details are not repeated.

S15, displaying the interface components according to the update information.

It should be noted that for a specific implementation process of step S15, reference may be made to the above description of the display module 103, and repeated descriptions are omitted.

In some embodiments, the predefined auxiliary parameter information includes a predefined auxiliary locating point on the interface coordinate system and a dimensional scaling factor of each interface component.

For step S11, in particular, the target object includes a person. A face recognition is performed on the scene picture, and it is determined whether the target object is present or not. It should be noted that for the process, reference may be made to the specific implementation of the detection unit 121, and repeated descriptions are omitted.

For step S12, in particular, the current position includes a target key point. The target object is tracked and a target key point in the face is determined according to a face feature of the target object in a case where it is determined through the face recognition that the target object is present. It should be noted that for the process, reference may be made to the specific implementation of the information determination unit