Alternatively, in some embodiments, before the method for three-dimensional reconstruction is performed, at least part of the camera and the plane mirror group is arranged manually by an operator.

In some embodiments, the arranging at least part of the camera and the plane mirror group may comprise adjusting at least one of the following as needed: the number of the plane mirror group, the position or posture of at least part of the plane mirror group relative to the camera, or the like. The example of this adjustment has been described above in conjunction with FIGS. 2B-2C so that it will not be repeated here. By arranging the camera and the plane mirror group accordingly according to the application requirement and the scenario feature, at least one of improving performance or reducing cost can be advantageously achieved.

Alternatively, in some embodiments, the camera and the plane mirror group are preset in place, and therefore the arrangement is unnecessary. For example, in a certain application scenario, the camera and the plane mirror group might have already been installed and adjusted in advance. In this case, the three-dimensional reconstruction on the target object can be directly performed without the arrangement of the camera and the plane mirror group.

In some embodiments, the method for three-dimensional reconstruction may further include calibrating at least one of the camera or the mirror, to obtain the relative position and posture information indicating the position and posture of the mirror relative to the camera (step 304).

For example, in some embodiments, the position and posture of the mirror relative to the camera, i.e., the relative position and posture information, may be determined in conjunction with the information sent by respective position and posture obtaining modules of the camera and the plane mirror group providing the mirror. Alternatively, in some embodiments, specific patterns containing marker features, such as a bar code, a QR code, or a checkerboard, may be arranged on the plane mirror group. By photographing these specific patterns using the camera, the relative position and posture information can be recognized. Still alternatively, in some embodiments, one of the camera and the mirror is pre-calibrated, and the relative position and posture information can be obtained only using a position and posture obtaining module on the other of the camera and the mirror.

Alternatively, in some embodiments, the relative position and posture between the camera and the mirror are preset and known, and therefore the step 304 is unnecessary. For example, in a certain application scenario, once the camera and the mirror are installed, the three-dimensional reconstruction may be performed many times without changing the arrangement of the two. In this case, it is not needed to re-obtain the relative position and posture information in each three-dimensional reconstruction.

However, if the relative position and posture between the camera and the mirror are changed by an arrangement, re-calibration is required.

In step 306, a composite image obtained by photographing a body of a target object and a mirror image of the target object is obtained.

In the prior art, in order to perform 3D reconstruction on a target object, a plurality of cameras are required to capture images from a plurality of angles of view. As mentioned above, a mirror image of the target object is equivalent to an image of the target object “photographed” at one or even several specific angles of view, that is, equivalently, each mirror replaces one or more cameras in the prior art (hereinafter referred to as mirror virtual cameras). By appropriately arranging the camera and the mirror, as shown in FIG. 6, the body of the target object and the mirror image of the target object in the mirror can be photographed onto a same image (referred to as the composite image herein). Therefore, image information at a plurality of angles of view that is required for the three-dimensional reconstruction of the target object can be obtained using the composite image.

In the related three-dimensional reconstruction technology, precision of three-dimensional modeling is directly related to the number of the cameras, and therefore the use of a plurality of cameras is generally needed to obtain images at a plurality of angles of view. In this application, three-dimensional modeling is performed by using the composite image obtained by photographing the body of the target object and the mirror image of the target object, which can reduce the requirement and dependence on the number of cameras and achieve high-precision processing with lower cost.

In step 308, three-dimensional reconstruction is performed by using the composite image and relative position and posture information representing the position and posture of the mirror relative to the camera.

An example of sub-steps of performing three-dimensional reconstruction (step 308) according to an embodiment of the present disclosure is described in detail below in conjunction with FIG. 4.

As illustrated in FIG. 4, random sampling may be performed on points within a given three-dimensional space including the target object (step 402).

In some embodiments, the given three-dimensional space including the target object may be suitably constrained to improve the efficiency of the sampling. In addition, the number N of the sampling points can be selected as needed. N is a positive integer.

Subsequently, a global feature corresponding to the sampling point may be extracted from the composite image (step 404).

FIG. 5 illustrates in detail an example of specific steps of extracting the global feature corresponding to the sampling point from the composite image according to an embodiment of the present disclosure.

As illustrated in FIG. 5, in step 502, feature extraction is performed on the composite image, to obtain a global feature map.

Specifically, in some embodiments, the composite image may be inputted into a feature extractor for the feature extraction.

In some embodiments, the feature extractor may include, but is not limited to, any one or combination of a neural network, automatic codec, SIFT, HOG, or the like.

As an output of the feature extractor, the global feature map for the composite image may be obtained.

In some embodiments, the global feature map may be formed by feature elements. Each feature element may be represented in a form of a multi-dimensional vector. The feature elements in the global feature map may correspond to pixel points on the image, respectively. Here, “correspondence” between a feature element and a pixel point refers to that the feature element can represent a feature of the corresponding pixel point. Those skilled in the art would readily appreciate that, the higher the resolution or the smaller the pixel point of an image, the more accurately the extracted global feature map can represent the image, but the greater the corresponding workload.

In some embodiments, to avoid significant computational overhead, performing global feature extraction on the image further includes, before inputting the image to the feature extractor, pre-processing, such as down-sampling, the image, to reduce the resolution of the image. For example, in some embodiments, an image with a resolution of 512*512 may be compressed into an image with a resolution of 64*64 before the image is inputted to a feature extractor.

In step 504, a geometric association between the sampling point and a pixel point of the composite image is determined. Therefore, a correspondence between a world coordinate system and an image coordinate system can be determined.

The pixel points in the composite image which correspond to the sampling points may include not only a pixel point obtained by photographing the body of the target object (hereinafter referred to as a body pixel point), but also a pixel point obtained by photographing the mirror image of the target object (hereinafter referred to as a mirror image pixel point). Therefore, establishing the geometric association between the sampling point and the pixel point of the composite image may specifically comprise: determining a geometric association between the sampling point and the body pixel point (sub-step 510), and determining a geometric association between the sampling point and the mirror image pixel point (sub-step 520).

In some embodiments, the geometric association between the sampling points and pixel points of the composite image may be determined by calculating a projection matrix of a body camera (real camera) and a projection matrix of a mirror virtual camera (mirror-provided virtual camera). The body camera projection matrix and the mirror virtual camera projection matrix may be calculated using the relative position and posture information known to the system or obtained by the step 304.

An example of the determination method of the projection matrix is explained below in conjunction with FIG. 5B.

For ease of calculation and description, a position of a camera C₀may be set as an origin O of a world coordinate system. By using the relative position and posture information, a normal vector n=(n_x, n_y, n_z) of a plane mirror group W₁and a distance d from a plane where the plane mirror group W₁is located to the camera C₀may be obtained.

A position of a mirror image of the camera C₀with respect to the plane mirror group W₁may be represented as:

T
1

=

2
⁢

(

(

T
0

·
n

)

⁢
n

+

d
⁢
n

)

[

Equation
⁢

1

]

- where T₀represents the position of the camera C₀in the world coordinate system. Since the camera C₀is assumed to be located at the origin O of the world coordinate system, the above equation may be further simplified as T₁=2dn. In conjunction with the above analysis, it can be concluded that the position T₁of the mirror image is a position of a mirror virtual camera C₁defined in this disclosure.

An posture rotation matrix R₁of the mirror virtual camera C₁may be represented as:

R
1

=

[

Equation
⁢

2

]

R
0

⁢

VR
0
T

=

R
0

⁢

(

-

n
x
2

+

n
y
2

+

n
z
2

-
2

⁢

n
x

⁢

n
y

-
2

⁢

n
x

⁢

n
z

-
2

⁢

n
x

⁢

n
y

n
x
2

-

n
y
2

+

n
z
2

-
2

⁢

n
y

⁢

n
z

-
2

⁢

n
x

⁢

n
z

-
2

⁢

n
x

⁢

n
z

n
x
2

+

n
y
2

-

n
z
2

)

⁢

R
0
T

- where V is a mirror image matrix, and R₀is a posture rotation matrix of the camera C₀. In this example, R₀may be represented as:

R
0

=

(

1

0

0

0

1

0

0

0

1

)

[

Equation
⁢

3

]

Based on the position and the posture rotation matrix in the world coordinate system, an extrinsic matrix M₀of the camera C₀may be represented as: M₀=[R₀T₀], and similarly, an extrinsic matrix M₁of the mirror virtual camera C₁may be represented as: M₁=[R₁T₁].

An intrinsic matrix I₀of the camera C₀is only related to intrinsic parameters of the camera, and may be generally obtained according to factory parameters or calibration of the camera itself. In general, the intrinsic matrix I₀may be represented as:

I
0

=

(

f
x

0

s
x

0

f
y

s
y

0

0

1

)

[

Equation
⁢

4

]

where f_xand f_yrepresent values related to a focal length of the camera C₀, and s_xand s_yrepresent values related to a principal point offset of the camera C₀in the imaging.

Therefore, according to the mirror imaging principle, an intrinsic matrix I₁of the mirror virtual camera C₁may be represented as:

I
1

=

(

-

f
x

0

s
x

0

f
y

s
y

0

0

1

)

[

Equation
⁢

5

]

Using the intrinsic and extrinsic matrices, a projection matrix P₀of the camera C₀may be represented as:

P
0

=

M
0

·

I
0

=

[

R
0

⁢
 

T
0

]

·

I
0

[

Equation
⁢

6

]

Similarly, a projection matrix P₁of the mirror virtual camera C₁may be represented as:

P
1

=

M
1

·

I
1

=

[

R
1

⁢
 

T
1

]

·

I
1

[

Equation
⁢

7

]

To simplify the description, in the above example, the camera C₀is set at the origin position in the world coordinate system. It should be understood by those skilled in the art that, there are no particular limitations on the coordinate position of the camera C₀. Moreover, if there are a plurality of real cameras, they may be respectively distributed at any position including the origin.

Although the above Equations 1 to 7 only exemplarily describe the calculation method of the projection matrix for one plane mirror group W₁, a similar calculation method is also applicable to calculation of a projection matrix of another plane mirror group (such as W₂) in the case where a plurality of plane mirror groups are included.

Furthermore, although the specific example of determining the geometric association between the sampling point and the pixel point of the composite image using the projection matrix is described in detail herein, this disclosure is not limited thereto.

In step 506, a global feature corresponding to the sampling point is determined from the global feature map based on the geometric association.

As described above, the feature elements in the global feature map may correspond to the pixel points on the composite image, respectively. Furthermore, according to the processing in the step 504, the geometric association between the sampling point and the pixel point of the composite image may be determined. Therefore, based on the geometric association, the correspondence between the sampling point and the feature element may be determined.

It should be noted that, in addition to the body pixel points obtained by photographing the body of the target object, the composite image may include mirror image pixel points obtained by photographing the mirror image of the target object. Therefore, in some embodiments, the number of the global feature corresponding to each sampling point may be greater than 1, such that the total number of the global features may be greater than the number of the sampling points. Due to the limitation from the angle of view and influence of a possible splicing gap, not each sampling point may surely be photographed by each of the body camera and the mirror virtual camera, but the total number of the global features may still be much greater than that in the case of photographing the body of the target object using the camera only.

As illustrated in FIG. 4, encoding may be performed on imaging-related geometric information of the sampling point, to generate geometric encoded information (step 406).

In some embodiments, the imaging-related geometric information of the sampling point may include a spatial coordinate of the sampling point and at least part of interior and exterior orientation information of a camera imaging the sampling point. The camera not only includes the body camera, but also includes the mirror virtual camera defined in this disclosure.

For example, in some embodiments, the imaging-related geometric information of the sampling point may include only the spatial coordinate of the sampling point. In these embodiments, the generated geometric encoded information may be related only to the sampling point itself. Pixel points or corresponding global features corresponding to a same sampling point, regardless of whether obtained by the body camera or the mirror virtual camera, will be associated with same geometric encoded information.

For example, in some other embodiments, the imaging-related geometric information of the sampling point may include not only the spatial coordinate of the sampling point but also the interior and exterior orientation information of the camera. In these embodiments, pixel points or corresponding global features obtained by imaging a same sampling point from different angles of view by the body camera or the mirror virtual camera are associated with their respective different geometric coding information.

In some embodiments, the generated geometric encoded information may be a multi-dimensional vector. For example, as an example, the geometric encoded information may include multi-dimensional vectors corresponding to the spatial coordinate of the sampling point and the interior and exterior orientation information of the camera, respectively.

The inventors of the present application have realized that, the geometric encoded information contains, for example, information of the above various aspects, so that it can represent the geometric feature more accurately than intuitive geometric information. Therefore, representing the geometric feature using the geometric encoded information is beneficial for improving the accuracy of three-dimensional reconstruction.

As illustrated in FIG. 4, the global feature and the corresponding geometric encoded information may be inputted into a model, to determine a geometric relationship between the sampling point and a surface of the target object (step 408).

Specifically, for each sampling point, the global feature corresponding to the sampling point and the corresponding geometric encoded information may be inputted into the model.

As analyzed above, the use of the mirror, similar to the use of an additional camera, can enable the effect of achieving more angles of view. Therefore, in the case of introducing the mirror to provide the mirror virtual camera, the number of the global feature corresponding to each sampling point may be an integer greater than 1. In some embodiments, a plurality of global features corresponding to different angles of view and corresponding geometric coding information can be inputted for one sampling point, which advantageously improves the accuracy of the determination.

In addition, if the geometric encoded information can include the interior and exterior orientation information of the camera to reflect the angle of view for the imaging, it would also be advantageous to further improve the accuracy of the determination.

Those skilled in the art can appreciate that, any model capable of determining the geometric relationship between respective sampling point and the surface of the target object according to the above input may be adopted. Based on the global feature for any sampling point and the corresponding geometric encoded information, the model adopted may output a determination result indicating the geometric relationship between the sampling point and the surface of the target object.

In some embodiments, the determination result may be numerical. For example, in some embodiments, the determination result may be a numerical value indicating a probability that the sampling point is located inside/outside the surface of the target object. For example, when the determination result is 1, it can be indicated that the sampling point is located inside the surface of the target object. In contrast, when the determination result is 0, it can be indicated that the sampling point is located outside the surface of the target object, and vice versa. In other cases, the determination result can be between 0 and 1.

In some embodiments, the method for three-dimensional reconstruction may further include three-dimensionally reconstructing a voxel envelope of the target object based on the geometric relationship between the sampling point and the surface of the target object.

In some embodiments, by determining the geometric relationships between all the sampling points and the surface of the target object, the three-dimensionally reconstructed target voxel envelope can be obtained. For example, in the case where the determination result is a numerical value indicating the probability that the sampling point is located inside/outside the surface of the target object, by extracting an iso-surface of 0.5, the surface of the target object can be determined.

As illustrated in FIG. 4, in some embodiments, performing three-dimensional reconstruction further includes training the model (step 410).

In some embodiments, the model may be represented using an implicit function f that outputs the probability that the sampling point is located inside/outside the surface of the target object according to the above input. The description is made below by taking this case as an example, but those skilled in the art can appreciate that the present disclosure is not limited thereto.

First, a discrimination error of the model may be calculated according to the determination result of the step 408.

In some embodiments, the discrimination error may be obtained by comparing the determination result outputted by the model with a real surface of the target object.

For example, in some embodiments, the real surface of the target object may be described using an implicit function f*.

f
*

(
X
)

=

{

1
,

X
⁢

is
⁢

inside
⁢

the
⁢

surface

0
,

otherwise

[

Equation
⁢

8

]

That is, if the point X is inside the surface of the target object, the function value of f* is 1, and if it is outside the surface, the function value of f* is 0. The real surface of the target object may be an iso-surface with the function value of f* of 0.5.

Similarly, the implicit function f describing the model may be represented as follows.

f
⁡
(
X
)

=

(

F

G
⁢
1

(
X
)

,
…

,

F

G
⁢
m

(
X
)

,

Z
1

(
X
)

,
…

,

Z
m

(
X
)

)

[

Equation
⁢

9

]

- where F_Gi(X) and Z_i(X) (1≤i≤m) respectively refer to an i-th global feature in one or more global features corresponding to the sampling point X and corresponding geometric encoded information, and m represents the number of the global feature corresponding to the sampling point X, m being a positive integer not greater than (L+1). As described above, when the imaging-related geometric information of the sampling point that is indicated by the geometric encoded information includes only the spatial coordinate of the sampling point, Z_i(X) may be the same.

Thus, in some embodiments, the discrimination error of the determination result may be calculated by calculating a difference between a function value of the implicit function f representing the model and the function value of the implicit function f* representing the real surface of the target object.

Subsequently, the model is iteratively optimized by, for example, updating parameters of the model, such that the discrimination error meets the precision requirement to complete the training of the model.

It should be appreciated by those skilled in the art that iterative optimization of the model may be performed by using any suitable method, including but not limited to gradient descent, random gradient descent, and the like.

The method for three-dimensional reconstruction according to the embodiment of the present disclosure can reduce the dependence on the number of cameras and enhance the adaptability to different application scenarios, and has low cost, high portability, extendibility and flexible adjustment capability, in addition, in the present application, an accurate and reliable information extraction and processing method is proposed, and the model is trained and used for the three-dimensional reconstruction of the target object based on the processed information. On this basis, the present disclosure can implement high-precision three-dimensional reconstruction on the target object using only a single camera or sparse cameras. The cost of the three-dimensional modeling can be reduced and/or the accuracy of the three-dimensional modeling can be improved.

It is noted that, the boundaries between the steps in the method described above are merely illustrative. In actual operations, the steps can be combined arbitrarily, and even combined into a single step. In addition, the execution order of the steps is not limited by the order described, and part of the steps may be omitted. The operational steps of the embodiments may also be combined with each other in any suitable order, thereby similarly implementing more or less operations than those described.

FIG. 7 illustrates a block diagram of an electronic device according to some embodiments of the present disclosure.

As shown in FIG. 7, the electronic device 7 of this embodiment comprises: a memory 71 and a processor 72 coupled to the memory 71, the processor 72 being configured to perform, based on instructions stored in the memory 71, the method for three-dimensional reconstruction according to the embodiment of the present disclosure.

The memory 71 may include, for example, a system memory, a fixed non-volatile storage medium, and the like. The system memory has thereon stored, for example, an operating system, an application, a boot loader, a database, another program, and the like.