As described above, for regions (a remote roadway, sidewalk, etc.) in which the number of feature points is expected to be small, the region dividing unit 1021 is capable of dividing the captured image 10 by using a larger size than that for regions (a nearby roadway, sidewalk, etc.) in which the number of feature points is expected to be large.

The region dividing unit 1021 changes the number of divisions and the division size in accordance with the characteristics of objects appearing in the captured image 10, and thus the number of feature points detected from a divided region in which a nearby object appears and the number of feature points detected from a divided region in which a remote object appears can be made equally close to each other. As a result, it is possible to improve the calibration accuracy attained by the image processing device 100 in the depth direction of the captured image 10.

Next, an example in which the region dividing unit 1021 performs different division based on images before and after a time series of a traveling vehicle (before a time series A and after time series B) will be described.

FIGS. 7 and 8 are diagrams showing examples of captured images 10 captured by the in-vehicle camera 101 at different times.

FIG. 7 is a diagram showing a state in which the captured image 10 captured before time series A is divided.

FIG. 8 is a diagram showing a state in which the captured image 10 captured after the time series B is divided. The captured image 10 shown in FIG. 8 is captured a time t after the captured image 10 captured in FIG. 7. That is, the captured image 10 shown in FIG. 8 is an image captured by the in-vehicle camera 101 in a state where the vehicle has moved forward for the time t.

The region dividing unit (region dividing unit 1021) divides the plurality of captured images 10 into different sizes on the basis of a prediction result of feature points corresponding to three-dimensional positions in the outside world detected from a plurality of captured images 10 captured by the in-vehicle camera (in-vehicle camera 101) as the vehicle moves. For example, the region dividing unit 1021 predicts where feature points corresponding to the three-dimensional positions detected from the captured image 10 captured at pre-time series A and the captured image 10 captured at post-time series B, respectively, are located. The region dividing unit 1021 divides the two captured images 10 using different division sizes so that the predicted positions of the feature points are included.

When the captured image shown in FIG. 7 is compared with the captured image 10 shown in FIG. 8, the height H1 of the upper region divided into eight divided regions in the image shown in FIG. 7 is the height H2 in the captured image 10 shown in FIG. 8, and the upper region of the captured image 10 is wider. In addition, the region on the left side of the image shown in FIG. 7 is divided by a width W11 including a roadway which is remote and a width W12 of a region including a sidewalk which is remote. Meanwhile, when the captured image 10 shown in FIG. 8 is captured, the vehicle moves forward and approaches the object. Therefore, the region on the left side of the image shown in FIG. 8 is wider than the width W11 shown in FIG. 7 as indicated by the width W21 including the remote roadway. Meanwhile, width W22, which includes the remote sidewalk in the region on the left side of the image shown in FIG. 8, is narrower than width W12 shown in FIG. 7.

Note that, in order to perform matching of divided regions of the captured image 10 captured at pre-time series A and the captured image 10 captured at post-time series B, divided regions of the captured image 10 captured at pre-time series A and divided regions of the captured image 10 captured at post-time series B must be present. Therefore, the number of divisions of the two captured images 10 is the same.

Because the region dividing unit 1021 changes the division size of the captured image 10 before and after the time series in this manner, the feature point detection unit 1022 is capable of reliably detecting feature points in the region where a noteworthy object in the direction of travel of the vehicle is present (for example, a roadway). In addition, when matching divided regions, the feature point matching unit 103 to be described below is capable of minimizing malfunctions in matching erroneous divided regions.

Returning again to FIG. 3, the description will be continued.

After the processing of step S11, the feature point detection unit 1022 selects one divided region resulting from the division by the region dividing unit 1021 (S12). The feature point detection unit 1022 performs feature point detection in the divided regions by using the individual threshold value for the divided region selected in step S12 (S13).

In feature point detection processing, detection threshold values for increasing or decreasing the number of feature points detected in the divided regions by the feature point detection unit 1022 is used. Usually, one value is set as the detection threshold value for feature points for one captured image 10. Meanwhile, the feature point detection unit 1022 according to the present embodiment is capable of individually setting a detection threshold value for each divided region. Therefore, the detection threshold value set for each divided region is referred to as an “individual threshold value”. The individual threshold value is a value set for each divided region by the threshold value setting unit 1023 described below.

Further, the individual threshold values set by the threshold value setting unit 1023 have a range of settable values. Here, luminance threshold values are used as the individual threshold values. By using luminance threshold values, it is possible to perform binarization processing for each pixel such that white is displayed in regions having a certain luminance or more and black is displayed in regions having a luminance less than the certain luminance. For example, increasing the luminance threshold value increases the black region in the captured image 10, and decreasing the luminance threshold value increases the white region in the captured image 10. When the luminance threshold value is increased to the maximum value, the captured image 10 is completely filled black, and when the luminance threshold is decreased to the minimum value, the captured image 10 is completely filled white. Therefore, a feature point cannot be detected in either case. Therefore, the range of the luminance threshold values which are effective for feature point detection is determined for each operation scene. The threshold value setting unit 1023 increases or decreases the individual threshold values within this range.

After step S13, the feature point detection unit 1022 counts the number of feature points in the divided regions detected in step S13 (S14). The number of feature points of the divided regions counted by the feature point detection unit 1022 is temporarily stored in the RAM 53 (see FIG. 13) or the like.

After the processing (S12 to S14) by the feature point detection unit 1022, the processing by the threshold value setting unit 1023 is started. The threshold value setting unit 1023 sets a threshold value to equalize the feature points detected by the feature point detection unit 1022. Therefore, the threshold value setting unit 1023 compares the number of feature points with a target value for each divided region for which the number of feature points has been counted in step S14, and determines whether or not the number of feature points is greater than the target value (S15). In steps S15 to S17, the threshold value setting unit (threshold value setting unit 1023) performs setting to raise the threshold values set for divided regions in which the number of feature points is greater than the target value, and performs setting to lower the threshold values set for divided regions in which the number of feature points is less than the target value. For this reason, the individual threshold values are set such that the number of feature points detected by the feature point detection unit 1022 is close to the target value. Here, the number of feature points will be described with reference to FIGS. 9 and 10.

FIG. 9 is a diagram showing an example of feature points detected from the first captured image 10.

FIG. 10 is a diagram showing an example of feature points detected from the second captured image 10.

The solid-line circles and broken-line circles shown in FIGS. 9 and 10 represent feature points. However, a solid-line circle represents a feature point for which a feature point indicating the same part has been detected and matched in two captured images 10, and a broken-line circle represents a feature point which has not been detected in the same part from the two captured images 10.

Normally, the feature point matching is performed by an operation to compare the feature amounts of one feature point of one captured image 10 (standard image) with the feature amounts of all feature points of the other captured image 10 (reference image) to find the feature point having the closest feature amount. The feature amount is a numerical value indicating the shape of a feature point, and is obtained by converting the arrangement of pixels (luminance values) around the feature point into a numerical value by using a certain procedure. Various shapes are assumed for the feature points; however, if the feature points are the same object, the feature points are expected to be the same. However, because the matching of the feature points is performed on the divided regions in the same position in the two captured images 10, the load can be made lighter than in the processing to match the feature points detected across the entire captured image 10, and thus the time until completion of the matching can be shortened.

The description returns to FIG. 3.

It is assumed that the upper left side (toward the back) of the captured image 10 shown in FIG. 9 is a divided region obtained through division using the division pattern shown in FIG. 6. In this case, because many feature points have been detected in the divided region on the upper left side (toward the back) of the captured image 10, the threshold value setting unit 1023 determines that the number of feature points in this divided region is greater than the target value (NO in S15). Therefore, the threshold value setting unit 1023 performs processing to increase the individual threshold values (S16). As a result, the detection threshold value in the divided region reaches a maximum value. That is, the detection threshold value is the maximum detection threshold value within the settable range. However, by dividing the current individual threshold value to the maximum value at predetermined intervals, the individual threshold values may be set at the intervals with which division is performed in step S16. Therefore, the individual threshold value may be set to a value smaller than the maximum value.

Meanwhile, it is assumed that the lower side (toward the front) of the captured image 10 shown in FIG. 9 is a divided region obtained through division using the division pattern shown in FIG. 6. In this case, because almost no feature point is detected in the divided region on the lower side (front side) of the captured image 10, the threshold value setting unit 1023 determines that the number of feature points in this divided region is equal to or less than the target value (NO in S15). Therefore, the threshold value setting unit 1023 performs processing to lower the feature point detection threshold value (S17). As a result, the detection threshold value in the divided region reaches the minimum value. That is, the detection threshold value is the minimum detection threshold value within the settable range. However, by dividing the current individual threshold value to the minimum value at predetermined intervals, the individual threshold values may be set at the intervals with which division is performed in step S17. Therefore, the individual threshold value may be set to a value greater than the minimum value.

After step S16 or S17, the feature point equalization detection unit 102 determines whether or not the processing of step S12 and subsequent steps has been performed on all the divided regions divided by the region dividing unit 1021 (S18). In a case where it is determined that there is a divided region that has not been processed (NO in S18), the feature point equalization detection unit 102 re-executes the processing from step S12 on the unprocessed divided region. Here, the feature point detection unit (feature point detection unit 1022) detects the feature points by using the threshold value set by the threshold value setting unit (threshold value setting unit 1023). Therefore, the number of feature points detected by the feature point detection unit 1022 is made to correspond to the target value. Meanwhile, in a case where it is determined that the processing has been performed on all the divided regions (YES in S18), the feature point equalization detection unit 102 ends the feature point equalization detection processing shown in FIG. 3.

Note that, in the feature point equalization detection processing shown in FIG. 3, after the individual threshold value is set for a certain divided region by means of the processing of steps S16 and S17, the feature point detection unit 1022 performs the feature point detection processing on another divided region by using the individual threshold value set in the previous processing. Such processing is performed in order to shorten the processing time for all the divided regions in a case where the feature point equalization detection processing shown in FIG. 3 is implemented.

However, if the image processing device 100 has spare processing capacity, it is more effective for the feature point detection unit 1022 to perform the feature point detection again in the same divided region. In this case, based on the individual threshold value set by the threshold value setting unit 1023 for a certain divided region, the feature point detection unit 1022 uses the set individual threshold value to re-detect the feature points in the same divided region. Thereafter, the individual threshold value setting processing by the threshold value setting unit 1023 is not performed, and the processing from step S12 may be performed on another divided region.

Returning to FIGS. 1 and 2, the description will be continued.

After the processing by the feature point equalization detection unit 102 shown in FIG. 3, the feature point matching unit (feature point matching unit 103) performs matching by associating the feature points detected in the plurality of captured images 10 with each other between the plurality of captured images 10 (S3). At this time, the feature point matching unit 103 performs matching to associate feature points detected as the same portion in the two captured images 10 outputted from the feature point equalization detection unit 102. Therefore, the feature point matching unit 103 is capable of matching the feature points detected in the two captured images 10 before division as in a conventional case.

Note that, in a case where divided regions are generated by dividing the two captured images 10, the feature point matching unit (feature point matching unit 103) performs matching by associating feature points for each divided region having the same position in the plurality of captured images 10. Therefore, the feature point matching unit 103 is capable of performing matching through association with feature points uniformly detected across the entire captured image 10, as compared with conventional processing in which matching is performed by associating feature points by taking the entire captured image 10 as a target.

The feature point matching processing is as described above with reference to FIGS. 9 and 10. For example, the feature point matching unit 103 utilizes the divided region divided by the region dividing unit 1021, and changes one feature point in the divided region in one captured image 10 to compare the feature point in the divided region corresponding to the other captured image 10. Because the range of the divided regions to be matched is limited by such processing, the feature point matching unit 103 is capable of avoiding erroneous matching with a feature point detected in another divided region, and thus the time required for the matching operation can be shortened.

The relative parameter calculation unit (relative parameter calculation unit 104) calculates relative parameters on the basis of the feature points associated among the plurality of captured images 10 (S4). Therefore, the relative parameter calculation unit 104 calculates relative parameters by using the camera parameters and the matched feature points. The relative parameters are parameters indicating the relationships between the relative positions and postures of one camera relative to another camera before and after movement of two in-vehicle cameras 101 or one moving in-vehicle camera 101. Further, when calculating the relative parameters, the relative parameter calculation unit 104 performs calculation so as to minimize the sum of reprojection errors of all the feature points matched in the two captured images 10.

Here, in a case where the in-vehicle camera (in-vehicle camera 101) is a stereo camera, the relative parameter calculation unit (relative parameter calculation unit 104) uses, as references, the axial position and the axial angle of one of two cameras constituting the stereo camera when the one camera captures a captured image 10, and uses, as the relative parameters, the relative axial position and the relative axial angle of the other camera when the other camera captures the other captured image 10. For example, taking the position of a right camera as a reference, a relative three-dimensional position ((x, y, z) axial position) of a left camera with respect to the reference, and a relative orientation ((x, y, z) axial angle) are used as the relative parameters.

Note that, in a case where the in-vehicle camera 101 is a monocular camera, captured images 10 which are captured before and after the movement of one moving in-vehicle camera 101 are used. However, the posture of the monocular camera hardly becomes constant before and after the movement. For example, under an ideal condition where the vehicle travels straight on a straight line, the posture of the in-vehicle camera 101 is constant. However, it is assumed that the posture of the in-vehicle camera 101 changes because the angle of the in-vehicle camera 101 actually changes to the left and right due to a minute steering wheel operation or because the angle of the in-vehicle camera 101 changes up and down as a result of application of the brakes or accelerator.

Therefore, in a case where the in-vehicle camera (in-vehicle camera 101) is a monocular camera, the relative parameter calculation unit (relative parameter calculation unit 104) uses, as references, the axial position and the axial angle of the monocular camera at the time of capturing one captured image 10 among a plurality of captured images 10 captured by the monocular camera as the vehicle moves, and uses, as relative parameters, the relative axial position and the relative axial angle of the monocular camera at the time of capturing the other captured image 10. For example, taking the position of a monocular camera captured at a certain moment as a reference, a relative three-dimensional position ((x, y, z) axial position) with respect to a reference of the monocular camera captured after a time t, and a relative orientation ((x, y, z) axial angle) are used as the relative parameters. As a result, even in a case where a monocular camera is used as the in-vehicle camera 101, the relative parameters are correctly calculated.

Next, the parallax image generation unit (parallax image generation unit 105) generates a parallax image of the plurality of captured images 10 on the basis of the relative parameters (S5). Therefore, the parallax image generation unit 105 creates a parallelized image which is obtained by parallelizing the two captured images 10 on the basis of the relative parameters. In the parallelized image, the epipolar line is horizontal and has the same Y coordinate. The parallax image generation unit 105 then performs stereo matching on the parallelized image to generate a parallax image.

Next, the three-dimensional distance recognition unit (three-dimensional distance recognition unit 106) recognizes three-dimensional distances in the outside world on the basis of the parallax image (S6). For example, the three-dimensional distance recognition unit 106 calculates three-dimensional distances in the outside world on the basis of the parallax value of the parallax image generated by the parallax image generation unit 105, the focal length of the camera included in the camera parameters, the pixel size of the light receiving element included in the in-vehicle camera