The subject region acquisition unit 55 acquires, from the entire image represented by the image signal D, a subject region RO including a subject to be acquired. The user can designate a subject (a face of a person, a bird, an airplane, a car, or the like) to be acquired by using the operating device 13. In addition, the user can designate a plurality of types of subjects to be acquired. The subject region acquisition unit 55 supplies information representing the subject region RO to the exposure value calculation unit 56. The entire image represented by the image signal D refers to the entire image represented by a signal output from an effective region of the imaging sensor 20 on which light is incident, and a signal output from an optical black region or the like is excluded. The subject region RO including the subject to be acquired is an example of a “specific subject region” according to the technology of the present disclosure. The specific subject region refers to a region including a subject that is optionally selected by an engineer or a user.

The exposure value calculation unit 56 performs exposure value calculation processing based on the image signal D and information indicating the set region RG and the subject region RO. Although described in detail below, the exposure value calculation unit 56 calculates a first exposure value E1 of the set region RG and a second exposure value E2 of the subject region RO, and calculates a third exposure value E3 as a final exposure value by performing weight addition of the first exposure value E1 and the second exposure value E2. The exposure value calculation unit 56 outputs the calculated third exposure value E3 to the main controller 50. Note that the exposure value is a value determined by the amount of light incident on the light-receiving surface 20A of the imaging sensor 20 in the exposure period and is controlled by the F number and the shutter speed (exposure time). The exposure value is also referred to as an EV value. The first exposure value E1 represents brightness of the set region RG. The second exposure value E2 represents brightness of the subject region RO.

The main controller 50 performs exposure control of controlling the opening diameter of the stop 32 and the shutter speed of the imaging sensor 20 based on the third exposure value E3. In a case of performing the exposure control, the main controller 50 decides a combination of the F number and the shutter speed at which the third exposure value E3 is obtained, using the program line diagram.

Note that, although not illustrated in FIG. 2, the processor 40 is configured to have a function of calculating a defocus amount based on the signal output from the phase difference detection pixels of the imaging sensor 20. The main controller 50 adjusts the focusing position by controlling the focus lens 31 based on the calculated defocus amount.

FIG. 3 illustrates an example of the set region RG. The image represented by the image signal D has a rectangular shape. For example, the set region RG has a rectangular shape, and a centroid O of the set region RG matches with a centroid of the image represented by the image signal D. The set region RG may have the same aspect ratio as the aspect ratio of the image represented by the image signal D. In the present embodiment, the set region RG is a display region, and the size or the aspect ratio of the set region RG is changed based on the size or the like of the recorded image.

FIG. 4 illustrates an example of subject region acquisition processing by the subject region acquisition unit 55. In the present embodiment, the subject region acquisition unit 55 acquires the subject region RO including a subject to be acquired by using a known template matching method. Specifically, the subject region acquisition unit 55 performs search using a rectangular template image according to the subject to be acquired, and acquires, as the subject region RO, a region having the highest rate of match with the template image from the entire image represented by the image signal D. FIG. 4 illustrates a case where the subject region RO is a face region of a person. In the present embodiment, the subject region RO is represented by a rectangular frame.

FIG. 5 illustrates a configuration example of the exposure value calculation unit 56. FIG. 6 conceptually illustrates exposure value calculation processing by the exposure value calculation unit 56. The exposure value calculation unit 56 includes a first exposure value calculation unit 61, a second exposure value calculation unit 62, a first determination unit 63, a second determination unit 64, a weight decision unit 65, and a weight addition unit 66.

The first exposure value calculation unit 61 performs photometry on the set region RG in the image represented by the image signal D, and calculates the first exposure value E1 based on the obtained photometry value. The second exposure value calculation unit 62 performs photometry on the subject region RO in the image represented by the image signal D, and calculates the second exposure value E2 based on the obtained photometry value. For example, the first exposure value calculation unit 61 and the second exposure value calculation unit 62 perform multi-divisional photometry.

The first determination unit 63 performs first determination of determining importance of the subject region RO with respect to the set region RG. In the present embodiment, the importance is a ratio R1 of the size of the subject region RO to the size of the set region RG. More specifically, the ratio R1 is a ratio of the area of the subject region RO to the area of the set region RG. The importance is higher as the ratio R1 increases.

The second determination unit 64 performs second determination of determining a positional relationship between the set region RG and the subject region RO. In the present embodiment, the second determination unit 64 determines whether or not the subject region RO is present in the set region RG. For example, the second determination unit 64 obtains a centroid P of the subject region RO, and determines whether or not the centroid P is present in the set region RG Hereinafter, “the subject region RO is present in the set region RG” refers to “the centroid P of the subject region RO is present in the set region RG”, and “the subject region RO is present outside the set region RG” refers to “the centroid P of the subject region RO is present outside the set region RG”.

In a case where the subject region RO is present outside the set region RG, the second determination unit 64 calculates a ratio R2 of a distance from the centroid O of the set region RG to the centroid P of the subject region RO to a distance from an outer periphery of the set region RG to the centroid P of the subject region RO.

FIG. 7 illustrates an example of a case where the subject region RO is present outside the set region RG. In FIG. 7, B indicates an intersection between a straight line connecting the centroid O and the centroid P and the outer periphery of the set region RG. In this case, the second determination unit 64 calculates a ratio R2 of the distance from the centroid O to the intersection B to the distance from the centroid O to the centroid P. In a case where the subject region RO is present outside the set region RG, R2<1.

FIG. 8 illustrates an example of a case where the subject region RO is present in the set region RG. In FIG. 8, B indicates an intersection between an extension line extending the straight line connecting the centroid O and the centroid P and the outer periphery of the set region RG. In this case, the second determination unit 64 calculates a ratio R2 of the distance from the centroid O to the intersection B to the distance from the centroid O to the centroid P. In a case where the subject region RO is present in the set region RG, R2≥1.

The weight decision unit 65 decides a first weight W1 for the first exposure value E1 and a second weight W2 for the second exposure value E2 based on a result of the first determination by the first determination unit 63 and a result of the second determination by the second determination unit 64. Specifically, the weight decision unit 65 decides a weight coefficient α based on the result of the first determination, and decides a correction coefficient β based on the result of the second determination. Then, the weight decision unit 65 decides the first weight W1 and the second weight W2 such that Expression (1) and Expression (2) are satisfied.

$\begin{matrix} W 1 = 1 - α \times β & (1) \end{matrix}$

$\begin{matrix} W 2 = α \times β & (2) \end{matrix}$

Here, the weight coefficient α is a coefficient representing the importance of the subject region RO with respect to the set region RG obtained by the first determination. The correction coefficient β is a coefficient representing the positional relationship between the set region RG and the subject region RO that is obtained by the second determination. In the present embodiment, the weight coefficient α is a coefficient depending on the above-described ratio R1. The correction coefficient β is a coefficient depending on the above-described ratio R2.

FIG. 9 illustrates an example of a relationship between the weight coefficient α and the ratio R1. The relationship between the weight coefficient α and the ratio R1 is defined in advance, and is stored in the memory 42, for example, as data. In the relationship illustrated in FIG. 9, α=1 in a case where R1≥1, and the weight coefficient α decreases as the ratio R1 decreases in a case where R1<1.

The weight decision unit 65 decides the weight coefficient α based on the ratio R1 obtained by the first determination by the first determination unit 63 and data representing the relationship between the weight coefficient α and the ratio R1. In a case where the size of the subject region RO is equal to or larger than the size of the set region RG, the weight coefficient α is set to a fixed value (in the example illustrated in FIG. 9, “1”). On the other hand, in a case where the size of the subject region RO is smaller than the size of the set region RG, the weight coefficient α decreases as the size of the subject region RO decreases. That is, the weight coefficient α monotonically increases as the ratio R1 increases.

Note that, in a range of 0≤R1<1, a relationship between the weight coefficient α and the ratio R1 may be linear. In addition, in the range of 0≤R1<1, a relationship between the weight coefficient α and the ratio R1 may be non-linear.

FIG. 10 illustrates an example of a relationship between the correction coefficient β and the ratio R2. The relationship between the correction coefficient β and the ratio R2 is defined in advance, and is stored in the memory 42, for example, as data. In the relationship illustrated in FIG. 10, R=1 in a case where R2≥1, and the correction coefficient β decreases as the ratio R2 decreases in a case where R2<1.

The weight decision unit 65 decides the correction coefficient β based on the ratio R2 obtained by the second determination by the second determination unit 64 and the data representing the relationship between the correction coefficient β and the ratio R2. In a case where the subject region RO is present in the set region RG, the correction coefficient β is set to a fixed value (in the example illustrated in FIG. 10, “1”). On the other hand, in a case where the subject region RO is present outside the set region RG, the correction coefficient R decreases as the distance between the subject region RO and the set region RG (for example, the distance between the centroid O and the centroid P) increases. That is, the correction coefficient β monotonically increases as the ratio R2 increases.

In a range of 0≤R2<1, a relationship between the correction coefficient β and the ratio R2 may be linear. In the range of 0≤R2<1, a relationship between the correction coefficient β and the ratio R2 may be non-linear. In addition, in the range of 0≤R2<1, the correction coefficient β may be set to a fixed value (for example, “0”).

As described above, in a case where it is determined in the second determination that the subject region RO is present in the set region RG, the weight decision unit 65 decides the correction coefficient β to 1. That is, in a case where it is determined in the second determination that the subject region RO is present in the set region RG, the weight decision unit 65 decides the first weight W1 and the second weight W2 regardless of the positional relationship between the set region RG and the subject region RO.

On the other hand, in a case where it is determined in the second determination that the subject region RO is present outside the set region RG, the weight decision unit 65 decides the correction coefficient β based on the positional relationship between the set region RG and the subject region RO. That is, in a case where it is determined in the second determination that the subject region RO is present outside the set region RG, the weight decision unit 65 decides the first weight W1 and the second weight W2 based on the positional relationship between the set region RG and the subject region RO.

The weight addition unit 66 calculates the third exposure value E3 by performing weight addition on the first exposure value E1 and the second exposure value E2 using the first weight W1 and the second weight W2. Specifically, the weight addition unit 66 calculates the third exposure value E3 by using a relational expression represented by Expression (3). Here, a relationship of 0≤W1≤1 and 0≤W2≤1 is satisfied.

$\begin{matrix} E 3 = E 1 \times W 1 + E 2 \times W 2 & (3) \end{matrix}$

FIG. 11 illustrates a flow of the overall processing executed by the imaging apparatus 10. FIG. 11 illustrates an example of a flow of processing in the video capturing mode.

First, the main controller 50 determines whether or not an imaging start instruction is issued by the user through an operation of the operating device 13 (step S10). In a case where an imaging start instruction is issued (YES in step S10), the main controller 50 controls the imaging controller 51 to cause the imaging sensor 20 to perform an imaging operation (step S11).

The image acquisition unit 52 acquires the image signal D for one frame that is output from the imaging sensor 20 (step S12). The region decision unit 54 decides the set region RG based on the output information from the operating device 13, and sets the set region RG in the image represented by the image signal D (step S13). The display controller 53 displays the image in the set region RG among the images represented by the image signals D on the display 15 (step S14).

The subject region acquisition unit 55 acquires the subject region RO including a subject to be acquired, from the image represented by the image signal D (step S15). The exposure value calculation unit 56 performs exposure value calculation processing based on the image signal D and the information indicating the set region RG and the subject region RO (step S16). The main controller 50 performs exposure control based on the third exposure value E3 calculated by the exposure value calculation processing (step S17).

FIG. 12 illustrates an example of a flow of the exposure value calculation processing. First, the first exposure value calculation unit 61 calculates the first exposure value E1 as described above (step S20). Next, the exposure value calculation unit 56 determines whether or not the subject region RO is acquired by the subject region acquisition unit 55 (step S21). In a case where the subject region RO is not acquired (NO in step S21), the exposure value calculation unit 56 sets the first exposure value E1 calculated in step S20 as the third exposure value E3 (step S22), and ends the exposure value calculation processing.

On the other hand, in a case where the subject region RO is acquired (YES in step S21), the second exposure value calculation unit 62 calculates the second exposure value E2 as described above (step S23). Next, the first determination unit 63 performs the first determination described above (step S24). The weight decision unit 65 decides the weight coefficient α based on a result of the first determination as described above (step S25). For example, processing of step S25 is performed by applying the ratio R1 obtained in the first determination to the relationship between the weight coefficient α and the ratio R1 (refer to FIG. 9).

Next, the second determination unit 64 performs the second determination described above (step S26). The weight decision unit 65 determines whether or not the subject region RO is present outside the set region RG based on a result of the second determination (step S27). In a case where the subject region RO is present outside the set region RG (YES in step S27), the weight decision unit 65 decides the correction coefficient β based on the positional relationship between the set region RG and the subject region RO as described above (step S28). On the other hand, in a case where the subject region RO is present in the set region RG (NO in step S27), the weight decision unit 65 decides the correction coefficient β to 1 (step S29). For example, processing of step S27 to step S29 is performed by applying the ratio R2 obtained in the second determination to the relationship between the correction coefficient β and the ratio R2 (refer to FIG. 10).

After step S28 or step S29, the weight decision unit 65 decides the first weight W1 and the second weight W2 based on the decided weight coefficient α and the decided correction coefficient β and Expression (1) and Expression (2) (step S30). In addition, the weight addition unit 66 calculates the third exposure value E3 based on the decided first weight W1 and the decided second weight W2 and Expression (3) (step S31). Through the above processes, the exposure value calculation processing is ended.

As described above, in the technology of the present disclosure, by deciding the first weight W1 for the first exposure value E1 of the set region RG and the second weight W2 for the second exposure value E2 of the subject region RO based on the first determination of determining the importance of the subject region RO with respect to the set region RG and the second determination of determining the positional relationship between the set region RG and the subject region RO, and performing weight addition on the first exposure value E1 and the second exposure value E2 using the first weight W1 and the second weight W2, the third exposure value E3 is calculated.

In the technology of the related art, since the positional relationship between the set region RG and the subject region RO is not considered in a case of calculating the third exposure value E3 which is a final exposure value, the third exposure value E3 does not depend on whether or not the subject region RO is present in the set region RG. For this reason, even in a case where the subject region RO is not present in the set region RG, the third exposure value E3 is influenced by the second exposure value E2 of the subject region RO. For example, when displaying the image of the set region RG, even in a case where the subject region RO, such as a person, is not present in the set region RG and a scene in the set region RG does not change, the brightness of the image to be displayed may change due to the influence of the subject region RO in a case where the subject region RO is present outside the set region RG as illustrated in FIG. 7. That is, in the technology of the related art, the presence of the subject region RO outside the set region RG may cause an inappropriate variation in brightness of the set region RG.

According to the technology of the present disclosure, the third exposure value E3 is calculated in consideration of the positional relationship between the set region RG and the subject region RO, and thus it is possible to prevent an inappropriate variation in brightness of the set region RG.

Hereinafter, various modification examples of the above-described embodiment will be described.

First Modification Example

In the embodiment described above, the weight decision unit 65 decides the first weight W1 and the second weight W2 based on the result of the first determination and the result of the second determination. On the other hand, the weight decision unit 65 may decide the first weight W1 and the second weight W2 in consideration of the first exposure value E1 and the second exposure value E2.

FIG. 13 illustrates a configuration of an exposure value calculation unit 56 according to a first modification example. As illustrated in FIG. 13, in the present modification example, the weight decision unit 65 decides the first weight W1 and the second weight W2 using the first exposure value E1 and the second exposure value E2 in addition to the result of the first determination and the result of the second determination. The other configurations of the exposure value calculation unit 56 according to the present modification example are the same as those of the above-described embodiment.

FIG. 14 illustrates a flow of exposure value calculation processing according to the first modification example. The exposure value calculation processing according to the present modification example is different from the exposure value calculation processing according to the above-described embodiment illustrated in FIG. 12 in that step S40 and step S41 are added between step S25 and step S26.

After the weight coefficient α is decided in step S25, the weight decision unit 65 determines whether or not the ratio R1 obtained in the first determination is smaller than a first threshold value (step S40). A fact that the ratio R1 is smaller than the first threshold value means that the size of the subject region RO is small with respect to the size of the set region RG and the importance of the subject region RO is low. That is, in step S40, it is determined whether or not the importance of the subject region RO is lower than a certain value.

In a case where the ratio R1 is not smaller than the first threshold value (NO in step S40), the weight decision unit 65 proceeds to processing of step S26. That is, in a case where the importance of the subject region RO is high, the same processing as in the embodiment described above is performed.

On the other hand, in a case where the ratio R1 is smaller than the first threshold value (YES in step S40), the weight decision unit 65 determines whether or not an absolute value of a difference between the first exposure value E1 and the second exposure value E2 is smaller than a second threshold value (step S41). A fact that the absolute value is smaller than the second threshold value means that a difference in brightness between the subject region RO and the background is small. That is, in step S41, it is determined whether or not the difference in brightness between the subject region RO and the background is smaller than a certain value.

In a case where the absolute value is not smaller than the second threshold value (NO in step S41), the weight decision unit 65 proceeds to processing of step S26. That is, even in a case where the importance of the subject region RO is low, the same processing as in the embodiment is performed in a case where the difference in brightness from the background is large.

On the other hand, in a case where the absolute value is smaller than the second threshold value (YES in step S41), the weight decision unit 65 proceeds to processing of step S29 and decides the correction coefficient β to 1. That is, in a case where the importance of the subject region RO is low and the difference in brightness from the background is small, the correction coefficient β is decided to 1 regardless of the positional relationship between the set region RG and the subject region RO, and the first weight W1 and the second weight W2 are decided.

As described above, in the present modification example, in a case where the ratio R1 of the size of the subject region RO to the size of the set region RG is smaller than the first threshold value and the absolute value of the difference between the first exposure value E1 and the second exposure value E2 is smaller than the second threshold value, the weight decision unit 65 decides the first weight W1 and the second weight W2 regardless of the positional relationship between the set region RG and the subject region RO.

In a case where the importance of the subject region RO is low and the difference in brightness from the background is small, the third exposure value E3 is less likely to be influenced by the second exposure value E2 of the subject region RO. However, in such a case, in a case where the correction coefficient β is decided based on the positional relationship between the set region RG and the subject region RO, the third exposure value E3 is changed in accordance with a change in the positional relationship, and unnecessary flickering may occur in brightness of the set region RG. In the present modification example, in a case where the importance of the subject region RO is low and the difference in brightness from the background is small, the first weight W1 and the second weight W2 are decided regardless of the positional relationship. Therefore, it is possible to prevent unnecessary flickering in brightness of the set region RG.

Second Modification Example

In the first modification example, the weight decision unit 65 decides the first weight W1 and the second weight W2 by using the first exposure value E1 and the second exposure value E2 in addition to the result of the first determination and the result of the second determination. On the other hand, the first weight W1 and the second weight W2 may be decided in consideration of the information of the subject included in the subject region RO.

The configuration of the exposure value calculation unit 56 according to the present modification example is the same as that of the first modification example, except that the weight decision unit 65 acquires the information of the subject included in the subject region RO from the subject region acquisition unit 55.

FIG. 15 illustrates a flow of exposure value calculation processing according to the second modification example. The exposure value calculation processing according to the present modification example is different from the exposure value calculation processing according to the first modification example illustrated in FIG. 14 in that step S42 is added between step S41 and step S29.

In a case where the absolute value of the difference between the first exposure value E1 and the second exposure value E2 is smaller than the second threshold value (YES in step S41), the weight decision unit 65 determines whether or not the subject included in the subject region RO is a specific subject (step S42). The type of the specific subject is stored in the memory 42 in advance as data. For example, the specific subject is an object such as a drone that moves at a high speed or a bird. The specific subject is a subject that can move at a high speed in the image represented by the image signal D. Note that the user may set the type of the specific subject by using the operating device 13. The specific subject is a subject that is optionally selected by an engineer or a user.

In a case where the subject included in the subject region RO is not the specific subject (NO in step S42), the processing proceeds to step S26. That is, even in a case where the importance of the subject region RO is low and the difference in brightness from the background is small, the same processing as in the embodiment is performed in a case where the subject included in the subject region RO is not the specific subject.

On the other hand, in a case where the subject included in the subject region RO is the specific subject (YES in step S42), the processing proceeds to step S29, and the correction coefficient β is decided to 1. That is, in a case where the importance of the subject region RO is low, the difference in brightness from the background is small, and the subject included in the subject region RO is the specific subject, the correction coefficient β is decided to 1 regardless of the positional relationship between the set region RG and the subject region RO, and the first weight W1 and the second weight W2 are decided.

As described above, in the present modification example, in a case where the ratio R1 of the size of the subject region RO to the size of the set region RG is smaller than the first threshold value, the absolute value of the difference between the first exposure value E1 and the second exposure value E2 is smaller than the second threshold value, and the subject included in the subject region RO is the specific subject, the weight decision unit 65 decides the first weight W1 and the second weight W2 regardless of the positional relationship between the set region RG and the subject region RO.

In a case where the subject included in the subject region RO is a specific subject that moves at a high speed, the positional relationship between the set region RG and the subject region RO is likely to change. As a result, unnecessary flickering may occur in brightness of the set region RG. In the present modification example, in a case where the subject included in the subject region RO is the specific subject, the first weight W1 and the second weight W2 are decided regardless of the positional relationship between the set region RG and the subject region RO. Therefore, it is possible to prevent unnecessary flickering in brightness of the set region RG.

Third Modification Example

In the embodiment, the subject region acquisition unit 55 acquires the subject region RO by a template matching method. On the other hand, the subject region RO may be acquired by using a machine-trained model.

FIG. 16 illustrates subject region acquisition processing according to the third modification example. As illustrated in FIG. 16, the machine-trained model LM is configured with a neural network including an input layer, an intermediate layer, and an output layer. The intermediate layer includes a plurality of neurons. The number of the intermediate layers and the number of neurons in each intermediate layer can be changed as appropriate. For example, the trained model LM is a CNN. The machine-trained model LM is obtained by performing machine learning using a plurality of images including a predetermined subject region as training data.

The subject region acquisition unit 55 inputs the image signal D to the machine-trained model LM. The machine-trained model LM specifies the subject region RO from the image represented by the image signal D, and outputs the specified subject region RO. For example, the machine-trained model LM outputs, as a heat map, a result obtained by performing segmentation for each pixel. For example, in a case where the subject to be acquired is a face of a person, a set of pixels of which the class belongs to a face of a person is specified as the subject region RO.

As described above, the subject region RO is acquired using the machine-trained model, and thus the subject region RO is acquired with high accuracy. Therefore, in the first determination, it is possible to determine the positional relationship between the set region RG and the subject region RO with higher accuracy. Thereby, it is possible to prevent an inappropriate variation in brightness of the set region RG with higher accuracy.

Other Modification Examples

In the embodiment, the set region RG is used as an image display region for displaying an image in the set region RG. On the other hand, the set region RG may be an image recording region for recording an image in the set region RG. In this case, the image in the set region RG is recorded in a storage device, such as the memory 42, as moving image data.

In addition, the set region RG may be an image cutout region for electronic zoom. In this case, as illustrated in FIG. 17, the size of the set region RG is changed in accordance with the zoom magnification of the electronic zoom. Specifically, the set region RG decreases as the zoom magnification of the electronic zoom increases.

In addition, the set region RG may be an image cutout region for camera shake correction. In this case, as illustrated in FIG. 17, the position of the set region RG is changed to cancel out the camera shake based on an output signal from an angular velocity sensor (not illustrated) such as a gyro sensor provided in the imaging apparatus 10.

In addition, in the embodiment, the first determination unit 63 determines the ratio R1 of the size of the subject region RO to the size of the set region RG as the importance of the subject region RO with respect to the set region RG. On the other hand, the importance is not limited to the ratio R1. For example, the importance may be a type of the subject. In this case, the weight decision unit 65 increases the weight coefficient α as the type of the subject has higher importance.

In addition, in the embodiment, the positional relationship between the set region RG and the subject region RO is determined based on the centroid P of the subject region RO. On the other hand, the criterion for determining the positional relationship is not limited to the centroid P of the subject region RO. For example, it may be determined that the subject region RO is present outside the set region RG in a case where at least a part of the subject region RO is present outside the set region RG, and it may be determined that the subject region RO is present in the set region RG in a case where the entire subject regions RO is present in the set region RG. On the contrary, it may be determined that the subject region RO is present in the set region RG in a case where at least a part of the subject region RO is present in the set region RG, and it may be determined that the subject region RO is present outside the set region RG in a case where the entire subject regions RO is present outside the set region RG.