In step S303, the smallest main area is selected from the detected plurality of main areas, and the smaller of the horizontal and vertical sizes of the smallest main area is set as MinA. That is, in the example of FIG. 5A, the smaller of the horizontal and vertical sizes of the pupil area 501 is set as MinA. Then, this value of MinA is set as the length of one side of one AF frame 504 (AF frame size).

In step S305, the horizontal size H of an area including all of the main areas is obtained from the horizontal coordinates and horizontal sizes of the detected main areas, as shown in FIG. 5B, and the number of AF frames in the horizontal direction is determined by dividing the obtained horizontal size H by the AF frame size MinA. Next, in step S307, the vertical size V of the area including all of the main areas is obtained from the vertical coordinates and vertical sizes of the detected main areas, as shown in FIG. 5B, and the number of AF frames in the vertical direction is determined by dividing the obtained vertical size V by the AF frame size MinA, and the AF frame setting process ends. Note that the order of the process of step S305 and the process of step S307 may be reversed, or they may be performed in parallel.

In this embodiment, the AF frame is a square using the length of the shorter side of the smallest main area, but the present invention is not limited to this. For example, the horizontal size and vertical size of the AF frame may be made different, or the size of the AF frame may be set based on the size of an area that includes all the main areas and the number of AF frames that the system control unit 209 can handle.

On the other hand, if the number of detected main areas is not plural, AF frames each having a predetermined size X are set for the detected face in step S304. The predetermined size X may be the pupil size estimated from the face, or may be set to a size that ensures good S/N ratio and sufficient focusing performance can be expected in consideration of a low-illumination environment. In this embodiment, the predetermined size X is set to an estimated pupil size. Then, in step S306, as shown in FIG. 4B, the number of AF frames is set using AF frames of size X that include the face area 401 and can handle cases where the face moves.

Furthermore, if a subject is not detected, in step S308, the size, setting positions, and number of AF frames are determined by a known method. Known methods include, for example, a method of setting one AF frame of a predetermined size in the center, a method of setting AF frames of a predetermined size at a plurality of arbitrary positions, and a method of dividing the screen into multiple areas and using each area as an AF frame. There are also various other methods, such as a user setting the AF frame using the operation unit 214, and the present invention is not limited to these methods.

Note that in the examples shown in FIGS. 4A and 4B and FIGS. 5A and 5B, a plurality of AF frames are set so as to cover the main area/areas, but the present invention is not limited to this, and for example, a plurality of AF frames may be set discretely within the main area or areas.

Once the AF frames are set by the above processing, the process proceeds to step S203 in FIG. 2.

The above-mentioned AF frame setting process does not have to be performed every time an image is input, and may be performed once every plural images are input. In that case, when the AF frame setting process is not performed, the set AF frames may be stored in the DRAM 206, and the AF frames stored at the closest timing may be read out and used.

AF Operation

FIG. 6 is a flowchart explaining the overall flow of the AF operation performed in step S203.

First, in step S401, a focus detection process is performed to detect the defocus amount and the process proceeds to step S402. The focus detection process will be described later.

In step S402, the subject detection information obtained in step S301 is used to perform a main AF frame selection process (described later), and the process proceeds to step S403. In step S403, the main AF frame selection result obtained in step S402 is used to calculate the actuation amount of the focus lens (described later), and the process proceeds to step S404. In step S404, the focus lens actuation amount obtained in step S403 is transmitted to the lens communication unit 210, the focus lens 103 is moved, and the AF operation ends.

Focus Detection Process

Next, the focus detection process performed in step S401 will be described with reference to FIG. 7.

First, in step S501, from the image data output from the image sensor 201, an area of one unprocessed AF frame from the AF frames set in step S202 is set as a focus detection area, and the process proceeds to step S502. In step S502, a pair of image signals (A image signal, B image signal) that are focus detection signals are acquired from pixels in the area of the image sensor 201 that corresponds to the focus detection area set in step S501, and the process proceeds to step S503. In step S503, an averaging process in the vertical direction is performed to average the pairs of image signals acquired in step S502 in the vertical direction, and the process proceeds to step S504. This process makes it possible to reduce the effects of noise in the image signals.

In step S504, a filtering process is performed to extract signal components of a predetermined frequency band from the signals obtained as a result of the averaging process in the vertical direction in step S503, and the process proceeds to step S505. In step S505, correlation amounts are calculated from the signals obtained by the filtering process in step S504, and the process proceeds to step S506. In step S506, correlation change amounts are calculated from the correlation amounts calculated in step S505, and the process proceeds to step S507. In step S507, an image shift amount is calculated from the correlation change amount calculated in step S506, and the process proceeds to step S508. In step S508, reliability indicating how reliable the image shift amount calculated in step S507 is calculated, and the process proceeds to step S509. In step S509, the image shift amount is converted into a defocus amount.

In step S510, it is determined whether or not there is an unprocessed AF frame among the AF frames set in step S202. If there is an unprocessed AF frame, the process returns to step S501 and the above processes are repeated for the next unprocessed AF frame. If all the set AF frames are processed, the focus detection process is ended.

Main Frame Selection Process

FIG. 8 is a flowchart explaining the main AF frame selection process performed in step S402.

First, in step S601, it is determined whether a subject has been detected by the subject detection unit 211. If not, the process proceeds to step S602, where the main AF frame is selected using a first selection method that does not use subject detection information, and the main AF frame selection process ends. As the first selection method, for example, a method of selecting a predetermined area in the screen (for example, an area including the center position) from the AF frame set in step S308 as the main AF frame is considered, but it is not limited to this, and known techniques may be used.

On the other hand, if a subject is detected, the process proceeds to step S603, where a main AF frame is selected from the AF frames for which the focus detection process has been performed using a second selection method, and the process proceeds to step S604.

Here, details of the main AF frame selection process using the second selection method performed in step S603 will be described with reference to the flowcharts of FIGS. 9 and 12.

First, in step S701 in FIG. 9, it is determined whether the subject's pupil is detected by the subject detection unit 211. If an pupil is detected, the process proceeds to step S702, and if no pupil is detected, the process proceeds to step S703. In step S702, the pupil area is added as a main AF frame selection area, and the process proceeds to step S703.

In step S703, it is determined whether or not the face of the subject is detected by the subject detection unit 211. If a face is detected, the process proceeds to step S704, and if a face is not detected, the process proceeds to step S705. In step S704, the face area is added as the main AF frame selection area, and the process proceeds to step S705.

In step S705, it is determined whether or not the torso (body) of the subject is detected by the subject detection unit 211. If the torso is detected, the process proceeds to step S706, and if the torso is not detected, the process proceeds to step S707. In step S706, the torso area is added as the main AF frame selection area, and the process proceeds to step S707.

In step S707, it is determined whether the number of AF frames including the main AF frame selection area is equal to or greater than a first threshold value. Here, the AF frame including the main AF frame selection area is an AF frame whose center is included in the main AF frame selection area, but is not limited thereto. For example, an AF frame may include at least a part of the main AF frame selection area, or an AF frame whose area overlaps the main AF frame selection area by a predetermined ratio or more. If the number of AF frames that include the main AF frame selection area is equal to or greater than a first threshold value, the process proceeds to step S708, otherwise the process proceeds to step S711.

In step S708, the defocus amount calculated for each AF frame including the main AF frame selection area is classified by predetermined depth to create a histogram. Then, in step S709, it is determined whether the peak value of the histogram created in step S708 is equal to or greater than a predetermined second threshold. In this embodiment, the maximum number of AF frames in the histogram is normalized by the total number of AF frames and converted into a percentage, which is used as the peak value. If the peak value is equal to or greater than the second threshold, the process proceeds to step S710, and if it is less than the second threshold, the process proceeds to step S711.

In step S710, the state is such that a first threshold value or more of AF frames exist in the main AF frame selection area, and in one depth (peak value bin) the second threshold value or more of percentage of the defocus amounts of the AF frames exist (hereinafter this state is referred to as the “first state”). In this case, a main AF frame selection process depending on the first state is performed.

On the other hand, in step S711, the number of AF frames included in the main AF frame selection area is less than the first threshold value, or even if it is equal to or greater than the first threshold value, the defocus amounts of the AF frames are scattered at multiple depths (hereinafter this state is referred to as the “second state”). In this case, a main AF frame selection process depending on the second state is performed.

In this embodiment, the main AF frame is selected in different ways in the different states described above.

FIG. 10 is a flowchart illustrating the main AF frame selection process under the first state performed in step S710.

First, in step S801, a loop process is started for all AF frames including the main AF frame selection area in order to select a main AF frame in the main AF frame selection area. In step S802, it is determined whether the AF frame being processed is an AF frame included in the peak value bin of the histogram. If so, the process proceeds to step S803, and if not, the loop process is repeated for another AF frame. In step S803, it is determined whether the AF frame being processed is closer to the center of the main AF frame selection area than the currently selected main AF frame, and if so, the process proceeds to step S804 to update the main AF frame, and if not, the loop process is repeated for another AF frame. The center of the main AF frame selection area may be, for example, the center of the maximum widths of the main AF frame selection area in the horizontal and vertical directions, or the center of gravity of the main AF frame selection area.

When the loop of step S801 ends, the main AF frame selection process under the first state ends. Through the above process, the AF frame that is closest to the center of the main AF frame selection area is selected as the main AF frame from among the AF frames included in the peak value bin.

FIG. 11 is a flowchart illustrating the main AF frame selection process under the second state performed in step S711.

In step S901, it is determined whether the center of the main AF frame selection area is within a predetermined depth, and if it is within the predetermined depth, the process proceeds to step S902, where an AF frame including the center of the main AF frame selection area is set as the main AF frame. On the other hand, if the center of the main AF frame selection area is not within the predetermined depth, the process proceeds to step S903.

In step S903, a loop process is performed for all frames to select a main AF frame from among the AF frames including the set main AF frame selection area. The initial value of the main AF frame is assumed to be information that a state that a main AF frame is not selected (e.g., total number of frames+1) can be determined, which is not shown. In step S904, it is determined whether the defocus amount of the AF frame being processed is within a predetermined depth and indicates that the AF frame being processed is on the closer side than the defocus amount of the currently selected main AF frame, and if these conditions are met, the main AF frame is updated in step S905.

In step S906, it is determined whether or not the main AF frame can be selected by the loop of step S903, and if not, in step S907, the AF frame including the center of the main AF frame selection area is set as the main AF frame.

When the above processing is completed, the process returns to step S603 in FIG. 8.

When the main AF frame is selected by the second selection method in step S603 through the above processes, it is determined in step S604 whether or not the subject's pupil is detected by the subject detection unit 211. If the pupil is detected, the process proceeds to step S605, and if not, the main AF frame selection process ends.

In step S605, a process to select an AF frame that represents a characteristic part such as an pupil is performed.

FIG. 12 is a flow chart explaining the characteristic part representative frame selection process performed in step S605.

First, in step S1001, the pupil area is set as the characteristic part area, and the process proceeds to step S1002. Note that in this embodiment, since a person is assumed to be the subject, the pupil is considered as the characteristic part and the pupil area is set as the characteristic part area, however, the present invention is not limited to this and any part can be set as the characteristic part.

In step S1002, in order to select the characteristic part representative frame, a loop process is performed for all AF frames including the set characteristic part area. The initial value of the characteristic part representative frame is set to information (e.g., total number of frames+1) that can determine that the characteristic part representative frame has not been selected, which is not shown. In step S1003, it is determined whether the defocus amount of the AF frame being processed is within a predetermined depth and indicating that the AF frame being processed is on the closer side than the currently selected characteristic part representative frame, and if the conditions are satisfied, the characteristic part representative frame is updated in step S1004. In step S1005, it is determined whether the characteristic part representative frame is selected through the loop S1002, and if it is not selected, in step S1006, the AF frame including the center of the characteristic part area is set as the characteristic part representative frame.

Then, the main AF frame selection process ends, and the process returns to step S402 in FIG. 6.

Lens Actuation Amount Calculation Process

FIG. 13 is a flowchart explaining the lens actuation amount calculation process in step S403.

First, in step S1301, the focus detection result of the main AF frame selected in step S402 is recorded, and the process proceeds to step S1302. Since the focus detection result of the main AF frame is recorded every time focus detection is performed, it is possible to trace back the history of past focus detection results.

In step S1302, it is determined whether the characteristic part representative frame is selected. If it is selected, the process proceeds to step S1303. If it is not selected, the process proceeds to step S1304.

In step S1303, the relative position of the image plane position of the characteristic part obtained from the main AF frame selected in step S603 and the focus detection result of the characteristic part representative frame selected in step S605 is recorded. The relative position of the characteristic part is obtained by subtracting the image plane position of the main AF frame from the image plane position of the characteristic part representative frame, and indicates the relative position of the image plane position of the characteristic part representative frame with respect to the image plane position of the main AF frame. Since the relative position of the characteristic part is recorded every time focus detection is performed and a characteristic part representative frame is selected, it is possible to trace back the history of past relative positions. When recording of the relative position of the characteristic part is completed, the process proceeds to step S1304.

In step S1304, the subject movement prediction unit 219 performs prediction process to predict the image plane position using the history of focus detection results, and the process proceeds to step S1305. Details of the prediction process will be described later.

In step S1305, the actuation target of the focus lens is set to the predicted image plane position, and the lens actuation amount calculation process ends.

FIG. 14 is a flowchart explaining the prediction process performed in step S1304.

First, in step S1401, prediction process for the entire subject is performed using the focus detection results of the main AF frame recorded in step S1301.

In the prediction process for the entire subject, a prediction curve is drawn based on the focus detection history to obtain the predicted position of the entire subject. A curve 1501 in FIG. 15 shows an example of a prediction curve, with the vertical axis indicating the image plane position and the horizontal axis indicating the time. The larger the image plane position in the vertical axis direction, the farther the distance is, and thus the history shown in FIG. 15 shows a situation of tracking a subject approaching the photographer (camera). As described above, the AF operation in step S203 is performed periodically in the servo shooting mode, and T1 to T5 each indicate the time when the AF operation in step S203 is performed.

A predicted image plane position is obtained by, for example, deriving a prediction curve by the bulk least squares method using the past image plane positions and the respective focus detection times, and calculating the image plane position at the prediction time based on this curve. The predicted time indicates the time of shooting if SW2 is ON, and indicates the current focus detection time if SW2 is not ON. In addition, the prediction method is not limited to the bulk least squares method, and in a case of using, for example, the recursive least squares method, it is not necessary to store the history of a plurality of past focus detection results, and the parameters calculated during the previous AF operation may be recorded to replace the focus detection history. When the prediction process for the entire subject is completed, the process proceeds to step S1402.

In step S1402, the relative position of the characteristic part is predicted using the history of the relative positions of the characteristic parts recorded in step S1303.

In predicting the relative position of the characteristic part, a prediction change amount is obtained by drawing a prediction curve using the history of past relative positions. A curve 1601 in FIG. 16 shows an example of a prediction curve, with the vertical axis indicating the relative position and the horizontal axis indicating the time, and the larger the relative position in the vertical axis direction, the further away the characteristic part is from the entire subject, and thus the history shown in