3 by repeating processing of determining whether or not persons appearing in captured images of different fixed cameras 3 are the same person using captured images of the plurality of fixed cameras 3 (in the example of FIG. 4, three fixed cameras 3a to 3c). Through this tracking processing, it is possible to know what kind of path each person has followed in the store S.

The processing of determining whether or not the persons appearing in the captured images of the above-described different fixed cameras 3 are the same person is performed, for example, in the following manner in consideration of consistency between the time of camera transition of the object and the current position of the object. As illustrated in the query X of FIG. 4, for persons (person 1 and person 2) appearing in the captured image of the fixed camera 3b at 15:00, matching is performed only with a person passing through the photographing range of the fixed camera 3a (appearing in the captured image of the fixed camera 3a) within 2 minutes (14:58 to 15:00). Thus, it is possible to know that the person 1 has moved from the photographing range of the fixed camera 3a to the photographing range of the fixed camera 3b. Furthermore, as illustrated in the query Y of FIG. 4, for the person (person 3) appearing in the captured image of the fixed camera 3c at 16:00, matching is performed only with a person passing through the photographing range of the fixed camera 3a within 10 minutes (15:50 to 16:00) (appearing in the captured image of the fixed camera 3a) and a person passing through the fixed camera 3b within 2 minutes (14:58 to 15:00) (appearing in the captured image of the fixed camera 3b). Thus, it is possible to know that the person 3 has moved from the photographing range of the fixed camera 3a to the photographing range of the fixed camera 3c.

Next, an outline of setting processing of multi-camera object tracking parameter performed by the above-described analysis server 1 will be described with reference to FIG. 5. FIG. 5 illustrates an input/output relationship of each block (that is, each of the blocks constituting the object tracking parameter setting support system 40) related to (automatic) setting processing of parameters (various types of spatial information) used for multi-camera object tracking and a block related to multi-camera object tracking processing with path limitation among functional blocks of the CPU 31 of the analysis server 1. An input circuitry 41, a mapping circuitry 43, a single camera object tracking circuitry 44, an in/out-area estimation circuitry 45, a region of interest estimation circuitry 46, an edge information estimation circuitry 48, an entrance/exit estimation circuitry 49, and a setting circuitry 50 in FIG. 5 correspond to an input circuitry, a mapping circuitry, a single camera object tracking circuitry, an in/out-area estimation circuitry, a region of interest estimation circuitry, a movable path estimation circuitry, an entrance/exit estimation circuitry, and a setting circuitry in the claims, respectively. Furthermore, the multi-camera object tracking circuitry 47 without path limitation and the multi-camera object tracking circuitry 51 with path limitation in FIG. 5 correspond to multi-camera object tracking circuitry in the claims.

The above-described input circuitry 41 receives an input of a captured image from each of the plurality of fixed cameras 3 (for example, in FIG. 4, the fixed cameras 3a to 3c) and data of a two-dimensional map (hereinafter, referred to as a “two-dimensional store map”) of a shop in which the plurality of fixed cameras 3 described above is arranged. The two-dimensional store map is, for example, a map of a store (two-dimensional store map 60) as illustrated in FIG. 6. Note that, in FIG. 6, the two-dimensional store map 60 is illustrated in a simplified manner for convenience of description.

The mapping circuitry 43 illustrated in FIG. 5 associates each point (each coordinate) in the captured image of each of the plurality of fixed cameras 3 with each point in the above-described two-dimensional store map 60.

Furthermore, the single camera object tracking circuitry 44 performs single camera object tracking, which is processing of tracking one or more tracking target persons within the photographing range of one camera, on the basis of a captured image from the above-described one camera (for example, the fixed camera 3a) among the plurality of fixed cameras 3. The in/out-area estimation circuitry 45 performs single camera object tracking by the single camera object tracking circuitry 44 on the captured image in a predetermined period of each of the plurality of fixed cameras 3, and estimates an in-area that is an area in which a person enters and an out-area that is an area in which a person exits in the photographing range of each of the plurality of fixed cameras 3 from statistical information of tracking results thereof. In addition, the region of interest estimation circuitry 46 performs the single camera object tracking by the single camera object tracking circuitry 44 on the captured image in the predetermined period of each of the plurality of fixed cameras 3, and estimates a region of interest which is a region through which the tracking target person is able to pass or a region through which the tracking target person is unable to pass in the photographing range of each of the plurality of fixed cameras 3 from the statistical information of tracking results.

In addition, the multi-camera object tracking circuitry 47 without path limitation in FIG. 5 performs multi-camera object tracking of tracking a plurality of tracking target persons across the photographing ranges of the plurality of fixed cameras 3 without limitation of the movable path with respect to the captured images of the predetermined period from the plurality of fixed cameras 3. Here, the above-described “movable path” means a path between the fixed cameras 3 on the two-dimensional store map 60 on which a person is able to move. Furthermore, the edge information estimation circuitry 48 estimates the above-described movable path from statistical information of multi-camera object tracking results by the above-described multi-camera object tracking circuitry 47 without path limitation. Note that the naming of the edge information estimation circuitry 48 is derived from the fact that each edge in a graph representing a path between the fixed cameras 3 on which a person is able to move is set as the above-described movable path in graph information representing a relationship between the fixed cameras 3 described later. Note that each node (Node) in the graph representing the above-described path between the fixed cameras 3 on which a person is able to move is each fixed camera 3. Reference numerals 55, 56, and 57 in FIG. 6 represent a path in an AB direction (that is, a path between the fixed camera 3a and the fixed camera 3b), a path in a BC direction (that is, a path between the fixed camera 3b and the fixed camera 3c), and a path in a AC direction (that is, a path between the fixed camera 3a and the fixed camera 3c), respectively.

In addition, the entrance/exit estimation circuitry 49 estimates an entrance area 58 and an exit area 59 (see FIG. 6) of the store S in which the plurality of fixed cameras 3 is arranged from the statistical information of the multi-camera object tracking results by the above-described multi-camera object tracking circuitry 47 without path limitation. Then, the setting circuitry 50 sets the in-area and the out-area estimated by the in/out-area estimation circuitry 45, the region of interest estimated by the region of interest estimation circuitry 46, the movable path estimated by the edge information estimation circuitry 48, the entrance and exit information (the entrance area 58 and the exit area 59) estimated by the entrance/exit estimation circuitry 49, and information of an association result (result of association between each point in a captured image of each fixed camera 3 and each point in the two-dimensional store map 60) performed by the mapping circuitry 43 as parameters used for multi-camera object tracking. Note that, for multi-camera object tracking by the multi-camera object tracking circuitry 47 without path limitation, information of single camera object tracking result by the single camera object tracking circuitry 44 is used. Thus, as described above, in a case where both the in-area and out-area and the region of interest estimated from statistical information of the single camera object tracking results and the movable path and the entrance and exit information (the entrance area 58 and the exit area 59) estimated from the statistical information of the multi-camera object tracking results without path limitation are set as parameters to be used for multi-camera object tracking, it is desirable to connect the multi-camera object tracking circuitry 47 without path limitation to the subsequent stage of the single camera object tracking circuitry 44 in series to reduce the calculation amount of the entire object tracking parameter setting support system 40.

Among the functional blocks of the CPU 31 of the analysis server 1 in FIG. 5, the input circuitry 41, the mapping circuitry 43, the single camera object tracking circuitry 44, the in/out-area estimation circuitry 45, the region of interest estimation circuitry 46, the multi-camera object tracking circuitry 47 without path limitation, the edge information estimation circuitry 48, the entrance/exit estimation circuitry 49, and the setting circuitry 50 constitute the object tracking parameter setting support system 40. In addition, the multi-camera object tracking circuitry 51 with path limitation in FIG. 5 performs multi-camera object tracking of tracking a plurality of tracking target persons across the photographing ranges of the plurality of fixed cameras 3 with limitation of the movable path on the basis of the captured images from the plurality of fixed cameras 3 on the basis of various multi-camera object tracking parameters set by the setting circuitry 50 of the object tracking parameter setting support system 40 described above. Note that the multi-camera object tracking processing by the multi-camera object tracking circuitry 51 with path limitation described above is actual multi-camera object tracking processing performed by the multi-camera object tracking system 10 on the basis of various multi-camera object tracking parameters set in advance by the object tracking parameter setting support system 40.

In order to ensure the accuracy and speed of the above-described actual multi-camera object tracking processing, it is very effective to utilize various types of spatial information including the information regarding the above-described movable path. In particular, in-information (information of an in-area which is an area where a person enters in a photographing range of each fixed camera 3) and out-information (information of an out-area which is an area where a person goes out in the photographing range of each fixed camera 3) to the photographing range of each fixed camera 3, and information of areas at the entrance and the exit of the store S (see the entrance area 58 and the exit area 59 in FIG. 6) are very important in addition to the above information regarding the movable path, and it is possible to implement accurate and high-speed multi-camera object tracking processing by performing the actual multi-camera object tracking processing with path limitation after setting these pieces of spatial information in advance.

As described above, in order to ensure the accuracy and processing speed of multi-camera object tracking, it is very effective to utilize various types of spatial information including information regarding a movable path. However, manually setting the above spatial information requires economical and temporal costs.

Therefore, in the object tracking parameter setting support system 40 of the present embodiment, the above-described various types of spatial information (including information regarding the movable path) are automatically obtained on the basis of the captured images from the plurality of fixed cameras 3 and the data of the two-dimensional store map 60, and the spatial information is described in graph information (see FIG. 7) expressing the relationship between the respective fixed cameras 3 (that is, various kinds of spatial information are automatically graphically structured and expressed). Then, the actual multi-camera object tracking processing with path limitation is performed using the graph information (the graph information is used as an input of the multi-camera object tracking processing), and thus it is possible to save time and effort to manually set the above-described spatial information, and to suppress economic and temporal costs. In addition, by using the above-described movable path, the in-information and the out-information, the information of the areas at the entrance and the exit, and the information of the region of interest, the path between the cameras on which a person is able to move, the area in which the person enters and the area in which the person exits (which may be a direction in which the person enters and a direction in which the person exits) in the photographing range of each fixed camera 3, the entrance and the exit of the store S, and the region through which the tracking target person is able to pass (or the region through which the tracking target person is unable to pass) are narrowed down, and the actual multi-camera object tracking processing with path limitation is performed, so that accurate and high-speed multi-camera object tracking processing can be performed.

Next, in addition to a flowchart of FIG. 8, processing performed by the multi-camera object tracking system 10 of the present embodiment will be described with reference to FIGS. 5, 6, 7, 9, 10, and 11. Note that processing from S1 to S6 in the flowchart of FIG. 8 is processing performed by the object tracking parameter setting support system 40. First, the input circuitry 41 (see FIG. 5) of the analysis server 1 receives an input of the captured image from each fixed camera 3 and the data of the two-dimensional store map 60 described above (see FIGS. 6 and 7) (S1).

Next, the mapping circuitry 43 of the analysis server 1 performs association (mapping) of each point (each coordinate) in the captured image of each fixed camera 3 with each point in the two-dimensional store map 60 (S2). More specifically, the mapping circuitry 43 associates each point (for example, each coordinate of a captured image of a 360 degree camera (fisheye camera)) in the captured image of each fixed camera 3 arranged in the store with each point in the two-dimensional store map 60 using conversion by a mathematical formula on the basis of the installation location and height of each fixed camera 3.

Next, with reference to the result of the association (mapping) performed in the above-described S2, the multi-camera object tracking circuitry 47 without path limitation and the single camera object tracking circuitry 44 of the analysis server 1 perform multi-camera object tracking without movable path limitation including single camera object tracking on the captured images (moving images in a certain period) in a predetermined period from the plurality of fixed cameras 3 (S3).

Next, the in/out-area estimation circuitry 45 and the region of interest estimation circuitry 46 of the analysis server 1 estimate (limit) the in-area, the out-area, and the region of interest in the photographing range of each fixed camera 3 from the statistical information of single camera object tracking results by the single camera object tracking circuitry 44 (S4).

First, estimation processing of the above-described in-area and out-area will be described. For example, as illustrated in FIG. 11, the in/out-area estimation circuitry 45 of the analysis server 1 estimates an in-area 62, which is an area where a person enters and an out-area 63, which is an area where a person exits, in a photographing range 61 of each fixed camera 3 from the statistical information of the single camera object tracking results by the single camera object tracking circuitry 44. Then, the in/out-area estimation circuitry 45 estimates an in-direction (direction in which a person enters (for example, “left”, “right”, and “down”)) and an out-direction (direction in which a person exits (for example, “right” and “down”)) for each node (for example, Node A (fixed camera 3a)) illustrated in FIG. 7 from estimation results of the in-area and out-area described above.

Next, estimation processing of a region of interest in S4 described above will be described. The region of interest estimation circuitry 46 of the analysis server 1 estimates a region of interest (ROI) which is a region through which the tracking target person is able to pass or a region through which the tracking target person is unable to pass in the photographing range 61 of each fixed camera 3 from the statistical information of the single camera object tracking results by the single camera object tracking circuitry 44. The region through which the tracking target person is unable to pass is, for example, a region where there is an obstacle (a product shelf or the like) in the photographing range of each fixed camera 3. In a case where the shape of the region is a square or a rectangle, the ROI (region of interest) is defined by a width (horizontal length), a height (vertical length), and an x coordinate and a y coordinate of the origin as illustrated in FIG. 7. Note that, as described above, when the shape of the ROI is a square or a rectangle, the origin is (the coordinates of) the upper left corner of the ROI, for example.

Next, the edge information estimation circuitry 48 of the analysis server 1 estimates the above-described movable path (edge information) and the entrance area 58 and the exit area 59 of the store S from the statistical information of the multi-camera object tracking results without path limitation by the above-described multi-camera object tracking circuitry 47 without path limitation (S5).

First, estimation processing of the above-described movable path will be described. For example, the edge information estimation circuitry 48 of the analysis server 1 estimates (determines) the above-described movable path by removing (excluding) paths equal to or less than a threshold in the statistical information 81 (see FIG. 9) of the above-described tracking results. More specifically, the edge information estimation circuitry 48 estimates the above-described movable path by removing, from candidates of the movable path, a path between the fixed cameras 3 through which fewer people have passed as a result of the multi-camera object tracking without path limitation by the above-described multi-camera object tracking circuitry 47 without path limitation. For example, in the case of the example illustrated in FIG. 9, since the threshold is 10, the edge information estimation circuitry 48 removes A→C (path from captured image A to captured image C (that is, a path from the fixed camera 3a to the fixed camera 3c)), B→D (path from captured image B to captured image D (that is, a path from the fixed camera 3b to the fixed camera 3d)), C→A (path from captured image C to image A (that is, a path from the fixed camera 3c to the fixed camera 3a)), and D→B (path from captured image D to captured image B (that is, a path from the fixed camera 3d to the fixed camera 3b)) from the candidates of the movable path, and estimates that the remaining path (A→B, A→D, B→A, B→C, C→B, C→D, D→A, and D→C) is the movable path. That is, the edge information estimation circuitry 48 estimates that the edge information (movable path) in the two-dimensional store map 60 of FIG. 9 is A⇔B, B⇔C, C⇔D, and A⇔D.

In the example illustrated in FIG. 9, the edge information (movable path) related to Node A (fixed camera 3a) is estimated to be A⇔B and A⇔D. In this example, since there are obstacles called shelves 82 in A⇔C (path connecting captured image A and captured image C (that is, the path connecting the fixed camera 3a and the fixed camera 3c)), a flow of people from the fixed camera 3a directly (linearly) toward the fixed camera 3c does not occur, and data of a tracking result (of people) passing through such a path (path) needs to be eliminated. Therefore, the edge information estimation circuitry 48 removes A⇔C (the path of A→C and the path of C→A) from the candidates of the edge information (movable path) of Node A (the fixed camera 3a).

Next, estimation processing of the entrance area 58 and the exit area 59 (see FIG. 10) in S5 described above will be described. The entrance/exit estimation circuitry 49 of the analysis server 1 estimates the entrance area 58 and the exit area 59 (see FIG. 10) of the store S in which the plurality of fixed cameras 3 is arranged from the statistical information of the multi-camera object tracking results by the above-described multi-camera object tracking circuitry 47 without path limitation. Then, from these estimation results, the entrance/exit estimation circuitry 49 estimates entrance information and exit information (whether or not there are an entrance and an exit within the photographing range of each fixed camera 3 and directions thereof) for each node (for example, Node A (fixed camera 3a)).

For example, the entrance/exit estimation circuitry 49 of the analysis server 1 estimates (determines) the entrance area 58 and the exit area 59 by removing (excluding) routes equal to or less than a threshold from routes (drawn with one stroke) in statistical information 83 (see FIG. 10) of the multi-camera object tracking results by the above-described multi-camera object tracking circuitry 47 without path limitation. More specifically, the entrance/exit estimation circuitry 49 narrows down the entrance and exit routes described above by removing a route between the fixed cameras 3 where fewer people have passed as a result of the multi-camera object tracking without path limitation by the multi-camera object tracking circuitry 47 without path limitation described above from candidates for the route from the entrance area 58 to the exit area 59 (hereinafter, referred to as “entrance and exit route”). Then, the entrance/exit estimation circuitry 49 estimates (determines) the in-area 62, in which many people enter, and the out-area 63 (see FIG. 11), in which many people exit, as the entrance area 58 and the exit area 59. Note that these in-area 62 and out-area 63 correspond to the start point and the end point in the narrowed entrance and exit route, respectively. For example, in the case of the example illustrated in FIG. 10, since the threshold is set to 10, the entrance/exit estimation circuitry 49 narrows the entrance and exit routes down to ABC, ADC, ABADC, and ADABC by removing BCD (the route from the fixed camera 3b toward the fixed camera 3d via the fixed camera 3c), BAC (the route from the fixed camera 3b toward the fixed camera 3c via the fixed camera 3a), and CBA (the route from the fixed camera 3c toward the fixed camera 3a via the fixed camera 3b) from the above-described candidates of the entrance and exit route. The respective fixed cameras 3 corresponding to the start point and the end point in each of the above-described narrowed entrance and exit routes (ABC, ADC, ABADC, and ADABC) are the fixed camera 3a and the fixed camera 3c. Using the above-described statistical information of the single camera object tracking results, the entrance/exit estimation circuitry 49 estimates (determines) the in-area 62 (see FIG. 11) having a large number of entering people in a photographing range 61a at the start point (fixed camera 3a) of the above-described narrowed entrance and exit route as the entrance area 58, and estimates (determines) the out-area 63 (see FIG. 11) having a large number of exiting people in a photographing range 61c at the end point (fixed camera 3c) of the above-described narrowed entrance and exit route as the exit area 59.

Next, the setting circuitry 50 of the analysis server 1 automatically constructs graph information 80 for input to the multi-camera object tracking using information (the in-direction and out-direction, the ROI (region of interest), the edge information (movable path), and the entrance information and exit information (entrance area and exit area)) of estimation results by the processing of S4 and S5, and sets (outputs) the graph information 80 and the information (hereinafter, referred to as “mapping information”) of the association result between each point of the captured image of each fixed camera 3 estimated in S2 and each point in the two-dimensional store map 60 as parameters used for multi-camera object tracking (S6).

Then, the multi-camera object tracking circuitry 51 with path limitation of the analysis server 1 (see FIG. 5) executes the multi-camera object tracking with limitation of the movable path using the above-described graph information 80 and the mapping information described above as input parameters (S7).

Here, a specific example of the above-described graph information 80 will be described in detail with reference to FIG. 7. For example, it is assumed that the fixed cameras arranged in the store S are three fixed cameras 3a, 3b, and 3c, they are arranged as illustrated in the two-dimensional store map 60 on the left side of FIG. 7, and there is an obstacle called the shelves 82 on the path connecting the fixed cameras 3a and 3c. In this case, the graph information 80 includes graph information regarding Node A (fixed camera 3a), graph information regarding Node B (fixed camera 3b), and graph information regarding Node C (fixed camera 3c). Then, in the case of the example illustrated in FIG. 7, the edge information (movable path) in the graph information regarding the Node A (fixed camera 3a) is A⇔B, the in-direction is left, right, and down, the out-direction is right and down, the entrance information is “entrance present (left)” (information that there is an entrance within the photographing range of the fixed camera 3a, and the entrance is on the left side), the exit information is “no exit” (information indicating that there is no exit within the photographing range of the fixed camera 3a), and the ROI (region of interest) is information regarding the region defined by the width, the height, and the x coordinate and the y coordinate of the origin.

As described above, according to the object tracking parameter setting support system 40 of the present embodiment, by inputting the captured images from the plurality of fixed cameras 3 and the data of the map (two-dimensional store map 60) of the store S in which these fixed cameras 3 are arranged, a movable path (edge information) is estimated on the basis of the input captured images from the plurality of fixed cameras 3 and the data of the two-dimensional store map 60, and information of the estimated movable path can be set as one of the parameters used for multi-camera object tracking. In this manner, it is possible to automatically set the information of the movable path necessary for performing the multi-camera object tracking with limitation of the movable path, and thus it is possible to save time and effort to manually set the information of the movable path and to suppress economic and temporal costs. Furthermore, by using the above-described information of the movable path, the multi-camera object tracking with limitation of the movable path can be performed on the captured images from the plurality of fixed cameras