JP2019023934A - Tracking device, camera, tracking method, and tracking program - Google Patents

Abstract

To make it possible to take appropriate action according to reliability of tracking.SOLUTION: A tracking device 100: comprises: an image acquisition unit 103 that repeatedly acquires images; an object reference information acquisition unit 104 that acquires object reference information on a tracking object included in the images; a surroundings reference information acquisition unit 104a that acquires surroundings reference information on the surroundings of the tracking object; a first similarity calculation unit 104a that calculates, for the images acquired in time series by the image acquisition unit 103, first similarity between a target image set in the images and the object reference information and detects a tracking object candidate image on the basis of the first similarity; a second similarity calculation unit 104a that calculates second similarity from the tracking object candidate image and surroundings reference information; and a reliability determination unit that determines reliability of the tracking object candidate image detected by the first similarity calculation unit 104a on the basis of the second similarity.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a tracking device, a camera, a tracking method, and a tracking program.

  In a tracking device that tracks a subject in a frame image input in time series, a technique for tracking the subject by calculating the similarity between the template image and the target image using a calculation method called normalized correlation is known. (See Patent Document 1).

Japanese Patent No. 3768073

  However, the conventional tracking device has a problem that, for example, it is difficult to detect a decrease in reliability such that the tracking target is shielded by a shielding object, and tracking becomes difficult.

A tracking device according to the present invention includes an image acquisition unit that repeatedly acquires an image, a target reference information acquisition unit that acquires target reference information related to a tracking target included in the image, and a peripheral reference that acquires ambient reference information about the periphery of the tracking target A first similarity between a target image set in the image and target reference information is calculated for the images acquired in time series by the information acquisition unit and the image acquisition unit, and tracking is performed based on the first similarity Based on the first similarity calculation unit that detects the target candidate image, the second similarity calculation unit that calculates the second similarity from the tracking target candidate image and the surrounding reference information, and the tracking based on the second similarity A reliability determination unit that determines the reliability of the target candidate image.
The camera according to the present invention is equipped with the tracking device.
The tracking method according to the present invention repeatedly acquires an image, acquires target reference information related to the tracking target included in the image, acquires ambient reference information about the periphery of the tracking target, and for images acquired in time series, A first similarity between the target image set in the image and the target reference information is calculated, a tracking target candidate image is detected based on the first similarity, and a second is determined from the detected tracking target candidate image and the surrounding reference information. And the reliability of the detected tracking target candidate image is determined based on the calculated second similarity.

  According to the present invention, it is possible to appropriately cope with tracking reliability.

It is a block diagram which illustrates the composition of the camera carrying the tracking device by one embodiment of the invention. It is a flowchart explaining the flow of the subject tracking process performed by a calculating part. It is a flowchart explaining the flow of the tracking process at the time of a subject lost. It is a flowchart explaining the detail of an initial frame process. It is a figure which illustrates a time-sequential frame image. It is a figure which illustrates a time-sequential frame image. It is a figure which illustrates a time-sequential frame image. It is a flowchart explaining the detail of an update process. It is a figure explaining supply of the tracking program with respect to a camera.

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a block diagram illustrating the configuration of a camera equipped with a tracking device according to an embodiment of the present invention. In FIG. 1, the camera 100 includes an operation member 101, a lens 102, an image sensor 103, a control device 104, a memory card slot 105, and a monitor 106.

  The operation member 101 includes various input members operated by the user, such as a power switch, a release button, a zoom button, a cross key, and the like. The lens 102 includes a plurality of optical lens groups, and forms a subject image on the imaging surface of the imaging element 103. FIG. 1 illustrates one lens that represents the optical lens group.

  The image sensor 103 is configured by a CMOS image sensor, for example, and captures a subject image formed by the lens 102. Data of the captured image is output to the control device 104. The control device 104 performs predetermined image processing on the image data.

  The memory card slot 105 is a slot for inserting a memory card as a storage medium. The image data output from the control device 104 is recorded on a memory card inserted in the memory card slot 105.

  The monitor 106 is configured by a touch operation liquid crystal panel mounted on the back surface of the camera 100, for example. The monitor 106 is used to set a monitor image (live view image) acquired in time series at predetermined intervals by the image sensor 103, a reproduced image based on image data stored in a memory card, and the camera 100. The setting menu screen can be displayed.

  The control device 104 includes a CPU, a memory, and other peripheral circuits, and functionally includes a calculation unit 104a described later. Note that the memory constituting the control device 104 includes a memory for storing a program, a work memory, and a memory used as a buffer memory.

<Description of flowchart>
2 and 3 are flowcharts for explaining the flow of subject tracking processing executed by the calculation unit 104a in the control device 104. FIG. The control device 104 activates the tracking program that performs the processes of FIGS. 2 and 3 when, for example, image data acquired in time series by the image sensor 103 is sequentially output from the image sensor 103. The computing unit 104a identifies the position of the tracking target in each frame image for the time-series frame images read from the image sensor 103 at a predetermined frame rate.

  In step S10 of FIG. 2, the calculation unit 104a performs initial frame processing for performing initial setting. FIG. 4 is a flowchart for explaining the details of the initial frame processing. The computing unit 104a inputs the frame image information of the first frame in step S10-1. For example, when the frame image is expressed in the RGB color system, the calculation unit 104a uses the following expressions (1) to (3) to calculate the hue (Hue), saturation (Saturation), and brightness (Value). Convert to a 3 component image.

ただし、Ｈ＜０の時、Ｈ＝Ｈ＋３６０である。 Here, among the three values of the RGB components, the maximum value is MAX, and the minimum value is MIN.

However, when H <0, H = H + 360.

In addition, the luminance image which consists of a luminance component (Y component) and the color difference image which consists of a color difference component (Cb component, Cr component) using following Formula (4)-(6) or following Formula (7)-(9). It may be converted into

Y = 0.2990R + 0.5870G + 0.1140B (4)
Cb = -0.1687R -0.3313G + 0.5000B + 128 (5)
Cr = 0.5000R − 0.4187G − 0.0813B + 128 (6)

Y = 0.25R + 0.50G + 0.25B (7)
Cb = -0.25R-0.50G + 0.75B + 128 (8)
Cr = 0.75R-0.50G-0.25B + 128 (9)

  In step S10-2, the calculation unit 104a sets the rectangular area designated as the tracking target by the user as the subject area. Specifically, when the user performs a touch operation on a desired subject position to be tracked on the screen of the monitor 106, the calculation unit 104a sets a predetermined rectangular range including the touch position as a subject area. For example, in the coordinate system used for the tracking calculation, the calculation unit 104a sets the upper left coordinate of the subject area as the subject position coordinate.

  In step S10-3, the calculation unit 104a acquires the image of the subject area determined in step S10-2 from the frame image information input in step S10-1 as an initial template image. In step S10-4, the calculation unit 104a acquires the image of the subject area determined in step S10-2 from the frame image information input in step S10-1 as a combined template image.

  In step S10-5, the computing unit 104a acquires the image of the subject area determined in step S10-2 from the frame image information input in step S10-1, as a non-update-type target template image. In step S10-6, the calculation unit 104a acquires the image of the subject area determined in step S10-2 from the frame image information input in step S10-1 as an update target template image.

  As described above, in the initial frame processing, the initial template image, the combined template image, the non-update type target template image, and the update type target template image are common. In the present embodiment, the initial template image and the synthesized template image are used in the tracking calculation, and the non-update type target template image and the update type target template image are used in the reliability determination process for the tracking calculation result. The non-update type target template image and the update type target template image constitute a target template image group.

  In step S10-7, the calculation unit 104a acquires a plurality of areas adjacent to the subject area determined in step S10-2 from the frame image information input in step S10-1 as a surrounding template image group. FIG. 5 to FIG. 7 are diagrams illustrating time-series frame images, which are panning shots of motorcycles (tracking targets) running on a circuit. As shown in FIG. 5, in the frame image, an area 51 adjacent above the subject area 40, an area 52 adjacent below the subject area 40, an area 53 adjacent to the left of the subject area 40, and adjacent to the right of the subject area 40 The area 54 to be configured constitutes a surrounding template image group.

  In step S10-8 in FIG. 4, the calculation unit 104a sets the lost flag to OFF, and proceeds to step S20 in FIG. The lost flag is set to ON when the tracking target is lost (when the tracking target cannot be detected in the frame image), and is OFF when the tracking target is not lost (when the tracking target is detected in the frame image). Flag.

  In step S20 of FIG. 2, the calculation unit 104a inputs frame image information for the second and subsequent frames, and proceeds to step S30. The processing content is the same as in the case of the first frame (step S10-1).

  In step S30, the arithmetic unit 104a determines whether or not the lost flag is OFF. When the lost flag is set to OFF, the calculation unit 104a makes an affirmative determination in step S30 and proceeds to step S40. Proceed to tracking process when lost. The tracking process when the subject is lost will be described later.

<Tracking calculation when the tracking target is not lost>
In step S40, the calculation unit 104a uses the initial template image and the synthesized template image to create a similarity map of the search area with the hue H, saturation S, and brightness V planes. The search area refers to a range in which the tracking target is searched in the input frame image information. For creating the similarity map, for example, as shown in the following equations (10) to (12), a sum of absolute differences SAD (Sum of Absolute Difference) between the target image and the template image is used. The target image is an image in the search area having the same size as the size of the template image.

SAD _H = Σ | H _{target image−} H _{template image} | (10)
SAD _S = Σ | S _{target image−} S _{template image} | (11)
SAD _V = Σ | V _{target image−} V _{template image} | (12)

In step S50, the calculation unit 104a multiplies the similarity map of the H, S, and V planes created in step S40 by a coefficient according to the distance from the subject position. The coefficient is calculated from the following equation (13).
Kyori (x, y) = Kyori ₀ + K (| x−M _x | + | y−M _y |) (13)
In the equation (13), a (x, y) is the two-dimensional coordinate values of the similarity map, M _x is the x-coordinate of the object position in the previous frame, M _y is the y coordinate of the object position in the previous frame. In this example, Kyori ₀ = 1.0 and K = 0.05. The reason why the weighting according to the distance is performed is to give a preferential treatment to a position close to the subject position in the previous frame.

In step S60, the calculation unit 104a integrates the similarity maps of the H, S, and V planes according to the following equation (14).
N = TempHeight × TempWidth × 255
MapHSV = 1 / N (GainH x DistMapH + GainS x DistMapS + GainV x DistMapV)
(14)

  In Equation (14), TempHeight is the vertical width of the template image, TempWidth is the horizontal width of the template image, DistMapH is the similarity map of the H plane multiplied by the distance weight coefficient, and DistMapS is the similarity of the S plane multiplied by the distance weight coefficient. The degree map, DistMapV, is a V-plane similarity map multiplied by a distance weight coefficient. In this example, GainH = 1.5, GainS = 1.5, and GainV = 0.1.

  The computing unit 104a further has the maximum similarity among the two integrated similarity maps (similarity map related to the initial template image and similarity map related to the combined template image) (in this embodiment, the absolute difference sum SAD is minimum). Is selected as a matching position, and a rectangular area having the same size as the template image with the matching position as an upper left coordinate is acquired as a subject candidate area.

Ｓ= ０．５×(ＮＣＣ ＋ １) ・・・・・・・・・・・・・・・（１６）
Object(i,j)は、被写体候補領域に対応し、Template(i,j)は、対象テンプレート画像群と周囲テンプレート画像群とに対応する。式（１５）は正規化相互相関（ＮＣＣ）の算出式であり、類似度が高いほど１に近づき、類似度が低いほど−１に近づく。式（１６）は、類似度が高いほど１に近づき、類似度が低いほど０に近づくように式（１５）の結果を変形したものである。 <Reliability determination for subject candidate area>
In step S70, the calculation unit 104a calculates the similarity between the subject template image group acquired in step S60 and the target template image group and the surrounding template image group by the following equations (15) and (16).

S = 0.5 × (NCC + 1) (16)
Object (i, j) corresponds to the subject candidate region, and Template (i, j) corresponds to the target template image group and the surrounding template image group. Formula (15) is a calculation formula for normalized cross correlation (NCC), and approaches 1 as the degree of similarity increases, and approaches -1 as the degree of similarity decreases. Expression (16) is obtained by modifying the result of Expression (15) so that it approaches 1 as the degree of similarity increases and approaches 0 as the degree of similarity decreases.

Calculation unit 104a further values similarity is the highest among the target template images S _maxP, the value of similarity is maximized in the surrounding template images and S _maxN, the following equation (17) Then, the relative similarity ^Sr is calculated.

  In FIG. 6, there is a pole that can be a shielding object 60 in the screen. The motorcycle (tracking target) running on the circuit approaches the shield 60, and the motorcycle (tracking target) is hidden by the shield 60 in FIG. In the frame image of FIG. 6, a region 51 adjacent above the subject region 40, a region 52 adjacent below the subject region 40, a region 53 adjacent to the left of the subject region 40, and a region 54 adjacent to the right of the subject region 40. Constitutes a surrounding template image group.

When the motorcycle (tracking target) is in a shielding state hidden by the shield 60, the subject candidate area 41 is set on the shield 60 as shown in FIG. There is a thing. In such a case, the similarity of the subject candidate area 41 to the target template image group is a relatively high value. However, in the present embodiment, the region 54 including the shielding object 60 in FIG. 6 one frame before is acquired as a surrounding template image. For this reason, the similarity of the subject candidate area 41 to the surrounding template image group (area 54 in this example) is higher than the similarity to the target template image group. Therefore, the value of the relative similarity S ^r becomes smaller, it can be determined blocking state correctly. According to equation (17), the value of the relative similarity S ^r if not shielded state approaches 1, the value of the relative similarity S ^r in the case of the shielding state is smaller.

In step S80, arithmetic operation unit 104a, the relative similarity ^{S r} calculated in step S70 it is determined whether or not a predetermined "determination threshold value 1" or more. Calculating unit 104a, the relative similarity S ^r is an affirmative decision in step S80 if the above "determination threshold value 1", the flow proceeds to step S90, and a negative decision in step S80 if less than "determination threshold value 1" step S140 Proceed to In the present embodiment, “determination threshold 1” = 0.4960. When affirmative determination is made in step S80, the subject candidate area selected in S60 has high reliability. When a negative determination is made in step S80, the reliability of the subject candidate area selected in S60 is low.

  In step S90, the calculation unit 104a employs the matching position selected in step S60 as a new subject position, and employs the subject candidate area as a new subject area.

  In step S100, the calculation unit 104a performs a process for updating the composite template image. FIG. 8 is a flowchart illustrating details of the update process. In step S100-1 of FIG. 8, the calculation unit 104a determines whether the similarity map for which the matching position has been selected in step S60 is a similarity map based on the initial template image or a similarity map based on the combined template image. To do. The calculation unit 104a proceeds to step S100-2 when determining that the map is a similarity map based on a composite template image, and proceeds to step S100-3 when the map is a similarity map based on an initial template image.

  In step S100-2, the calculation unit 104a calculates the image obtained by multiplying the pixel value of the new subject area adopted in step S90 by the weighting factor A, and the pixel value of the initial template image to the weighting factor B (where A ≧ B, and The image multiplied by A + B = 1) is added, the added image is updated as a composite template image, and the process proceeds to step S110 in FIG.

  In step S100-3, the calculation unit 104a includes an image obtained by multiplying the pixel value of the new subject area adopted in step S90 by the weight coefficient B, and an image obtained by multiplying the pixel value of the initial template image by the weight coefficient A. Are added, the image after the addition is updated as a composite template image, and the process proceeds to step S110 in FIG.

  In step S110 in FIG. 2, the calculation unit 104a performs an update process of the update target template image. The new update-type target template image includes an image obtained by multiplying the pixel value of the non-update-type target template image by the weight coefficient C, and an image obtained by multiplying the pixel value of the new subject area adopted in step S90 by the weight coefficient D. Add to create. As a result, an update-type target template image that reflects the current shape of the subject and that is not too far from the initial state can be acquired. Note that C ≧ D and C + D = 1.

  In step S120, the calculation unit 104a updates a plurality of areas adjacent to the new subject area adopted in step S90 as a new surrounding template image group. In step S130, the calculation unit 104a determines whether or not processing for all frame images has been completed. The arithmetic unit 104a makes an affirmative determination in step S130 when the process has been performed on all frame images, and ends the process of FIG. The arithmetic unit 104a makes a negative determination in step S130 if the process has not been performed on all frame images, and returns to step S20. When returning to step S20, new frame image information is input, and the above-described processing is repeated.

  In step S140, which proceeds after making a negative determination in step S80, the calculation unit 104a determines that there is no tracking target in the frame image, sets the lost flag to ON, and proceeds to step S130.

<Tracking calculation when the tracking target is lost>
FIG. 3 is a flowchart for explaining the tracking process when the subject is lost. In step S150 of FIG. 3, the calculation unit 104a uses the initial template image and the synthesized template image to convert the similarity map of the search area to each of the hue H, saturation S, and brightness V as in step S40. Create in plain. However, the search area is set wider than when the lost flag is OFF (when the tracking target is not lost). Thereby, even when the tracking target appears at a position away from the lost (shielded) coordinates, the tracking target can be captured in the search area.

  In step S160, as in step S50, the calculation unit 104a multiplies the similarity map of each of the H, S, and V planes created in step S150 by a coefficient according to the distance from the subject position. However, the value of K in the above equation (13) is set to a smaller value than when the lost flag is OFF. As a result, the absolute difference sum SAD when the subject appears at a position away from the lost coordinates becomes small, and can be selected as the subject candidate region. In the present embodiment, K = 0.02.

  In step S170, as in step S60, the calculation unit 104a integrates the similarity maps of the H, S, and V planes according to the above equation (14), and calculates the similarity from the two integrated similarity maps. Is selected as the matching position at the xy coordinate where the maximum is (the absolute difference sum SAD is minimum in the present embodiment). Then, a rectangular area having the same size as the template image with the selected matching position as the upper left coordinate is acquired as a subject candidate area.

In step S180, the arithmetic unit 104a, as in the case of step S70, the calculating the relative similarity ^{S r} by equation (15) to (17).

In step S190, the arithmetic unit 104a relative similarity ^{S r} calculated in step S180 it is determined whether or not a predetermined "determination threshold value 2" or more. If the relative similarity S ^r is equal to or greater than “determination threshold 2”, the calculation unit 104a makes an affirmative determination in step S190 and proceeds to step S200. If the relative similarity S ^r is less than “determination threshold 2”, the operation unit 104a makes a negative determination in step S190. Proceed to In the present embodiment, a value (= 0.4990) larger than “determination threshold 1” is used for “determination threshold 2”. When the determination in step S190 is affirmative, the subject candidate area selected in S170 has high reliability. When a negative determination is made in step S190, the reliability of the subject candidate area selected in S170 is low.

  In step S200, the calculation unit 104a employs the matching position selected in step S170 as a new subject position, and employs the subject candidate area as a new subject area.

  In step S210, the arithmetic unit 104a performs a process of updating the composite template image, as in the case of step S100. In step S220, the calculation unit 104a performs an update process of the update-type target template image as in the case of step S110. In step S230, the calculation unit 104a performs a process of updating the surrounding template image group as in the case of step S120.

  In step S240, the calculation unit 104a determines that the tracking target has returned within the frame (has escaped the shielding state), sets the lost flag to OFF, and proceeds to step S130 in FIG.

According to the embodiment described above, the following operational effects can be obtained.
(1) The camera 100 repeatedly acquires a frame image, the image sensor 103, and a calculation unit 104a that acquires a template image (initial template image, composite template image, target template image group) related to the tracking target included in the frame image. The calculation unit 104a that acquires a surrounding template image group related to the periphery of the tracking target, and the first target image and template image set in the frame image with respect to the frame image acquired in time series by the calculation unit 104a A first similarity calculation unit (calculation unit 104a) that calculates a similarity (SAD) and detects the subject candidate region 41 based on the first similarity, and a first similarity calculation unit (calculation unit 104a). The second similarity (NCC) is calculated from the detected subject candidate area 41 and the surrounding template image group. Based on the second similarity (NCC) calculated by the second similarity calculation unit (calculation unit 104a) and the second similarity calculation unit (calculation unit 104a). A calculation unit 104a that determines the reliability of the subject candidate area 41 detected by the calculation unit 104a). As a result, appropriate measures can be taken according to the determined reliability.

(2) When the reliability of the subject candidate area 41 is higher than a predetermined value, the position of the subject candidate area 41 in the frame image is set as the tracking target position, and when the reliability is lower than the predetermined value, it is determined that the subject is not in the image. By doing so, it is possible to appropriately track the tracking target.

(3) When the reliability of the subject candidate area 41 is higher than the predetermined value, the position of the tracking target is specified and the process proceeds to the next frame image. When the reliability is lower than the predetermined value, it is determined that there is no subject. Since the process proceeds to the process for the next frame image, the tracking process can be appropriately performed.

(4) The second similarity calculation unit (calculation unit 104a) is a third unit between the subject candidate area 41 and the template image (target template image group) detected by the first similarity calculation unit (calculation unit 104a). The similarity (NCC) is calculated, and the calculation unit 104a is detected by the first similarity calculation unit (calculation unit 104a) based on the third similarity (NCC) and the second similarity (NCC). Since the reliability of the subject candidate area 41 is determined, compared to the case where the reliability determination is performed only by using the similarity of the same type as the similarity (SAD) calculated by the first similarity calculation unit (calculation unit 104a). Reliability can be determined.

(5) The similarity calculation for the first similarity calculation unit (calculation unit 104a) to calculate the first similarity (SAD) and the second similarity calculation unit (calculation unit 104a) are the second Since the similarity calculation for calculating the similarity (NCC) and the third similarity (NCC) is made different from each other, it is more reliable than the case of determining reliability by using only the same kind of similarity calculation. Can determine gender. The computing unit 104a also calculates the first similarity based on the value obtained by dividing the third similarity (NCC) by the added value of the third similarity (NCC) and the second similarity (NCC). The reliability of the subject candidate area 41 detected by the calculation unit (calculation unit 104a) is determined. Thereby, it becomes easy to determine whether or not the tracking target is in a shielding state hidden by the shielding object 60.

(6) When the arithmetic unit 104a divides the third similarity (NCC) by the added value of the third similarity (NCC) and the second similarity (NCC) is greater than the threshold value It is determined that the reliability is high, and it is determined that the reliability is low when the reliability is lower than the threshold. Specifically, since it can be determined that the reliability is high when not in the shielding state and the reliability is low when in the shielding state, the reliability determination can be performed appropriately.

(7) The first similarity calculation unit (calculation unit 104a) uses the calculation unit 104a to increase the reliability of the search area for detecting the subject candidate area 41 based on the first similarity (SAD) in the frame image. If it is determined that the reliability is lower than that determined. Thereby, for example, when the tracking target appears at a coordinate position away from the coordinate position in the shielding state, the possibility of capturing the tracking target in the search area can be increased.

(8) The calculation unit 104a acquires a plurality of surrounding template image groups for the left and right and top and bottom positions of the tracking target, and the second similarity calculation unit (the calculation unit 104a) includes the subject candidate area 41 and the plurality of surroundings. The plurality of second similarity degrees (NCC) are calculated with each of the template image groups 51 to 54, and the calculation unit 104a has the highest similarity among the plurality of second similarity degrees (NCC). Based on the similarity of 2, the reliability of the subject candidate area 41 detected by the first similarity calculation unit (calculation unit 104a) is determined. In general, in the frame image before the tracking target is hidden by the shielding object 60, the shielding object 60 is included in any of the left, right, upper, and lower sides of the tracking target. For this reason, by using the second similarity having the highest similarity among the plurality of second similarities (NCC) with respect to the left and right and up and down directions, it is possible to appropriately determine the shielding state in which the reliability is lowered.

The following modifications are also within the scope of the present invention, and one or a plurality of modifications can be combined with the above-described embodiment.
(Modification 1)
In the embodiment described above, the tracking device has been described by taking the camera 100 as an example, but a tracking program may be supplied to the camera 100 later. The supply of the tracking program to the camera 100 can be transmitted from the personal computer 205 storing the program to the camera 100 by infrared communication or short-range wireless communication as exemplified in FIG.

  The program may be supplied to the personal computer 205 by setting a storage medium 204 such as a CD-ROM storing the program in the personal computer 205, or to the personal computer 205 by a method via the communication line 201 such as a network. You may load. When passing through the communication line 201, the program is stored in the storage device 203 of the server 202 connected to the communication line.

  Further, a storage medium 204B such as a memory card storing the program may be set in the camera 100. As described above, the tracking program can be supplied as various types of computer program products such as provision via a storage medium or a communication line.

(Modification 2)
In the above description, in the tracking calculation for detecting the subject candidate area, the absolute difference sum (SAD) is used for determining the similarity between the target image and the template image (initial template image, composite template image), and detection is performed. In the calculation for determining the reliability of the subject candidate area 41, normalized cross-correlation (NCC) is used for similarity determination between the subject candidate area 41 and the template image (target template image group, surrounding template image group). . Instead of performing different similarity determinations in the tracking calculation and the reliability determination calculation, the tracking calculation and the reliability determination calculation have the same similarity determination (absolute difference sum (SAD) or normalized cross correlation (NCC)). ) May be configured.

(Modification 3)
Size of initial template image and composite template image used for similarity determination between target image and template image, target template image group and surrounding template image group used for similarity determination between subject candidate region 41 and template image Although an example in which the size of the template images is the same has been described, the sizes of these template images are not necessarily the same. For example, the size of the target template image group and the surrounding template image group may be smaller than the sizes of the initial template image and the synthesized template image.

  Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

40 ... Subject area 41 ... Subject candidate areas 51-54 ... Ambient template image group 60 ... Shielding object 100 ... Camera 103 ... Image sensor 104 ... Control device 104a ... Calculation unit 201 ... Communication line 202 ... Servers 204, 204B ... Storage medium 205 …Personal computer

Claims (11)

An image acquisition unit that repeatedly acquires images;
A target criterion information acquisition unit that acquires target criterion information related to the tracking target included in the image;
A surrounding reference information acquisition unit for acquiring surrounding reference information about the periphery of the tracking target;
A first similarity between the target image set in the image and the target reference information is calculated for the image acquired in time series by the image acquisition unit, and tracking is performed based on the first similarity. A first similarity calculation unit for detecting a target candidate image;
A second similarity calculation unit that calculates a second similarity from the tracking target candidate image and the surrounding reference information;
A reliability determination unit that determines the reliability of the tracking target candidate image based on the second similarity;
A tracking device comprising: The tracking device according to claim 1,
When the reliability is higher than a predetermined value, the position of the tracking target candidate image in the image is set as the position of the tracking target, and when the reliability is lower than the predetermined value, it is determined that the subject is not in the image.
A tracking device characterized by that. The tracking device according to claim 2,
When the reliability is higher than a predetermined value, the position of the tracking target is specified and the process proceeds to the next image. When the reliability is lower than the predetermined value, it is determined that there is no subject and the next image is determined. Proceed to processing,
A tracking device characterized by that. The tracking device according to claim 2 or 3,
The second similarity calculation unit calculates a third similarity between the tracking target candidate image and the target reference information;
The tracking device according to claim 1, wherein the reliability determination unit determines the reliability of the tracking target candidate image based on the third similarity and the second similarity. The tracking device according to claim 4,
A similarity calculation for the first similarity calculation unit to calculate the first similarity, and a second similarity calculation unit to calculate the second similarity and the third similarity. Is different from the similarity calculation for
The reliability determination unit determines the reliability of the tracking target candidate image based on a value obtained by dividing the third similarity by an addition value of the third similarity and the second similarity. A tracking device characterized by that. The tracking device according to claim 5,
The reliability determination unit determines that the reliability is high when the divided value is larger than a threshold value, and determines that the reliability is low when the divided value is smaller than the threshold value. . The tracking device according to claim 6,
The first similarity calculation unit is configured to detect a range in which the tracking target candidate image is detected based on the first similarity in the image, when the reliability determination unit determines that the reliability is high. The tracking device is characterized in that it is widened when it is determined that the reliability is low. In the tracking device according to any one of claims 1 to 7,
The surrounding reference information acquisition unit acquires a plurality of surrounding reference information regarding the left and right and top and bottom positions of the tracking target,
A second similarity calculating unit calculates a plurality of the second similarities between the tracking target candidate image and each of the plurality of surrounding reference information;
The tracking device according to claim 1, wherein the reliability determination unit determines the reliability of the tracking target candidate image based on a second similarity having the highest similarity among the plurality of second similarities.   A camera comprising the tracking device according to claim 1. Acquire images repeatedly,
Obtaining target reference information regarding the tracking target included in the image,
Obtaining ambient reference information about the periphery of the tracking target;
A first similarity between the target image set in the image and the target reference information is calculated for the images acquired in time series, and a tracking target candidate image is detected based on the first similarity. And
A second similarity is calculated from the detected tracking target candidate image and the surrounding reference information,
Determining reliability of the detected tracking target candidate image based on the calculated second similarity;
The tracking method characterized by this. A process of repeatedly acquiring images,
Processing for acquiring target reference information related to the tracking target included in the image;
Processing for obtaining ambient reference information about the periphery of the tracking target;
A first similarity between the target image set in the image and the target reference information is calculated for the images acquired in time series, and a tracking target candidate image is detected based on the first similarity. Processing to
A process of calculating a second similarity from the detected tracking target candidate image and the surrounding reference information;
A process of determining the reliability of the detected tracking target candidate image based on the calculated second similarity;
A tracking program that causes a computer to execute.

Images