JP2022019339A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To allow stable continuation of tracking even when an object having a similar posture or similar appearance characteristics approaches.SOLUTION: An information processing apparatus according to the present invention that solves the above-mentioned problem is an information processing apparatus that detects at least one or more objects from an image, and has: estimation means that, based on a learned model that has learned image characteristics indicating the shielding relationship between a shielding object and a shielded object, estimates, for each of the objects detected from the image, the shielding relationship with the other object detected from the image; and specification means that, based on the shielding relationship estimated by the estimation means, specifies, for each of the objects detected from the image, the correspondence relationship with an object detected in an image picked up at a time different from that of the image.SELECTED DRAWING: Figure 3

Description

The present invention relates to a technique for tracking a subject.

As a technique for tracking a specific subject in an image, there are techniques that utilize luminance and color information, template matching, and the like. In recent years, a technology using Deep Neural Network (hereinafter abbreviated as DNN) has been attracting attention as a highly accurate tracking technology.

Non-Patent Document 1 is one of the methods for tracking a specific subject in an image. The image showing the tracking target and the image to be the search range are input to the Convolutional Neural Network (hereinafter abbreviated as CNN) having the same weight. By calculating the cross-correlation between the features obtained from the CNN, the position where the tracking target exists in the image in the search range is specified. While such a tracking method can accurately identify the position of the tracking target, when objects similar to the tracking target overlap on the screen, a failure to track an erroneous target is likely to occur.

In order to avoid this, as typified by the method of Patent Document 1, a histogram is created from the color characteristics and depth information of the region of the detected object, and its change is examined to determine whether or not the object is shielded. There is a method to do.

US Patent Application No. 10185877 (B2) Public Relations

Bertinetto et al. , Fully-Convolutional Siamese Network Tracking, arXiv 2016

However, in the method shown in Patent Document 1, when objects having similar postures or objects having similar external features overlap on the screen, it is difficult to make a determination because the histograms of features such as colors and textures are unlikely to differ. There are challenges. For example, in a group competition of sports, the clothes and postures of a plurality of people existing in a narrow range are often the same, and a failure to track different people as the same person may occur. The present invention has been made in view of such a problem, and an object of the present invention is to stably continue tracking even when objects having similar appearance characteristics and postures are close to each other.

The information processing device according to the present invention that solves the above problems is an information processing device that detects at least one or more objects from an image, and learns image features showing a shielding relationship between a shielded object and a shielded object. Based on the learned model, the estimation means for estimating the shielding relationship of each object detected from the image with other objects detected from the image, and the shielding relationship estimated by the estimation means. For each object detected from the image, there is a specific means for specifying a correspondence relationship with the object detected in the image captured at a time different from the image.

According to the present invention, tracking can be stably continued even when objects having similar appearance characteristics and postures are close to each other.

Schematic diagram illustrating an example of object detection The figure which shows the hardware configuration example of an information processing apparatus. Block diagram showing a functional configuration example of an information processing device Flow chart showing the processing procedure executed by the information processing device The figure which shows the result example of the processing of an information processing apparatus The figure which shows the result example of the processing of an information processing apparatus Flow chart showing the processing procedure executed by the information processing device Example of derivation of information about obstruction Block diagram showing a functional configuration example of an information processing device The figure which shows the result example of the processing of an information processing apparatus Flow chart showing the processing procedure executed by the information processing device The figure which shows the example of the learning process of an information processing apparatus The figure which shows the result example of the processing of an information processing apparatus The figure which shows the result example of the processing of an information processing apparatus The figure which shows the result example of the processing of an information processing apparatus The figure which shows the example of the learning process of an information processing apparatus

<Embodiment 1>
The information processing apparatus according to the embodiment will be described with reference to the drawings. It should be noted that the same operation is performed between the drawings with the same reference numeral, and the description thereof will be omitted. In addition, the components described in this embodiment are merely examples, and the scope of the present invention is not limited to them.

In this embodiment, a function of detecting and tracking a person from a moving image or a still image frame continuously shot will be described. The scope of application does not limit the category of the object to be detected / tracked, but the first embodiment limits the target to a person. In the present embodiment, a person is detected for each image that is continuous in time, and the tracking of the person is realized by associating which person is the same person as which person among the continuous images. In this embodiment, in particular, assuming shooting of a sporting event or the like, it is assumed that the clothes, movement directions, etc. of the person are similar, and the person approaches and intersects frequently. In such a case, erroneous correspondence is likely to occur only by associating people with similar external features such as the position of the person or the color of clothes in each image. Such a failure is called mismatching here.

In this embodiment, attention is paid to the information on the occlusion output by the trained model that has learned the occlusion-related patterns when the objects are overlapped from the viewpoint of the photographer. By using the shielding relationship output by the trained model together with the identification of the correspondence relationship of the objects, it is possible to suppress the failure of associating the person in the foreground with the person in the back and improve the tracking accuracy.

FIG. 1 is a diagram schematically showing this. Images 2100, 22120, and 2140 in FIG. 1A show three frames of still images of a moving image of two playing cards with the same pattern crossing each other on a table from above (time). They are arranged from left to right in chronological order). It is not possible to determine how the cards in each image have moved just by observing the images 2100, 2120, and 2140. On the other hand, FIG. 1B is an example of shooting the same situation at a higher frame rate than that of FIG. 1A. That is, it is a group of images taken at shorter time intervals. By observing the images of FIG. 1B in chronological order, it is possible to associate which card moved to where in the next image. As a whole, it can be relatively easily estimated that the card 2201 on the left side has passed over the card 2202 on the right side and has moved to the right side. As shown in image 2210 and image 2230, by observing the appearance that occurs transiently at the moment of intersection between objects, it is possible to determine which card passes through the front side and which is on the back side in image 2220. It becomes. In this determination, special sensors such as a 2.5-dimensional depth image and high-cost information such as optical flow are not always required. It can be solved as a pattern recognition problem between the shielding relationship and the appearance feature, which is what kind of appearance is likely to occur when objects overlap in the foreground and the back. This is also the case in scenes such as the intersection of people shown in FIGS. 1 (C) and 1 (D). In FIGS. 1 (C) and 1 (D), it is assumed that the clothes, postures, and the like of the person are visible and the moving directions are the same. Even in such a case, if the state of appearance before and after the intersection of the objects is focused on and observed, it is determined that the person 2401 is on the front side and the person 2402 is on the back side in the image 2420 of FIG. 1 (D). In the subsequent images, if this shielding relationship is maintained, when the person 2401 shields the person 2402 once, the person 2401 on the front side is tracked and the shielding relationship and the image feature of the person 2401 on the back side are retained. do. Then, when the shielding is canceled, it is possible to detect the person 2402 in the back side again while continuing the tracking of the person 2401. The above is an explanation showing the outline of the principle of the present embodiment. Detailed processing will be described later.

FIG. 2 is a hardware configuration diagram of the information processing apparatus 1 that tracks the tracking target by image recognition in the present embodiment. The CPU H101 controls the entire apparatus by executing the control program stored in the ROM H102. The RAM H103 temporarily stores various data from each component. Also, the program is expanded so that the CPU H101 can be executed. The storage unit H104 stores the data to be processed according to the present embodiment, and stores the data to be tracked. As the medium of the storage unit H104, an HDD, a flash memory, various optical media, or the like can be used. The input unit H105 is composed of a keyboard, a touch panel, a dial, and the like, and receives input from the user, and is used when setting a tracking target or the like. The display unit H106 is composed of a liquid crystal display or the like, and displays a subject and a tracking result to the user. In addition, this device can communicate with other devices such as a photographing device via the communication unit H107.

FIG. 3 is a block diagram showing a functional configuration example of the information processing apparatus. In FIG. 3, the processes executed by the CPU H101 are shown as functional blocks. The information processing device 1 has an image acquisition unit 201, an object detection unit 202, a shielding information generation unit 203, an extraction unit 204, and a mapping unit 205, and is connected to an external storage unit 206. The storage unit 206 may be inside the information processing device 1. Each function is briefly explained. The image acquisition unit 201 acquires an image of a moving image or a continuous still image obtained by capturing a specific object (a person in the present embodiment) by an image pickup device. The object detection unit 202 detects an image feature indicating a predetermined object set in advance from the image acquired by the image acquisition unit 201. For example, a region of a person in an image is detected based on a trained model in which image features showing a human body (head or moving body) are learned in advance using images of people in various postures. The occlusion information generation unit 203 is based on a trained model that has learned the image features showing the occlusion relationship between the obstructed object and the obstructed object, and for each object detected from the image, another object detected from the image. Estimate the occlusion information indicating the occlusion relationship with. The occlusion information is a likelihood (obstruction / occlusion score) indicating the possibility that the object of interest is obscured by another object. For example, if one object is likely to be occluded by another, the occluded / occluded score approaches 1. For one object, if it is likely that it is obscuring another object, the obscured / obscured score is close to zero. A shielded / shielded score showing such a shielding relationship is estimated using a trained model. The extraction unit 204 stores the shielding information in the storage unit 206 for each object detected for a certain image. The association unit 205 associates an object detected between a plurality of images. That is, based on the shielding information, for each object detected from a certain image, the correspondence relationship with the object detected in the image captured at a different time from the certain image is specified. Objects can be tracked by correctly associating objects detected from each of the images captured at different times. Further, by assuming that the shielding relationship between the objects is maintained for a certain period of time, it is possible to associate the objects with each other using the shielding relationship. The storage unit 206 stores the obscured score of each detected object. The details of the processing of each functional unit will be described with reference to the flowchart of FIG.

FIG. 4 is a flowchart showing the flow of processing of the present embodiment. In the following description, the notation of the process (step) is omitted by adding S at the beginning of each process (step). However, the information processing apparatus does not necessarily have to perform all the steps described in this flowchart. The process shown in the flowchart of FIG. 4 is executed by the CPU H101 of FIG. 2, which is a computer, according to a computer program stored in the storage unit H104.

In S301, the information processing apparatus 1 starts a loop process that repeats for each moving image frame. In S302, the image acquisition unit 201 sequentially acquires image frames of moving images and continuous still images of a person. Subsequent processing is sequentially performed for each image from S301 to S311. The image acquisition unit 201 may acquire an image captured by an image pickup device connected to the information processing device, or may acquire an image stored in the storage unit H104. It is an example of the image frame acquired by the moving image frame 3100, 3110, 3120, 3130, 3140 in FIG. 5A.

Next, in S303, the object detection unit 20 detects at least one or more predetermined objects from the acquired image based on the image characteristics of the predetermined object (here, a person). Known techniques for detecting an object in an image include a method using Liu (Liu, SSD: Single Shot Multibox Detector. In: ECCV2016). The result of detecting the candidate object in the image is shown in FIG. 5 (A). FIG. 5 (A) is a Bounding Box (hereinafter referred to as BB) showing an object region in which rectangular frames 3101, 3102, 3103, 3111, 3112, 3113, 3121, 3122, 3131, 3132, 3141, 3142 are detected.

In S304, the occlusion map generation unit 203 generates a occlusion map showing the occlusion information regarding whether or not each image is shielded (shielding relationship) for each area. The occluded map generation unit 203 estimates a visible region (shielded object region) of the occluded object for each image. Here, it is determined whether each person overlaps with another person, and if they overlap, whether they are on the back side or the front side, and the result is output as a score (likelihood) of the shielded state for each area. This is a kind of recognition task of semantic region division, and can be realized by using a known method such as the method of Chen et al. (Chen, DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs, 2016).

FIG. 6A shows a schematic diagram illustrating the process of generating the shielding map and an example of the result. The neural network 402 is a neural network that determines the shielding state of each pixel of the input image from the input image. When the RGB image 401 is input, the neural network 402 outputs a result of estimating whether or not there is a person in the image and whether or not the person is shielded as a shielding map 404. The map outputs an obscured score of 0 when it is estimated to be an unobstructed person and an area other than the person, and 1 in the area of the obscured person. Black areas in the obstruction map 404 indicate higher obstruction scores. That is, it indicates that the black area was presumed to be the area of the shielded person. The neural network 402 is trained in advance so that such an output can be performed for the input image (learning will be described later). The shielding map 404 shown in the figure shows an example of an ideal output state as an estimation result.

In addition to the RGB image 401, a derivative form in which the 2.5-dimensional depth image 405 is separately acquired by using a dedicated sensor or the like is also conceivable. The information of 4 channels in which the depth image 405 is connected to the RGB image 401 of 3 channels is learned and recognized as an image input instead of the RGB image. This makes it possible to make the information in the shielded area more accurate.

Next, in S305, the information processing apparatus 1 executes the loop processing of S306 to S307 for each object detected in S303. In S305 to S308, the extraction unit 204 extracts information indicating the shielding relationship for each detected object from the generated shielding map and stores it in the storage unit 206. In S306, the extraction unit 204 extracts information indicating the shielding relationship for each detected object from the shielding map. Specifically, the shielded scores of the shielded map 404 in the person detection frame 407 of FIG. 6A are averaged. The closer the occluded score is to 1, the more likely the object is occluded, and the closer the occluded score is to 0, the more likely the object is unobscured. Assuming that the position of the detection frame is displaced and noise is included in the shielding map 404, the weighted average is acquired with an emphasis on the vicinity of the center of the frame as shown in FIG. 6B. The operation 412 in the figure means the element product (Hadamard product) for each partial region of each image. Map 413 is a two-dimensional Gaussian function having a peak in the center and a sum of image blocks of 1 (the vertical and horizontal sizes are deformed according to the person detection frame). Examples of the acquisition results are shown in FIG. 6A with the symbol occ as shielded score values 409 and 410. It is determined that the detection frame on the left side has a high obstruction score because it is a person on the back side, and the detection frame on the right side has a low obstruction score because it is on the front side. Examples of the results of the above processing on the input image of FIG. 5 are shown as FIGS. 5 (B) (shielding map) and FIG. 5 (C) (shielded score estimation result of each frame). From the start to the end of the crossing, the shielded score corresponding to the person 3101 located on the back side is relatively higher than that of the person 3102.

In S307, the storage unit 206 stores the shielding score of each detected object acquired by the extraction unit 204. At the same time, information on the position and size of each detected object and features related to the appearance of the object such as color and texture histograms are also stored. Here, these multiple types of quantities are collectively referred to as feature quantities of the detected object. In addition to this, the intermediate layer information of the neural network or the like may be used as the feature amount related to the appearance. (For example, "Hariharan, et. Al, Hypergranulars for Object Segmentation and Fine-grounded Localization, in CVPR2015").

In S308, the information processing apparatus 1 ends the loop that repeats for each image and the loop that repeats for the person detected in each image. This loop ends when obstruction information is acquired for all the people detected from the image for each image. Next, in S309 to S310, the mapping unit 205 maps objects between the preceding and following images (however, in the case of the first moving image frame, this is not performed because there is no past frame). First, in S309, the association unit 205 acquires the shielded score, the position size, and the visible feature amount, which are the feature amounts of the past objects stored in the storage unit 206. Next, in S310, the mapping unit 205 associates the object detected in the past moving image frame with the object detected in the frame currently being processed.

FIG. 7 shows a detailed processing flow of the matching unit 205 in S310. In S501, the mapping unit 205 first creates a pair of all combinations between the detected object of the current frame and the object detected in the previous frame. If n and m people are detected in the previous and next frames, a total of n × m pairs is generated. Next, in S502, the matching unit 205 calculates the similarity for all the object pairs. As the degree of similarity, an index based on the difference in the feature amount between the detected objects can be used. As an example, the similarity between the past detected object c ₁ and the current detected object c ₂ is calculated by the following equation.
(Formula 1)
L (c ₁ , c ₂ ) = -W ₁ || BB ₁ -BB ₂ ||
-W ₂ || f ₁ -f ₂ ||-W ₃ || occ ₁ -occ ₂ ||
Here, BB is a vector that summarizes four variables (center coordinate value x, center coordinate value y, width, height) of each object, and f indicates the characteristics of each object. || x || is the L ^p norm of x. occ is the obscured score of each object. W1, W2, and W3 are balance coefficients of 0 or more set by adjusting empirically or machine learning, respectively. Here, the variation of each feature amount may be statistically obtained in advance and each feature amount may be normalized. Even when objects intersect with each other, by tracking the person on the side that shields the other object, the side that shields the other object immediately before the person on the shielded side is confirmed in the image again. When trying to associate with the person of, the value of the third term of the formula 1 becomes small, and the similarity is calculated to be low. That is, by this process, it is less likely that people with different shielding relationships are matched with each other, and erroneous tracking matching can be suppressed.

Next, in S503, the mapping unit 205 performs mapping (matching) for specifying the correspondence between the objects based on the degree of similarity between the past object and the current object. There are several matching methods. For example, there is a method of preferentially matching candidates with high similarity, a method of using a Hungarian algorithm, and the like. Here, the former is used.

In S503, if the mapping is not completed for all the objects in the current frame, the mapping unit 205 associates the pair with the maximum similarity as the same person in S506. The pair of objects that have been associated are omitted from the matching candidates. When the magnitude of the maximum similarity among the pairs remaining at that time falls below a predetermined threshold value during the above processing, it means that there are no more similar object pairs remaining. In that case, the association is terminated without forcibly associating (S505).

The above processes S301 to S311 are performed for each moving image frame. As a result, as shown in the result example in FIG. 5 (D), a person is detected in the moving image, and a series of tracking results of where each object has moved can be obtained (symbol A for the same person between frames). , B, C are attached to indicate the tracking result).

<Modification example>
In this embodiment, the distances of each index such as the shielded score and the appearance are weighted and summed based on the difference as the degree of similarity of matching between the object pairs. Here, for example, it is conceivable to use KL divergence. It is also conceivable to perform metric learning to obtain a more accurate distance index. Also, instead of using a single similarity only once, the similarity is first judged based on the appearance characteristics, and those that meet the conditions are then judged based on the similarity of the score in the shielded state, etc. A method or a stepwise judgment method can be considered. Furthermore, using the method of a known classifier such as a neural network or a support vector machine, it is also possible to learn and discriminate the explanatory variable as a feature quantity and the objective variable as a result of whether or not they are the same object, and judge matching based on this value. It is possible. As described above, the correspondence between objects between frames is not limited to a specific form.

As yet another derivative form, in order to stabilize the estimated value of the shielded score, it is conceivable to use the value obtained by moving average the past scores as shown in the following equation.
(Formula 2)
occ ^EMA _(t) = (1-α) × occ ^EMA _(t-1) + α × occ _(t)
The above equation is a value called an exponential moving average value, occ ^EMA _(t) is the exponential moving average value of the obscured score at time t, occ _(t) is the obscured score at time t, and α is 0 <α≤1. It is a coefficient of. For objects that have been tracked in multiple frames in the past, calculate the exponential moving average value using the above formula, and use the exponential moving average obscured score instead of the original obscured score when comparing similarities. .. As a result, when the overlapping state gradually occurs over a plurality of frames at the time of intersection, matching can be performed based on the average value of the shielded scores of the plurality of frames, so that the correspondence between the objects is more stable.

As yet another derivative form, a form in which matching is performed using not only the degree of similarity between the preceding and following frames but also the features and positions of a plurality of past frames n steps before is conceivable. By using this method, even if an object is shielded once and a frame that cannot be tracked occurs, it can be tracked again if the shielding is removed in the subsequent frames. In this form, for example, the average value of the feature amount of the object is obtained by going back to n frames, and the similarity with the object detected from the current frame is calculated based on this. Alternatively, the similarity between the past n-frame object and the current frame object is obtained, and the object having the highest average value of the obtained n similarity is associated with the object. It is also conceivable to make a bidirectional judgment using the results of not only the past but also the future frame of n steps. This form is inferior in real-time processing because the result is not known until the future frame is processed, but it is more accurate than the method of looking only at the past.

As yet another derivative form, a form for dealing with a detection failure can be considered. In object detection / tracking, it is possible that the posture of an object has changed to a special shape, and so on, causing a temporary failure at the stage of object detection. When such undetection occurs, it is determined that the object existing in the previous frame does not correspond in the current frame at the time of associating between the frames. Then the tracking will be interrupted there. In order to prevent such a failure, the following measures can be taken. That is, if a person who does not support matching occurs, that information is stored in a list and added to the matching candidates when matching the next frame (if it is not yet supported even after a certain period of time has passed). It determines that the object itself no longer exists and removes it from the list. This is called timeout processing here).

As described above, various methods can be considered for associating objects across moving image frames, and the association is not limited to a specific form.

<Variations of the form of shielding information and learning methods>
In the present embodiment, the visible area (shielded object area) of the shielded object is estimated as the shield map. Various derivative forms can be considered for this form as well. An example is shown in FIG. Here, as shown in FIG. 8 (B), it is not necessary to estimate the visible region of the object on the back side as shown in the image 801. For example, the entire area of the object on the back side is estimated as in the image 802. Further, the area of the shielded object on the front side is estimated as in the image 803 (an object that does not overlap with another object such as the area 440 in the figure is also estimated including the area on the front side as another form. It is conceivable that such a single object is not included in the front area). Further, instead of estimating the foreground region as in the images 801 to 803, it is also conceivable to estimate the position of the center and the center of gravity of the object as in the image 804. In the case of image 804, a large positive value is estimated in the region near the center of the shielded object, and a small negative value is estimated in the region near the center of the shielded object (the central region of the object here is as shown in the figure). A form that makes an estimation of a Gaussian function-like region is conceivable).

Here, a method of learning information on the shielded state will be described with reference to FIG. 6D. The neural network 402 shown by the above-mentioned method of Chen et al. Outputs a shielded score map 403 of a shielded object to an RGB image 401 which is an input image. An example of the result of 403 is shown in 430. The method of CHEN et al. Is a method of estimating the foreground region of a specific category object, but here, a teacher value 431 of the shielding information is given so that a map similar to the teacher value 431 can be obtained by estimation of the neural network 402. Do learning. Specifically, the map 403 of the output result and the teacher value 431 are compared, and the loss value calculation 432 is performed by a known method such as cross entropy or squared error. Adjust the weight parameter of the neural network 402 by the error back propagation method or the like so that the loss value gradually decreases (this process is the same as the method of Chen et al., So details are omitted). The input image and teacher value should be given in sufficient quantity. Since it is costly to create a teacher value for an area of overlapping objects, it is conceivable to use CG or to create learning data by using an image composition method in which an object image is cut out and superimposed. The above is the learning method.

Furthermore, in the present embodiment, an index called an obscured score obtained within the frame of the object obtained above is obtained. FIG. 8C shows various examples of acquisition of shielding information. FIG. 8 (C1) is an embodiment of the present embodiment. In addition to this, (C2) the difference value between the obscured score on the back side and the obscured score on the front side is acquired in the object frame, (C3) the score at the center of the object is referred to only one point, and so on. Be done. Also, when acquiring within the frame, when acquiring within the frame of the object, the area that overlaps with other object frames may be omitted from the acquisition because it is not clear which object the area is. Be done.

Further, a method of simultaneously performing the above-mentioned <estimation of the shielding state> and <acquisition of the shielding score of each object> can be considered. An example is a method called an anchor, which is a known method used in Liu's method and the like. Since a set of candidate frames for objects is obtained in this method, it is conceivable to use this to estimate and associate the shielded score of whether each candidate frame is a shielded object or a shielded object (details of this form are implemented). Since it is shown in the third form, the description is omitted here).

Further, it is also possible to acquire each of a plurality of forms of shielding information as shown above and use this as a multidimensional feature related to shielding for associating objects in the subsequent stage. Alternatively, the ratio of the shielded area of the object may be estimated and used by machine learning from the above-mentioned multidimensional characteristics related to shielding. In this case, the multidimensional feature related to the shielding may be used as an explanatory variable, the ratio of the area where the object is shielded as the target variable, and regression estimation may be performed by a known technique such as logistic regression.

<Embodiment 2>
In the present embodiment, a person is detected and tracked in the same manner as in the first embodiment. The hardware configuration is the same as that of FIG. 2 of the first embodiment. The block diagram showing the functional configuration example in this embodiment is shown in FIG. 9A. A new shielding state determination unit 301 has been added to the configuration of the first embodiment. In the first embodiment, it may not be possible to detect a person in the frame of the person during tracking due to the overlap of the people. For example, as shown in the moving image frame 4120 in FIG. 10A, when the overlapping area between people is large, it is often the case that the person on the back side cannot be detected. In such a case, the shielded state determination unit 301 determines that the person exists but is in the shielded state.

Unlike undetected due to temporary detection failure of the object detection unit as described in the first embodiment, when a group of people is moving in the same direction at the same speed, the undetected state continues for a long time. .. Further, the position on the screen that has been removed from the shielded state may be separated from the position where the shielded state has started. For this reason, it is desirable to positively determine that the state is shielded and to increase the success rate of tracking by performing processing according to the estimated state.

The overall processing flow of this embodiment is the same as that of FIG. 4 of the first embodiment, but the details of the processing of S310 are different as follows. Here, only the processing of S310, which is different from the first embodiment, will be described. A detailed flow of the S310 process performed by the shielding state determination unit 301 will be described with reference to FIG. First, as in the past, in S601, the current frame and the previous frame are associated with each other. If there is an object in the front frame that was not associated with S602, it is possible that it has entered the shielded state. Therefore, in S603, it is checked whether the height of the obstruction score of the object up to that point is equal to or higher than the threshold value. This is because if the frame rate of the moving image frame is sufficiently high, the shielded score often increases before and after it becomes undetected due to shielding. Further, in S604, it is investigated whether the number of detected objects in the current frame is reduced in the peripheral area of the object, and if the above two results are true, it is estimated that the object has entered the shielded state and the shielded state. Store in the list of (S605). For the objects stored in the list of shielded states, the features and positions at the time of the previous detection are also stored. This increases the possibility of tracking when the obstruction is removed and detected again.

S606 to S610 are processes for determining whether or not the object in the shielded state has reappeared. If there is an object in the current frame that was not associated with S603, it is possible that the object can be detected again after leaving the shielded state. Therefore, in S607, it is checked whether the height of the obstruction score of the object is equal to or higher than the threshold value. Further, in S608, it is examined whether or not the number of detected objects in the current frame has increased in the peripheral region of the object. When both results are true and the object has a high similarity to any of the objects stored in the list of shielded states (S608), the object has escaped from the shielded state and reappeared. I presume. At that time, the associated object is removed from the list of shielded states (S609). For the object removed from the list of shielded states, the feature amount and the position detected from the current input image are acquired.

Here, as a device of the matching process, for example, when matching an object between frames, matching with a person in a shielded state reduces the penalty of the degree of similarity depending on the distance. Take a long time-out to wait for reappearance. It is conceivable that the threshold value for associating with an object in a shielded state is set lower than the threshold value for matching between ordinary objects.

Furthermore, although the explanation is made assuming the overlap of two people, the overlap may occur between three or more people. In this case, if it is determined that the shielded state has been entered, it is added to the list of shielded states, and when it reappears, it is associated with the previous frame and removed from the list of shielded states each time. As a result, it is possible to deal with three or more people to some extent.

<Embodiment 3>
In this embodiment, a mode for tracking a single object specified by the user will be described. Here, the tracking target is not limited to a specific category such as a human body, and a form of tracking an unspecified object specified by the user is dealt with. For example, it may be an animal such as a dog or a vehicle such as a car.

The figure of the functional block is shown in FIG. 9B. The tracking object designation unit 302 is newly added to the previous configuration. Here, the functions of the tracking object designation unit 302 and the object detection unit 202 can be easily realized by using the method of Non-Patent Document 1. The tracking object designation unit 302 is a functional unit that allows the user to specify the frame position of the tracking target object in the moving image frame. This initializes the characteristics of the object to be tracked. The object detection unit 202 identifies the image region having the highest degree of coincidence with the target object in each moving image. An example of the identified results is shown in FIG. 12 (A). The frame 5111 on the moving image frame 5110 in FIG. 12A is a frame of the tracking object instructed by the user. In the moving image frame 5120, this object is moving to the right side in the screen, and is detected as the frame 5121 by the object detection unit 202. The method of Non-Patent Document 1 is excellent as an object tracking method, but an erroneous switch easily occurs between similar objects. Therefore, in the present embodiment, as in the previous embodiments, the information regarding the shielding state of the tracking object is estimated, and it is determined whether or not an erroneous switch has occurred.

For this purpose, a neural network 6300 as shown in FIG. 13B is used as the shielding information generation unit 203. This is because when you input the image 6301 of the detected tracking object (here, the aspect ratio is normalized to the square image for the sake of simplicity), the image pattern is seen and it is shielded (Yes). It is a classifier that outputs the classification result 6302 of whether or not it is (No). The definition of the presence or absence of shielding is defined as whether or not the area of the object is shielded by what percentage or more. The values of these two classes are given as teacher values to train the neural network 6300. This technique is not detailed because it is a widely known method similar to a normal image classification task. Further, if the teacher value (target variable) is given as the ratio of the shielded area instead of the binary value of the presence or absence of the shield and regression learning is performed, the ratio of the shield can be estimated as in the estimation result 6303. For this regression learning, a square error or the like is used as the loss value given at the time of learning.

14 (A) and 14 (B) show the results of estimating the degree of shielding of the tracking object candidate by the shielding information generation unit 203. With respect to the detection result of the object shown in FIG. 14 (A), the object detection unit 202 added the reference numeral occ to FIG. 14 (B) to indicate the estimated value of the shielded area. In the figure, the fluctuation range of the obscured score is smaller than a predetermined value (for example, a value such as 0.3), and it can be determined that tracking has not failed (here, as used in the first embodiment as well as the obscured score). The success / failure of tracking may be judged by also using the similarity of the features of different positions and appearances).

On the other hand, in FIG. 14C, the object 7201 passes through the other side of the object 7202 from the moving image frame 7220 to 7230, and as a result, the object detection unit 202 mistakenly sets the position of the object in the moving image frame 7230 as the frame 7231. I'm estimating. Since the occlusion score in this case greatly fluctuates from 0.4 to 0.0 as shown in FIG. 14 (D), it can be determined that erroneous tracking has occurred due to the intersection. If it is known that erroneous tracking has occurred, it is possible to stop the detection at that point or take measures such as correcting it at a later stage.

The above is the description of this embodiment.

As shown in FIGS. 5110 to 5150, the learning of the shielding information generation unit 203 is learned so that the shielding state of various objects can be estimated so that the shielding state of an unspecified object can be determined. Is desirable.

As another derivative form, in FIG. 13B, an image 6301 cut by an object frame is shown as an input image. However, since it is important to observe not only the object but also its surroundings in order to determine whether or not it is in a shielded state, it is conceivable to input a wider range as an input image (in that case, at the time of estimation). Also cut out the same range and enter it).

As another form of derivation, as shown in FIG. 13C, a form in which an object is detected and the degree of shielding is estimated at the same time using a candidate frame called an anchor as in the above-mentioned Liu method is also considered. Be done. The anchor frame is a set of candidate frames having a plurality of sizes and aspect ratios as shown in FIG. 13 (D) (here, three types of anchor frames are shown). As shown in the result image 6450 of FIG. 13C, the anchor frame is arranged in each block area in the image. When the image is input, the neural network 6400 generates an obscured score map 6430 indicating whether or not the object is present at each anchor in each block region. The shielded score map 6430 is three maps corresponding to three types of anchor frames. An example of the estimation result is shown in FIG. 13 (C) 6450 (since the above method is widely known, refer to the above-mentioned Liu method for details).

Here, as a derivative form of the present embodiment, the shielded score map 6440 of the object is generated at the same time as the estimation of whether or not the object exists. This is a map that estimates what the coverage ratio will be for each anchor frame if there is an object there. The map also consists of three maps corresponding to the number of types of anchors (at the time of learning, in each block of the image, the teacher value of the shielded score may be given to each anchor frame to learn the neural network 6400). An example of the result is shown in FIG. 13 (C) 6460. An example of finally integrating the two estimation maps is illustrated as an example of integration result 6470.

Although the above description is an example of object detection, since the method of Non-Patent Document 1 is also an anchor candidate frame-based method, it is possible to construct a derivative form in which an object is tracked and its obstruction score is estimated at the same time. Is.

<Embodiment 4>
In this embodiment, a mode for tracking a single object specified by the user will be described. The figure of the functional block is the same as the embodiment 3 and is FIG. 9B. In the conventional embodiments, when comparing the degree of similarity, the feature quantities are compared between the immediately preceding and immediately preceding frames, and the preceding and following n frames are used for comparison. Was done. In the present embodiment, more accurate mapping is performed by replacing this part with machine learning.

The recurrent neural network is a technique capable of processing time-series data to perform identification, classification, and the like, and a long short term memory network (hereinafter referred to as LSTM) known by the method of Byeon et al. Is a typical method. (Byeon et al., Scene labeling with LSTM recurrent neural networks, CVPR 2015). With this method, it is possible to discriminate changes in the characteristics of objects over time and associate them with each other. FIG. 15 shows a schematic diagram of the configuration of this embodiment and an example of the results. Here, one object 9102 is designated as a tracking target and is tracked by a method such as Bertinetto et al. (An erroneous switch occurs in a moving image frame of t = 2). The LSTM units 9501 to 9504 of FIG. 15C receive the features 9401 to 9404 of the object being tracked at each time, determine whether the tracking is successful or unsuccessful, and output 9701 to 9704. Output. Here, a plurality of LSTM units are written in the figure, but here, the state of each time of the same unit is shown instead of the existence of a plurality of units. The LSTM unit at each time sends a recurrent input 9802 to the LSTM unit at the next time. The LSTM unit changes the internal state as necessary based on the recurrent input 9802 including the characteristics of the object at that time and the past information up to that point. This makes it possible to determine whether or not the current tracking is successful, taking into account how the pattern of the object changes over time.

As the feature amount of the input to the LSTM unit, the feature amount of the neural network described in the third embodiment can be used. For example, the input value to the final layer 6320 of the neural network 6310 for determining the obscured score of the object in FIG. 13B is used. Here, the input value (multidimensional vector) is used instead of the output value (single-value scalar) of the layer. This is to capture a variety of shielding information into the LSTM by using the same multidimensional features used to determine the shielding state. From this, it can be expected that various shielding patterns can be determined.

When learning the LSTM unit, the teacher value is given as to whether the tracking at each moment is successful or unsuccessful, and each weight parameter of the LSTM is adjusted. As another form, as shown in FIG. 16D, if learning is performed by giving whether or not the teacher is in the shielded state as a teacher value instead of success or failure in tracking, it is determined whether or not the teacher is in the shielded state. It is also possible to let them.

As another embodiment, as in the derived embodiment described in the third embodiment, the neural network detects the tracking object based on the anchor frame and simultaneously determines the position and the shielded score of the object in FIG. 13 (C) as 9401. The feature amount 6420 of 6400 may be used. In this form, the tracking and detection of an object and the shielding score of the object can be determined simultaneously and at high speed.

<Embodiment 5>
In this embodiment, a mode for tracking a single object specified by the user will be described. The basic functional configuration is the same as that of the first embodiment. In this embodiment, perspective information between relative objects is used as the shielding information of the objects.

An example is shown in FIG. 16 (A1). Here, an RGB image 801 is prepared as a learning image. Further, a distance image 833 corresponding to the RGB image 801 is obtained by a laser range finder device, stereo measurement, or the like. The distance image 833 represents the absolute value of the distance from the camera in gray scale, and the whiter the color, the closer the distance is to the object. In the present embodiment, the RGB image 801 is used as the input image, the distance image is used as the teacher value 831, and the weight of the neural network 402 is learned. However, obtaining an output result 830 that is exactly the same as the distance image 831 as an absolute value is a relatively difficult problem for pattern recognition, and the shielding information used in the present embodiment does not need to have such high accuracy. Therefore, in this embodiment, learning is performed to estimate the relative perspective relationship between nearby objects.

For example, as shown in the output result 830 of the figure, the estimation results 8301 and 8302 of the distance between the person 8011 and 8012 are not correct as absolute values. The estimation results 8301 and 8303 corresponding to the person 8011 and the person 8013 distant from the person 8011 also have an incorrect perspective relationship. However, the correct result is obtained if it is limited to the order relationship between the two persons 8011 and 8012 in the vicinity. In this way, the <relationship of perspective order> of <local objects> is learned so that it can be estimated correctly, and these are aggregated and used as the shielding information of the objects.

The above is realized by applying the following measures to the loss value calculation during learning. FIG. 16 (A2) shows a teacher value region 831a which is an enlargement of a region near the symbol * on the teacher value 831 of FIG. 16 (A1). The corresponding output result region 830a is also shown. Here, attention is paid to each pixel i and pixel j on the region 831a, and the loss of the pixel pair is obtained depending on whether or not the perspective relationship is correct. Here, since the perspective relationship between the pixels i and the pixels j on the region 830a matches the teacher value, no loss occurs. On the other hand, if the estimation result is as in the region 830b, the perspective relationship is not correct and a loss is recorded. Such a determination is made for all pixel pairs within a predetermined distance. Finally, the value obtained by dividing the number of pairs with incorrect perspective by the total number of pairs is taken as the total loss value. The neural network 802 learned in this way has completed learning, and the output result 834 of the estimated distance is shown in FIG. 16 (B).

Next, the output results 834 of the relative distance are aggregated to obtain the shielding likelihood of the object. Here, the person detection frames 8351 and 8352 that have been separately detected are used for totaling for each detection frame. The values of the respective distances in each frame are averaged to be ^dave ₁ and ^dave ₂ . Next, the value of this distance is compared and normalized between adjacent object frames, and converted into a score value occ of the shielded likelihood. For example, convert with the following formula.
(Formula 3)
^occ _i = Sigmoid (Log (dave _i / ^dave _j ))
= 1 / (1 + d ^ave _j / d ^ave _i ),
^occ _j = 1 / (1 + dave _i / ^dave _j ),
However, Sigmoid (x) = 1 / (1 + exp (-x)).
Here, i and j are two adjacent detection object frames having an overlapping portion. When three or more objects overlap, the obstruction score occ _i may be obtained by the above formula, and the maximum value among them may be used as the obstruction score of the object.

The above is the processing content until the relative distance estimation is performed and the shielded score is totaled. Since the tracking process using the shielded score is the same as that in the first embodiment, it is omitted here.

The following can be considered as derivative learning ideas. (1) The loss is totaled only for the pair in which the difference between the teacher values of the distance is equal to or more than the predetermined threshold value Θ. This makes it possible to learn robustly against noise when observing a distance image. (2) Perform (1) and set the margin area. For example, not only whether the perspective relationship of the pair is correct or incorrect, but also when the perspective relationship is correct and the values are not relatively separated by a predetermined threshold value Θ or more, a loss is generated. (3) A case where the output value for a pixel pair in which the difference between the teacher values of the distance is less than the threshold value Θ is larger than the threshold value Θ also causes a loss as an error. This suppresses noise-like output.

As described above, there may be various forms, but any form can be used as long as the distance can be learned relative to and locally, and the form is not limited to one. The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network for data communication or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. Further, the program may be recorded and provided on a computer-readable recording medium.

1 Information processing device 201 Image acquisition unit 202 Object detection unit 203 Shielding information generation unit 204 Feature quantity acquisition unit 205 Correspondence unit 206 Storage unit

Claims (16)

An information processing device that detects at least one or more objects from an image.
For each object detected from the image, the shielding relationship with other objects detected from the image is based on a trained model that has learned the image features showing the shielding relationship between the object to be shielded and the shielded object. An estimation means for estimating the occlusion information indicating
It is characterized by having, for each object detected from the image, at least based on the shielding information, a specific means for specifying the correspondence relationship with the object detected in the image captured at a time different from the image. Information processing device. The estimation means provides the shielding information indicating the shielded partial region of the object for each object detected from the image based on the trained model for estimating the shielded area of the object in the input image. The information processing apparatus according to claim 1, wherein the information processing apparatus is estimated. The specifying means specifies a correspondence relationship between an image feature of the object, the shielding information estimated by the estimation means, and an object detected in an image captured at a time different from the image. The information processing apparatus according to claim 1 or 2, wherein the information processing apparatus is characterized in that. The estimation means estimates the shielding information, which is a likelihood indicating that each object detected from the image is shielded by another object.
The information processing apparatus according to any one of claims 1 to 3, further comprising a holding means for holding the likelihood in association with an image feature of the object. The shielding information is characterized in that, for each region of the image, the region of the object being shielded shows a larger likelihood, and the other regions show a smaller likelihood. The information processing apparatus according to claim 4. The specific means captures each object detected from the image at a time different from the image based on the position in the image, the image feature detected from the image, and the shielding relationship in the image. The information processing apparatus according to any one of claims 1 to 5, wherein the correspondence relationship with the object detected from the image is specified. Further having an acquisition means for acquiring an area for each object from the image,
The estimation means according to any one of claims 1 to 6, wherein the estimation means estimates the shielding information indicating the presence or absence of a shielded object in the area of each object acquired by the acquisition means. The information processing device described. Is there an object that is shielded based on the correspondence between each object detected from the image identified by the specific means and the object detected in the image captured at a time different from the image? Judgment means to determine whether or not,
The information processing apparatus according to any one of claims 1 to 7, further comprising a storage means for storing the shielded object. The determination means determines as the first object an object that does not correspond to each object detected from the image among the objects detected in the image captured before the image.
The information processing apparatus according to claim 8, wherein the storage means stores that the first object is shielded at the time when the image is captured. The determination means determines, among the objects detected from the image, an object that does not correspond to the object detected in the image captured before the image as the second object.
The storage means has a similarity with the first object determined to be shielded in the image captured before the image by the storage means with respect to the second object detected from the image. The information processing apparatus according to claim 9, wherein when the value is larger than a predetermined threshold value, it is stored that the first object is not shielded at the time when the image is captured. When two objects are detected in the first image and one object is detected in the second image captured after the first image.
The estimation means estimates the shielding information indicating that the object detected in the second image is shielding another object.
The specific means is detected in the second image with respect to an object that is shielding another object among the objects detected in the first image based on the shielding information estimated by the estimation means. The information processing apparatus according to any one of claims 1 to 10, wherein it is specified that the object is the same as the object. When two objects are detected from the third image captured after the second image,
The estimation means shields the other object from the two objects detected from the third image with respect to the object associated with the image feature of the object detected from the second image. Estimate the shielding information indicating
The specific means refers to an object shielded by another object among the objects detected from the first image, and an object detected from the second image among the objects detected from the third image. The information processing apparatus according to claim 11, wherein an object different from the object associated with the image feature is specified to be the same object as the shielded object in the second image. The information processing apparatus according to any one of claims 1 to 12, wherein the trained model is a neural network. One of claims 1 to 13, wherein the trained model is a model in which image features showing a shielding relationship of the object are trained based on a plurality of images captured at shorter time intervals. The information processing apparatus according to item 1. A program for making a computer function as each means included in the information processing apparatus according to any one of claims 1 to 14. An information processing method that detects at least one or more objects from an image.
Based on the trained model that learned the image features showing the shielding relationship between the shielded object and the shielded object, for each object detected from the image, the shielding relationship with other objects detected from the image is determined. An estimation process for estimating the shielding information to be shown, and
It is characterized by having a specific step of specifying a correspondence relationship between each object detected from the image based on at least the shielding information and the object detected in the image captured at a time different from the image. Information processing method.

Images