KR20170001223A - Information extracting system, information extracting apparatus, information extracting method of thereof and non-transitory computer readable medium - Google Patents

Abstract

Provided are an information extracting system, an information extracting apparatus, an information extracting method thereof and a non-temporary computer readable medium. An information extracting system according to an embodiment of the present invention comprises: a plurality of imaging devices; an information extracting device which extracts information about objects included in an image captured through each imaging device from the plurality of imaging devices; and a server which receives characteristic information of the objects from the information extracting device to store the information. The information extracting device integrates a plurality of single images received from the plurality of imaging devices to generate an integrated image; detects objects included in each of the single images; extracts characteristic information regarding each of the detected objects; tracks the positions of the detected objects in the integrated image; and generates object information about each object by using the extracted characteristic information and the tracked position information.

Description

TECHNICAL FIELD The present invention relates to an information extraction system, an information extraction apparatus, an information extraction method thereof, and a non-transitory computer readable recording medium,

The present invention relates to an information extracting system, an information extracting apparatus, an information extracting method thereof, and a non-transitory computer readable recording medium, and more particularly to an information extracting system for recognizing an object from a plurality of images and extracting object information, An information extracting apparatus, an information extracting method thereof, and a non-transitory computer readable recording medium.

Recently, there is an increasing interest in a technique for extracting and using information of a person existing in a recognition area from a camera image. For example, this technology can be applied to digital signage to allow a user to reproduce a specific advertisement, or to be applied to a security system to detect anomalous indications.

When a single camera image is used, a wide angle camera is mainly used to realize a wide recognition area. However, when a wide angle camera is used, image distortion occurs in the wide angle correction process. There is a problem that the method of using a single camera image has a low recognition performance because the distortion becomes worse as it goes to the peripheral portion of the image.

Even if images of a plurality of cameras are used, if there is no overlapping part between camera images, a matching algorithm according to the arrangement relationship of the cameras affects them. Since the method of determining the same person is used when the person existing in one image disappears and the other person appears in another image after the camera is installed after using the positional relationship between the cameras, There is a problem that the count performance is degraded. Further, there is a problem that many usability is also restricted.

There is a method of extracting user information after synthesizing into one image and extracting user information from individual images and then using the relative positional relationship between cameras to extract the same person Respectively.

In the case of using a synthesized image, it is relatively easy to track a recognized person or to count the number of persons detected. However, in the process of combining images received from a plurality of cameras into one image, image distortion occurs. This degrades the recognition performance of the learning-based recognizer. Therefore, there is a problem that performance is deteriorated when detecting the position of a person at the beginning or extracting information of a person who is sensed.

In the case of using an individual image, since the camera image which is not distorted is used, recognition performance is not deteriorated even if a learning-based recognizer is used. However, there is a need for a method of using an individual image to determine whether the person detected by the individual engine is the same person, and to convert the information of the detected person into a result detected by one recognizer. In addition, there is a problem that the rate of occurrence of errors is remarkably increased in the process of determining whether a person is the same person when a person is present at a portion where the individual images overlap or when a person moves at a high speed at a portion where the individual images overlap.

An object of the present invention is to provide an information extraction system capable of reliably extracting object information while minimizing deterioration of recognition performance when images of multiple cameras are used for a wide recognition range, An information extracting apparatus, and an information extracting method.

According to an aspect of the present invention, there is provided an information extracting system including a plurality of photographing apparatuses, information extracting means for extracting information of an object included in an image photographed by each photographing apparatus in the plurality of photographing apparatuses, And a server for receiving and storing the characteristic information of the object from the information extracting device and the information extracting device, wherein the information extracting device is configured to combine a plurality of single images received from the plurality of photographing devices to generate an integrated image, The method includes detecting the object included in each of the plurality of single images, extracting characteristic information for each of the detected objects, tracking the position of the detected object in the integrated image, And object information of each object is generated using the position information.

According to another aspect of the present invention, there is provided an information extraction apparatus including a communication unit for receiving a plurality of single images, an image integration unit for combining the plurality of single images to generate an integrated image, An object detecting unit for detecting an object included in each of the plurality of single images and extracting characteristic information for each of the detected objects, an object tracking unit for tracking the position of the detected object in the integrated image, And an information generating unit for generating object information of each object using the information and the tracked position information.

The image integrating unit may generate a mapping table that stores a correspondence relationship between the pixel positions of the plurality of single images and the pixel positions in the combined image.

In addition, the mapping table may convert a region in which the pixels of the object are distributed to a rectangular pixel region, and store the corresponding relationship between the pixel region in the plurality of single images and the pixel region in the combined image.

The object tracking unit may convert position information of the object detected in the plurality of single images into position information in the integrated image using the mapping table, and track the object using the converted position information .

The object detecting unit may detect current position information in the plurality of single images using the position information and the mapping table in the integrated image of the object to be tracked, The position information in the integrated image can be corrected using the current position in the plurality of single images and the mapping table.

The object tracking unit may correct the positional information in the integrated image every predetermined period.

The object detection unit may convert position information of the object tracked in the integrated image into position information in the plurality of single images using the mapping table, Can be extracted.

The characteristic information may include at least one of a gaze direction, an age, a gender, race, and emotion of the object.

In addition, the communication unit may transmit the generated object information to the external device every predetermined period.

According to another aspect of the present invention, there is provided an information extracting method of an information extracting apparatus, comprising: receiving a plurality of single images; combining the plurality of single images to generate an integrated image; The method comprising the steps of: detecting an object included in each of the plurality of single images; tracking the position of the detected object in the integrated image; extracting characteristic information on the detected object from the plurality of single images; And generating object information of each object using the extracted characteristic information and the tracked position information.

The method may further include generating a mapping table storing a correspondence relationship between pixel positions of the plurality of single images and pixel positions of the combined image.

In addition, the mapping table may convert a region in which the pixels of the object are distributed to a rectangular pixel region, and store the corresponding relationship between the pixel region in the plurality of single images and the pixel region in the combined image.

The step of tracking may further include the steps of converting position information of an object detected in the plurality of single images into position information in the integrated image using the mapping table, And tracing.

Detecting current position information in the plurality of single images using the position information in the integrated image of the tracked object and the mapping table; and detecting a current position in the plurality of single images, And the step of correcting the position information in the integrated image using the mapping table.

And, the correcting step may correct the positional information in the integrated image every predetermined period.

The extracting may include converting the position information of the object tracked in the integrated image into position information of the plurality of single images using the mapping table, And extracting the characteristic information.

The characteristic information may include at least one of a gaze direction, an age, a gender, race, and emotion of the object.

The method may further include transmitting the generated object information to an external device at predetermined intervals.

According to another aspect of the present invention, there is provided a non-transitory computer readable recording medium including a program for executing an information extracting method of an information extracting apparatus, Generating an integrated image by combining the received plurality of single images, detecting an object included in each of the plurality of single images, tracking the position of the detected object in the integrated image, Extracting characteristic information for each object from the plurality of single images and generating object information of each object using the extracted characteristic information and the tracked position information, .

According to the above-described various embodiments, it is possible to trace an object stably even when the object moves fast at a portion where a plurality of images overlap, or when a large number of objects exist at a portion where a plurality of images overlap.

1 illustrates an information extraction system according to an embodiment of the present invention,
FIG. 2 is a schematic block diagram for explaining a configuration of an information extracting apparatus according to an embodiment of the present invention;
3 is a detailed block diagram illustrating a configuration of an information extraction apparatus according to an embodiment of the present invention,
4 is a view for explaining a formula for arranging a plurality of photographing apparatuses so that photographed images overlap each other;
5 is a view for explaining the operation of the information extracting apparatus according to an embodiment of the present invention,
6 is a diagram for explaining a pixel correspondence relationship between a single image and an integrated image,
7A and 7B are diagrams for explaining a mapping table according to various embodiments of the present invention;
8 is a diagram for explaining an operation of detecting a new object in a single image,
9 is a view for explaining an operation of correcting position information of an object being tracked in an integrated image using position information in a single image,
10 is a diagram for explaining an operation of extracting characteristic information of an object,
11 is a flowchart illustrating an information extraction method of an information extraction apparatus according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. The following terms are defined in consideration of the functions of the present invention, and may vary depending on users, operators, customs, and the like. Therefore, the definition should be based on the contents throughout this specification.

1 is a diagram illustrating an information extraction system 1000, in accordance with an embodiment of the present invention. 1, the information extraction system 1000 includes an information extraction apparatus 100, a plurality of photographing apparatuses 200, and a server 300. [

The information extracting apparatus 100 extracts information of objects included in the images photographed by the plurality of photographing apparatuses 200. [ The information extracting apparatus 100 detects an object included in each single image using each of a plurality of single images. In addition, the information extraction apparatus 100 extracts characteristic information about each of the detected objects using a single image. Since the single image having no image distortion due to the merging of images is used, the information extracting apparatus 100 can improve the performance of object detection and object characteristic information extraction.

The information extracting apparatus 100 combines a plurality of single images received from a plurality of photographing apparatuses to generate an integrated image. The information extraction device 100 tracks the positions of the objects detected in the plurality of single images in the integrated image. By tracking the object in the integrated image, the information extracting apparatus 100 can reduce an error occurring in object counting and the like.

By performing detection, tracking, and characteristic extraction of an object on an image suitable for each process using the correspondence relationship between the plurality of single images and the pixels of the integrated image, the information extraction apparatus 100 can use only a plurality of single images, It is possible to take only the merits of a method using only an integrated image.

The specific configuration of the information extraction apparatus 100 will be described again with reference to FIGS. 2 and 3. FIG.

A plurality of photographing apparatuses 200 are arranged so that photographed images overlap each other. In order to generate an integrated image, a plurality of single images must overlap with each other. For example, as shown in FIG. 4, two photographing apparatuses can be arranged. In Fig. 4, the angle of view (FoV) of the single photographing apparatus is?, And the two photographing apparatuses are arranged so as to face each other staggeringly. In the extended recognition area, the effective angle of view (Effective FoV) based on the intermediate point of the two photographing apparatuses is?. In addition, the minimum recognition distance is represented by Z _min , and the maximum recognition distance is represented by Z _max .

When the recognition area (?, Z _min , Z _max ) to be supported by the information extraction system 1000 is determined, an angle? Tilt is determined to be tilted so that the plurality of photographing apparatuses 200 face each other. The determined angle? Tilt to be tilted can be expressed by Equation (1).

Here, ∠tilt is an angle at which a plurality of photographing apparatuses 200 should be tilted so as to face each other. Further, " alpha " means that one photographing apparatus 200-1 is located at the minimum recognition distance _Zmin The angle of inclination from the horizontal line in which the plurality of photographing apparatuses 200 are disposed. alpha can be expressed by the following equation (2).

In the formula (2), Z _min represents the minimum recognition distance, and d ₀ and d ₁ are the number of images captured by each of the plurality of photographing apparatuses 200-1 and 200-2 to determine the effective view angle γ of the plurality of photographing apparatuses 200 And a distance from the point where it is made. For example, when positioned at the midpoint of the plurality of photographing apparatuses 200, d ₀ and d ₁ have the same value.

d ₀ can be expressed as Equation (3) below using the minimum recognition distance Z _min and the effective angle of view γ.

Further, alpha ₀ is an alpha value when the tilt angle is 0, and can be expressed as Equation (4) below.

In Equation (4),? Represents the angle of view (FoV) of the single photographing device.

Equation (1) may be expressed by Equation (5) using Equations (2) and (4).

In FIG. 1, a plurality of photographing apparatuses 200 are illustrated as being implemented as separate CCTV cameras. However, the present invention is not limited thereto. For example, the present invention may be embodied in various forms such as a camera attached to a digital signage have. It may also be implemented as a single device with the information extraction device 100.

The server 300 receives and stores object characteristic information and position information from the information processing apparatus 100. The server 300 can generate information suitable for people passing through the area where the image is photographed using the received object information. The server 300 can transmit the generated information to a display device (e.g., a large-sized TV, a digital signage, etc.) located in the area where the image is photographed. In another embodiment, when the information extracting apparatus 100 itself has a display unit, the information extracting apparatus 100 itself may generate and display appropriate information.

Through the information extraction system 1000 as described above, objects can be accurately detected, tracked, and object information can be extracted with a wide recognition area.

2 is a block diagram schematically showing a configuration of an information extraction apparatus 100 according to an embodiment of the present invention. 2, the information extraction apparatus 100 includes a communication unit 110, an image integration unit 120, an object detection unit 130, an object tracking unit 140, and an information generation unit 150.

The communication unit 110 performs a function of transmitting / receiving an external device. For example, the communication unit 110 can receive a plurality of single images. Further, the communication unit 110 can transmit the information of the extracted object to the outside. The communication unit 110 transmits the information of the object to the outside when requested by the user or every predetermined period.

An image stitching unit 120 combines a plurality of received single images to generate an integrated image. The image integration unit 120 may generate a mapping table that stores the correspondence between the pixels of the plurality of single images and the pixel positions in the integrated image. Through the location correspondence relationship information stored in the mapping table, contents processed in each of the single image and the integrated image can be transmitted to each other.

An object detection unit (130) detects an object included in a plurality of single images. The object detection unit 130 determines whether an object exists in the individual single image. If the object exists, the object detection unit 130 converts the position information of the object into position information in the integrated image using the mapping table. The converted position information is transmitted to the object tracking unit 140.

In addition, the object detecting unit 130 extracts feature information for each object using a single image. For example, the characteristic information extracting function included in the object detecting unit 130 may be performed on a separate chip. It is preferable that the functions performed by the object detection unit 130 are performed using a single image rather than an integrated image.

The object tracking unit 140 tracks the position of the detected object in the integrated image. The object tracking unit 140 can track the position of the object in the integrated image using a pattern matching algorithm or an optical flow detector to reduce the amount of computation. In order to remove the cumulative error generated in the tracking process, the object tracking unit 140 may correct position information using a single image every predetermined period.

The information generation unit 150 generates object information of each detected object using the extracted object characteristic information, tracked position information, and the like. For example, the object information may include at least one of an object's gaze direction, age, sex, race, emotion, attention time, and dwell time in the recognition area.

3 is a block diagram for explaining the configuration of the information extraction apparatus 100 according to an embodiment of the present invention in detail. 3, the information extraction apparatus 100 includes a communication unit 110, an image integration unit 120, an object detection unit 130, an object extraction unit 140, an information generation unit 150, a storage unit 160, An operation input unit 170, a display unit 180, and a control unit 190.

The communication unit 110 performs a function of transmitting / receiving an external device. The communication unit 120 is composed of a wireless or wired communication module and performs a wireless or wired communication function with the display device 200. Specifically, the communication unit 110 can receive a plurality of single images and transmit information of the extracted objects.

The communication unit 110 can use HDMI (High Definition Multimedia Interface), LVDS (Low Voltage Differential Signaling), LAN (Local Area Network), or the like as a wired communication method. The communication unit 110 may use various methods such as NFC (Near Field Communication), wireless LAN (Wireless LAN), IR (InfraRed) communication, Zigbee communication, WiFi, and Bluetooth. Although various wireless and wired communication methods that the communication unit 110 can use as described above have been described, the present invention is not limited to the communication methods described above.

An image stitching unit 120 combines a plurality of received single images to generate an integrated image. In order to generate an integrated image in the image integration unit 120, a plurality of overlapping portions must exist in the single image. The proportion of overlapping portions is not limited to a specific ratio. For example, the image integrating unit 120 may generate an integrated image using a plurality of single images overlapped by an area of 25 to 30%.

The image integration unit 120 may generate a mapping table that stores the correspondence between the pixels of the plurality of single images and the pixel positions in the integrated image. Through the location correspondence relationship information stored in the mapping table, contents processed in each of the single image and the integrated image can be transmitted to each other. By generating the mapping table, the image integration unit 120 can generate an integrated image for the next input image frame using a smaller amount of calculation.

When generating the mapping table, the image integrating unit 120 may store the correspondence of the coordinate values or the index values of the individual pixels, but may convert the distribution area of the pixels into the rectangular area to generate the mapping table. When the information converted into the rectangular area is used, there is an effect that the number of data to be compared when converting into position information in other images is reduced.

If the mapping table generation unit 120 fails to generate the mapping table, the mapping table generation unit 120 repeats the mapping table generation process until a normal integrated image is generated. After the generation of the mapping table is successful, the process of detecting, tracing, and extracting an object starts from the next input image frame.

An object detection unit (130) detects an object included in a plurality of single images. The object detection unit 130 determines whether an object exists in the individual single image. If the object exists, the object detection unit 130 converts the position information of the object into position information in the integrated image using the mapping table. The converted position information is transmitted to the object tracking unit 140.

The object detecting unit 130 converts the position information in the integrated image of the tracked object into position information in the single image using the mapping table. Accordingly, the object detection unit 130 can know the current position of the tracked object in a single image. The object detection unit 130 may transmit the current position of the single image of the object to the object tracking unit 140 to correct the position information of the object in the combined image.

In addition, the object detecting unit 130 extracts feature information for each object using a single image.

The object tracking unit 140 tracks the position of the detected object in the integrated image. The object tracking unit 140 can track the position of the object in the integrated image by using various methods such as pattern matching.

The object tracking unit 140 converts the position information of the object detected in the single image into the position information in the integrated image using the mapping table. Then, the object tracking unit 140 tracks the object in the integrated image using the converted position information.

When an object is tracked in an integrated image, the number of objects can be accurately counted, and an arithmetic operation for determining whether the object is the same in a plurality of single images can be omitted.

The object tracking unit 140 may correct the position information using a single image every predetermined period. The object tracking unit 140 can recognize the current position information of the object in the integrated image by converting the current position information of the object in the single image captured by the object detection unit 130 using the mapping table. Accordingly, the object tracking unit 140 can remove the accumulated error generated in the tracking process. As another example, the object tracking unit 140 may perform an operation for correcting the position information if the matching score of the object to be tracked falls below a preset value. The matching score of an object refers to a value obtained by comparing the identity of a pattern of an object being tracked in each image frame. For example, the object tracking unit 140 may calculate a correlation score of pixels where objects are being tracked for each frame to obtain a matching score.

The information generation unit 150 generates object information of each detected object using the extracted object characteristic information, tracked position information, and the like. For example, the object information may include at least one of an object's gaze direction, age, sex, race, emotion, attention time, and dwell time in the recognition area.

The information generation unit 150 may collect the information of each object at a predetermined period to generate integrated information. The generated integrated information can be transmitted to the outside through the communication unit 110. [ For example, using the generated object information, the information extraction apparatus 100 can determine which advertisement or information is suitable for the recognition area. As another example, the information extraction apparatus 100 itself may determine suitability by receiving the object information from the external device without making a suitability determination.

The storage unit 160 stores various programs and data necessary for the operation of the information extraction apparatus 100. For example, using the mapping table stored in the storage unit 160, the information extraction apparatus 100 can recognize a correspondence relationship between positions in a plurality of single images and positions in an integrated image. In addition, the storage unit 160 may include a random access memory (RAM) and a read-only memory (ROM). As is well known in the art, ROM is responsible for transferring data and instructions to the CPU in a uni-directional manner, and RAM is typically used to transfer data and instructions in a bi-directional manner.

The operation input unit 170 allows the information extraction device 100 to exchange commands with a user. For example, this may be a keypad, a touch screen, a remote controller, a mouse, and the like. The operation input unit 170 may be implemented in combination with the display unit 180 when the apparatus is implemented as an apparatus requiring a display such as a touch screen. For example, the operation input unit 170 may receive a user command to cause the information of the object to be transmitted to the outside, or to correct the position information of the object being tracked.

The display unit 180 can display the extracted object information. In addition, the display unit 180 may display a plurality of received single images or integrated images. For example, when the information extracting apparatus having the display unit 180 is disposed near the recognition area, the selected advertisement screen and the like can be displayed on the display unit 180 by analyzing the extracted object information. The display unit 180 in the information extracting apparatus 100 according to an embodiment of the present invention is not necessarily implemented.

The display unit 180 may be designed with various display panels. That is, the display unit 180 may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an electronic paper (E-paper), a plasma display panel (PDP), a Vacuum Fluorescent Display (VFD), a Field Emission Display (FED), and an Electro Luminescence Display (ELD). The display panel is mainly made of a light emitting type, but does not exclude a reflective type display. In addition, the display unit 180 may be implemented as a flexible display, a transparent display, or the like.

The control unit 190 controls the overall configuration of the information extraction device 100. [ The control unit 190 may include a ROM (Read Only Memory) and a RAM (Random Access Memory) for storing a module for controlling the CPU (Central Processing Unit) and the information extracting apparatus 100, and data.

For example, the control unit 190 can generate and control a clock signal for timing to transmit object information to an external device, timing to correct position information in the integrated image, and the like. As another example, the control unit 190 may perform a relay function to transmit position information of the object, matching table information, and the like among the components of the information extraction apparatus 100.

Hereinafter, the operation of the information extraction apparatus 100 will be described in more detail with reference to FIGS. 5 to 10. FIG.

5 is a diagram for explaining the operation of the information extraction apparatus 100 according to an embodiment of the present invention. Referring to FIG. 5, information transmitted between the components of the information extraction apparatus 100 will be mainly described.

First, a first single image (Image_Cam 1) and a second single image (Image_Cam 2) received from an external first image capturing apparatus and a second image capturing apparatus are transmitted to the image integrating unit 120. The image integrating unit 120 integrates the plurality of received single images to generate an integrated image (Image_stitched).

When the integrated image is generated, the image integrating unit 120 determines which pixel of each single image corresponds to which pixel of the combined image, and which pixel of the combined image corresponds to which pixel of each single image have. This information is stored in each mapping table.

For example, the object detecting unit 130 may perform the detecting operation and the characteristic extracting operation with separate modules. The object detection unit 130 determines whether an object exists using each single image. When the object is included in a single image, the object detection unit 130 converts the position of the object in the single image to the position in the integrated image using the mapping table, and transfers the position to the object tracking unit 140.

The object tracking unit 140 tracks the position of the object using the integrated image. The object tracking unit 140 can correct the position of the object in the integrated image using the object position in the single image determined by the object detection unit 130 every predetermined period. Accordingly, the object tracking unit 140 can remove the accumulated error generated in the tracking process.

The object tracking unit 140 may transmit the tracked object position information to the object detection unit 130. [ The object detection unit 130 extracts the characteristic information of the object using the position information of each single image and the tracked object. For example, the characteristic information of the object may include at least one of a gaze direction of an object, an age, a sex, race, emotion, attention time, and dwell time in the recognition area.

The characteristic information of the extracted object and the position information of the tracked object are collected by the information generation unit 150. [ The information generation unit 150 generates object information using the collected information. The information generation unit 150 may transmit object information generated externally at regular intervals.

6 is a diagram illustrating a process of constructing a mapping table using a corresponding pixel position when a plurality of single images are combined to form an integrated image.

For example, as shown in FIG. 6, the pixel a in the first single image may correspond to the pixels c, d, and e of the combined image. Similarly, the pixel b in the first single image may correspond to the pixel d, f of the combined image. Pixels are not necessarily 1: 1 matched in a single image and an integrated image. This is because in the image integration process, there is generally a multiple connection relationship between pixels. The image integration unit 120 stores the matching relationship as a mapping table.

7A and 7B are diagrams showing examples of the mapping table. For example, the mapping table of FIG. 7A stores mapping relationships of pixels in the first single image and the integrated image. It is shown which pixel in a single image corresponds to which pixel in the integrated image and which pixel in the single image corresponds to which pixel in the integrated image. For example, it can be seen that the pixel a of the first single image corresponds to the pixels c, d, and e in the integrated image. Also, it can be seen that the pixel d of the combined image corresponds to the pixels a and b of the first single image. Through such a mapping table, the information extraction apparatus 100 can reduce the amount of calculation required for integrating subsequent image frames.

In the mapping table of FIG. 7B, instead of storing all (x, y) coordinate values or index values of all the mapped pixels, the area where the mapped pixels are distributed is represented as one rectangular area and stored. When all the information of the mapped pixels is recorded, the amount of memory usage is increased and the amount of computation can be increased when checking corresponding pixels. Therefore, the information extraction apparatus 100 can use the mapping table of the format shown in FIG. 7B .

8 to 10 are views for explaining the operation of the object detecting unit 130 in more detail. Hereinafter, the case where the object is a person will be described as an example.

Fig. 8 shows an operation for detecting a new object in a single image. Referring to FIG. 8, the object detection unit 130 detects a face position using a face detector (FD) on a single image. Since the position of the detected face is a position in a single image, the object detection unit 130 converts the position of the face into the position in the integrated image using the mapping table. Then, the object detecting unit 130 transmits the converted position information to the object tracking unit 140. [

FIG. 9 shows an operation for correcting an error occurring in the object tracking process. Referring to FIG. 9, the object detecting unit 130 'receives face position information from the object tracking unit 140 in the integrated image. The object detection unit 130 'obtains positional information on a single image using a mapping table. Then, the object detecting unit 130 'sets a search range around the face position in the single image, and finds the actual face position in a single image again. The object detecting unit 130 'converts the position of the actual face determined again to the object tracking unit 140 by converting the position information of the integrated image. Accordingly, the information extraction apparatus 100 can correct an error generated in the object tracking process.

10 is a diagram showing an example of an operation of extracting the characteristic information of the object from the object detecting unit 130 ". Referring to FIG. 10, the object detection unit 130 'receives position information on an integrated image of an object tracked from the object tracking unit 140. FIG. The object detection unit 130 " converts the received position information into position information in a single image using a mapping table. The object detecting unit 130 'accurately grasps the position of the object using the position information and the single image, and extracts various characteristic information of the object.

As another example, the object detecting unit 130 " may extract only the characteristic information at the object detecting step, and then perform only the tracking operation for accurately counting the number through the object tracking unit 140. [ In this embodiment, there is no need to receive position information in the integrated image after object tracking, in order to extract the object characteristic information.

Through the information extracting apparatus 100 according to various embodiments as described above, even when a large number of people exist in a portion where a plurality of single images overlap, or when a moving speed of a person is high, It is possible to accurately extract the characteristic information and reduce the probability of failing to track the object.

11 is a flowchart for explaining an information extracting method of the information extracting apparatus 100 according to an embodiment of the present invention.

Referring to FIG. 11, the information extraction apparatus 100 first receives a plurality of single images from the outside (S1110). Then, the information extraction apparatus 100 integrates the received images to generate an integrated image (S1120). The information extracting apparatus 100 may generate an integrated image, and may generate a mapping table storing a plurality of pixels of a single image and a corresponding relationship of pixels in the integrated image. The mapping table may correspond to coordinate information or index information of each pixel, but it may also be a case where a region in which pixels are distributed is converted into a rectangular pixel region and the corresponding relationship is stored.

The information extraction apparatus 100 can detect an object using a plurality of single images and extract characteristic information of the object (S1130). In another embodiment, the information extraction apparatus 100 first detects an object using a plurality of single images, receives the position information in the integrated image after tracking the object, converts the position information into position information in a single image, It is also possible to extract the characteristic information of the object using the position information.

The information extraction apparatus 100 can track the detected object using the generated integrated image (S1140). Specifically, the information extraction apparatus 100 converts the position information of the object detected in the plurality of single images into position information in the integrated image using the mapping table. Then, the information extraction apparatus 100 tracks the object in the integrated image using the converted position information.

In addition, the information extracting apparatus 100 can correct the position information in the integrated image of the object being tracked. The correction of the position information may be performed every predetermined period or when the mapping score of the object being tracked falls below a predetermined value.

Specifically, the information extraction apparatus 100 detects the position information in a single image using the current position information and the mapping table in the integrated image of the object being tracked. Then, the information extraction apparatus 100 sets a search area around the position of the detected single image and determines the position of the accurate current object. The information extracting apparatus 100 corrects the position of the object in the integrated image by using the accurate position information of the current object determined in the single image and the mapping table.

The information extraction apparatus 100 generates object information using the extracted characteristic information of the object and the tracked position information (S1150). For example, the characteristic information of the object may be at least one of an object's gaze direction, age, sex, race, emotion, attention time, and dwell time in the recognition area. The information extraction apparatus 100 can transmit the generated object information to the outside every predetermined period.

Through such an information extraction method, it is possible to support a wide range of recognition areas, and to reliably extract object information while minimizing deterioration of recognition performance.

In addition, the program code for performing the information extraction method according to various embodiments as described above can be stored in various types of recording media. More specifically, it may be a random access memory (RAM), a flash memory, a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an Electrically Erasable and Programmable ROM (EEPROM), a register, a hard disk, a removable disk, And may be stored in various types of recording media readable by a terminal, such as a memory, a CD-ROM, and the like.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be construed as limiting the scope of the invention as defined by the appended claims. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present invention.

1000: information extraction system 100: information extraction device
110: communication unit 120: image integration unit
130: Object detection unit 140: Object tracking unit
150: information generating unit 160:
170: user interface unit 180: display unit
190: control unit 200: photographing apparatus
300: server

Claims (20)

In an information extraction system,
A plurality of photographing apparatuses;
An information extracting device for extracting information of an object included in an image photographed by each photographing device in the plurality of photographing devices; And
And a server for receiving and storing characteristic information of the object from the information extraction device,
The information extracting apparatus comprises:
A plurality of single images received from the plurality of photographing apparatuses are combined to generate an integrated image, the object included in each of the plurality of single images is detected, the characteristic information for each of the detected objects is extracted, And the object information of each object is generated using the extracted characteristic information and the tracked position information. In the information extracting apparatus,
A communication unit for receiving a plurality of single images;
An image integrating unit for synthesizing the plurality of single images received to generate an integrated image;
An object detecting unit detecting an object included in each of the plurality of single images and extracting characteristic information about each of the detected objects;
An object tracking unit for tracking the position of the detected object in the integrated image; And
And an information generating unit for generating object information of each object using the extracted characteristic information and the tracked position information. 3. The method of claim 2,
The image integrating unit,
And generates a mapping table storing a correspondence relation between pixel positions of the plurality of single images and pixel positions of the combined image. The method of claim 3,
Wherein the mapping table comprises:
Wherein a region in which the pixels of the object are distributed is converted into a rectangular pixel region and a corresponding relationship between the pixel region in the plurality of single images and the pixel region in the combined image is stored. The method of claim 3,
The object-
Converting the position information of the object detected in the plurality of single images into position information in the integrated image using the mapping table and tracking the object using the converted position information, . The method of claim 3,
Wherein the object detection unit comprises:
Detecting current position information in the plurality of single images using the position information in the integrated image of the tracked object and the mapping table,
The object-
And corrects the position information in the integrated image using the detected current position in the plurality of single images and the mapping table. The method according to claim 6,
The object-
And corrects the position information in the integrated image every predetermined period. The method of claim 3,
Wherein the object detection unit comprises:
The position information of the object tracked in the integrated image is converted into position information in the plurality of single images using the mapping table and characteristic information of the object is extracted using the converted position information Information extraction device. 3. The method of claim 2,
The characteristic information may include,
Wherein the information includes at least one of a gaze direction, an age, sex, race, and emotion of the object. 3. The method of claim 2,
Wherein,
And transmits the generated object information to the external device every predetermined period. In the information extraction method of the information extraction device,
Receiving a plurality of single images;
Integrating the received plurality of single images to generate an integrated image;
Detecting an object included in each of the plurality of single images;
Tracking the position of the detected object in the integrated image;
Extracting characteristic information about each of the detected objects from the plurality of single images; And
And generating object information of each object using the extracted characteristic information and tracked position information. 12. The method of claim 11,
And generating a mapping table storing a correspondence relationship between pixel positions of the plurality of single images and pixel positions of the combined image. 13. The method of claim 12,
Wherein the mapping table comprises:
Wherein an area in which the pixels of the object are distributed is converted into a rectangular pixel area and a correspondence relation between the pixel area in the plurality of single images and the pixel area in the integrated image is stored. 13. The method of claim 12,
Wherein the tracing comprises:
Converting position information of an object detected in the plurality of single images into position information in the integrated image using the mapping table; And
And tracking the object using the converted position information. 13. The method of claim 12,
Detecting current position information in the plurality of single images using position information in the integrated image of the tracked object and the mapping table; And
And correcting the position information in the integrated image using the detected current positions of the plurality of single images and the mapping table. 16. The method of claim 15,
Wherein the correcting comprises:
Wherein the position information in the integrated image is corrected every predetermined period. 13. The method of claim 12,
Wherein the extracting comprises:
Converting position information of the tracked object in the integrated image into positional information in the plurality of single images using the mapping table; And
And extracting characteristic information of the object using the converted position information. 12. The method of claim 11,
The characteristic information may include,
Wherein the information includes at least one of a gaze direction, an age, sex, race, and emotion of the object. 12. The method of claim 11,
And transmitting the generated object information to an external device at predetermined intervals. A non-transitory computer readable recording medium including a program for executing an information extraction method of an information extraction apparatus,
The information extracting method of the information extracting apparatus,
Receiving a plurality of single images;
Integrating the received plurality of single images to generate an integrated image;
Detecting an object included in each of the plurality of single images;
Tracking the position of the detected object in the integrated image;
Extracting characteristic information about each of the detected objects from the plurality of single images; And
And generating object information of each object using the extracted characteristic information and tracked position information.

Images