JP5685958B2 - Image display system - Google Patents

Description

  The present invention relates to a technique for processing and displaying a captured video, and more particularly to a technique for displaying a processed video according to the state of a viewer who is shooting.

  Digital signage, which is an advertising medium for displaying advertisements using a display device such as a display or a projector, has begun to be installed in various places. By using digital signage, not only will it be possible to provide rich content using video and audio, but it will also be possible to efficiently deliver advertisements according to the location of digital signage. Expansion is expected.

  Recently, various improvements have been made on digital signage, and a technique for changing an image to be displayed in accordance with the movement of a viewer existing before digital signage has been proposed (see Patent Document 1).

  In the technique described in Patent Document 1, a composite image is generated based on human recognition information and motion information. However, since the tracking process is not performed, each person's viewing time cannot be grasped. Therefore, it is impossible to realize a scenario based on browsing time. In addition, since individual management based on IDs is not performed, it is possible to perform only one-side composition for all viewers, which is insufficient for interactive display for each individual. In addition, as in the technique described in Patent Document 1, detecting a human movement has a problem that a processing load is high and a quick display change cannot be performed.

Japanese Patent No. 4238371

  In the technique described in Patent Document 1, a composite image is generated based on human recognition information and motion information. However, since the tracking process is not performed, each person's viewing time cannot be grasped. Therefore, it is impossible to realize a scenario based on browsing time. In addition, since individual management by ID is not performed, only one-sided composition can be performed for all members, which is insufficient for interactive display corresponding to each individual. In addition, as in the technique described in Patent Document 1, detecting a human movement has a problem that the processing load is high and a quick display change cannot be performed.

  Accordingly, an object of the present invention is to provide an image display system that can interactively display to each individual according to a scenario based on the browsing time of the viewer and can quickly change the display. To do.

  In order to solve the above problems, in the first aspect of the present invention, there is provided a camera for photographing a person, an image processing device for synthesizing a photographed image sent from the camera, and a display for displaying the synthesized image synthesized. An image display system provided with the image processing apparatus storing scenario data storing means for storing scenario data for determining the timing of combining one or more persons on the video and the content, and the content used for the combining And a face image captured in one frame of the video sent from the camera, and the position / rectangular size of the face detection frame is detected as face detection frame data for each detected face image. The face detection means for outputting, and the tracking means for associating the face detection frame data acquired from the face detection means with the face detection frame data of other frames And a scenario data associating means for associating a face object including the face detection frame data detected by the face detecting means with a person included in the scenario data, and according to the association, the face object Is assigned to a person in the scenario data, the position and size of the content are changed in accordance with the position and size of the face detection frame data of the face object, and a composite image is generated by combining the content on the frame. An image display system provided with an image generation means is provided.

  According to the first aspect of the present invention, scenario data that determines the timing of composition of one or more persons on a video and content is prepared, a face detection frame is detected from a frame of the captured video, and face detection is performed. Perform frame tracking, assign the obtained face object to the person in the scenario data, change the position and size of the content according to the position and size of the face detection frame data of the face object, and synthesize the content on the frame Since it did in this way, the image | video which carried out the synthesis | combination in real time can be displayed on the person reflected in the image | photographed image | video.

  According to the second aspect of the present invention, in the image display system according to the first aspect of the present invention, the scenario data includes a plurality of commands, and for each frame, a person is extracted from the frame, and the face is extracted. When the state of the area ratio between the feature portion and the preset region is larger than a predetermined threshold is removed for a predetermined number of frames by removing the detection frame data portion and identifying the feature portion, It further includes an instruction determination unit that determines that there is an instruction from a viewer, and the composite image generation unit is configured to handle the scenario data after switching the command when there is an instruction from the instruction determination unit. According to the association by the attaching means, the face object is assigned to the person of the command after the switching in the scenario data. To.

  According to the second aspect of the present invention, the scenario data has a plurality of commands, and for each frame, a person is extracted from the frame, and the face detection frame data portion is removed to identify the feature portion. If the state where the area ratio between the feature part and the preset area is larger than a predetermined threshold continues for a predetermined number of frames, it is determined that there is an instruction from the viewer and the command is switched. After that, since the face object is assigned to the person of the command after switching in the scenario data, it is possible to display the synthesized image in real time on the person according to the instruction of the person shown in the captured video it can.

  According to a third aspect of the present invention, in the image display system according to the second aspect of the present invention, the instruction determination unit includes a background-removed image obtained by extracting a background from the frame, and extracting the person. This is performed by extracting an overlapping portion of the HSV image obtained by performing HSV conversion on the frame.

  According to the third aspect of the present invention, the extraction of a person from a frame is performed by overlapping a background-removed image obtained by removing the background from the frame and an HSV image obtained by performing HSV conversion on the frame. Therefore, when an instruction from the viewer is determined, it is possible to more accurately extract and determine the skin color part that is a feature of the person.

  According to a fourth aspect of the present invention, in the image display system according to any one of the first to third aspects of the present invention, the tracking means performs the face detection for each face detection frame data acquired from the face detection means. An evaluation value indicating a position / rectangular size difference between the face detection frame data acquired from the frame processed immediately before from the detection unit and the face detection frame data detected in the currently processed frame is calculated, and the evaluation value The smallest previous face detection frame data is associated with each face detection frame data this time, and a face object is generated for each face image newly detected by the face detection unit, and is obtained from the tracking unit. The state of the face object is transitioned to the browsing state according to a predetermined state transition table with reference to the association result between the previous and current face detection frame data. A state transition management means, wherein the scenario data associating means obtains a face object that is in a browsing state or a standby state by the state transition management means, and associates with a person included in the scenario data. It is what is performed.

  According to the fourth aspect of the present invention, for each detected face detection frame data, the position of the face detection frame data acquired from the previously processed frame and the face detection frame data detected in the currently processed frame An evaluation value indicating a rectangular size difference is calculated, tracking is performed by associating the previous face detection frame data with the smallest evaluation value with each face detection frame data, and a face is detected for each newly detected face image. Create an object, refer to the result of matching the previous and current face detection frame data obtained from the tracking means, and change the state of the face object to the viewing state according to the predetermined state transition table, and browse The face object that is in the state or standby state is associated with the person included in the scenario data, so the face image with high possibility of noise is not synthesized It is possible to perform the synthesis only the proper face image.

  Further, according to a fifth aspect of the present invention, in the image display system according to the fourth aspect of the present invention, the composite image generating means is configured to display the person of the scenario data in order from the face object whose state transition management means has detected the browsing state. It is characterized by assigning to.

  According to the fifth aspect of the present invention, since the state transition management means assigns to the person in the scenario data in order from the face object from which the browsing state is detected, each content can be obtained even when there are a plurality of detected persons. Can be accurately combined with the corresponding person.

  In the sixth aspect of the present invention, in the image display system according to any one of the first to third aspects of the present invention, the face detection means sets the output face detection frame data as a face object. The tracking unit is configured to track the face detection frame data in the next frame for each of the face detection frame data acquired from the face detection unit, and set the same face object. The associating means acquires the face object set by the face detecting means, and performs associating with a person included in the scenario data.

  According to the sixth aspect of the present invention, the output face detection frame data is immediately set as a face object, so that the face object can be set quickly and can be quickly associated with the scenario data. It becomes.

  According to a seventh aspect of the present invention, in the image display system according to any one of the first to sixth aspects of the present invention, the scenario data includes a correspondence relationship between a person and content based on an elapsed time from the start of scenario reproduction, The correspondence relationship between the content and the location of the content in the content storage unit is defined, and the composite image generation unit extracts the content corresponding to the person from the content storage unit according to the elapsed time. , And performing synthesis.

  According to the seventh aspect of the present invention, the scenario data defines the correspondence between the person and the content based on the elapsed time from the scenario playback start, and the correspondence between the content and the location of the content in the content storage means. Therefore, according to the scenario, the corresponding content is synthesized with the person according to the timing, and a smooth synthesized image can be reproduced.

  According to the present invention, an image obtained by synthesizing captured images according to a scenario can be displayed interactively for each individual according to the scenario based on the browsing time of the viewer, and can be quickly displayed. There is an effect.

The figure explaining the structure of the image display system 1 in this embodiment. 1 is a hardware block diagram of an image processing device 2 that constitutes an image display system 1. FIG. FIG. 3 is a functional block diagram realized by a computer program installed in the image processing apparatus 2. The flowchart explaining the process which the image processing apparatus 2 analyzes a flame | frame. The flowchart for demonstrating a tracking process. The flowchart for demonstrating face detection frame data matching processing. The figure explaining the state transition table in this embodiment. The figure for demonstrating a human body and a face detection result. The flowchart explaining the process in which the image processing apparatus 2 produces a synthesized image. The figure which shows an example of the scenario data of an XML format in case a target is one person. The figure which shows an example of the scenario data of the XML format in case a target is two persons. The figure which shows the mode of the composition of the content using a face detection frame. The figure which shows the display state of a synthesized image in case a target is one person. The figure which shows the display state of the synthesized image in case there are two targets. The figure which shows an example of the scenario data of the XML format in the case of a touch button correspondence. The figure which shows the display state of the synthesized image in the case of a touch button corresponding | compatible. The figure which shows the mode of the image process in an instruction | indication determination process. The functional block diagram implement | achieved by the computer program mounted in image processing apparatus 2 '. The flowchart for demonstrating a face detection process and a tracking process.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating a configuration of an image display system 1 according to the present embodiment, FIG. 2 is a hardware block diagram of an image processing apparatus 2 that configures the image display system 1, and FIG. It is a functional block diagram implement | achieved by the performed computer program.

  As illustrated in FIG. 1, the image display system 1 includes a display 3 that is a display device such as a liquid crystal display. The display 3 may display not only the photographed image but also the advertisement by dividing the display area.

  In this case, the image display system 1 also functions as a digital signage by installing the display 3 in a street or a store. When the image display system 1 functions as digital signage, a server that controls advertisement video displayed on the display 3 is required.

  An angle is set on the display 3 so that the face of the person who is watching the video reproduced on the display 3 is photographed, and a video for photographing a person who is viewing the advertisement video reproduced on the display 3 A camera 4 is installed.

  The video captured by the video camera 4 is input to the image processing apparatus 2 using a USB port or the like. The image processing apparatus 2 analyzes the frame included in the video transmitted from the video camera 4 and displays the display 3. And the face of the person who browsed the video reproduced on the display 3 are detected, and a log relating to the viewer (for example, the viewing time of the display 3) is stored.

  In the apparatus constituting the image display system 1 illustrated in FIG. 1, commercially available apparatuses can be used for the display 3 and the video camera 4, but the image processing apparatus 2 has features that are not found in the prior art. The image processing apparatus 2 will be described in detail.

  The image processing apparatus 2 can be realized by using a general-purpose computer, and the image processing apparatus 2 includes hardware similar to that of the general-purpose computer. In FIG. 2, as the hardware, a CPU 2a (CPU: Central Processing Unit) is provided. ), A ROM 2b (ROM: Read-Only Memory) on which the BIOS is mounted, a RAM 2c (RAM: Random Access Memory) that is a main memory of the computer, and a large-capacity data storage device 2d (for example, a hard disk) ), An input / output interface 2e for data communication with an external device (video camera 4), a network interface 2f for network communication, and a display output interface 2g for sending information to a display device (display 3) , Character input device 2h (for example, keyboard) and poi Computing device 2i (e.g., a mouse) describes.

  A computer program for operating the CPU 2a is installed in the data storage device 2d of the image processing apparatus 2, and the image processing apparatus 2 is provided with the means shown in FIG. 3 by this computer program. The data storage device 2d is capable of storing various data necessary for the image display system, and stores scenario data that determines the timing of combining one or more persons on the video with the content. It also serves as a scenario data storage means and a content storage means for storing content used for composition.

  Here, the content stored in the content storage means will be described. The content is a material image when a composite image is obtained by combining with the frame of the captured video. FIG. 12A shows an example of content. The content is not particularly limited, and various forms can be used. In the example of FIG. 12A, a case where a wig is prepared as the content is shown. This content has rectangular information (position, width, and height in the x and y directions), and alignment with the captured frame is possible using this rectangular information.

  As shown in FIG. 3, the input of the image processing device 2 is an image captured by the video camera 4, and the output of the image processing device 2 is a processed image obtained by processing the captured image.

  The image processing apparatus 2 includes a background removing unit 20 that removes a background image of a frame of a video shot by the video camera 4 and a background removing unit 20 as means for analyzing the frame of the video shot by the video camera 4. Face detection means 21 for detecting a human face from the frame from which the background image has been removed, human body detection means 22 for detecting a human body from the frame from which the background image has been removed by the background removal means 20, and the face detection means 21 have detected. For each face image newly detected by the face detecting means 21, the tracking means 23 for associating the face with the front and back frames, the moving picture analyzing means 24 for detecting the designated face image from the frame by using a moving picture analysis method such as a particle filter. A face object is generated and a pair of the previous face detection frame data obtained from the tracking means 23 and the current face detection frame data. Referring to the attached result, the state of the face object is transitioned according to a predetermined state transition table, and the state transition management unit 25 that stores a log corresponding to the state transition of the face object and the face detection unit 21 detect the state. A composition obtained by processing each frame of a video shot by the video camera 4 according to the scenario data, the scenario data associating means 83 for associating the face object whose state has been changed by the transition management means 25 and the prepared scenario data. The composite image creating means 84 for creating an image, the instruction determining means 85 for determining whether or not the viewer is instructing the touchless button defined at a predetermined position, and the scenario according to the determination by the instruction determining means 85 Command switching means 86 for switching commands in data is provided, and in the present embodiment, Personal attribute estimation means 26 for estimating the personal attributes (age and gender) of the person from the face image detected by the face detection means 21 in order to include in the log data the attributes (age and gender) of the person who has viewed the display 3; Log file output means 27 for outputting the log stored in the state transition management means 25 in a file format, synthetic target definition means 80 for defining the target (person or place) to be processed in the scenario data, content (image, A synthesized content defining unit 81 for defining voice, CG, etc.) in the scenario data, and an animation scenario defining unit 82 for defining the processing content in the scenario data.

  The scenario data can be created in advance by another system and stored in the data storage device 2d as the scenario data storage means, but the synthesis target definition means 80, the synthesis content definition means 81, and the animation scenario definition means 82. Can also be created. The composite target definition means 80, the composite content definition means 81, and the animation scenario definition means 82 are used to create scenario data that indicates how to process each frame of a video obtained by shooting. . The format of the scenario data is not particularly limited, but in the present embodiment, XML (eXtensible Markup Language) is adopted. In the present embodiment in which XML is used as the scenario data, the synthesis target definition unit 80, the synthesis content definition unit 81, and the animation scenario definition unit 82 can be realized by a text editor. Therefore, scenario data is created when the text editor is activated and the administrator inputs characters using the character input device.

  FIG. 10 is a diagram illustrating an example of scenario data in the XML format when the number of targets is one. From now on, the synthetic target definition unit 80, the synthetic content definition unit 81, and the animation scenario definition unit 82 will be described in detail with reference to the scenario data of FIG. The synthetic target definition means 80 defines a target by setting four attributes of target ID, type, absolute coordinate, and transition setting. In the example of FIG. 10, it corresponds to a range surrounded by two tags, <Simulation Targets> on the third line and </ Simulation Targets> on the ninth line. Regarding the type, two types of person and place can be set, but in the example of FIG. 10, only the person is set using the <Human> tag. The transition setting can be set only for a person, and sets whether or not to associate with a new person when the associated person disappears. In the example of FIG. 10, the tag of the 5th line sets three attributes of target ID, type, and transition setting, the target ID is “0”, the type is “human”, and transition setting (IsTransfer) Is “false (not set)”.

  The composite content definition unit 81 defines the content by setting three attributes of a content ID, a content path, and an overlap setting. In the example of FIG. 10, it corresponds to a range surrounded by two tags, <Simulation Contents> on the 10th line and </ Simulation Contents> on the 18th line. In the example of FIG. 10, the content ID (ContentsID) is defined for seven contents from “0” to “6”. As shown in FIG. 10, a content path (ContentsPath) and an overlap setting (OverlapOrder) are set in units of one line for each content.

  The animation scenario definition means 82 defines an animation scenario by setting seven attributes of command ID, command type, start key, end key, key type, target ID, and content ID. In the example of FIG. 10, it corresponds to a range surrounded by two tags of <Animation Commands> on the 19th line and </ Animation Commands> on the 34th line. In the example of FIG. 10, the command ID (CommandID) is defined for seven commands from “0” to “6”. As shown in FIG. 10, for each command, a command type, a start key, an end key, a key type, a target ID, and a content ID are set in units of two lines. The command type indicates what type of effect is generated based on what frame, and layer synthesis, α blend synthesis, and audio reproduction start are prepared. Of these, layer composition and alpha blend composition indicate the type of image composition, layer composition overwrites content, and alpha blend composition is content and frame according to the set α ratio. Is synthesized through transmission. In the example of FIG. 10, layer composition (LayerMontage) is set as the command type (CommandType). The start key and end key set the start time and end time of each command. In the present embodiment, the scenario data time is managed as “0.0” at the start of the scenario and “1.0” at the end of the scenario. Therefore, the start key (StartKey) of the command that starts first is “0.0”, and the end key (EndKey) of the command that ends last is “1.0”. The key type is to set a target as a reference for the start key and the end key, and three types of own, base, and global are prepared. own is based on the browsing time of the face object corresponding to each target ID, base is based on the browsing time of the face object corresponding to target ID = 0, and global is based on the time when the first frame of the captured video is acquired. To do. In the example of FIG. 10, since “own” is set as the key type (KeyType), the time point when the face object appears in the frame (the time point when the face object is determined to be “viewing start”) is set to “0.0”. The start key and end key will be recognized.

  The <CycleInterval> tag of the scenario data sets the time from the start to the end of the scenario in seconds. In the example of FIG. 10, “10” is set in the <CycleInterval> tag on the first line. Therefore, it is shown that it is 10 seconds from the start to the end of the scenario. The scenario is managed in real time that is 10 times the value of the start key and end key. The <IsAutoLoop> tag of the scenario data sets whether to perform loop processing (repeated processing). In the example of FIG. 10, “true” is set in the <IsAutoLoop> tag on the second line. , Loop processing is performed. The <CycleInterval> tag and <IsAutoLoop> tag can also be set using a text editor. In this way, the scenario data created by the composite target definition means 80, the composite content definition means 81, and the animation scenario definition means 82 is stored in the data storage device 2d as the scenario data storage means.

  When the image processing device 2 analyzes the frames of the video captured by the video camera 4 in time series, the data storage device 2d of the image processing device 2 can view the display as a log file that can be used for browsing measurement. A browsing time log file in which time is stored, a position log file in which the position of a person who has viewed the display is stored, and a person attribute log file in which the personal attributes (for example, age and gender) of the person who has viewed the display are stored Log file output that stores the total number of people in front of the display, the number of people who are not browsing the display, the number of people who have viewed the display, and outputs these log files Means 27 is provided in the image processing apparatus 2. In the present invention, it is not essential to create a log file, but the face object and the browsing start time in the process of creating the log file are used to generate a composite image.

  First, each means provided for analyzing and processing a frame of a video shot by the video camera 4 will be described while explaining a process in which the image processing apparatus 2 analyzes a frame of a video transmitted from the video camera 4. To do.

  FIG. 4 is a flowchart for explaining processing in which the image processing apparatus 2 analyzes a frame of a video transmitted from the video camera 4. Although details of each processing will be described later, when one frame of video is input to the image processing device 2, the image processing device 2 performs background removal processing S1 on the frame, and about the frame after the background removal processing S1. Then, face detection processing S2 and human body detection processing S3 are performed.

  The image processing apparatus 2 performs the face detection process S2 and the human body detection process S3 on the frame after the background removal process S1, and then uses the result of the face detection process S3 as a frame to be processed this time. A state transition in which a tracking process S4 is performed for associating the face detected from the N frame with the face detected from the previous frame N-1 frame, and the state of the face object is changed based on the result of the tracking process S4. The management process S5 is executed.

  First, the background removal process S1 will be described. The means responsible for the background removal processing S1 is the background removal means 20 of the image processing apparatus 2. The image processing apparatus 2 executes the background removal process S1 because the position and angle of the video camera 4 provided at the upper part of the display 3 is fixed as shown in FIG. This is because a captured image includes a background image that does not change, and by removing this background image, a human body / face can be detected with high accuracy.

  As the background removal process executed by the background removal unit 20 of the image processing apparatus 2, existing technology can be used, and the video captured by the video camera 4 may change in the morning, noon, and night, for example. It is preferable to use a dynamic background update method that takes into account the temporal change of.

  Dynamic background update methods that take into account temporal changes in the background include, for example, “Shinji Morita, Kazumasa Yamazawa, Nobuhiko Terasawa, Naokazu Yokoya:“ Network-enabled remote monitoring system using omnidirectional image sensors ”, electronic The method described in the Journal of Information and Communication Engineers (D-II), Vol. J88-D-II, No. 5, pp. 864-875, (2005.5) can be used.

  Next, the face detection process S2 executed by the face detection unit 21 of the image processing apparatus 2 will be described. Various face detection methods including the face detection method described in Patent Document 1 have been disclosed as face detection methods performed in the face detection process S2, but in this embodiment, as weak classifiers, The face detection method by Adaboost algorithm using black and white Haar-Like feature is adopted. For the face detection method by Adaboost algorithm using black and white Haar-Like features as weak classifiers, see “Paul Viola and Michael J. Jones,“ Rapid Object Detection using a Boosted Cascade of Simple Features ”, IEEE CVPR, 2001. ", Rainer Lienhart and Jochen Maydt," An Extended Set of Haar-like Features for Rapid Object Detection ", IEEE ICIP 2002, Vol. 1, pp. 900-903, Sep. 2002.

  Face detection frame data is obtained for each face image included in the frame by executing the face detection method using the Adaboost algorithm using the black and white Haar-Like feature as a weak classifier. The position of the face detection frame used when detecting the image (for example, the coordinates of the upper left corner) and the rectangular size (width and height) are included.

  Next, the human body detection process S3 executed by the human body detection unit 22 of the image processing apparatus 2 will be described. As a method of detecting a human body, a method of detecting a human body using an infrared sensor and utilizing a human body temperature is well known. However, in this embodiment, the human body detection method performed in the face detection process S2 is weak. A human body detection method based on the Adaboost algorithm using HOG (Histogram of Oriented Gradients) features is adopted as a discriminator. For the human body detection method using the Adaboost algorithm using the HOG (Histogram of Oriented Gradients) feature as a weak classifier, see “N. Dalal and B. Triggs,” Histograms of Oriented Gradientstional Conference on Computer Vision, pp. 734-741. , 2003. "

  By executing the human body detection method using the Adaboost algorithm using the HOG feature as a weak classifier, human body detection frame data is obtained for each human body included in the frame, and when this human body detection frame data is detected, The position (for example, the coordinates of the upper left corner) and the rectangular size (width and height) of the human body detection frame used in the above are obtained.

  FIG. 8 is a diagram for explaining the human body and face detection results. The person photographed in the frame 7 of FIG. 8 includes a total of six persons 7a to 7f, and the human body detection means 22 of the image processing apparatus 2 detects each person 7a to 7f, and each person 7a to 7f is detected. Human body detection frame data 70a to 70f corresponding to 7f are output. Further, the face detection means 21 of the image processing apparatus 2 detects the faces of the persons 7a to 7c in which both eyes are photographed, and outputs face detection frame data 71a to 71c corresponding to each face.

  Next, the tracking process S4 executed by the tracking unit 23 of the image processing apparatus 2 will be described. In the tracking process S4, a process for associating the face detection frame data detected by the face detection unit 21 from the N-1 frame with the face detection frame data detected by the face detection unit 21 from the N frame by the tracking unit 23 of the image processing apparatus 2. Is executed.

  From here, the tracking process S4 performed by the tracking means 23 of the image processing apparatus 2 will be described in detail. FIG. 5 is a flowchart for explaining the tracking process S4 executed by the tracking unit 23 of the image processing apparatus 2.

  The tracking unit 23 of the image processing apparatus 2 first acquires the face detection frame data and the human body detection frame data obtained from the N frame from the face detection unit 21 and the human body detection unit 22, respectively, in order to perform the tracking process S4 for the N frame. (S10).

  In the next tracking process S4, the face detection frame data obtained from the N frame is used as the face detection frame data of the N-1 frame. Therefore, the tracking unit 23 of the image processing apparatus 2 obtains the frame from the N frame. The face detection frame data thus obtained is stored in the RAM 2c or the data storage device 2d.

  When the tracking unit 23 of the image processing apparatus 2 acquires the face detection frame data and the human body detection frame data of N frames, the tracking means 23 performs display browsing determination for each human body detection frame data of N frames (S11).

  As described above, since the human body detection frame data includes the position and the rectangular size of the human body detection frame, and the face detection frame data includes the position and the rectangular size of the face detection frame, the face detection frame is included. The human body detection frame data can be determined as the human body detection frame data of the person who is browsing the display 3, and the human body detection frame data which does not include the face detection frame is the human body detection frame data of the person who is not browsing the display 3. Can be determined as data.

  When the tracking means 23 of the image processing apparatus 2 makes a display browsing determination for each human body detection frame data of N frames in this way, it is displayed in front of the display 3 as a number of people log file when the N frames are captured. The total number of persons who are present, that is, the number of human body detection frame data detected by the human body detection means 22, and the number of persons who are not browsing the display 3, that is, human body detection frame data not including a face detection frame. Number of people browsing the display 3, that is, the number of human body detection frame data including the face detection frame is generated, and a data storage is performed by assigning N frame number and the like Store in device 2d.

  When the tracking unit 23 of the image processing apparatus 2 performs display browsing determination for each human body detection frame data of N frames, the face detection frame data detected by the face detection unit 21 from the N−1 frame and the face detection unit 21. Executes face detection frame data association processing S12 for associating the face detection frame data detected from the N frames.

  FIG. 6 is a flowchart for explaining the face detection frame data association processing S12. In this embodiment, in the flow illustrated in FIG. 6, the evaluation value obtained by using the evaluation function of Equation 1 described below. Is used to associate the face detection frame data.

Note that the evaluation function f1 () in Equation 1 is an evaluation function using the nearest neighbor method, and the evaluation value obtained by the evaluation function f1 () is an evaluation indicating the difference between the position / rectangular size of the face detection frame data. Value. Further, the evaluation value obtained by the evaluation function f2 () of Expression 1 is obtained from the face image included in the face detection frame specified by the face detection frame data to the evaluation value obtained from the evaluation function f1 (). The difference between the SURF feature amounts indicating the features of the image becomes an evaluation value added by weighting.

  In order to associate the face detection frame data detected from the N-1 frame with the face detection frame data detected from the N frame, the tracking unit 23 of the image processing apparatus 2 first counts the number of face detection frame data obtained from the N frame. Only the loop processing L1 is executed.

  In this loop process L1, the tracking means 23 of the image processing apparatus 2 first executes the loop process L2 by the number of face detection frame data detected from the N-1 frame, and in this loop process L2, the loop process L1 The evaluation value is calculated by substituting the position / rectangular size of the face detection frame data to be processed and the position / rectangular size of the face detection frame data to be processed by the loop processing L2 into the evaluation function f1 () of Equation 1. In step S120, an evaluation value indicating a position / size difference from the face detection frame data to be subjected to the loop processing L1 is calculated for each face detection frame data detected from the N-1 frame.

  The tracking unit 23 of the image processing apparatus 2 calculates an evaluation value indicating a position / size difference from the face detection frame data to be processed by the loop processing L1 for each face detection frame data detected from the N-1 frame. Then, the minimum value of the evaluation value is searched (S121), and after calculating the difference between the minimum value of the evaluation value and another evaluation value (S122), it is determined whether there is the difference value equal to or less than the threshold value (S123). ).

  Then, the tracking unit 23 of the image processing apparatus 2 includes the difference between the minimum value of the evaluation value indicating the position / size difference from the face detection frame data to be processed by the loop processing L1 and the other evaluation values. When there is a difference equal to or less than the threshold, the tracking unit 23 of the image processing apparatus 2 executes the loop process L3 for the number of face detection frame data whose evaluation value is within the threshold.

  In this loop process L3, the face image within the face detection frame specified by the face detection frame data to be processed by the loop process L1 and the N-1 frame face detection frame data to be processed by the loop process L3 are specified. The difference of the SURF feature quantity with the face image in the face detection frame to be obtained is obtained, the difference of the SURF feature quantity is substituted into the evaluation function f2 () of Formula 1, and the evaluation value obtained by adding the difference of the SURF feature quantity is It is calculated for each face detection frame data detected from the N-1 frame (S124).

  The evaluation value obtained using the evaluation function f1 () using only the nearest neighbor method is used to calculate the evaluation value obtained by adding the difference of the SURF feature values using the evaluation function f2 () expressed by the mathematical formula 1. If the difference value between the minimum value and other evaluation values is equal to or smaller than the threshold value, it is considered that the face detection frames having similar sizes are close to each other (for example, the person 7a, b in FIG. 8), and the nearest neighbor method This is because the N-1 frame face detection frame data associated with the N frame face detection frame data cannot be determined from the above evaluation values.

  By using the evaluation function f2 () expressed by Equation 1 and calculating an evaluation value obtained by adding the difference between the SURF feature amounts, an evaluation value in which the facial features are added is calculated. When face detection frames having similar sizes are close to each other, the N-1 frame face detection frame data having a similar face is associated with the N frame face detection frame data.

  Then, the tracking unit 23 of the image processing apparatus 2 uses the N-1 frame face detection frame data having the minimum evaluation value obtained from the evaluation function of Formula 1 as the N frame face to be subjected to the loop processing L1. A process of associating with the detection frame data is executed (S125). If the evaluation value using the evaluation function f2 () shown in Equation 1 is not calculated, the evaluation value used in this process is the value obtained from the evaluation function f1 () shown in Equation 1. Thus, when the evaluation value using the evaluation function f2 () shown in Equation 1 is calculated, the evaluation value used in this process is the value obtained from the evaluation function f2 () shown in Equation 1. Become.

  When the loop processing L1 ends and the tracking means 23 of the image processing apparatus 2 associates the N frame face detection frame data with the N-1 frame face detection frame data, the N-1 frame face detection frame data overlaps. Then, it is confirmed whether it is associated with the face detection frame data of N frames (S126).

  When the face detection frame data of the N-1 frame overlaps and is associated with the face detection frame data of the N frame, the tracking unit 23 of the image processing apparatus 2 overlaps the N-1 frame. By referring to the evaluation value of the face detection frame data of the frame and recursively executing the process of associating the smaller evaluation value with the face detection frame data of the N frame, finally the face detection frame data of the N frame N-1 frame face detection frame data to be associated with is determined (S127).

  From here, it returns to description of the flow illustrated in FIG. When the tracking process S4 is completed, the state transition management unit 25 of the image processing apparatus 2 obtains the state transition obtained in advance from the tracking process S4 and refers to the result of association between the previous and current face detection frame data, and is determined in advance. The state transition management process S5 is executed to change the state of the face object according to the table and store a log corresponding to the state transition of the face object. If there is a predetermined state transition in the state transition management process S5, A corresponding predetermined log file is stored in the data storage device 2d.

  In the state transition management unit 25 of the image processing apparatus 2, in order to manage the state transition of the face object, a state transition table in which a state of the face object and a rule for state transition are defined in advance is defined. The tracking unit 23 of the image processing device 2 refers to this state transition table and changes the state of the face object based on the result of the face detection frame data association processing S12.

  From here, an example of the state transition table is illustrated, and the state transition management process S5 executed by the state transition management unit 25 of the image processing apparatus 2 will be described while explaining the state transition table.

  FIG. 7 is a diagram illustrating the state transition table 6 in the present embodiment. The state transition table 6 illustrated in FIG. 7 defines the state of the face object and the transition from the state of the N-1 frame to the state of the N frame. The vertical axis of the state transition table 6 indicates the state of the N-1 frame. The horizontal axis indicates the state of N frames, and the condition for state transition is described at a location where the vertical axis and the horizontal axis intersect. In the state transition table, “-” indicates an illegal state transition.

  In the state transition table 6 illustrated in FIG. 7, None, candidate face, current face, standby face, noise face, and end face are defined as face object states, and the state transition defined in the state transition table is described. However, each state will be described.

  None, which is one of the face object states, means a state in which no face object exists, and there is no N-1 frame face detection frame data associated with N frame face detection frame data (condition 1 in FIG. 7). ), The state transition management unit 25 of the image processing apparatus 2 selects a face object having an attribute value such as an ID for identifying the face object, face detection data of the N frame, data related to the state assigned to the face object, and the like. A new face is generated and the state of the face object is set as a candidate Face.

  The candidate face that is one of the face object states means a state in which the newly detected face image may be noise, and the candidate face is provided as one of the face object states. This is because in the case of a complex background, erroneous detection of a face image is likely to occur even if background removal processing is performed, and the newly detected face image may be noise.

  The face object in the candidate face state is given, as data related to the candidate face state, a state ID indicating the candidate face state, the date and time when the state transition is made to the candidate face, and a counter.

  The states that can be changed from the candidate face are the candidate face, the current face, and the noise face, and a predetermined number of face detection frames corresponding to the face objects that are in the candidate face state are continuous within a predetermined setting time. If the tracking is successful (condition 2-2 in FIG. 7), the state of the face object changes from the candidate face to the current face.

  The reason why a counter is provided for the attribute of the face object in the candidate face state is to count the number of times that the face detection frame corresponding to the face object in the candidate face state can be tracked continuously within the set time. Then, the state transition management unit 25 of the image processing apparatus 2 determines that when the state of the face object including the N-1 frame face detection data associated with the N frame face detection frame data is the candidate Face, The face detection frame data attached to the face object is updated to N frame face detection frame data, and the counter of the face object is incremented.

  Then, when executing the state transition management process S5, the state transition management unit 25 of the image processing apparatus 2 refers to the date and time when the state transition is made to the candidate face for each face object that is the candidate face, and within the set time When the value of the counter has reached a predetermined setting value, the state of the face object is changed to the current Face. Further, the state transition management unit 25 of the image processing apparatus 2 causes the state of the face object that has not reached the set value at the time when the set time has elapsed to change to the noise face (FIG. 7). Condition 2-3), the face object for which the set time has not elapsed does not change state (condition 2-1 in FIG. 7).

  The noise face that is one of the states of the face object is a state in which the face image detected by the face detection unit 21 of the image processing apparatus 2 is determined to be noise, and the face object that has transitioned to the noise face is considered to have disappeared. It is made and is not used for the subsequent state transition management process S5.

  The current face, which is one of the face object states, is a state in which a person corresponding to the face object can determine that the display 3 is in the browsing state. It is time to browse.

  When the state transition management unit 25 of the image processing apparatus 2 changes the state of the face object from the candidate Face to the current Face, the face detection frame data of the face object is updated to N frame face detection frame data and the Face As the data related to the above, a face ID indicating the current face state and the date and time when the state is changed to the current face are assigned to the face object.

  In addition, when the state of the face object is changed to “Face” in order to store the person attributes (for example, age and gender) of the person browsing the display as a log, the state transition management unit 25 of the image processing apparatus 2 The person attribute estimating means 26 is operated to acquire a person attribute obtained from the face detection frame specified by the face detection frame data of the face object whose state is currently changed to Face, and the object ID and person attribute of the face object are described. The attribute log file thus stored is stored in the data storage device 2d.

  The person attribute estimation means 26 provided in the image processing apparatus 2 will not be described in detail, but automatic identification of a person's attribute (age / gender) from a person's face image is an automatic cigarette sale. For example, a technique disclosed in Japanese Patent Application Laid-Open No. 2007-080057 can be used.

  Further, the state transition management unit 25 of the image processing apparatus 2 stores a position log file for storing the position of the person who is browsing the display 3 in time series when the state of the face object is currently changed to Face. Newly generated in the device 2d. The position log file at the time of generation is given the object ID of the face object whose state has been changed to Face and the face detection frame data included in the face object whose state has been changed to Face.

  The states that can be changed from the current face state are the current face and the standby face. The state transition management unit 25 of the image processing apparatus 2 is configured so that the state of the face object including the N-1 frame face detection data associated with the N frame face detection frame data is currently Face (condition 3-1). The face detection frame data attached to the face object is updated to face detection frame data in N frames, and the face detection frame data is added to the position log file specified by the object ID of the face object.

  Further, when the state transition management unit 25 of the image processing apparatus 2 performs the state transition management process S5, N-1 frame face detection frame data that is not associated with the N frame face detection frame data is assigned. When the state of the face object is currently “Face”, the moving image analysis unit 24 is operated to perform processing for detecting a face image corresponding to the face detection frame data of the N−1 frame from the N frame by the moving image analysis method.

  In the present embodiment, the moving image analysis unit 24 of the image processing apparatus 2 first has N frames that are already associated with N-1 frame face detection frame data that has not been associated with N frame face detection frame data. The occlusion state is determined between the face detection frame data, and it is confirmed whether the target person's face is completely occluded.

The moving image analysis means 24 of the image processing apparatus 2 executes processing for determining the occlusion state of the face object according to Equation 2 for all face objects that exist at this time and are currently in the face, candidate face, and standby face states. To do.

  When the moving image analysis unit 24 of the image processing apparatus 2 executes the process of determining the occlusion state of the face object according to Equation 2, the process branches based on the determination result.

  When it is determined that there is a high possibility that the person to be tracked is an occlusion in a completely hidden state (when the criterion 1 of Expression 2 is satisfied), tracking by the particle filter is performed, and the position of the target face object・ Detect the rectangle size. The particle filter is described in many literatures such as “Takekazu Kato:“ Particle filter and its implementation ”, IPSJ Research Report, CVIM-157, pp.161-168 (2007).” .

  Also, if it is judged that the possibility is high due to occlusion in which the person being tracked is half-hidden (corresponding to criterion 2 in Formula 2), tracking is performed using the LK method (Lucus-Kanade algorithm) The position / rectangular size of the face object is detected. The LK method is described in “Lucas, BD and Kanade, T .:“ An Iterative Image Registration Technique with an Application to Stereo Vision ”, Proc. DARPA Image Understanding Workshop, pp. 121-130, 1981.” ing.

  When it is determined that there is a high possibility that the person to be tracked does not have occlusion (when the criterion 3 in Expression 3 is satisfied), the moving image analysis unit 24 of the image processing apparatus 2 performs tracking using the CamShift method. To detect the position / rectangular size of the target face object. The CamShift method is described in “G. R. Bradski:“ Computer vision face tracking for use in a perceptual user interface, ”Intel Technology Journal, Q2, 1998.”.

  When the target face image can be detected from N frames by any one of these methods, the state transition management unit 25 of the image processing apparatus 2 uses the face detection data of the face object that is currently in the face state as these methods. If the target face image cannot be tracked by any of these methods, the state of the face object that is currently in the face state is changed to the standby face (see FIG. 7 condition 3-2).

  A standby face, which is one of the states of a face object, is a state in which a face image corresponding to the face object cannot be detected even using the moving image analysis means 24 provided in the image processing apparatus 2.

  Further, when the state transition management unit 25 of the image processing apparatus 2 changes the state of the face object to the standby face, the face detection frame data of the face object is not updated, and the state of the standby face is used as data related to the standby face. Is given to the face object, the date and time when the face object has made a transition to the current Face, and the date and time when the face object has made a transition to the standby Face.

  The state in which state transition is possible from the standby face is the current face or the end face. The state transition management unit 25 of the image processing apparatus 2 searches for a face object including face detection frame data of N frames before a predetermined time elapses after the state transition to the standby face, and the state of the face object Is a standby face, the state of the face object is changed from the standby face to the current face (condition 4-1 in FIG. 7).

  When the state of the face object is changed from the standby face to the current face, the state transition management unit 25 of the image processing apparatus 2 indicates that the date and time when the face object has changed to the current face is the state of the standby face. Sometimes the date and time assigned to the face object is used.

  Further, when executing the process for managing the state transition of the face object, the tracking unit 23 of the image processing apparatus 2 changes the state of the face object that has passed a predetermined time from the state transition to the standby face to the end face. The state is changed (condition 4-3 in FIG. 7), and the state of the face object for which the set time has not elapsed is not changed (condition 4-2 in FIG. 7).

  The end face, which is one of the face object states, is a state corresponding to a person who can no longer be detected by the image processing apparatus 2, and the face object whose state is the end face is considered to have disappeared. It is not used in the transition management process S5.

  Note that the state transition management unit 25 of the image processing apparatus 2 indicates the object ID of the face object and the date and time when the face object is currently transitioned to Face before the face object is transitioned to end Face. A browsing time log file describing the browsing start time and the browsing end time, which is the date and time when the face object changes to the standby face, is generated and stored in the data storage device 2d.

  As described above in detail, the image processing apparatus 2 has five states of the face object generated for each face detected by the face detecting unit 21: None, candidate Face, current Face, standby Face, noise Face, and end Face. Can be stored as a log according to the state transition of the face object by changing the state of the face object according to the tracking result of the face corresponding to the face object. become.

  According to the above-described contents, while the face object state is currently Face, the face corresponding to the face object can be continuously detected. It becomes browsing time.

  Further, by defining the candidate Face as the state of the face object, even when a face is erroneously detected due to noise, the influence on the browsing time of the display 3 is eliminated. Also, by defining the standby face as the state of the face object, even if the same face is detected after losing sight of the face, it can be handled as the same face.

  FIG. 9 is a flowchart illustrating a process in which the image processing apparatus 2 creates a composite image based on a video frame transmitted from the video camera 4. When the image processing device 2 is activated and scenario data to be used is designated, first, the scenario data association unit 83 reads the designated scenario data from the data storage device 2d (S21). Then, the scenario data association unit 83 interprets the scenario data and starts creating an image according to the scenario data (S22).

  Next, the scenario data association unit 83 acquires the face object data generated by the state transition management unit 25 (S23). The face object data includes an object ID, face detection frame data (position, rectangular size), and browsing time.

  Subsequently, the scenario data association unit 83 performs a process of associating the face object data acquired from the state transition management unit 25 with the scenario data (S24). Specifically, the object ID of the face detection frame data included in the face object data is associated with the target ID. When a plurality of face detection frame data is acquired from the state transition management unit 25, the date and time when the state transition to the candidate face is the earliest date and time is set to “0”, and thereafter the date and time when the state transition to the candidate face is performed. In order from the earliest, “1”, “2” and “3” are set while increasing the number by one. In the example of FIG. 10, since only one target ID “0” is set in the scenario data, the scenario data association unit 83 is identified by the object ID associated with the target ID “0”. The face detection frame data is targeted.

  Next, the composite image creating means 84 performs processing for creating a composite image for display in units of frames (S25). Specifically, first, the start time is set to “0.0”, and the <Animation Commands> of the scenario data is referred to at this time “0.0”. Then, as shown in FIG. 10, from the start key “0.0” to the end key “0.1”, the key type “own”, the command type “LayerMontage (layer composition)”, the target ID “0”, the content Since the ID is “0”, the composite image creating unit 84 creates a composite image by layer-combining the content ID “0” with the face detection frame data with the target ID “0”.

  Since the content path of content ID “0” (material for composition) can be identified by referring to <Simulation Contents> of the scenario data, it can be determined from the storage location in the data storage device 2d identified by the path. Get content. As described above, since the rectangular information is set in the content, the composite image creating unit 84 changes the rectangular information and the size of the content so that the rectangular information and the rectangular size of the face detection frame data match, Layer composition of the changed content is performed at a position where the position of the rectangle information after the change and the position of the face detection frame data match. Specifically, the size of the rectangle set in the content is changed to match the rectangle of the face detection frame data set in the face image as shown in FIG. The content is resized according to the change ratio, and the composition process is performed so that the two rectangles match. As a result, for example, when the content is a wig as shown in FIG. 12A, a composite image (FIG. 12C) in a state where the wig is covered with a human face is obtained. The composite image creating unit 84 displays the obtained composite image on the display 3. As a result, a composite image in which the frame of the captured video is processed is displayed on the display 3.

  When the creation of the composite image for one frame is completed, the scenario data association unit 83 determines whether or not the scenario is in progress (S26). Specifically, the <CycleInterval> tag in the scenario data is referenced with the elapsed time from the start of image creation according to the scenario data. If the elapsed time is less than the scenario time, it is determined that the scenario is in progress and the elapsed time If the time is longer than the scenario time, it is determined that the scenario is over. If it is determined that the scenario is in progress, the scenario data association unit 83 returns to S23 and acquires face object data.

  In S24, the scenario data association unit 83 performs processing for associating the next face object data acquired from the state transition management unit 25 with the scenario data. Also at this time, as in the first loop, the earliest date and time when the state transition to the candidate face is set to “0”, and thereafter “1” and “2” in order of the date and time when the state transition to the candidate face occurs. Set "3" while increasing the number by one. Then, according to the scenario data, the scenario data association unit 83 targets the face detection frame data specified by the object ID associated with the target ID “0”.

  Next, in S25, the composite image creating unit 84 performs a process of creating a composite image in units of frames. Specifically, the elapsed time is acquired, and the <Animation Commands> of the scenario data is referenced with the acquired elapsed time. When the acquired elapsed time is smaller than “0.1”, as shown in the scenario data of FIG. 10, between the start key “0.0” and the end key “0.1”, as in the case described above. Therefore, since the key type is “own”, the command type is “LayerMontage (layer composition)”, the target ID is “0”, and the content ID is “0”, the composite image creating unit 84 performs the face detection frame with the target ID “0”. A composite image is created by layer combining the content ID “0” with the data. When the acquired elapsed time exceeds “0.1”, the key type “own”, the command type is used from the start key “0.1” to the end key “0.3” as shown in FIG. Since “LayerMontage”, target ID “0”, and content ID “1”, the composite image creation unit 84 combines the content ID “1” with the face detection frame data of the target ID “0”. By doing so, a composite image is created. In this way, the process of executing the scenario data is repeated according to the elapsed time until it is determined in S26 that the scenario is ended.

  If it is determined in S26 that the scenario has been completed, the scenario data association unit 83 determines whether or not to repeat the process (S27). Specifically, the <IsAutoLoop> tag in the scenario data is referenced, and if “true” is set, it is determined that the process is repeated. If it is determined that the process is to be repeated, the scenario data association unit 83 resets the elapsed time to “0”, starts measuring the elapsed time again, returns to S22, and follows the scenario data. Start creating an image. In this manner, the composite image obtained from each frame of the video is sequentially displayed on the display, so that it is displayed as a processed video.

  FIG. 13 is a diagram showing a display state of a composite image obtained according to the scenario data of FIG. First, the composite image balloon "Hello" has been synthesized is displayed on the frame (a). The frame at the scenario begins (StartKey = "0.0") until the elapsed time of 1 second (EndKey = "0.1"), all balloon "Hello" (ContentsID = "0") is a synthetic image synthesized is displayed It will be. At this time, by referring to <SimulationContents> in ContentsID = "0", and specifying a location "fukidashi001.jpg" in the content storage unit of the content, it acquires the content of the balloon "Hello".

  Next, when the elapsed time is 1 second (StartKey = "0. 1"), a synthesized image in which the balloon "isn't your hair nice today?" (ContentsID = "1") is displayed (I). For frames from an elapsed time of 1 second (StartKey = "0. 1") to an elapsed time of 3 seconds (EndKey = "0.3."), A composite image in which all the balloons are nice today's hair is synthesized? Will be displayed. At this time, by referring to <SimulationContents> with ContentsID = "1", the location "fukidashi002.jpg" in the content storage means of the content is specified, and the content of the balloon "isn't your hair today nice?"

  When the elapsed time is 3 seconds (StartKey = "0.3"), a composite image in which the balloon "Is it a little more casual" (ContentsID = "2") is combined is displayed (c). At this time, by referring to <SimulationContents> with ContentsID = "2", the location "fukidashi003.jpg" in the content storage unit of the content is specified, and the content of the balloon "Is it a little more casual" is acquired.

  For the frames from the elapsed time of 3 seconds (StartKey = "0.3") to the elapsed time of 6 seconds (EndKey = "0.6"), a composite image in which all the balloons "I want to make it a little more casual" is displayed. Become. When the elapsed time is 6 seconds (StartKey = "0.6"), a composite image in which the black wig (Contents ID = "5") is combined is displayed (D). At this time, by referring to <SimulationContents> with ContentsID = "5", the location "afro.jpg" in the content storage unit of the content is specified, and the content of the black wig is acquired. For frames from an elapsed time of 6 seconds (StartKey = "0.6") to an elapsed time of 7 seconds (EndKey = "0.7"), a composite image in which all black wigs are combined is displayed.

  When the elapsed time is 7 seconds (StartKey = "0.7"), a composite image in which the balloon "This is good" (ContentsID = "3") and the blond wig (ContentsID = "6") is displayed. (E). At this time, by referring to <SimulationContents> with ContentsID = "6", the location "blond.jpg" in the content storage means of the content is specified, and the content of the blond wig is acquired. For frames from an elapsed time of 7 seconds (StartKey = "0.7") to an elapsed time of 9 seconds (EndKey = "0.9"), all the balloons are “this is good” and a composite image of a blonde wig is displayed. Will be. When the elapsed time is 9 seconds (StartKey = "0.9"), a balloon "Nedou will" (ContentsID = "4") and a composite image in which a blonde wig is synthesized are displayed (F). At this time, by referring to <SimulationContents> with ContentsID = "4", the location "fukidashi005.jpg" in the content storage means of the content is specified, and the content of the balloon "Nendodo" is acquired. For frames from an elapsed time of 9 seconds (StartKey = "0.9") to an elapsed time of 10 seconds (EndKey = "1.0"), a composite image in which all black wigs are combined is displayed.

  Next, a case where there are two targets will be described. FIG. 11 is a diagram illustrating an example of scenario data in XML format when there are two targets. A significant difference from the example of FIG. 10 is that two target IDs are set in a <Human> tag sandwiched between <Simulation Targets> tags. In the example of FIG. 11, target IDs “0” and “1” are set. In the <Animation Commands> tag, one target ID is set for each command ID. In the example of FIG. 11, the target ID “0” is set for the command IDs “0”, “2”, “3”, and “5”, and the target ID “1” is set for the command IDs “1”, “4”, and “6”. "Is set.

  When the number of targets is two, the process is executed according to the flowchart of FIG. 9 as in the case where the number of targets is one. If there are two targets, the scenario data association unit 83 acquires a plurality of pieces of face object data generated by the state transition management unit 25 in S23. Therefore, in S24, the acquired face object data is converted into scenario data. Associate with. In the example of FIG. 11, two target IDs “0” and “1” are set in the scenario data. Therefore, the scenario data associating unit 83 associates the object IDs “0” and “1” with each other. The face detection frame data specified by the ID is targeted. As a result, even if the face detection frame of the target face object is moved so as to be replaced by a frame, the composite image creating unit 84 can synthesize content in accordance with each face detection frame.

  FIG. 14 is a diagram showing a display state of a composite image obtained according to the scenario data of FIG. First, is displayed combined image on the left side of the frame balloon "Hello" is synthesized, at the elapsed time of 1 second, the composite image in the right frame balloon "Hello" is synthesized is displayed (A). When the elapsed time reaches 3 seconds, a composite image in which a blond wig and a black wig are combined is displayed (not shown). When the elapsed time reaches 4 seconds, a composite image in which the balloon “your hair today”, the blonde wig and the black wig are combined is displayed (I). When the elapsed time reaches 6 seconds, a composite image in which a blonde wig and a black wig are combined is displayed (not shown). When the elapsed time is 7 seconds, a composite image in which the balloon “your hair today”, the blonde wig, and the black wig are combined is displayed (c). When the elapsed time is 9 seconds, a composite image in which a balloon wig and a black wig are combined is displayed ("D").

  Next, a case where interactive display is performed using a virtual touchless button will be described. In order to perform interactive display, scenario data corresponding to the interactive display is required. FIG. 15 is a diagram illustrating an example of scenario data in the XML format in the case of corresponding to a touchless button (one target). From now on, the synthetic target definition unit 80, the synthetic content definition unit 81, and the animation scenario definition unit 82 will be described in detail with reference to the scenario data of FIG. The synthetic target definition means 80 defines a target by setting four attributes of target ID, type, absolute coordinate, and transition setting. In the example of FIG. 15, it corresponds to a range surrounded by two tags, <Simulation Targets> on the first line and </ Simulation Targets> on the seventh line. Regarding the type, two types of person and place can be set, but in the example of FIG. 15, only the person is set using the <Human> tag. The transition setting can be set only for a person, and sets whether or not to associate with a new person when the associated person disappears. In the example of FIG. 15, the tag in the third row sets three attributes of target ID, type, and transition setting, the target ID is “0”, the type is “human”, and transition setting (IsTransfer) Is “false (not set)”.

  The composite content definition unit 81 defines the content by setting three attributes of a content ID, a content path, and an overlap setting. In the example of FIG. 15, it corresponds to a range surrounded by two tags, <Simulation Contents> on the eighth line and </ Simulation Contents> on the eleventh line. In the example of FIG. 15, two contents whose content ID (ContentsID) is “0” and “1” are defined. As shown in FIG. 15, a content path (ContentsPath) and an overlap setting (OverlapOrder) are set for each content in units of one line.

  The animation scenario definition means 82 defines an animation scenario by setting seven attributes of command ID, command type, start key, end key, key type, target ID, and content ID. In the example of FIG. 15, it corresponds to a range surrounded by two tags, <Animation Commands> on the 12th line and </ Animation Commands> on the 17th line. In the example of FIG. 15, two commands with command IDs (CommandID) “0” and “1” are defined. As shown in FIG. 15, for each command, a command type, a selector ID (SelectorID), a start key, an end key, a key type, a target ID, and a content ID are set in units of two lines. The command type indicates what type of effect is generated based on what frame, and layer synthesis, α blend synthesis, and audio reproduction start are prepared. Of these, layer composition and alpha blend composition indicate the type of image composition, layer composition overwrites content, and alpha blend composition is content and frame according to the set α ratio. Is synthesized through transmission. In the example of FIG. 15, layer composition (LayerMontage) is set as the command type (CommandType).

  The selector ID is a condition for specifying a command to be executed, and a command with a command ID corresponding to the selector ID is executed. A set of one or more commands specified by the selector constitutes one scenario. Therefore, there is a one-to-one correspondence between selectors and scenarios. In the example of FIG. 15, one scenario is composed of one command. When the selector ID = 0, the command with the command ID = 0 is executed, and when the selector ID = 1, the command ID = 1. Indicates that the command will be executed. The start key and end key are used to set the start time and end time of each command, as in the example of FIG. In the present embodiment, the scenario data time is managed as “0.0” at the start of the scenario and “1.0” at the end of the scenario. In the example of FIG. 15, unlike the example of FIG. 10, the start key (StartKey) of all commands is “0.0”, and the end key (EndKey) of the last command is “1.0”. The key type is to set a target as a reference for the start key and the end key, and three types of own, base, and global are prepared. own is based on the browsing time of the face object corresponding to each target ID, base is based on the browsing time of the face object corresponding to target ID = 0, and global is based on the time when the first frame of the captured video is acquired. To do. In the example of FIG. 10, since “own” is set as the key type (KeyType), the time point when the face object appears in the frame (the time point when the face object is determined to be “viewing start”) is set to “0.0”. The start key and end key will be recognized.

  In the example of FIG. 15, the <CycleInterval> tag and the <IsAutoLoop> tag shown in FIG. 10 are surrounded by two tags, <Human> on the second line and </ Human> on the fourth line. Are included in the target ID tag. Even when the <CycleInterval> tag and <IsAutoLoop> tag are included in the target ID tag, the <CycleInterval> tag of the scenario data is the time from the start to the end of the scenario in seconds, as in the example of FIG. The <IsAutoLoop> tag of the scenario data sets whether to perform loop processing (repetition processing). In the example of FIG. 15, since CycleInterval = “10” is set in the target ID tag, it indicates that it is 10 seconds from the start to the end of the scenario. The scenario is managed in real time that is 10 times the value of the start key and end key. In the example of FIG. 15, since IsAutoLoop = “true” is set in the target ID tag, this indicates that loop processing is performed. In this way, the scenario data created by the composite target definition means 80, the composite content definition means 81, and the animation scenario definition means 82 is stored in the data storage device 2d as the scenario data storage means.

  Furthermore, the animation scenario definition means 82 defines an animation selector by setting four attributes: selector ID, select type, select number, and trigger ID. In the example of FIG. 15, it corresponds to a range surrounded by two tags of <AnimationSelectors SelectorsNum = “2” InitialSerectorID = “0”> on the 18th line and </ AnimationSelectors> on the 21st line. In the example of FIG. 15, two selectors having selector IDs (Selector ID) “0” and “1” are defined. As shown in FIG. 15, a selector ID (SelectorID), a select type (SelectType), a select number (SelectNum), and a trigger ID (TriggerID) are set for each selector in units of one line. The selector ID is an ID that identifies the selector. The select type (SelectType) defines a method for specifying the selector ID, and there are four types: selectnum, incrementnum, decrementnum, and returnnum. selectnum selects the directly specified selector ID. The incrementnum selects a selector ID obtained by adding a select number (SelectNum) to the currently selected selector ID. Decrementnum selects the selector ID obtained by subtracting the select number (SelectNum) from the currently selected selector ID. returnnum selects the selector ID corresponding to the trigger ID. The select number (SelectNum) is a number used for incrementnum and decrementnum. The trigger ID is an ID that is generated in response to a predetermined event. In the example of FIG. 15, the trigger ID is generated when a touchless button is designated.

  Furthermore, the animation scenario definition means 82 sets an animation trigger by setting seven attributes of a touchless button ID, a button image path, a mask file path, a button position, a button size, a continuous time threshold value, and a mask ratio threshold value. Define In the example of FIG. 15, it corresponds to a range surrounded by two tags of <AnimationTriggers> on the 22nd line and </ AnimationTriggers> on the 31st line. In the example of FIG. 15, two buttons having a touchless button ID (TouchlessButton ID) “0” and “1” are defined. As shown in FIG. 15, touchless button ID (TouchlessButton ID), button image path (ButtonImagePath), mask file path (MaskFilePath), button position (PositionX, PositionY), button size (ButtonWidth) in units of 4 lines for each button. , ButtonHeight), continuous time threshold (Thtime), and mask ratio threshold (Thmask). The touchless button ID is an ID that identifies the touchless button. The touchless button ID has a one-to-one correspondence with the trigger ID. When the touchless button ID is determined, the trigger ID is also set to the same value. The button image path is a path for specifying a position where an image of a touchless button to be displayed on the display 3 is recorded. The mask file path is a path for specifying a position where a mask image for masking to round the corners of the button shape is recorded. The button position is an xy coordinate on the screen on which the touchless button is displayed. In the button instruction determination described later, when a button area is set, since it is necessary to invert the left and right of the image, the x coordinate is converted and used accordingly. The button size is the width and height of the touchless button. The continuous time threshold value is a threshold value of time for determining that the touchless button is continuously pressed. The mask ratio threshold is an area ratio threshold for determining that the touchless button is pressed.

  The case where the scenario data shown in FIG. 15 is designated will be described using the flowchart of FIG. When the image processing device 2 is activated and designated as scenario data using the scenario data shown in FIG. 15, first, the scenario data association unit 83 reads the designated scenario data from the data storage device 2d (S21). Then, the scenario data association unit 83 interprets the scenario data and starts creating an image according to the scenario data (S22).

  Simultaneously with the start of image creation, the image processing apparatus 2 displays the button image of the touchless button on the display 3. The viewer is photographed by the video camera 4, and the photographed image is always acquired. When the viewer tries to touch the button image, the image is photographed.

  When the creation of the image is started, the scenario data association unit 83 acquires the face object data generated by the state transition management unit 25 (S23). The face object data includes an object ID, face detection frame data (position, rectangular size), and browsing time.

  Subsequently, the scenario data association unit 83 performs a process of associating the face object data acquired from the state transition management unit 25 with the scenario data (S24). Specifically, the object ID of the face detection frame data included in the face object data is associated with the target ID. When a plurality of face detection frame data is acquired from the state transition management unit 25, the date and time when the state transition to the candidate face is the earliest date and time is set to “0”, and thereafter the date and time when the state transition to the candidate face is performed. In order from the earliest, “1”, “2” and “3” are set while increasing the number by one. In the example of FIG. 15, since only one target ID “0” is set in the scenario data, the scenario data association unit 83 is identified by the object ID associated with the target ID “0”. The face detection frame data is targeted.

  Next, the composite image creating means 84 performs processing for creating a composite image for display in units of frames (S25). Specifically, first, the start time is set to “0.0”, and the <Animation Commands> of the scenario data is referred to at this time “0.0”. As shown in FIG. 15, in <Animation Commands>, there are two commands with command IDs “0” and “1”, both of which are a start key “0.0” and an end key “1.0”. Up to now, the key type is “own”, the command type is “LayerMontage (layer composition)”, and the target ID is “0”, and only the content ID is different between “0” and “1”. As described above, the command ID is determined by the selector ID. However, since the initial selector ID (InitialSerectorID) is set to “0” as shown in the 18th line in FIG. The creation unit 84 creates a composite image by layer-combining the content with the content ID “0” with the face detection frame data with the target ID “0”.

  Since the content path of content ID “0” (material for composition) can be identified by referring to <Simulation Contents> of the scenario data, it can be determined from the storage location in the data storage device 2d identified by the path. Get content. As described above, since the rectangular information is set in the content, the composite image creating unit 84 changes the rectangular information and the size of the content so that the rectangular information and the rectangular size of the face detection frame data match, Layer composition of the changed content is performed at a position where the position of the rectangle information after the change and the position of the face detection frame data match. Specifically, the size of the rectangle set in the content is changed to match the rectangle of the face detection frame data set in the face image as shown in FIG. The content is resized according to the change ratio, and the composition process is performed so that the two rectangles match. As a result, for example, when the content is a wig as shown in FIG. 12A, a composite image (FIG. 12C) in a state where the wig is covered with a human face is obtained. The composite image creating unit 84 displays the obtained composite image on the display 3. As a result, a composite image in which the frame of the captured video is processed is displayed on the display 3.

  When the creation of the composite image for one frame is completed, the scenario data association unit 83 determines whether or not the scenario is in progress (S26). Specifically, refer to “CycleInterval” in the <Target ID> tag in the scenario data with the elapsed time from the start of image creation according to the scenario data, and if the elapsed time is less than the scenario time, If the elapsed time is equal to or longer than the scenario time, it is determined that the scenario is ended. If it is determined that the scenario is in progress, the scenario data association unit 83 returns to S23 and acquires face object data.

  In S24, the scenario data association unit 83 performs processing for associating the next face object data acquired from the state transition management unit 25 with the scenario data. Also at this time, as in the first loop, the earliest date and time when the state transition to the candidate face is set to “0”, and thereafter “1” and “2” in order of the date and time when the state transition to the candidate face occurs. Set "3" while increasing the number by one. Then, according to the scenario data, the scenario data association unit 83 targets the face detection frame data specified by the object ID associated with the target ID “0”.

  Next, in S25, the composite image creating unit 84 performs a process of creating a composite image in units of frames. Specifically, the elapsed time is acquired, and the <Animation Commands> of the scenario data is referenced with the acquired elapsed time. At this time, if the selector ID is “0”, the content of the content ID “0” is selected. If the selector ID is “1”, the content of the content ID “1” is converted to the face detection frame of the target ID “0”. A composite image is created by layering the data. In this way, the process of executing the scenario data is repeated according to the elapsed time until it is determined in S26 that the scenario is ended.

  If it is determined in S26 that the scenario has been completed, the scenario data association unit 83 determines whether or not to repeat the process (S27). Specifically, the <IsAutoLoop> tag in the scenario data is referenced, and if “true” is set, it is determined that the process is repeated. If it is determined that the process is to be repeated, the scenario data association unit 83 resets the elapsed time to “0”, starts measuring the elapsed time again, returns to S22, and follows the scenario data. Start creating an image. In this manner, the composite image obtained from each frame of the video is sequentially displayed on the display, so that it is displayed as a processed video.

  On the other hand, in parallel with the processing of S22 to S25, the instruction determination unit 85 always determines whether or not the touch button has been instructed (S28). Specifically, the determination is performed by performing image processing on the frame of the captured video. FIG. 17 is a diagram showing an image obtained by the image processing of the instruction determination unit 85. In FIG. 17, a white portion indicates a pixel having a pixel value “1”, and a black portion indicates a pixel having a pixel value “0”. First, the instruction determination unit 85 acquires the background removed image processed by the background removal unit 20 in S1 of FIG. Then, each pixel value of the multi-value background removed image is binarized with a predetermined threshold value. FIG. 17A shows a binarized background removed image. Subsequently, the instruction determination unit 85 performs HSV conversion on the input frame image, binarizes it with a predetermined threshold value, and obtains a binarized HSV image. FIG. 17B shows a binarized HSV image. A color portion such as skin color is extracted by this HSV conversion. Then, an And calculation process is performed on the binarized background removed image and the binarized HSV image. And the person image which extracted the person is obtained. FIG. 17C shows a person image.

  Next, the instruction determination unit 85 acquires the face detection frame acquired by the face detection unit 21 in S2 of FIG. The instruction determination unit 85 generates a binary face mask image masking the acquired face detection frame. FIG. 17D shows a face mask image. Then, the instruction determination unit 85 performs an And calculation process on the face mask image and the person image. As a result, a feature image excluding the face portion from the person is obtained. FIG. 17E shows a feature image.

  Next, the instruction determination unit 85 determines the button position (PositionX, PositionY) and button size (ButtonWidth, ButtonHeight) of the touchless button in a range surrounded by two tags <AnimationTriggers> and </ AnimationTriggers> in the scenario data. ) Information. Then, from the button position and the button size, the button area and the total number of pixels included in the button are obtained. Further, a ratio Rmask between the number of pixels of the pixel value “1” (expressed in white in the drawing) in the button area of the feature image and the total number of pixels included in the button is calculated. FIG. 17F shows the relationship between the feature image and the button area. In FIG. 17F, a rectangular area surrounded by a white line is a button area. As illustrated in FIG. 15, when a plurality of touchless buttons are set, the instruction determination unit 85 performs a process of calculating the ratio Rmask for each button.

  The instruction determination unit 85 performs the processing as described above for each frame, and calculates the ratio Rmask of each touchless button. Then, the mask threshold Thmask set in the scenario data is compared with the ratio Rmask of each touchless button. Then, based on the comparison result, it is determined which of the following states 1) to 3).

(Judgment of the state by the ratio and duration of the feature on the touchless button)
1) When Rmask <Thmask Normal state 2) When Rmask> Thmask and Time <Thtime Hand is defeated 3) When Rmask> Thmask and Time> Thtime Instructed state

  In 2) and 3) above, Time is the time calculated by multiplying the number of frames by the reciprocal of the frame rate, and Thtime is a threshold of time set in the scenario data. In the case of 1), since the area of the hand on the touchless button is small, it is determined that the hand is not in a state of being touched by the touchless button but in a normal state. In the case of the above 2), since the area of the hand on the touchless button is large, the hand is in a state of being touched, but since the time is short, it is not possible to determine even the instruction for the touchless button. In the case of 3), since the area of the hand on the touchless button is large and the time is long, it is determined that the instruction is for the touchless button. In the scenario data shown in FIG. 15, in both of the two touchless buttons, Thtime is set to 1.0 second and Thmask is set to 0.1 (10%).

  That is, in S28, the instruction determination unit 85 performs processing for each frame to determine whether the hand area ratio for each touchless button is equal to or greater than the threshold value, and the hand area ratio is equal to or greater than the threshold value. When a certain touchless button continues for a predetermined number of frames or more, it is determined that the touchless button has been instructed.

  If the instruction determination unit 85 determines in S28 that the touchless button has been instructed, the command switching unit 86 performs command switching (S29). Specifically, the selector ID is changed. In the example of FIG. 15, “0” is changed to “1”, and “1” is changed to “0”. By changing the selector ID, the command ID is also changed, and the command is switched.

  FIG. 16 is a diagram showing a display state of a composite image obtained according to the scenario data of FIG. In FIGS. 16A and 16B, a rectangle with rounded corners indicates a touchless button. In the example of FIG. 16, touchless buttons labeled “Blonde” and “Black hair” are displayed. When the viewer is recognized, first, as shown in the scenario data 18th line in FIG. 15, since the InitialSelectorID that defines the initial selector ID is “0”, the selector ID is set to “0”. The command ID is set to “0”. Then, according to the <Command ID> tag on the 13th and 14th lines, the content ID “0” is set for the frame from the scenario start time (StartKey = “0.0”) to the elapsed time of 10 seconds (EndKey = “1.0”). Are layered. At this time, by referring to <SimulationContents> with ContentsID = "0", the location "afro.jpg" in the content storage unit of the content is specified, and the content of the black wig is acquired. And as shown to Fig.16 (a), a black wig is synthesize | combined with a viewer's image. When there is no command switching by the instruction of the touchless button, a composite image in which all black wigs are synthesized for the frames from the start of the scenario (StartKey = "0.0") to the elapsed time 10 seconds (EndKey = "1.0"). Will be displayed.

  When the viewer touches the “blonde” touchless button and the instruction determination unit 85 determines that the touchless button is instructed according to the condition of <AnimationTriggers>, the instruction determination unit 85 determines that the touchless button ID (TouchlessButton ID) is set to “1”, the command switching means 86 sets the trigger ID corresponding to the touchless button ID on a one-to-one basis to “1”, the selector ID is set to “1”, and the command ID is set to “1”. Switch to 1 ”. Then, according to the <Command ID> tag in the 13th and 14th lines, the content ID “1” is set for the frame from the scenario start time (StartKey = “0.0”) to the elapsed time of 10 seconds (EndKey = “1.0”). Are layered. At this time, by referring to <SimulationContents> with ContentsID = "1", the location "blond.jpg" in the content storage means of the content is specified, and the content of the blond wig is acquired. And as shown in FIG.16 (b), it replaces with a black wig and a blond wig is synthesize | combined with a viewer's image. After this, if there is no command switching by the instruction of the touchless button, all blonde wigs are composed for the frames from the start of the scenario (StartKey = "0.0") to the elapsed time 10 seconds (EndKey = "1.0"). The synthesized image thus displayed is displayed.

<Configuration not using state transition management means>
In the image display system of the above embodiment, the state transition management unit 25 is used, and when it is determined that the detected face image is noise, the browsing state is not determined, but the state transition management unit 25 is used. It is also possible to determine that all detected face images are in the browsing state. Next, a configuration not using the state transition management unit 25 will be described.

  FIG. 18 is a functional block diagram realized by a computer program installed in the image processing apparatus 2 ′ when the state transition management unit 25 is not used. 18, components having the same functions as those in FIG. 3 are denoted by the same reference numerals, and detailed description thereof is omitted.

  An image processing apparatus 2 ′ illustrated in FIG. 18 includes a tracking unit 23 ′ instead of the tracking unit 23 illustrated in FIG. This tracking means 23 'also has a function corresponding to the moving picture analysis means 24 shown in FIG.

  The image processing apparatus 2 ′ illustrated in FIG. 18 performs the processes of S1 and S3 in the same manner as the image processing apparatus 2 among the processes of S1 to S5 illustrated in FIG. Further, the face detection process and the tracking process are executed in cooperation. As described above, the state transition management process in S5 is not performed.

  FIG. 19 is a flowchart showing face detection processing and tracking processing. First, after performing the background removal processing S1, in processing N frames, it is determined whether or not the number of face detection frames in N-1 frames is greater than 0 (S31). When the number of N-1 frame face detection frames is larger than 0, the tracking unit 23 'executes the tracking process (S32).

  The tracking means 23 'tracks each face detection frame in the N-1 frame and specifies a corresponding face detection frame in the N frame. As the tracking unit 23 ′, a known tracking method such as “particle filter”, “LK method”, or “CamShift method” executed by the moving image analysis unit 24 can be employed.

  When the tracking process of the face detection frame from the N-1 frame to the N frame is finished, the face detection unit 21 performs the face detection process in the N frame (S33). The face detection process in S33 is the same as the face detection process in S2 shown in FIG. If it is determined in S31 that the number of face detection frames in the N-1 frame is not greater than 0, the face detection unit 21 does not perform the tracking process from the N-1 frame to the N frame and the face detection unit 21 The face detection process is performed.

  Subsequently, it is determined whether or not the number of N frame face detection frames newly detected in the face detection process S33 is greater than 0 (S34). The newly detected N frame face detection frame is obtained by excluding the face detection frame tracked from the N-1 frame to the N frame from the face detection frames detected in the N frame.

  Next, the face detection means 21 assigns an object ID to each face detection frame data newly detected in the N frame, and sets a face object composed of the face detection frame data, the object ID, and the tracking time ( S35). The face object is specified by the object ID, and the face detection frames associated by tracking are specified by the same object ID. The initial value of the tracking time is set to zero.

  Subsequently, it is determined whether or not the number of face detection frames in the N frame is greater than 0 (S36). In S36, it is determined whether or not a face detection frame in which an object ID has already been issued exists in the N frame regardless of whether or not a new detection has been performed in the N frame.

  When the face detection frame exists, the tracking time is calculated for the face object of each face detection frame (S37). Specifically, the tracking time of each face object up to N frames is calculated by adding the time corresponding to one frame to the tracking time calculated up to the immediately preceding N-1 frame. When the tracking time is calculated, N is incremented (S38), and the process proceeds to the next N frame. If the result of determination in S36 is that there is no face detection frame, there is no target to be tracked in N frames, so tracking time is not calculated and N is incremented (S38). ), And shifts to processing for the next N frame.

  The face detection unit 21 and the tracking unit 23 ′ of the image processing apparatus 2 ′ repeatedly execute the process shown in FIG. 19 for each frame on which the background process has been performed by the background removal unit 20.

  The face object given in the process shown in FIG. 19 is associated with the scenario data by the scenario data association unit 83 in S24 shown in FIG. In the processing shown in FIG. 19, the object ID of the face object is set in increments of “0”, “1”, “2”, and “3” in the order in which the face detection frames are detected.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above embodiments, and various modifications can be made. For example, in the above-described embodiment, when the instruction determination unit extracts a person, a background-removed image obtained by removing the background from the frame and an overlapping portion of the HSV image obtained by performing HSV conversion on the frame are displayed. Although extraction is performed, other methods may be used as long as the person being photographed can be extracted.

  In the above embodiment, a composite image is generated and displayed according to scenario data. However, sound source data is prepared, sound source data is reproduced according to scenario data, and sound is output from a speaker. good. In addition, when there are several or more visitors, processing for switching content or switching content according to gender / age may be executed.

  INDUSTRIAL APPLICABILITY The present invention is applicable to industries that display images on a display using a computer and digital signage that displays advertisements as video.

DESCRIPTION OF SYMBOLS 1 Image display system 2, 2 'Image processing apparatus 20 Background removal means 21 Face detection means 22 Human body detection means 23, 23' Tracking means 24 Movie analysis means 25 State transition management means 26 Person attribute estimation means 27 Log file output means 3 Display 4 video camera 6 state transition table 80 composite target definition means 81 composite content definition means 82 animation scenario definition means 83 scenario data association means 84 composite image creation means 85 instruction determination means 86 command switching means

Claims (7)

An image display system comprising a camera for photographing a person, an image processing device for synthesizing a captured video sent from the camera, and a display for displaying the synthesized video that has been synthesized,
The image processing apparatus includes:
A scenario in which scenario data including setting items indicating whether or not to associate with a new person is determined when the timing of combining with content is determined for each of two or more persons on the video and the associated person disappears Data storage means;
Content storage means for storing content used for composition;
Face detection means for detecting a face image captured in one frame of a video transmitted from the camera and outputting the position / rectangular size of the face detection frame as face detection frame data for each detected face image; ,
Tracking means for associating the face detection frame data acquired from the face detection means with face detection frame data of another frame;
When the face object including the face detection frame data detected by the face detection unit is associated with a person included in the scenario data, and the setting item is set to be associated with a new person And a scenario data associating means for associating with a new person after the person associated with the face object disappears ,
According to the association, the face object is assigned to the person of the scenario data, the position and size of the content is changed according to the position and size of the face detection frame data of the face object, and the content is placed on the frame. A composite image generating means for generating a composite image;
An image display system comprising: The scenario data has a plurality of commands,
For each frame, a person is extracted from the frame, a part of the face detection frame data is removed to identify a characteristic part, and an area ratio between the characteristic part and a preset region is a predetermined threshold value When the larger state continues for a predetermined number of frames, it further includes instruction determination means for determining that there is an instruction from the viewer,
The synthetic image generation means switches the face object in the scenario data in accordance with the association by the scenario data association means after switching the command when an instruction is given by the instruction determination means. 2. The image display system according to claim 1, wherein the image display system is assigned to a person of a later command.   The instruction determination unit extracts the person by extracting an overlap portion between a background-removed image obtained by removing a background from the frame and an HSV image obtained by performing HSV conversion on the frame. The image display system according to claim 2. For each face detection frame data acquired from the face detection means, the tracking means detects the face detected in the face detection frame data acquired from the frame processed last from the face detection means and the currently processed frame. An evaluation value indicating a position / rectangular size difference from the detection frame data is calculated, and the previous face detection frame data having the smallest evaluation value is associated with each face detection frame data of this time,
Generating a face object for each face image newly detected by the face detection means;
A state transition management unit that refers to the result of association between the previous and current face detection frame data obtained from the tracking unit and transitions the state of the face object to a browsing state according to a predetermined state transition table; Have
The scenario data association unit acquires a face object in a browsing state or a standby state by the state transition management unit, and performs association with a person included in the scenario data. The image display system according to any one of claims 1 to 3.   The image display system according to claim 4, wherein the composite image generation unit assigns the scenario data to a person in order from a face object for which the state transition management unit has detected a browsing state or a standby state. The face detection means sets the output face detection frame data as a face object,
The tracking means performs tracking of the face detection frame data in the next frame for each of the face detection frame data acquired from the face detection means, and sets the same face object.
4. The scenario data associating means acquires the face object set by the face detecting means and associates with a person included in the scenario data. The image display system according to any one of the above. The scenario data defines a correspondence relationship between a person and content based on an elapsed time from the start of scenario reproduction, and a correspondence relationship between the content and the location of the content in the content storage unit, and the composite image generation unit 7. The image display system according to claim 1, wherein content corresponding to the person is extracted from the content storage unit and synthesized according to the elapsed time.

Images