JP2019121105A - Control device and control program - Google Patents

Abstract

To measure the reaction of a person who is in the direction to which a user has paid attention.SOLUTION: A reaction measurement system 1 acquires an attention direction to which a user has paid attention from an image captured by a visible camera 4, and measures the reaction of an audience 3 positioned in the acquired attention direction to the user using an infrared camera 6.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a control device and a control program.

  In Patent Document 1, an imaging unit provided on a ceiling part of a predetermined space for photographing the space, a temperature sensor for detecting a temperature at a predetermined position, a drive mechanism for turning the temperature sensor in a predetermined direction, and the imaging A main control unit that controls the drive mechanism and a communication unit that performs predetermined communication based on the control of the main control unit, while receiving image data acquired by the unit and a detected temperature detected by the temperature sensor There is disclosed a fall detection system characterized by comprising

  Patent Document 2 detects a second sensor for detecting an amount of state of an event generated on the side of a living body or an event generated on the living body itself, and detects a state amount of a causal event causing an abnormal state of the event. The sensor by the second sensor is a detection target, triggered by the fact that the state amount of the event detected by the first sensor and the first sensor satisfies a predetermined condition for considering the occurrence of the abnormal state as a trigger. And a detection start means for starting detection of a state quantity of an event to be detected.

Patent No. 5207040 Patent No. 4575133 gazette

  2. Description of the Related Art Conventionally, development of a method for measuring an inner state of a person such as an emotion has been advanced using a measuring device such as a camera for capturing an image and a sensor for detecting motion.

  However, in the conventional measurement method, the number and direction of the measuring devices are adjusted in advance, in order to measure the inner surface state for all the people present in a predetermined range, for example, a room or a hall. Therefore, in the conventional measurement method, the direction of the measuring device is adjusted during the measurement of the inner surface state to measure the inner surface state of only the person in a specific direction in the room or the hall or to select the measurement target person of the inner surface state. It is difficult to do.

  An object of the present invention is to provide a control device and a control program capable of measuring a reaction of a person who is in a direction in which a user pays attention.

  In order to achieve the above object, the invention of the control device according to claim 1 is an acquisition unit for acquiring a direction of interest focused by the user, and the use of the measurement target person located in the direction of interest acquired by the acquisition unit. Control means for controlling the measuring device for measuring the reaction to the subject.

  The invention according to claim 2 is that the user is a presenter who speaks to the audience, the acquisition means acquires the direction in which the presenter talks as the attention direction of the user, and the control means is Controlling the measuring device to measure a reaction of the audience located in the direction in which the presenter acquired by the acquisition means speaks to the speaker's remark.

  In the invention according to claim 3, the acquisition means acquires the direction in which the user is facing as the attention direction of the user.

  In the invention according to claim 4, the control means controls the measuring device such that the person to be measured whose reaction is to be measured by the measuring device changes in accordance with a change in the direction in which the user faces.

  In the invention according to claim 5, the acquisition means acquires the attention direction of the user from a statistic related to the direction in which the user is facing in a predetermined period, and the control means is configured to The measuring device is controlled to measure the reaction of the person to be measured located in the direction of interest to the user.

  In the invention according to claim 6, the acquisition means acquires, as the user's attention direction, the direction in which the cumulative time that the user has faced is the longest in the predetermined period.

  In the invention according to claim 7, the acquisition means acquires, as the attention direction of the user, the direction in which the cumulative number of times the user has faced is the largest in the predetermined period.

  In the invention according to claim 8, the acquisition means acquires an average of directions in which the user has faced as the attention direction of the user in the predetermined period.

  In the invention according to claim 9, the acquisition means acquires the direction in which the face or body of the user is facing as the direction in which the user is facing.

  In the invention according to claim 10, the acquisition means acquires the propagation direction of the voice emitted by the user collected by a microphone as the direction in which the user talks.

  In the invention according to claim 11, a plurality of the microphones are provided, and the acquisition means acquires the propagation direction of the voice emitted by the user from the intensity distribution of the voice collected by each of the microphones.

  In the invention according to claim 12, the control means causes the reaction information indicating the reaction of the person to be measured to the utterance of the user to be displayed on a display device installed at a position viewed by the user. Control the display device.

  In the invention according to claim 13, the reaction information is information indicating a degree of interest of the person to be measured for the utterance of the user.

  In the invention according to claim 14, the control means controls the display device such that the degree of interest of the person to be measured with respect to the utterance of the user is displayed in association with the position of the person to be measured. .

  The invention of a control program according to claim 15 causes a computer to function as each means of the control device according to any one of claims 1 to 14.

  According to the inventions of claims 1 and 15, it is possible to measure the reaction of the person who is in the direction in which the user pays attention.

  According to the second aspect of the present invention, it is possible to measure the reaction of only the audience in the direction in which the user speaks from among the entire audience.

  According to the third aspect of the present invention, it is possible to obtain the user's attention direction from the image obtained by photographing the user.

  According to the fourth aspect of the present invention, it is possible to select the person to be measured according to the direction of the user at each time point.

  According to the fifth aspect of the present invention, even when the user unintentionally turns to a direction different from the direction of attention, comparison with the case where the person to be measured is selected according to the direction of the user at each time point Thus, the change of the selection range of the person to be measured can be suppressed.

  According to the sixth aspect of the present invention, it is possible to obtain the user's attention direction from the accumulated time in each direction in which the user is facing.

  According to the seventh aspect of the present invention, it is possible to acquire the user's attention direction from the cumulative number of times in each direction to which the user has faced.

  According to the eighth aspect of the invention, even when the cumulative number or the cumulative time in which the user is facing in the plurality of directions is the same, it is possible to acquire the attention direction of the user. Have.

  According to the ninth aspect of the present invention, even when the direction of the user's face and body is different, it is possible to obtain the direction in which the user is facing.

  According to the tenth aspect of the present invention, it is possible to obtain the direction in which the user talks from the collected voice.

  According to the invention as set forth in claim 11, it is possible to obtain the direction in which the user speaks with high accuracy, as compared with the case where the direction in which the user speaks is acquired using one microphone. .

  According to the twelfth aspect of the invention, the user can select the attention direction while confirming the reaction information of the person to be measured.

  According to the invention as set forth in claim 13, the user has the effect of being able to speak while grasping the remarks of interest of the person to be measured.

  According to the invention of claim 14, the user has the effect of being able to speak while grasping where and what person being measured is interested in what utterance.

It is a figure which shows the structural example of the reaction measurement system which concerns on 1st Embodiment. It is a figure showing the example of principal part composition of the electric system concerning the control device concerning a 1st embodiment. It is a flow chart which shows an example of a flow of control processing concerning a 1st embodiment. It is a flow chart which shows the modification of the flow of control processing concerning a 1st embodiment. It is a figure which shows the structural example of the reaction measurement system which concerns on 2nd Embodiment. It is a figure which shows the example of a principal part structure of the electric system which concerns on the control apparatus which concerns on 2nd Embodiment. It is a flow chart which shows an example of the flow of control processing concerning a 2nd embodiment. It is a flow chart which shows the modification of the flow of control processing concerning a 2nd embodiment.

  Hereinafter, the present embodiment will be described with reference to the drawings. In addition, the same code | symbol is provided to the component and process in which the function is the same throughout all drawings, and the overlapping description is abbreviate | omitted.

First Embodiment
FIG. 1 is a block diagram showing an example of a reaction measurement system 1 according to the present embodiment. The reaction measurement system 1 is a system that measures an inner state such as an emotion of a person (hereinafter, referred to as a “measurement target person”) located in a direction of interest where a user who uses the reaction measurement system 1 pays attention.

  For example, as shown in FIG. 1, when the presenter 2 gives a presentation to the audience 3, the presenter 2 uses the reaction measurement system 1 to make the audience 3 located in the attention direction the presenter 2 pays attention to Reaction is obtained. That is, the presenter 2 is an example of a user who uses the reaction measurement system 1, and the audience 3 is an example of a person to be measured whose response is to be measured.

  The reaction measurement system 1 includes, for example, a visible camera 4, a control device 5, an infrared camera 6, and a display device 7.

  The visible camera 4 is a camera for capturing an image of an object represented by light visible to human eyes, and is attached at a position for capturing the presenter 2. In this case, the image captured by the visible camera 4 may include the audience 3 but at least the presenter 2 is included. There is no restriction on the installation method of the visible camera 4, and if an image including the presenter 2 can be obtained, the photographer who takes a photograph of the presenter 2 takes a hand even though it is fixed at a predetermined position by a mounting bracket or the like. It may be taken with you. An image captured by the visible camera 4 is transmitted to the control device 5.

  Although the presenter 2 may be photographed by a plurality of visible cameras 4, the cost of the reaction measurement system 1 increases as the number of visible cameras 4 increases. It is preferable to take a picture.

  The control device 5 receives an image captured by the visible camera 4 and acquires an attention direction in which the presenter 2 is focusing from the received image. Then, since the control device 5 measures the reaction of the audience 3 located in the acquired attention direction of the presenter 2, the imaging direction of the infrared camera 6 so that the imaging direction of the infrared camera 6 approaches the attention direction of the presenter 2 Control.

  The infrared camera 6 is attached to a predetermined place such as a wall, a ceiling, a floor or a stand so that the imaging direction can be changed according to the control of the control device 5, and the infrared camera 6 is positioned in the direction instructed by the control device 5 The infrared image of the audience 3 to be taken.

  From the infrared image, the temperature of the part of the audience 3 not covered with clothes or the like such as the face can be obtained. Therefore, using the known method of measuring the inner surface state of a person from the temperature of the face as shown in Non-Patent Document 1, the control device 5 uses the infrared image taken by the infrared camera 6 to present the presenter 2 Get the reaction of the audience 3 located in the attention direction that you noted.

[Non-patent document 1]: Zhilei Liu, Shangfei Wang. "Emotion Recognition Using Hidden Markov Models from Facial Temperature Sequence" Affective Computing and Intelligent Interaction (ACII), Lecture Notes in Computer Science (LNCS), vol 6975, pp. 240-247 , 2011

  In the reaction measurement system 1, the infrared camera 6 is used as an example of a measurement device that measures the reaction of the audience 3. However, any device that measures the reaction of the audience 3 may be used instead of the infrared camera 6. An apparatus may be used.

  It is known that the inner state of a person also appears in, for example, facial expressions, gestures, and postures. Therefore, in place of the infrared camera 6, a visible camera for capturing the facial expression of the audience 3, and a three-dimensional sensor for detecting human movement using infrared rays such as Kinect (registered trademark) The reaction may be measured. Alternatively, the reaction of the audience 3 may be measured by combining the infrared camera 6 and at least one of a visible camera and a three-dimensional sensor.

  Although the audience 3 may be photographed by a plurality of infrared cameras 6, since the cost of the reaction measurement system 1 increases as the number of infrared cameras 6 increases, the audience 3 is photographed by one infrared camera 6 Is preferred.

  The display unit 7 is connected to the control unit 5 and displays reaction information indicating the reaction of the audience 3 measured by the control unit 5. By arranging the display device 7 within the range of the view of the presenter 2, the presenter 2 who is giving a presentation will be displayed the reaction of the audience 3 located in the direction in which the presenter 2 focused.

  The control device 5 of the reaction measurement system 1 described above is configured using, for example, a computer. FIG. 2 is a diagram showing an example of a main configuration of an electric system in the control device 5 configured using the computer 10. As shown in FIG.

  The computer 10 has a central processing unit (CPU) 11, a read only memory (ROM) 12, a random access memory (RAM) 13, a non-volatile memory 14, and an input / output function as acquisition means and control means according to the present embodiment. An interface (I / O) 15 is provided. The CPU 11, the ROM 12, the RAM 13, the non-volatile memory 14, and the I / O 15 are connected to one another via the bus 16. The operation system used by the computer 10 is not limited.

  The non-volatile memory 14 is an example of a storage device in which the stored information is maintained even if the power supplied to the non-volatile memory 14 is cut off. For example, a semiconductor memory may be used but it may be a hard disk.

  For example, the visible camera 4, the infrared camera 6, the display device 7, and the communication device 8 are connected to the I / O 15.

  The communication device 8 includes, for example, a communication protocol for connecting a communication line such as the Internet and the control device 5, and performs data communication between the control device 5 and another external device connected to the communication line.

  In addition to the device shown in FIG. 2, for example, another device such as an input unit for receiving an instruction from the user and notifying the CPU 11 may be connected to the I / O 15. The input unit includes, for example, a button, a touch panel, a keyboard, and a mouse. In addition, the visible camera 4, the infrared camera 6, the display device 7, and the communication device 8 do not necessarily have to be connected to the I / O 15. For example, the display device 7 and the communication device 8 may be connected as needed.

  Next, the operation of the control device 5 in the reaction measurement system 1 will be described with reference to FIG.

  FIG. 3 is a flowchart showing an example of the flow of control processing executed by the CPU 11 after the control device 5 is activated or when an instruction to start operation is received from the user via an input unit (not shown).

  A control program defining the control process is stored in advance in, for example, the ROM 12 of the control device 5. The CPU 11 of the control device 5 reads a control program stored in the ROM 12 and executes control processing.

  First, in step S <b> 10, the CPU 11 determines whether an image obtained by photographing the presenter 2 has been received from the visible camera 4. When the image is not received, the process of step S10 is repeatedly executed to monitor the reception status of the image. On the other hand, when an image is received from the visible camera 4, the process proceeds to step S20.

  In step S20, the CPU 11 acquires the direction angle in the direction in which the presenter 2 is facing from the image received in step S10, and stores the acquired direction angle in the RAM 13. Specifically, the CPU 11 extracts the face of the presenter 2 from the image using, for example, face pattern recognition or a known image recognition method using a face feature vector, and the direction of the extracted face is determined in advance. The angle (orientation angle) that is made with the selected direction (reference direction) is acquired as the direction in which the presenter 2 is facing, and stored in the RAM 13.

  When the presenter 2 speaks to the audience 3 who is in a specific position, the presenter 2 tends to face the face of the audience 3 and thus the extracted face direction is opposite to that of the presenter 2 It can be understood as the direction in which

  The direction in which the presenter 2 is facing does not necessarily have to be determined by the orientation of the face of the presenter 2, for example, the orientation of the body of the presenter 2, the orientation of the presenter's neck, or the line of sight of the presenter 2. It may be determined by the direction. Here, as an example, the orientation of the face of the presenter 2 is described as the direction in which the presenter 2 is facing.

  In step S30, for example, the CPU 11 is in the respective directions to which the presenter 2 is opposite, acquired while tracing back in the past by a predetermined period (hereinafter referred to as "determination period") from the execution start time of step S30. Then, statistical processing is performed, and the attention direction of the presenter 2 is acquired from the obtained statistic.

  In the judgment period, the statistics of the attention direction of the presenter 2 acquired by the image obtained from the visible camera 4 in the corresponding period approach the actual attention direction of the presenter 2 by the actual device of the reaction measurement system 1 It is a period set in advance by computer simulation or the like based on experiments and the design specifications of the reaction measurement system 1. The length of the set determination period is stored, for example, in the non-volatile memory 14, and the CPU 11 acquires the determination period by referring to the non-volatile memory 14.

  Here, as an example, the start time of the determination period is set as the execution start time of step S30, but it may be the time when an image is received from the visible camera 4 in step S10, for example.

  As a statistic for each direction in which the presenter 2 is opposite, which is acquired in the determination period, for example, an average of the directions in which the presenter 2 is opposite is used. Specifically, the average value of the direction angles in the direction in which the presenter 2 is opposite is taken as the attention direction of the presenter 2.

  In addition, the attention direction of the presenter 2 may be acquired using not only the average value of the direction angle of the direction in which the presenter 2 is opposite but another statistic. For example, the CPU 11 may acquire, for each direction, the time in which the presenter 2 is facing, and may set the direction angle of the direction in which the accumulated time in which the presenter 2 is facing is the longest in the determination period. Further, the CPU 11 may acquire the number of times the presenter 2 is facing for each direction, and may set the direction angle of the direction in which the cumulative number of times the presenter 2 is facing is the largest in the determination period.

  Furthermore, in the determination period, the average value of the direction angles in the direction in which the presenter 2 made a relative direction, the direction angle in the direction in which the cumulative time that the presenter 2 made a pair is the longest, and The average value of at least two directional angles obtained by a plurality of statistical processes, such as corners, may be used as the direction of attention of the presenter 2 in the determination period.

  The user who uses the reaction measurement system 1 selects in advance which calculation method is used to obtain the attention direction of the presenter 2 from among a plurality of prepared calculation methods of the attention direction of the presenter 2 in the determination period select. The calculation method selected by the user is stored, for example, in the non-volatile memory 14. The CPU 11 refers to the calculation method stored in the non-volatile memory 14 and acquires the attention direction of the presenter 2 in the determination period.

  In addition, whenever the CPU 11 acquires the direction angle of the direction in which the presenter 2 is facing in step S20, the old direction angle not included in the determination period among the direction angles of the presenter 2 stored in the RAM 13 It is determined whether or not exists. If there is a direction angle older than the determination period, the CPU 11 erases the direction angle, then calculates the attention direction of the presenter 2 in the determination period so that the remaining capacity of the RAM 13 does not run short. In this case, if each direction angle stored in the RAM 13 is used, the direction of attention of the presenter 2 in the determination period is inevitably calculated, and the direction angle stored in the RAM 13 is within the determination period. The direction of attention of the presenter 2 in the determination period can be obtained without determining whether it is a direction angle or not one by one.

  In step S40, the CPU 11 controls the imaging direction of the infrared camera 6 so that the infrared camera 6 captures an image of the attention direction of the presenter 2 acquired in step S30.

  Specifically, when the infrared camera 6 is provided in advance with a function of changing attribute values related to the shooting direction such as pan angle, tilt angle, and zoom ratio, the CPU 11 changes at least one of these attribute values. A change instruction is sent to the infrared camera 6, and the imaging direction of the infrared camera 6 is controlled so as to capture an image of the presenter 2's attention direction. Also, even if the infrared camera 6 itself is not provided in advance with a function to change the attribute value related to the shooting direction, the infrared camera 6 is attached to a camera platform (not shown) having a pan angle and tilt angle changing function. If so, the CPU 11 transmits a change instruction to change at least one of these attribute values to a pan head (not shown), and sets the shooting direction of the infrared camera 6 so as to shoot an image in the attention direction of the presenter 2 Control.

  In step S50, the CPU 11 transmits an imaging instruction to the infrared camera 6 whose imaging direction is controlled in step S40, and captures an infrared image of the audience 3 located in the attention direction to which the presenter 2 focuses. The CPU 11 stores the captured infrared image of the audience 3 in the RAM 13.

  In step S60, the CPU 11 measures the reaction of the audience 3 to which the presenter 2 focused, from the temperature of the face of the audience 3 according to the contents of the above-mentioned "non-patent document 1", for example. The CPU 11 stores the measured reaction as, for example, the RAM 13 as reaction information.

  There is no restriction on the type of reaction of the audience 3 measured in step S60, and the control device 5 measures the degree of interest representing the degree of interest in the speech of the presenter 2 as an example. The degree of interest is measured by using the phenomenon that the temperature of the audience 3 rises as the degree of interest increases. Therefore, the body temperature is classified into a plurality of divisions in advance, and the CPU 11 associates the division to which the body temperature of the audience 3 belongs to each region corresponding to the audience 3 in the infrared image of the audience 3 photographed. Specifically, for example, classification is performed in the order of “A division”, “B division”,... From the higher temperature division, and the division corresponding to the temperature of the audience 3 for each area corresponding to the audience 3 Correspond

  The division associated with each area corresponding to the audience 3 is an example of reaction information indicating the degree of interest of the audience 3 photographed in the infrared image on the presenter 2.

  In the reaction measurement system 1, since the reaction of only the audience 3 located in the direction in which the presenter 2 focused is measured, the reaction of the audience 3 such as the infrared camera 6 is compared with the case where the reaction of the entire audience 3 is measured. The number of measuring devices for measuring

  In step S <b> 70, the CPU 11 controls the display device 7 so that the response information of the audience 3 to the presenter 2 is displayed on the display device 7. As an example, the CPU 11 is a screen in which the position of the audience 3 is associated with reaction information indicating the degree of interest in the audience 3 so that the presenter 2 can know where and how much the audience 3 has the degree of interest. Is displayed on the display unit 7. Specifically, the CPU 11 causes the display device 7 to display a screen in which a floor plan in which the position of the audience 3 is known, such as a seating chart, is color-coded according to the degree of interest of the audience 3.

  Therefore, by looking at the screen displayed on the display device 7, the presenter 2 can proceed with the presentation while confirming the reaction of the noted audience 3. That is, if the reaction of the audience 3 is not good, the topic may be changed to a story in which the reaction of the audience 3 is increased, which is useful for improving the quality of the presentation.

  In the reaction measurement system 1, the display device 7 is used as a notification means for notifying the presenter 2 of the reaction of the audience 3. However, the presenter 2 may be notified using a device other than the display device 7. For example, the reaction of the audience 3 to the presenter 2 may be notified by voice through an ear monitor worn on the presenter 2's ear.

  In step S80, the CPU 11 determines whether an instruction to end the control processing has been received via the input unit. If the end instruction has not been received, the process proceeds to step S10, and reception of a new image from the visible camera 4 is monitored. That is, by repeatedly executing the processing of steps S10 to S80 until the end instruction is received, the reaction of the audience 3 located in the direction in which the presenter 2 focused is measured. On the other hand, when the end instruction is received, the control process shown in FIG. 3 is ended.

  As described above, according to the reaction measurement system 1 according to the present embodiment, the direction in which the presenter 2 is facing is acquired from the image of the presenter 2 and the reaction of the audience 3 located in the acquired direction is measured. Do.

  In the control process shown in FIG. 3, although the direction obtained by performing statistical processing on each of the opposite directions of the presenter 2 acquired in the determination period is regarded as the attention direction of the presenter 2, the presentation is made at each time point The direction in which the person 2 is facing may be taken as the attention direction of the presenter 2.

  FIG. 4 is a flowchart showing an example of the flow of control processing executed by the CPU 11 in this case.

  The flowchart shown in FIG. 4 differs from the flowchart shown in FIG. 3 in that step S30 is deleted, and the other processes are the same as in FIG.

  In step S20 of the flowchart shown in FIG. 4, when the CPU 11 acquires the direction angle of the direction in which the presenter 2 is facing from the image received in step S10, the CPU 11 executes step S40.

  In step S40, the CPU 11 controls the imaging direction of the infrared camera 6 so as to capture the image in the direction in which the presenter 2 acquired in step S20 is facing. That is, the CPU 11 acquires and obtains the attention direction of the presenter 2 using only the direction of the presenter 2 obtained from the image received immediately before in step S10 without considering the past direction of the presenter 2 The shooting direction of the infrared camera 6 is adjusted to the attention direction.

  As a result, the shooting direction of the infrared camera 6 follows in real time the current direction in which the presenter 2 is facing.

  Therefore, by executing the control process of FIG. 4, the audience 3 to be measured for the reaction changes in accordance with the change in the direction in which the presenter 2 is opposite.

Second Embodiment
In the reaction measurement system 1 according to the first embodiment described above, the direction in which the presenter 2 faces is acquired from the image obtained by capturing the presenter 2. However, the method of acquiring the direction in which the presenter 2 is opposite is not limited to this.

  In the second embodiment, a reaction measurement system 1A will be described which acquires the direction in which the presenter 2 is opposite from the propagation direction of the sound emitted by the presenter 2.

  FIG. 5 is a block diagram showing an example of a reaction measurement system 1A according to the present embodiment. As shown in FIG. 5, the reaction measurement system 1A includes, for example, a microphone 9, a control device 5A, an infrared camera 6, and a display device 7.

  In the reaction measurement system 1A, the microphone 9 is used instead of the visible camera 4 of the reaction measurement system 1 shown in FIG. 1, and the control device 5 is replaced with the control device 5A.

  The microphone 9 is a device for collecting the voice of the presenter 2. For example, the microphone 9 is fixed at a predetermined position by a mounting bracket or the like, but may be a microphone 9 attached to clothes of the presenter 2 like a pin microphone. .

  Further, the number of microphones 9 included in the reaction measurement system 1A is not limited, and may be one or more. In the example shown in FIG. 5, the voices of the presenter 2 are collected by two microphones 9.

  The microphone 9 may collect the voices of the audience 3, but it is preferable to use a directional microphone rather than an omnidirectional microphone so that only the voice of the presenter 2 is collected.

  The control device 5A receives the voice collected by each of the microphones 9, specifies the propagation direction of the received voice, and acquires the propagation direction of the voice as the direction in which the presenter 2 speaks. Then, the control device 5A recognizes the direction in which the presenter 2 speaks as the attention direction of the presenter 2 and measures the reaction of the audience 3 located in the attention direction of the presenter 2; The imaging direction of the infrared camera 6 is controlled so as to approach the attention direction of the person 2.

  The control device 5A of the reaction measurement system 1A described above is configured using, for example, a computer. FIG. 6 is a diagram showing an example of a main configuration of an electric system in the control device 5A configured using the computer 10.

  The configuration example shown in FIG. 6 differs from the configuration example of the control device 5 shown in FIG. 2 in that the microphone 9 is connected to the I / O 15 instead of the visible camera 4, and the other configuration is the control device 5. Is the same as Similar to the control device 5, a device different from the device shown in FIG. 6, such as an input unit, may be connected to the I / O 15. Further, the microphone 9, the infrared camera 6, the display device 7, and the communication device 8 do not necessarily need to be connected to the I / O 15. For example, the display device 7 and the communication device 8 may be connected as needed.

  Next, the operation of the control device 5A in the reaction measurement system 1A will be described using FIG.

  FIG. 7 is a flowchart showing an example of the flow of control processing executed by the CPU 11 when the control device 5A is activated or when an instruction to start operation is received from the user via an input unit (not shown).

  A control program defining the control process is stored in advance in, for example, the ROM 12 of the control device 5A. The CPU 11 of the control device 5A reads a control program stored in the ROM 12 and executes control processing.

  The flowchart shown in FIG. 7 differs from the flowchart of the control process in control device 5 shown in FIG. 4 in that steps S10 and S20 are replaced with steps S10A and S20A, respectively, and the other processes are shown in FIG. It is the same as the flowchart shown. Therefore, the processes of steps S10A and S20A will be described below.

  In step S10A, the CPU 11 determines whether voice has been received from each of the microphones 9 or not. When the voice is not received, the process of step S10A is repeatedly executed to monitor the voice reception state. On the other hand, when the voice is received from each of the microphones 9, the process proceeds to step S20A.

  Since the microphone 9 is set to collect the voice of the presenter 2, the received voice is considered to be the voice emitted by the presenter 2.

  In step S20A, the CPU 11 acquires the propagation direction of the voice emitted by the presenter 2 using the intensity distribution of the voice emitted by the presenter 2 received from each of the microphones 9 in step S10A.

  Here, “intensity distribution of sound” indicates the size of sound received by the microphone 9 every direction from the position of the microphone 9 toward the surface of a sphere virtually set with the position of the microphone 9 as a center. It is a distribution, for example, obtained from received audio data. That is, the intensity distribution of voice represents the magnitude of voice by the distance from the center of the sphere virtually set, and a line (center line) from the center of the sphere virtually set the direction of voice toward the surface of the sphere In the direction of

  The CPU 11 identifies the center line in the direction in which the magnitude of the sound is largest for each intensity distribution of the sound obtained from each of the microphones 9, and announces the direction from the position of the presenter 2 toward the intersection of each center line It is acquired as the propagation direction of the voice emitted by person 2. The position information of the presenter 2 may be acquired from, for example, a GPS (Global Positioning System) unit possessed by the presenter 2. Further, in a situation where it can be considered that the position of the presenter 2 does not change, the position information of the presenter 2 may be stored in advance in the non-volatile memory 14.

  When the number of the microphones 9 is one, the direction in which the magnitude of the sound is the largest from the position of the microphone 9 may be the propagation direction of the sound emitted by the presenter 2.

  The propagation direction of the voice emitted by the presenter 2 is considered to be the opposite direction of the presenter 2.

  After this, the CPU 11 executes the processing after step S30 already described, aligns the shooting direction of the infrared camera 6 with the direction in which the presenter 2 focused in the determination period, and the audience located in the direction where the presenter 2 focuses The reaction of 3 is measured, and the measurement result is displayed on the display unit 7.

  As described above, according to the reaction measurement system 1A according to the present embodiment, the direction in which the presenter 2 is facing is acquired from the voice emitted by the presenter 2, and the reaction of the audience 3 located in the acquired direction is measured. Do.

  In the control process shown in FIG. 7, although the direction obtained by statistically processing each of the facing directions of the presenter 2 acquired in the judgment period is set as the attention direction of the presenter 2, the presentation is made at each time point The direction in which the person 2 is facing may be treated as the attention direction of the presenter 2.

  FIG. 8 is a flowchart showing an example of the flow of control processing executed by the CPU 11 in this case.

  The flowchart shown in FIG. 8 differs from the flowchart shown in FIG. 7 in that step S30 is deleted, and the other processes are the same as in FIG.

  In step S20A of the flowchart shown in FIG. 8, when the CPU 11 acquires the direction angle of the direction in which the presenter 2 is facing from the voice received in step S10A, the CPU 11 executes step S40.

  In step S40, the CPU 11 controls the imaging direction of the infrared camera 6 so as to capture the image in the direction in which the presenter 2 acquired in step S20A is facing. That is, the CPU 11 acquires and obtains the attention direction of the presenter 2 using only the direction of the presenter 2 obtained from the voice received immediately before in step S10A without considering the past direction of the presenter 2 The shooting direction of the infrared camera 6 is adjusted to the attention direction.

  As a result, the shooting direction of the infrared camera 6 follows in real time the current direction in which the presenter 2 is facing.

  Therefore, by executing the control process of FIG. 8, the audience 3 to be measured for the reaction changes in accordance with the change in the direction in which the presenter 2 is opposite.

  In the reaction measurement system 1A described above, the audience 3 who is the subject of the reaction is selected from the direction in which the presenter 2 speaks, but the method of selecting the audience 3 who is the subject of the reaction is not limited thereto. For example, the microphone 9 for collecting the voice of the audience 3 is separately provided, the position of the audience 3 who has issued the voice for asking a question etc. from the intensity distribution of the voice acquired by the microphone 9 is identified, and the direction of the audience 3 identified The reaction direction of the audience 3 may be measured in accordance with the imaging direction of the infrared camera 6.

  As mentioned above, although this invention was demonstrated using each embodiment, this invention is not limited to the range as described in each embodiment. Various changes or improvements can be added to each embodiment without departing from the scope of the present invention, and a form to which the changes or improvements are added is also included in the technical scope of the present invention. For example, the order of processing may be changed without departing from the scope of the present invention.

  Moreover, in each embodiment, although reaction measurement system 1 and 1A were used for the measurement of the reaction of the audience 3 in a presentation, the application scope of reaction measurement system 1 and 1A is not restricted to a presentation.

  For example, when inspecting a factory or a company, the direction in which the inspection member paid attention may be acquired from the image of the visible camera 4 and the reaction of a person located in the direction in which the inspection member paid attention may be measured. In addition, in the school, the reaction of the students located in the direction in which the teacher pays attention may be measured and used to grasp the understanding level of the class content. Thus, the reaction measurement system 1, 1A is applied at a scene where it is desired to measure the reaction of a person who is in the direction in which the user pays attention.

  In each embodiment, an embodiment in which the control processing is realized by software has been described as an example, but processing equivalent to the flowcharts shown in FIG. 3, FIG. 4, FIG. 7, and FIG. It may be implemented in a specific integrated circuit and processed by hardware. In this case, control processing can be speeded up.

  Moreover, although the form by which the control program was installed in ROM12 was demonstrated by each embodiment mentioned above, it is not limited to this. The control program according to the present invention can also be provided in the form of being recorded on a computer readable storage medium. For example, the control program according to the present invention may be provided in the form of being recorded on an optical disc such as a CD (Compact Disc) -ROM or a DVD (Digital Versatile Disc) ROM. Furthermore, the control program according to the present invention may be provided in the form of being recorded in a semiconductor memory such as a USB memory and a flash memory. Furthermore, the control devices 5 and 5A may obtain the control program according to the present invention from an external device connected to a communication line via a communication line such as the Internet.

1 (1A) ... reaction measurement system 2 ... presenter 3 ... audience 4 ... visible camera 5 (5A) ... control device 6 ... infrared camera 7 ... display device 8 ... · · Communication device 9 · · · microphone 10 · · · computer 11 · · · CPU
12 ・ ・ ・ ROM
13 ・ ・ ・ RAM
14 ・ ・ ・ nonvolatile memory

Claims (15)

Acquisition means for acquiring the attention direction the user has noticed;
A control unit that controls a measurement device that measures a reaction of the person to be measured located in the direction of interest acquired by the acquisition unit with the user;
Control device equipped with The user is a presenter who speaks to the audience,
The acquisition means acquires the direction in which the presenter speaks as the attention direction of the user,
The control device according to claim 1, wherein the control means controls the measurement device to measure a reaction of the audience located in the direction in which the presenter acquired by the acquisition means speaks to the speaker's remark. The control device according to claim 2, wherein the acquisition unit acquires a direction in which the user is facing as the attention direction of the user. The control device according to claim 3, wherein the control means controls the measurement device such that the person to be measured whose response is measured by the measurement device changes in accordance with a change in the direction in which the user faces. The acquisition means acquires the attention direction of the user from statistics regarding the direction in which the user is facing in a predetermined period,
The control device according to claim 3, wherein the control means controls the measurement device to measure a reaction of the measurement target person located in the attention direction of the user to the user. The control device according to claim 5, wherein the acquiring unit acquires a direction in which the accumulated time in which the user is opposed is the longest in the predetermined period as the attention direction of the user. The control device according to claim 5, wherein the acquisition unit acquires a direction in which the number of times the user has made a cumulative count with respect to each other in the predetermined period is the largest as the attention direction of the user. The control device according to claim 5, wherein the acquisition means acquires an average of directions in which the user has made a relative direction as the attention direction of the user in the predetermined period. The control device according to any one of claims 3 to 8, wherein the acquisition means acquires a direction in which the user's face or body is facing as a direction in which the user is facing. The control device according to claim 2, wherein the acquisition means acquires a propagation direction of the voice emitted by the user collected by a microphone as a direction in which the user speaks. A plurality of the microphones are provided,
The control device according to claim 10, wherein the acquisition means acquires the propagation direction of the voice emitted by the user from the intensity distribution of the voice collected by each of the microphones. The control unit controls the display device such that reaction information indicating a reaction of the person to be measured to the utterance of the user is displayed on a display device installed at a position viewed by the user. The control device according to any one of the above. The control device according to claim 12, wherein the response information is information indicating a degree of interest of the person to be measured with respect to a statement of the user. The control device according to claim 13, wherein the control means controls the display device such that the degree of interest of the person to be measured with respect to the utterance of the user is displayed in association with the position of the person to be measured.   The control program for functioning a computer as each means of the control apparatus in any one of Claims 1-14.