JP2017228216A - Information processing apparatus, control method therefor, program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operability of an operation that is inputted by using multiple input positions, in a system for recognizing input on the basis of a proximity state between an object and a touch target surface.SOLUTION: An information processing apparatus comprises: a three-dimensional information acquisition part 210 for acquiring information corresponding to a distance from a predetermined operation surface regarding an input position that is inputted in a space above the predetermined operation surface; a recognition part 213 for recognizing a first operation inputted by using one input position of which the distance from the predetermined operation surface is less than a first threshold, and a second operation inputted by using multiple input positions of which the distances to the predetermined operation surface are less than the first threshold value, on the basis of the acquired information; and a display control part 125 configured to control output for responding to the recognized operation. The recognition part 213 does not respond to the first operation while there are multiple input positions of which the distances from the predetermined operation surface are smaller than a second threshold value that is greater than the first threshold value and there is one input position of which the distance from the predetermined operation surface is smaller than the first threshold.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for recognizing operations input by a plurality of linked positions.

  In a system that recognizes a touch operation, a multi-touch operation that inputs a predetermined instruction by touching and interlocking a plurality of positions on an operation surface with an object such as a fingertip or a pen tip is widely used. Even in a situation where the user intends to perform a multi-touch operation, an ideal situation where a plurality of objects can touch the operation surface at the same time without any time error is rare. In actual multi-touch input, there is a slight time difference between the timings at which a plurality of objects touch the operation surface.

  Therefore, in Patent Literature 1, when the distance between the two touch positions or the difference in touch start time is small enough to satisfy the condition, the input of the two touch positions is not intended to be two operation inputs, but one multi It is determined that the input of the touch operation is intended.

JP 2013-25621 A

  In a system in which an image or UI (user interface) is projected on an arbitrary table surface or wall surface to be used as an operation surface, and the image or UI is operated by touching the image or UI with a finger, pen, etc., various cameras and sensors are used. A method for detecting the proximity state to the operation surface is used. For example, a distance between a tip of a predetermined object such as a hand or a pen and an operation surface may be acquired as a proximity state, and a state where the distance is smaller than a predetermined value may be recognized as a touch input. In a system in which the distance between the tip of the object and the operation surface is smaller than a predetermined value is regarded as a touch input, it is substantially before the object touches the operation surface and while the object exists in the air. Touch input is recognized. Therefore, even if the user arranges the timing at which each object is physically brought into contact with the operation surface, if the position of the object is in the air, there is a difference in the timing at which the touch input is detected. There is also sex. Further, in such a system, an error may occur in the result of distance measurement between the tip of the object and the operation surface, affected by the accuracy of the sensor and the positional relationship between the objects. Therefore, there is a possibility that a difference occurs in the timing at which it is detected that the distance between the tip and the operation surface is smaller than a predetermined value depending on the object. As a result of the time difference thus generated, it may take a long time to recognize that only one of the two objects is being touch-input, and a single touch operation may be recognized during that time.

  In Patent Document 1, when it is detected as a touch input that the proximity state between the object and the operation surface satisfies a predetermined condition, sufficient consideration is given to specifying a plurality of input positions intended for one multi-touch operation input. I didn't.

  The present invention has been made in view of the above, and improves the operability of an operation input using a plurality of input positions in a system that recognizes an input based on a proximity state of an object and a touch target surface. With the goal.

  In order to solve the above problems, an information processing apparatus according to the present invention includes an acquisition unit that acquires information corresponding to a distance from the predetermined operation surface with respect to an input position input to a space on the predetermined operation surface; Based on the information acquired by the acquisition means, the first operation input using one input position whose distance from the predetermined operation surface is less than the first threshold, and the distance to the predetermined operation surface is Recognizing means for recognizing a second operation input using a plurality of input positions below a first threshold, and output control means for controlling an output for responding to the operation recognized by the recognizing means, The output control means includes a plurality of input positions whose distance from the predetermined operation surface is less than a second threshold value that is greater than the first threshold value, and the distance from the predetermined operation surface satisfies the first threshold value. One input position below During, characterized in that it does not respond to the first operation.

  According to the present invention, the operability of an operation input using a plurality of input positions in a system that recognizes an input based on the proximity state of the object and the touch target surface is improved.

The figure showing an example of the table top interface system using information processor 100 The figure showing an example of hardware constitutions and functional composition of information processor 100 The flowchart which shows an example of the flow of the main process which the information processing apparatus 100 performs The flowchart which shows an example of the flow of the process which specifies the input position effective for multi-touch operation. The flowchart which shows an example of the flow of the process which determines the touch state of an input position The flowchart which shows an example of the flow of the process which recognizes the touch operation by an input position. The figure showing the example of a user's hand state, the input position corresponding to it, and the example of various threshold values The figure showing the example of a user's hand state, the input position corresponding to it, and the example of various threshold values The flowchart which shows an example of the flow of the main process which the information processing apparatus 100 performs The flowchart which shows an example of the flow of the main process which the information processing apparatus 100 performs The figure which represents notionally two threshold values set in a modification

  Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. Note that the configurations described in the embodiments are examples, and the scope of the present invention is not necessarily limited to these specific configurations.

<First Embodiment>
First, as a first embodiment, an example of processing for recognizing a touch operation performed by a user on an item projected on a table surface of a table top interface system will be described. In particular, it is assumed that both a single touch operation input at one input position and a multi-touch operation input at a plurality of input positions can be recognized.

  FIG. 1A is an example of the appearance of a tabletop interface system in which the information processing apparatus 100 according to the present embodiment is installed. The operation surface 101 is a table portion of the table top interface, and the user can input a touch operation by touching the operation surface 101. However, in this embodiment, a touch sensor is not mounted on the operation surface 101. For this reason, the information processing apparatus 100 does not determine whether the operation surface 101 and an operation body such as a finger or a pen operated by the user are actually in contact with each other, but a state close enough to be regarded as being in contact (hereinafter referred to as touch input). The touch operation can be recognized by detecting the touch state.

  In the present embodiment, the distance image sensor 102 is installed above the operation surface 101 so as to look down at the operation surface. A distance image is an image in which information corresponding to the distance from the reference position (for example, the center of a lens, etc.) of the imaging means that captures the distance image to the subject surface imaged by the pixel is reflected in the value of each pixel. It is. In the present embodiment, the distance image sensor 102 captures the distance value from the distance image sensor 102 to the operation surface 101 or the object surface above the distance image. The captured distance image is input to the information processing apparatus 100 as a distance image. The information processing apparatus 100 acquires the three-dimensional position of the user's hand 106 by analyzing the distance image and recognizes the input operation. Therefore, the user operates a space gesture by moving a predetermined object such as a hand within a range that can be imaged by the distance image sensor 102 in a space on the operation surface (a space between the operation surface 101 and the distance image sensor 102). Can be entered. In the present embodiment, a sensor that acquires distance information using a reflection pattern (or reflection time) of infrared light is used. However, for example, a distance image can be obtained by installing a stereo camera system or an infrared light emitting element and an infrared light receiving element. Further, in the space including the operation surface 101, any means capable of obtaining three-dimensional position information including the height direction of the operation body is not limited to a mode for capturing a distance image, for example, an electrostatic sensor, a pressure sensor, The present embodiment can also be implemented by a method of obtaining three-dimensional position information using a temperature sensor.

  In the present embodiment, the visible light camera 103 is installed so as to look down on the operation surface 101 from above. The information processing apparatus 100 can function as a document camera that controls the visible light camera 103 to capture an image of an object placed on the operation surface 101 and obtain a read image thereof. Therefore, the information processing apparatus 100 detects and further identifies an object existing in the space on the operation surface 101 based on the visible light image obtained by the visible light camera 103 and the distance image obtained by the distance image sensor 102. The object includes, for example, a user's hand, a paper medium, a document such as a book, and other three-dimensional objects. However, in the case of the system illustrated in FIG. 1A, the angle of view of the distance image sensor 102 and the visible light camera 103 does not include the user's head existing around the table. Therefore, in the obtained distance image, the end of the image intersects with some part of the user's arm (portion beyond the shoulder).

  The projector 104 projects an image on the upper surface of the operation surface 101. In this system, the user performs an operation by touch or space gesture on the item 105 included in the projected image. As described above, in the present embodiment, the distance image acquired by the distance image sensor 102 is used for detecting the hand 106 and recognizing the operation. By using the distance image, there is an advantage that even if the color of the user's hand changes due to the projection light of the projector 104, it is difficult to be affected. The display device of the present system can be configured by using the operation surface 101 as a liquid crystal display instead of the projector 104. In that case, even if a method of detecting a human hand from an image by detecting a skin color region from a visible light image, the hand can be detected without being affected by the projection light.

  Note that the distance image sensor 102 and the visible light camera 103 themselves are not necessarily installed above as long as an image obtained by viewing the operation surface 101 from above is obtained. For example, the distance image sensor 102 and the visible light camera 103 may be configured so as to capture reflected light from a mirror installed above. Similarly, in the example of FIG. 1A, the projector 104 projects onto the operation surface 101 so as to look down from diagonally above, but the projection light projected in different directions is used using a mirror or the like. It may be reflected on the operation surface 101. The same applies to the case where the operation surface 101 is installed along the vertical direction.

  In the present embodiment, the x, y, and z axes shown in FIG. 1 are defined in the three-dimensional space on the operation surface 101 to handle position information. In the example of FIG. 1A, the point 107 is the origin of the coordinate axes. Here, as an example, the two dimensions parallel to the upper surface of the table are the xy plane, and the direction perpendicular to the upper surface of the table and extending upward (normal direction) is the positive direction of the z axis. In the present embodiment, the z-axis direction corresponds to the height direction in the world coordinate system. However, in the present embodiment, a non-horizontal surface such as a whiteboard or a wall surface is used as the operation surface 101, or the operation surface has irregularities, or a virtual generated using a known AR (augmented reality) system. It can also be applied to a surface.

  FIG. 1B is a diagram illustrating the relationship between the distance image captured by the distance image sensor 102 and the system. In the present embodiment, the distance image sensor 102 has a function of acquiring a distance image in which three-dimensional position information of a predetermined space included in the angle of view is reflected. In the present embodiment, the distance image sensor 102 emits infrared light from the distance image sensor 102 toward the operation surface 101, and the reflected light reflected by the surface of the subject is received by the light receiving element of the sensor.

  The distance image used in this embodiment reflects the distance to the subject surface corresponding to the time taken to reflect each pixel by measuring the phase delay of the received reflected light in the pixel value. Is. In the present embodiment, since the distance image sensor 102 captures a distance image so as to look down the operation surface 101, the distance information represented by the pixel value of the distance image is in the depth direction in the direction of looking down the operation surface 101 from the distance image sensor 102. Distance. In other words, each pixel value of the distance image includes information for obtaining position information (z coordinate) in the distance direction from the operation surface 101. However, in the present embodiment, the distance image sensor 102 is installed so as to have an angle with respect to the z axis. Therefore, in this embodiment, the pixel value of the distance image is not used as it is as the z coordinate, but coordinate conversion using parameters according to the sensor and the installation environment is performed as described later. Then, as will be described later, the uvd coordinates of the position information defined in the distance image are converted into xyz coordinates of the world coordinates and used. In the present embodiment, since the operation surface is assumed to be a table surface parallel to the ground, the distance from the operation surface 101 represented by the z coordinate of the input position corresponds to a distance value representing the height from the operation surface 101. . Therefore, the z coordinate may be expressed as height.

  In FIG. 1B, the distance image 108 represents an example of the content of the distance image captured by the distance image sensor 102. However, in FIG. 1B, the distance information reflected in the pixel value is omitted, and only the edge of the subject is clearly shown. The operation surface 101 and the origin 107 correspond to FIG. The image end 111 corresponds to the angle of view of the distance image sensor 102.

  As shown in FIG. 1B, a two-dimensional coordinate system based on the u axis and the v axis is set in the distance image. In the example of FIG. 1B, the resolution of the distance image is 640 [dot] × 480 [dot]. Assume that the position coordinates in the distance image of the input position 110 are (u, v), and the pixel value corresponding to the distance from the distance image sensor 102 to the input position is d. In the present embodiment, coordinate conversion based on the lens characteristics of the distance image sensor 102 and the relative positional relationship with the operation surface 101 is performed on the position information defined in the distance image in this way. As a result, the coordinates of each pixel are mapped to the real-world coordinate system defined in the table, and the three-dimensional position (x, y, z) in the real space can be acquired for the input position 110. The transformation matrix used for coordinate transformation is acquired by performing adjustment work in advance when the distance image sensor 102 is installed. Note that the distance d used for calculating the three-dimensional coordinates is not limited to a single pixel value, but a noise removal process and an averaging process are performed on pixels corresponding to several pixels near the input position 110 in the hand region. And may be specified.

  A hatched area 109 is an image of the user's hand 106 in the distance image 108 (hereinafter simply referred to as the hand area 109). In the present embodiment, by performing threshold processing on the z-coordinate, an area where a subject existing at a position higher than the operation surface 101 that is the table surface is detected as a hand area. In this embodiment, one input position is detected for each detected hand. The input position is a position estimated that the user is pointing with a finger, and is represented by three-dimensional coordinates. In the present embodiment, it is premised that when the user designates a point for a touch operation, a “pointing pose” in which only one finger is extended is taken. This is because a pose with one finger extended is a natural posture for many people to point to one point. Therefore, as the input position, a position estimated to be an end portion of the hand region 109 is specified. In a pointing pose, the edge hits the fingertip. Specifically, the hand region 109 is detected from the distance image, and the coordinate indicating the pixel located farthest from the image end 111 in the hand region 109 is regarded as one point corresponding to the fingertip. In the case of the hand region 109 shown in FIG. 1B, the point farthest from the image end 111 is specified as the input position 110. Note that the method of specifying the input position is not limited to this, and for example, the five fingers of the hand may be detected from the hand region to specify the end portion of the predetermined finger.

  In the present embodiment, the center of the portion where the hand region 109 intersects the image end 111 in each frame of the distance image repeatedly captured according to the frame rate is defined as the hand entry position. In the case of the hand region 109 in FIG. 1B, the intrusion position 112 is specified. The intrusion position 112 is also acquired as three-dimensional position information in real space by converting the position information defined in the distance image. In the present embodiment, when a plurality of hand regions are detected from the distance image, the input position and the intrusion position are specified for each of them. Then, the touch operation is recognized for each of the plurality of hand regions.

  Hereinafter, in the present specification, a case where the user's hand 106 and its finger are used as an example of an operation body used for input of a touch operation by the user and an end thereof will be described. However, in the present embodiment, the present invention can be applied not only to fingers but also instruments such as a stylus and a robot arm as the operating body. Note that the end of the operating body refers to a part used to point to the input position for the touch operation. However, if the part belongs to a part of the operating body and allows input of the touch operation, the end of the operating body has a protruding shape. It does not need to be limited to the end.

  In the present embodiment, the distance between the operation body and the operation surface 101 is acquired using the distance image captured by the distance image sensor 102, and when the distance is smaller than the first threshold, the operation body is the operation surface 101. Recognize that you touched. Further, a second threshold value is provided separately from the distance threshold value for recognizing that the operating body touches the operation surface 101. When a plurality of operating objects are detected, a threshold determination by the second threshold is performed before the determination by the first threshold so that the plurality of operating objects can input one multi-touch operation in conjunction with each other. After determining whether it is used, it is determined whether touch input has been made by each operating tool. FIG. 2A is a hardware configuration diagram of a tabletop interface including the information processing apparatus 100 according to the present embodiment. A central processing unit (CPU) 200 uses the RAM 202 as a work memory, executes an OS and a program stored in the ROM 201 and the storage device 203, and performs various processing operations and logical determinations. Further, each component connected to the system bus 204 is controlled. The storage device 203 is a hard disk drive, an external storage device connected by various interfaces, and the like, and stores programs and various data related to the operation recognition processing of the embodiment.

  The distance image sensor 102 captures a distance image of a space on the operation surface 101 including a table on which the item is displayed and a user's hand operating the item according to the control of the CPU 200, and the captured distance image is input to the system bus 204. Output. In the present embodiment, the distance image acquisition method will be described based on a reflection time method (Time-of-Flight method) that is less affected by ambient light or table surface display. It is also possible to use a pattern method or the like. The projector 104 projects and displays an image item to be operated on the table according to the control of the CPU 200.

  In the above-described system, the visible light camera 103, the distance image sensor 102, and the projector 104 are external devices connected to the information processing apparatus 100 via an input / output interface, respectively. Configure the processing system. However, these devices may be integrated with the information processing apparatus 100. The distance image sensor 102 may be an electrostatic sensor, a pressure sensor, a temperature sensor, or the like, and acquires contact information such as three-dimensional position information of the input position, contact pressure of the input position, and a contact area.

  FIG. 2B is an example of a block diagram illustrating a software configuration of the information processing apparatus 100. Each of these functional units is realized by the CPU 200 developing a program stored in the ROM 201 in the RAM 202 and executing processing according to each flowchart described later. The execution result of each process is held in the RAM 202. Further, for example, when hardware is configured as an alternative to software processing using the CPU 200, arithmetic units and circuits corresponding to the processing of each functional unit described here may be configured.

  The three-dimensional information acquisition unit 210 acquires a distance image captured by the distance image sensor 102 at regular intervals according to the frame rate, and stores the acquired distance image in the RAM 202 as needed. The target acquired by the three-dimensional information acquisition unit 210 and exchanged with each functional unit is actually a signal corresponding to the image data, but will be described as simply “acquiring a distance image” in the present specification.

  The operating tool acquisition unit 211 performs threshold determination and noise reduction processing on each pixel of the distance image acquired by the three-dimensional information acquisition unit 210, and detects a hand region (including an arm) in the distance image. The hand region is a pixel group in which a hand used as an operating tool by the user is captured as a subject in the input distance image. Then, an ID is assigned to the detected hand region and is held in the RAM 202. When assigning IDs, if the hand regions detected in the frames of different distance images are regarded as the same hand region, the same ID is assigned. The condition that is regarded as the same hand region is that it is detected in a frame of a distance image that is close in time and that the intrusion position 112 of the hand region is short. On the other hand, if they are not regarded as the same hand area, numbers are assigned in the order in which the hand areas are detected (ID = 1, 2,...).

  The input position acquisition unit 212 detects one point corresponding to the end of the hand region as an input position based on the contour information of the hand region detected by the operating tool acquisition unit 211. In the present embodiment, as shown in FIGS. 1A and 1B, coordinates indicating a pixel that is farthest from the image end 111 in the contour of the hand region are detected as the input position. Then, the position information in the distance image of the detected input position is converted into xyz coordinates of world coordinates, the same ID as that of the hand region is given, and held in the RAM 202. Note that three-dimensional position information (coordinate values) and contact information (contact pressure, contact area, etc.) of the fingertip position close to the operation surface 101 are acquired by an electrostatic sensor, a pressure sensor, or the like, and the RAM 202 is used as input position information. It is also possible to adopt a method of holding in.

  The specifying unit 213 specifies an input position that satisfies a predetermined condition at the input position detected by the input position acquiring unit 212 and an input position that is effective for the multi-touch operation. In the present embodiment, among a plurality of input positions detected by the input position acquisition unit 212, when there are a plurality of input positions whose distance from the operation surface 101 is smaller by a predetermined second threshold, multi-touch is selected from them. A valid input position is specified as an operation input. As the second threshold value used here, a distance larger than a first threshold value (sometimes referred to as a touch threshold value) used for determining that the input position is in the touch state is set. In this way, in this embodiment, whether or not there is a possibility that the input position may be used for the multi-touch operation before determining whether or not each touch position is in the touch state by the two-stage threshold determination. It is possible to obtain the result of determining. For example, even if only one of the two input positions is in the state of transition to the touch state, if there is a high possibility that they will be used for multi-touch operation, the single touch operation based on the input position transitioned to the touch state Suppress recognition. And it waits for the other input position to change to a touch state, and it becomes possible to recognize multi-touch operation according to having changed to the multi-touch state.

  In the present embodiment, the operation specifying unit 213, when a plurality of input positions whose distances from the operation surface 101 are smaller than the second threshold satisfy a predefined condition, input them as effective inputs for a multi-touch operation. Specify as location. Then, the identified input position is associated with the ID, and the identification result is held in the RAM 202 as pairing information. Hereinafter, the process of specifying a plurality of input positions effective for the multi-touch operation is referred to as “pairing” the plurality of input positions.

  The recognition unit 214 recognizes an input touch operation. First, based on the three-dimensional coordinates of the input position, it is determined whether the proximity state of each input position to the operation surface 101 is considered to be a touch input, and the determination result is held in the RAM 202. In this embodiment, when the distance (z coordinate corresponding to the height) between the input position and the operation surface 101 is smaller than a predetermined threshold, it is determined that the touch input is being performed (hereinafter referred to as a touch state). . Then, by tracking the position on the operation surface 101 (corresponding to the xy coordinates of the input position) indicated by the input position while in the touch state, touch events such as touch, release, move, flick, pinch, rotate, etc. Is generated. Then, the generated touch event is notified to the display control unit 215.

  The display control unit 215 generates drawing data reflecting the result of each process (such as a touch event) executed in response to a user operation notified by the recognition unit 214, and outputs the drawing data to the projector 104, thereby operating the operation surface 101. Control what is displayed on the screen. The user operation is defined by an input position, a movement of the input position, and the like, and can input a predetermined instruction to the information processing apparatus 100. Further, the display control unit 215 notifies the user of the pairing result by reflecting the result of pairing the input position in the drawing data. For example, the same color is projected on the paired input position, the same shape (circle, polygon, etc.) or symbol is projected, and the number or character is displayed for visual notification. Note that the notification of the pairing result is not limited to the visual sense, and may be a notification by voice, a feeler, or the like. For example, a notification sound may be sounded when pairing is completed, or the input position may be vibrated.

  FIG. 3 is a flowchart showing the flow of main processing of the information processing apparatus according to this embodiment. In the present embodiment, when the information processing apparatus 100 is turned on, first, in step S301 of FIG. 3, the display control unit 215 generates a screen on which UI components are arranged based on the data acquired from the storage device 203. And output to the projector 104. Then, the projector 104 projects a screen on the operation surface 101.

  Next, in step S <b> 302, the three-dimensional information acquisition unit 210 acquires a distance image from the distance image sensor 102. The distance image sensor 102 captures a distance image of the operation area of the operation surface 101 at a predetermined time interval (frame rate) while the information processing apparatus is turned on, and the acquired distance image is displayed. Input to the information processing apparatus. Therefore, this process is repeated with the frame rate of the distance image.

  In step S303, the operating tool obtaining unit 211 detects the user's hand area (including the arm). If a hand area is detected (YES in step S303), the process proceeds to step S304. On the other hand, if the hand area is not detected (NO in step S303), the process returns to step S302, and the processes in steps S302 and S303 are repeated until the hand area is detected.

  In step S <b> 304, the operating tool acquisition unit 211 assigns an ID to the hand region detected in step S <b> 303 and stores the ID in the RAM 202. For example, IDs are given numbers in the detected order (1, 2,...). In step S <b> 305, the input position acquisition unit 212 detects the input position of the hand area detected in step S <b> 303, assigns the same ID as the hand area to the detected input position, and holds it in the RAM 202. In step S306, the specifying unit 213 specifies an input position effective for the multi-touch operation at the input position detected in step S305. The contents of the process of step S306 will be described later with reference to the flowchart of FIG. In step S307, the recognition unit 214 performs a process for determining the touch state of the input position detected in step S305. The contents of the process of step S307 will be described later with reference to the flowchart of FIG.

  In step S308, the recognition unit 214 recognizes the operation based on the input position detected in step S305 based on the processing results in steps S306 and S307. The contents of the process of step S308 will be described later with reference to the flowchart of FIG. In step S309, the display control unit 215 updates the display of the object (image, data, UI, etc.) on the screen projected by the projector 104 in accordance with the operation recognized in step S308. In step S310, it is determined whether or not the information processing apparatus has been turned off. If the power is turned off (YES in step S306), the information processing apparatus 100 ends all the processes. On the other hand, if the power is not turned off (NO in step S306), the process returns to step S302.

  FIG. 4 is a flowchart showing an example of the flow of the input position specifying process executed in step S304 of the present embodiment. First, in step S401, one input position to be processed is selected from the detected input positions. In step S402, the specifying unit 213 determines whether the input position to be processed has already been paired. If the input position is already paired (YES in step S402), the process proceeds to step S403. If the input position is not paired (NO in step S402), the process proceeds to step S406.

  In step S403, the specifying unit 213 determines whether the height of the input position is greater than or equal to a third threshold value. The third threshold value is a threshold value that represents a reference for the distance from the operation surface and is used for determining whether or not to cancel pairing. The size of the third threshold is determined by multi-touch since the operating body is sufficiently separated from the operation surface 101 when any of the plurality of paired input positions transitions to a state of the third threshold or higher. It is set to such an extent that the possibility of continuing the operation is low. If the height of the input position to be processed is greater than or equal to the third threshold (YES in step S403), the process proceeds to step S404. If the height of the input position is smaller than the third threshold (NO in step S403), the process proceeds to step S405. In step S <b> 404, the specifying unit 213 cancels the pairing of the input position, and holds information representing it in the RAM 202.

  In step S405, the specifying unit 213 determines whether or not the pairing determination has been completed at all the input positions detected in step S305. If the pairing determination has been completed at all input positions (YES in step S405), the process in FIG. 4 is terminated, and the process proceeds to step S307. If pairing determination has not been completed at all input positions (NO in step S405), the process returns to step S401 and the process is repeated.

  When it is determined NO in step S402, in step S406, the specifying unit 213 determines whether or not the height of the input position to be processed is equal to or less than the second threshold value. The second threshold value is a threshold value that represents a reference for the distance from the operation surface. When there are a plurality of input positions whose height is equal to or less than the second threshold, there is a possibility that a plurality of operating objects will be sufficiently close to the operation surface 101, and then a multi-touch operation may be input in conjunction with the size. Set to a level that can be considered. If the height of the input position is less than or equal to the second threshold (YES in step S406), the process proceeds to step S407. If the height of the input position is greater than the second threshold (NO in step S406), the process proceeds to step S405. In the present embodiment, the third threshold value is larger than the second threshold value in order to reliably detect that the paired input position has shifted to a position sufficiently away from the operation surface 101 in consideration of distance detection errors and the like. Value. However, the second threshold value and the third threshold value may be the same value.

  In step S407, the specifying unit 213 determines whether there are two input positions whose height is equal to or less than the second threshold value. More specifically, among the information held in the RAM 202, the z-coordinate of the input position not to be processed among the plurality of input positions detected in step S305 is referred to, and the height is the second threshold value. It is determined whether the following cancels another. If there are two input positions whose height is equal to or less than the second threshold (YES in step S407), the process proceeds to step S408. If there are no two input positions whose height is less than or equal to the second threshold (NO in step S407), the process proceeds to step S405. In step S408, the specifying unit 213 determines whether the distance between two input positions whose height is equal to or smaller than the second threshold is smaller than a reference distance among the plurality of detected input positions. The reference distance here is a reference distance of a range in which two input positions can move while interlocking for multi-touch operation, and is defined in the xy plane. An appropriate distance may be selected as the reference distance according to the type of multi-touch operation that can be recognized by the information processing apparatus 100. For example, it is assumed that the multi-touch operation that can be recognized by the information processing apparatus 100 includes a reduction operation that moves the two input positions so as to be narrowed from a relatively wide state. In this case, at least the state where the two input positions are separated to the extent necessary for recognizing the reduction operation also needs to satisfy the condition regarding the reference distance determined in step S408. If the distance between the two input positions is smaller than the reference distance (YES in step S408), the process proceeds to step S409. If the distance between the two input positions is not smaller than the reference distance (NO in step S408), the process proceeds to step S405. However, depending on the operating environment, the process of step S408 may be omitted. For example, when the operation surface 101 is small enough to allow a single operator to perform a predetermined multi-touch operation, the determination result in step S408 described above is self-evident and can be omitted. In addition, for example, when the number of operators who perform operations simultaneously is limited to one, since there is a high degree of certainty that the multi-touch operation is performed when there are a plurality of input positions whose height is equal to or less than the second threshold value, Processing may be omitted. In step S409, the specifying unit 213 pairs the two input positions determined to satisfy the distance condition in step S408, and holds the pairing information in the RAM 202.

  FIG. 5 is a flowchart showing the flow of the determination process of the touch state of the input position, which is executed in step S307 of the present embodiment. First, in step S501, one input position to be processed is selected from the detected input positions. In step S502, the recognition unit 214 determines whether or not the input position to be processed has already been touched. If the input position has already been touched (YES in step S502), the process proceeds to step S503. If the input position is not touched (NO in step S502), the process proceeds to S506. In step S503, the recognition unit 214 determines whether the height of the input position is equal to or greater than a fourth threshold value. The fourth threshold value is a threshold value of the height of the input position for determining that the touch state has been released in response to the operating body sufficiently moving away from the operation surface 101, and may be referred to as a release threshold value. If the height of the input position is greater than or equal to the fourth threshold (YES in step S503), the process proceeds to step S504. If the height of the input position is smaller than the fourth threshold (NO in step S503), the process proceeds to S505. In step S504, the recognition unit 214 cancels the touch state of the input position, holds in the RAM 202 that the input position is in the non-touch state, and proceeds to step S505.

  In step S505, the recognition unit 214 determines whether or not the determination of the touch state has been completed at all the input positions detected in step S305. If the determination of the touch state has been completed at all input positions (YES in step S505), the process in FIG. 5 is terminated, and the process proceeds to step S308. If the determination of the touch state is not completed at all input positions (NO in step S505), the process returns to the start of the process in FIG. 5 and proceeds to step S501.

  On the other hand, in step S506, the recognition unit 214 determines whether the distance (corresponding to the height) from the operation surface of the input position to be processed is equal to or less than the first threshold value. The first threshold value is a distance threshold value for determining that a touch input has been made by detecting that the operating tool has sufficiently approached the operation surface 101. If the distance from the operation surface is equal to or less than the first threshold (YES in step S506), the process proceeds to step S507. If the distance from the operation surface is greater than the first threshold (NO in step S506), the process proceeds to step S505. In step S507, the recognition unit 214 sets the input position to the touch state, holds in the RAM 202 that the input position is in the touch state, and proceeds to step S505. In the present embodiment, it is assumed that the fourth threshold value is larger than the first threshold value so that the touch state is not accidentally released until the operating tool is sufficiently separated from the operation surface 101. However, these distances may coincide.

  FIG. 6 is a flowchart showing the flow of recognition processing for the input position touch operation executed in step S308 of the present embodiment. First, in step S601, one input position to be processed is selected from the detected input positions. In step S602, the recognition unit 214 refers to the determination result of the touch state of the input position to be processed. If the input position is in the touch state (YES in step S602), the process proceeds to step S603. If the input position is not in the touch state (NO in step S602), the process proceeds to step S606.

  Next, in step S603, the recognition unit 214 refers to the determination result of the pairing state of the input position to be processed. If the input position is in the pairing state (YES in step S603), the process proceeds to step S604. If the input position is not in the pairing state (NO in step S603), the process proceeds to S607. In step S604, the recognition unit 214 refers to the determination result of the touch state of the other input position that is paired with the input position to be processed. If the other input position that is paired is in the touch state (YES in step S604), the process proceeds to step S605. If the other input position that is paired is not in the touch state (NO in step S604), the process proceeds to step S606.

  In step S605, the recognition unit 214 recognizes the multi-touch operation based on the input position. Then, information (such as a touch event) indicating the recognition result is notified to the display control unit 215 and the process proceeds to S606. In step S606, the recognition unit 214 determines whether or not the recognition of the touch operation has been completed at all the input positions detected in step S305. At this time, the input position read as the pairing partner in step S604 and used for the multi-touch operation recognition in step S606 is treated as an input position that has been recognized for touch operation. If the recognition of the touch operation has been completed at all the input positions (YES in step S606), the process of FIG. 6 ends and the process proceeds to step S309. If the recognition of the touch operation is not completed at all input positions (NO in step S606), the process proceeds to step S601. On the other hand, if NO in step S603, the process proceeds to step S607. In step S607, the recognition unit 214 recognizes a single touch operation based on the input position. Information (such as a touch event) representing the recognition result is notified to the display control unit 215, and the process proceeds to step S606.

  As described above, in this embodiment, in step S604, even if the input position to be processed is in the touched state, if the input position paired with the input position is not in the touched state, the input in the touched state Does not perform single-touch operation recognition based on position. Therefore, before and after the multi-touch operation is performed, an operation in which a single touch event unintended by the user is uttered can be reduced. In the flowcharts of FIGS. 4 to 6, the condition when the number of input positions to be paired is set to two has been described. However, the present embodiment is not limited to two, and three or more input positions are set. It can also be applied when recognizing linked multi-touch operations. When a multi-touch operation using N (N: integer greater than or equal to 3) input positions is input, it is determined in step S407 whether there are N input positions whose distance from the operation surface is equal to or less than the second threshold value. . In step S408, for example, it may be determined whether the distance between the gravity center position determined from the N input positions and each input position is smaller than the reference distance. At this time, the reference distance may be determined by the number of assumed operators. Further, in the flowchart of FIG. 6, an operation with 1 to “N−1” input positions is waited for without being recognized until all paired input positions reach the touch state.

  Next, operations performed by the information processing apparatus 100 according to the present embodiment will be described with reference to FIGS. 7 and 8 in association with specific examples of the user's hand state. 7A to 7E show a multi-touch operation (with two hands) when the user extends the hand 700 and the hand 701 over the operation surface 101 and then touches (or closes) the fingertip to the operation surface 101. ) In a stepwise manner. However, the hand 700 and the hand 701 are not shown in FIGS. 7B to 7E in order to show the movement of the input position in an easy-to-understand manner. A first threshold 703 and a second threshold 702 are set in advance. Each is set at a height (distance from the operation surface) relative to the operation surface. As an example, the first threshold 703 = 10 mm and the second threshold 702 = 20 mm. The characteristic processing at each stage and the correspondence between each step in the flowchart described above are shown by writing the step number in parentheses.

  First, a state where the user's finger is at a position away from the operation surface 101 will be described with reference to FIG. FIG. 7A shows the space on the operation surface 101 at time t1 as viewed from the side. Two hand regions of the operated hand 700 and the operated hand 701 are detected from the distance image acquired from the distance image sensor 102 at time t1 (step S303). A hand ID for identifying the hand region is given to the detected hand region. Specifically, ID = 1 is assigned to the hand area of the hand 700, and ID = 2 is assigned to the hand area of the hand 701 (step S304). An input position is detected from the detected hand area. Specifically, P1 is detected as the input position of the hand region of the operating hand 700, and P2 is detected as the input position of the hand region of the operating hand 701. ID = 1 is assigned to the input position P1, and ID = 2 is assigned to the input position P2. Then, the uvd coordinates of the position information in the distance image between the input positions P1 and P2 are converted into xyz coordinates of world coordinates (step S305). Specifically, the coordinates of the input position P1 are converted from (u1, v1, d1) to (x1, y1, z1), and the coordinates of the input position P2 are converted from (u2, v2, d2) to (x2, y2, z2). (Step S305).

  Next, an input position effective for the multi-touch operation is specified, and the process proceeds to a process of pairing each other (step S306). First, the input position P1 is set as a processing target (step S401). The input position P1 is not paired (NO in step S402), and the height z1 of the input position P1 is greater than the second threshold 702 (NO in step S406). Since the pairing determination is not completed at all the input positions detected in the distance image acquired this time (NO in step S405), the unprocessed input position P2 is set as a processing target (step S401). The input position P2 is not paired (NO in step S402), and the height z2 of the input position P2 is larger than the second threshold 702 (NO in step S406). If the pairing determination has been completed at all input positions detected in the distance image acquired this time (YES in step S405), the pairing determination process ends.

  Thus, the process proceeds to the process for determining the touch state of the input position (step S307). However, in the state of FIG. 7A, the operating hand 700 and the operating hand 701 are both maintained at a position sufficiently higher than the operation surface 101, and the input positions P1 and P2 are in the touch state. is not. Therefore, since the touch operation is not recognized (step S308), detailed description is omitted.

  Next, a state where the height z3 of the designated position P3 of the operating hand 700 is equal to or less than the second threshold value 702 will be described with reference to FIG. FIG. 7B shows the space on the operation surface 101 at time t2 as viewed from the side. A distance image at time t2 is acquired from the distance image sensor 102 (step S302), and two hand regions of the operated hand 700 and the operated hand 701 are detected from the acquired distance image (step S303). ). ID = 1 is assigned to the hand area of the operating hand 700, and ID = 2 is assigned to the hand area of the operating hand 701 (step S304). P3 is detected as the input position of the hand region of the operating hand 700, and P4 is detected as the input position of the hand region of the operating hand 701. Then, the coordinates of the input position P3 are converted from (u3, v3, d3) to (x3, y3, z3), and the coordinates of the input position P4 are converted from (u4, v4, d4) to (x4, y4, z4). (Step S305).

  In the specifying process (step S306), first, the input position P3 is set as a processing target (step S401). The input position P3 is not paired (NO in step S402), the height z3 of the input position P3 is equal to or less than the second threshold value 702 (YES in step S406), and the input position whose height is equal to or less than the second threshold value 702 is There are not a plurality (NO in step S407). Since the pairing determination has not been completed at all the input positions (NO in step S405), the unprocessed input position P4 is selected as a processing target (step S401). The input position P4 is not paired (NO in step S402), and the height z4 of the input position P4 is greater than the second threshold 702 (NO in step S406). Since the pairing determination has been completed at all input positions detected in the distance image acquired this time (YES in step S405), the pairing determination process ends. Then, the process proceeds to the determination process of the touch state of the input position (step S307). In FIG. 7B, both the operating hand 700 and the operating hand 701 are maintained at positions sufficiently higher than the operation surface 101, and the input positions P1 and P2 are not in the touch state. Therefore, since the touch operation is not recognized (step S308), detailed description is omitted.

  Next, a state where the height z5 of the designated position P5 of the operated hand 700 and the height z6 of the designated position P6 of the operated hand 701 are equal to or smaller than the second threshold value 702 will be described with reference to FIG. I will explain. FIG. 7C is a view of the space on the operation surface 101 at time t3 as viewed from the side. A distance image at time t3 is acquired from the distance image sensor 102 (step S302), and two hand regions of the operating hand 700 and the operating hand 701 are detected from the acquired distance image (step S303). ). ID = 1 is assigned to the hand area of the operating hand 700, and ID = 2 is assigned to the hand area of the operating hand 701 (step S304). P5 is detected as the input position of the hand region of the operating hand 700, and P6 is detected as the input position of the hand region of the operating hand 701. Then, the coordinates of the input position P5 are converted from (u5, v5, d5) to (x5, y5, z5), and the coordinates of the input position P6 are converted from (u6, v6, d6) to (x6, y6, z6). (Step S305).

  In the specifying process (step S306), first, the input position P6 is set as a processing target (step S401). The input position P6 is not paired (NO in step S402), and the height z5 of the input position P5 is equal to or less than the second threshold 702 (YES in step S406). In FIG. 7C, there are two input positions whose height is equal to or less than the second threshold value 702 (YES in step S407). Here, it is assumed that the interval between the input positions P5 and P6 is smaller than the reference distance (YES in step S408). The input positions P5 and P6 are paired, and ID1 of the input position P5 and ID2 of P6 of the input position are associated and held as pairing information (step S409). Since the pairing determination has not been completed at all the input positions detected in the distance image acquired this time (NO in step S404), the unprocessed operation position P6 is set as the processing target (step S401). The operation position P6 is already paired with the operation position P5 (YES in step S402), and the height z6 of the operation position P6 is smaller than the third threshold 802 (NO in step S403). When the pairing determination has been completed at all the operation positions detected in the distance image acquired this time (YES in step S405), the pairing determination process ends.

The touch state of the input position is determined (step S307), and both the input positions P5 and P6 are not in the touch state as in FIGS. 7A and 7B. Therefore, in the state of FIG. 7C, the operating hand 700 and the operating hand 701 are maintained at a position sufficiently higher than the operation surface 101, and the input positions P5 and P6 are not in the touch state. Therefore, the touch operation is not recognized.
Next, a state where the height z7 of the designated position P7 of the operating hand 700 is equal to or less than the first threshold value 703 will be described with reference to FIG. FIG. 7D shows the space on the operation surface 101 at time t4 as viewed from the side. A distance image at time t4 is acquired from the distance image sensor 102 (step S302), and two hand regions of the operating hand 700 and the operating hand 701 are detected from the acquired distance image (step S303). ). ID = 1 is assigned to the hand area of the operating hand 700, and ID = 2 is assigned to the hand area of the operating hand 701 (step S304). P7 is detected as the input position of the hand region of the operating hand 700, and P8 is detected as the input position of the hand region of the operating hand 701. Then, the coordinates of the input position P7 are converted from (u7, v7, d7) to (x7, y7, z7), and the coordinates of the input position P8 are converted from (u8, v8, d8) to (x8, y8, z8). (Step S305).

  In the specifying process (step S306), first, the input position P7 is set as a processing target (step S401). The input position P7 has been paired (YES in step S402), and the height z7 of the input position P7 is smaller than the pairing elimination threshold 802 (NO in step S403). Further, since the pairing determination has not been completed at all the input positions detected in the distance image acquired this time (NO in step S405), the unprocessed input position P8 is next processed (step S401). ). The input position P8 has already been paired (YES in step S402), and the height z8 of the input position P8 is smaller than the pairing elimination threshold 802 (NO in step S403). Pairing determination is completed at all input positions detected in the distance image acquired this time (YES in step S405). Accordingly, the pairing determination process is terminated while the pairing information is maintained.

  Also in the process of determining the touch position of the input position (step S307), first, the input position P7 is set as a processing target (step S501). The input position P7 has not been determined to be touched so far (NO in step S502). However, the height z7 of the input position P7 is equal to or less than the first threshold value 703 at time t4 (YES in step S506), and the input position P7 is set in the touch state and held in the RAM 202. Since the touch state determination is not completed at all the input positions (NO in step S505), the unprocessed input position P8 is set as a processing target (step S501). The input position P8 has not been touched so far (NO in step S502), and the height z8 of the input position P8 is greater than the first threshold 703 even at time t4 (NO in step S506). Since the touch state determination has been completed at all input positions detected in the distance image acquired this time (YES in step S505), the touch state determination process ends.

  Next, in the process of recognizing the touch operation at the input positions P7 and P8 (step S308), the input position P7 is set as a processing target (step S601). The input position P7 is in the touched state (YES in step S602), the input position P7 has been paired (YES in step S603), and the input position P8 that is the pairing partner of the input position P7 is not in the touched state (step NO in S604). Since the touch operation recognition is not completed at all the input positions (NO in step S606), the unprocessed input position P8 is the next processing target (step S601). The input position P8 is not in the touch state (NO in step S602), and the touch operation recognition process is completed at all input positions (YES in step S606), so the touch operation recognition process is terminated. In this way, in the state of FIG. 7D, the input position P7 of the operating hand 700 is in the touch state, but it is paired with the input position P8 and there is a possibility that the multi-touch operation is input. Since it is considered high, recognition of single touch operations is not performed. Since the touch operation is not recognized, the projection content is not changed (step S309).

  Next, a state where the pointing position P9 of the operating hand 700 and the pointing position P10 of the operating hand 701 are paired and both are in a touch state will be described with reference to FIG. FIG. 7E shows the space on the operation surface 101 at time t5 as viewed from the side. A distance image at time t5 is acquired from the distance image sensor 102 (step S302), and two hand regions of the operating hand 700 and the operating hand 701 are detected from the acquired distance image (step S303). ). ID = 1 is assigned to the hand area of the operating hand 700, and ID = 2 is assigned to the hand area of the operating hand 701 (step S304). P9 is detected as the input position of the hand region of the operating hand 700, and P10 is detected as the input position of the hand region of the operating hand 701. The coordinates (x7, y7, z7) of the input position P7 and the coordinates (x8, y8, z8) of the input position P8 are acquired (step S305). Since the input position P9 and the input position P10 are paired at this time and are in the touch state, detailed descriptions of step S306 and step S307 are omitted.

  In the process of recognizing the touch operation at the input positions P9 and P10 (step S308), the input position P9 is first processed (step S601). Since the input position P9 is in a touch state (YES in step S602) and has been paired with the input position P10 (YES in step S603), the input position P10 is also in a touch state (YES in step S604). Therefore, the input positions P9 and P10 are recognized as a multi-touch operation (step S605). For example, it is recognized as an enlargement / reduction of a display image based on a change in the distance between two points, or a display image rotation operation based on a relative rotation of two points. The display is updated based on the recognition result of the multi-touch operation at the input positions P9 and P10 (step S309).

  As described above with reference to FIG. 7, in the present embodiment, by providing the second threshold at a position higher than the first threshold, the input position is paired with another input position before the input position transitions to the touch state. Determine whether a ring is necessary. For this reason, it is possible to reduce the occurrence of a single touch event unintended by the user in a situation where there is a high possibility that a multi-touch operation is intended.

  Next, FIGS. 8A to 8D show a multi-touch operation from a state where the user inputs a multi-touch operation (operation with two hands) by bringing the fingertip into contact with (or close to) the operation surface 101. A state in which the fingertip being released is separated from the operation surface 101 is shown step by step. However, in order to show the movement of the input position in an easy-to-understand manner, the illustration of the hand 800 and the hand 801 is omitted in FIGS. A third threshold value 802 and a fourth threshold value 803 are set in advance. Each height is set based on the operation surface. As an example, the third threshold 802 = 25 mm and the fourth threshold = 15 mm. The characteristic processing at each stage and the correspondence between each step in the flowchart described above are shown by writing the step number in parentheses.

  First, the state of FIG. 8A is based on the fingertip of the hand 800 operated by the user (input position P11) and the fingertip of the operated hand 801 (input position P12) close to the operation surface 101. Multi-touch operation is being performed. FIG. 8A shows the space on the operation surface 101 at time t6 as viewed from the side. First, a distance image at time t6 is acquired from the distance image sensor 102 (step S302), and two hand regions of an operating hand 800 and an operating hand 801 are detected from the acquired distance image ( Step S303). ID = 3 is assigned to the hand area of the operating hand 800, and ID = 4 is assigned to the hand area of the operating hand 801 (step S304). The input position P11 of the hand region of the operating hand 800 and the input position P12 of the hand region of the operating hand 801 are detected. The detected input position is assigned the same ID as the hand area, P11 is ID = 3, and P12 is ID = 4. The coordinates (x11, y11, z11) of the input position P11 and the coordinates (x12, y12, z12) of the input position P12 are acquired (step S305).

  Then, the input position P11 and the input position P12 are paired, and the input position IDs 3 and 4 are associated and held as pairing information (step S306). The input position P11 and the input position P12 are in a touch state (step S307), recognized as a multi-touch operation (step S308), and the display is updated based on the recognition result of the multi-touch operation (step S309).

  Next, in the state of FIG. 8B, the fingertip (input position P13) of the hand 800 operated by the user is in the touched state, and the fingertip (input position P14) of the operated hand 801 is in the touched state. It is in a state of disappearing. FIG. 8B is a view of the space on the operation surface 101 at time t7 as seen from the side. First, the distance image at time t7 is acquired from the distance image sensor 102 (step S302). Similarly to FIG. 8A, a hand region is detected from the acquired distance image, and an input position is acquired from the detected hand region. The input position of the operating hand 800 is P13 (x13, y13, z13), and the input position of the operating hand 801 is P14 (x14, y14, z14). Since all the input positions are lower than the third threshold value 802, the pairing state is maintained (step S306).

  In the touch state determination process (step S307), since the height z14 of the input position P14 is not less than the fourth threshold value 803 (YES in step S503), the touch state of the input position P14 is canceled (step S504). Since the input position P14 of the pairing partner is not in the touch state (NO in step S604), neither the multi-touch operation nor the single touch operation is recognized (step S308), and the display is not updated (step S309). As described above, in this embodiment, as long as the pairing state is maintained, the single touch operation is not recognized even if one input position is in the touch state.

  Next, the state of FIG. 8C is a state in which the fingertip (input position P15) of the hand 800 operated by the user and the fingertip (input position P16) of the operated hand 801 are not in the touch state. It is. FIG. 8C shows the space on the operation surface 101 at time t8 as viewed from the side. A distance image at time t8 is acquired from the distance image sensor 102 (step S302). Similarly to FIG. 8A, a hand region is detected from the acquired distance image, and an input position is acquired from the detected hand region. The input position of the operating hand 800 is P15 (x15, y15, z15), and the input position of the operating hand 801 is P16 (x16, y16, z16). Since the height z15 of the input position P15 and the height z16 of the input position P16 are both smaller than the third threshold 802 (NO in step S403), the pairing state is continued. Since the height z15 of the input position P15 is greater than or equal to the fourth threshold value 803 (YES in step S503), the touch state of the input position P15 is released (step S504).

  Next, FIG. 8D shows a state in which the pairing of the fingertip (input position P17) of the hand 800 operated by the user and the fingertip (input position P18) of the operated hand 801 is released. It is. FIG. 8D is a view of the space on the operation surface 101 at time t9 as seen from the side. First, a distance image at time t9 is acquired from the distance image sensor 102 (step S302). Similarly to FIG. 8A, a hand region is detected from the acquired distance image, and an input position is acquired from the detected hand region. The input position of the operating hand 800 is P17 (x17, y17, z17), and the input position of the operating hand 801 is P18 (x18, y18, z18). In the specific process (step S306), since the height z18 of the input position P18 is equal to or greater than the third threshold value 802 (YES in step S403), the pairing between the input position P17 and the input position P18 is canceled (step S404).

  As described above with reference to FIG. 8, in this embodiment, by providing the third threshold value at a position higher than the fourth threshold value, it is determined whether to continue pairing even after the input position leaves the effective area of the touch operation. Can be done. Therefore, it is possible to reduce the occurrence of a single touch event that is not intended by the user after the multi-touch operation in an environment where both the single-touch operation and the multi-touch operation can be recognized.

<Modification 1>
As shown in the flowchart of FIG. 6, in the touch operation recognition process according to the first embodiment, when the input position to be processed is in a touch state and is paired, another paired input position It is determined whether or not the touch state is also present (step S604). In Modification 1, the process of step S604 is omitted from the series of processes of the first embodiment. That is, if the two input positions are in the pairing state, the recognition of the multi-touch operation is started (step S605) when one of the two input positions is in the touch state (step S603 deYES). When the two input positions are in the pairing state, there is a high possibility that a multi-touch operation will be performed after that, so it is estimated that the z coordinate of the other input position will soon reach the range where the touch state is determined. Therefore, in the first modification, the multi-touch operation recognition is started based on the movement of the xy coordinates of the two input positions without waiting for the z-coordinate of the other input position to be determined as the touch state. In the first modification, it is possible to obtain an effect of improving the sensitivity of the response to the multi-touch operation for the user while suppressing the occurrence of a single touch event that is not intended by the user.

  Here, with reference to FIG. 7D and FIG. 8B, a difference in processing between the first embodiment and the modification example 1 will be described. In FIG. 7D, the fingertip of the operating hand 700 (input position P7) and the fingertip of the operating hand 701 (input position P8) are paired, and only the input position P7 is in a touch state. This is the state at time t4. As described above, in the first embodiment, in the state of FIG. 7D, the input positions P7 and P8 are paired (step S306), and only the input position P7 is in the touch state (NO in step S604). Therefore, multi-touch operation is not recognized. On the other hand, in the case of the modified example 1, the input position P7 is in the touch state (YES in step S602), and the multi-touch operation is recognized in step S605 in response to having been paired (YES in step S603).

  On the other hand, in the state of FIG. 8B, the fingertip (input position P13) of the hand 800 operated by the user is in the touched state, and the fingertip (input position P14) of the operated hand 801 is not in the touched state. This is the state at time t7. The input positions P13 and P14 have been paired. As described above, in the first embodiment, since the input position P14 is no longer in the touch state (NO in step S604), neither the multi-touch operation nor the single touch operation is recognized. On the other hand, in the first modification, the input position P13 is still in the touch state (YES in step S602), and in response to being paired with the input position P14 (YES in step S603), the multi-touch operation in step S605 is performed. Recognition is performed.

  Although the second threshold value 702 and the first threshold value 703 are different values, a value close to a certain extent is set, such as 20 mm and 10 mm in the example of FIG. Therefore, when two input positions are below the second threshold value 702 and one of them is below the first threshold value 703, it can be estimated that the other input position is also in a position very close to the first threshold value 703. In the first modification, in such a situation, it is considered that there is a high possibility that both two input positions are used for the multi-touch operation, and recognition of the multi-touch operation is started from that point (that frame). For this reason, the user may feel that the recognition sensitivity has increased, such as a quick response at the start of the multi-touch operation, and the ability to continue the multi-touch operation even if the finger is slightly lifted from the operation surface 101. The first embodiment and the first modification thereof may be specified according to the user's operability preference, such as whether or not to obtain the response speed.

<Modification 2>
The first embodiment is based on a system that acquires the three-dimensional position of the user's fingertip from the distance image and regards that the distance between the operation surface and the fingertip is below a predetermined value as input by touch. On the other hand, in the second modification, an application example to a system that detects an input position by installing a proximity sensor that can be detected by distinguishing between contact and proximity of an object on an operation surface instead of a distance image sensor will be described.

  FIG. 9 is a flowchart showing a flow of main processing of the information processing apparatus in the second modification. The difference from FIG. 3 is that step S901 is executed instead of the processing for the distance image in steps S302 to S305. Note that the processing executed in other steps is the same as that in the first embodiment, and a description thereof will be omitted.

  In step S901, an input position is detected by a proximity sensor. When the input position is detected (YES in step S901), the position information of the input position is held, and the process proceeds to step S306. The position information of the acquired input position is (x, y) coordinates and height information. The height information includes two states: a state in which the operation surface is in contact and a state in which the operation surface is not in contact but is in proximity. For example, when the proximity sensor is a capacitive sensor, the fingertip height is not strictly detected. For example, a threshold is provided for the magnitude of the change in capacitance that occurs at the input position, and when there is a change greater than or equal to the threshold, the operating tool is in contact. Distinguish between contact and proximity by considering them close. For example, a binary value in which the state in contact with the operation surface is “1” and the state in proximity to the operation surface is “0” is stored as information corresponding to the height. Information corresponding to the height may be held in multiple values or numerical values. However, as in the first embodiment, the height information may be information corresponding to the distance from the operation surface, and is not limited to the vertical direction. For example, when an arbitrary wall surface is set as the operation surface, the distance from the operation surface may correspond to a coordinate value on a horizontal coordinate axis.

FIG. 11A is a diagram conceptually illustrating three threshold values handled by the first threshold value of the second modification. As described above, in the sensor that performs proximity detection by the electrostatic capacitance method, the height at which proximity can be detected may differ depending on the operation body. However, in FIG. 11A, in order to visually show the concept of the modified example 2, for convenience, a range closer to the operation surface than the threshold 1100 of a certain distance is a range in which the proximity sensor can detect the position of the operation body. Suppose there is. Further, a threshold 1101 is defined as the distance between the operating body and the operating surface that the operating body physically touches the operating surface. The threshold value 1100 has a role corresponding to the second threshold value in the first embodiment, and the threshold value 1101 has a role corresponding to the first threshold value in the first embodiment. Second Threshold First Threshold By setting as shown in FIG. 11A, in the second modification, pairing is performed when a plurality of input positions in close proximity are detected, and pairing is performed. When only one of the input positions shifts to the touch state, the single touch operation is not performed. Then, the multi-touch operation is recognized in response to the transition of all input positions to the touch state. As described above, the first embodiment can be applied not only to a table top interface using a distance image sensor but also to a system using a proximity sensor. By using the proximity sensor, the cost of the apparatus may be reduced and the position detection accuracy of the input position may be improved depending on the installation conditions and use conditions of the apparatus.

<Modification 3>
Modification 3 shows an application example to a system in which an input position is detected by installing a pressure-sensitive sensor capable of detecting pressing by an object on the operation surface instead of the above-described distance image sensor and proximity sensor. Here, it is premised that an operation input in a state where pressure is applied to one input position and an operation input in a state where pressure is applied to a plurality of input positions can be distinguished and recognized.

  FIG. 10 is a flowchart showing the flow of the main process of the information processing apparatus in this modification. The difference from FIG. 3 is that step S901 is executed instead of the processing for the distance image in steps S302 to S305. Note that the processing executed in other steps is the same as that in the first embodiment, and a description thereof will be omitted.

  In step S1001, the input position on the operation surface is detected by the pressure sensor. When the input position is detected (YES in step S1001), the position information of the input position is held, and the process proceeds to step S306. The position information of the input position is (x, y) coordinates and contact pressure information. The contact pressure information includes two states: a state in which the operation surface is in contact and a state in which the operation surface is in contact with and further pushed in (a pressing force exceeding the reference is applied). For example, the state of being in contact with the operation surface is set to “0”, and the state of being pushed in close proximity to the operation surface is held as “1”. Note that the contact pressure information may be held in multiple values or numerical values.

  FIG. 11B is a diagram conceptually showing two threshold values, the first threshold value and the second threshold value used in the third modification. As described above, the pressure sensor detects the pressure applied to the input position by the operating body, and does not detect the distance by which the operation surface is pushed. However, in FIG. 11B, in order to visually show the concept of the third modification, a threshold is provided for the distance that the operation surface is conceptually pushed in by applying pressure to the operation surface for convenience. Here, the distance 0 representing the stage at which the operating body physically touches the operation surface is set as the second threshold 1102, and the distance pushed when a pressing force equal to or higher than the reference is applied is set as the first threshold 1103. The threshold value 1102 has a role corresponding to the second threshold value in the first embodiment, and the threshold value 1103 has a role corresponding to the first threshold value in the first embodiment. However, in the case of the modification 3, the touch input is recognized even from the threshold value 1102 to the threshold value 1103. The pushing distance is equal to or greater than the first threshold 1103 when the pressing force applied at the input position is equal to or greater than the reference pressure. In consideration of the user's feeling of operation, it is preferable that the threshold 1102 and the second threshold are adjusted in the vicinity of the operation surface, and the threshold 1103 and the first threshold are adjusted in the range in which the contact pressure at the input position can be detected. However, in the threshold adjustment, the threshold 1103 first threshold is set to a value corresponding to a contact pressure larger than the threshold 1100 second threshold. As described above, the first embodiment can be applied not only to a table top interface using a distance image sensor but also to a system using a pressure sensor. In particular, the pressure sensor detects the magnitude of the pressing force on the operation surface at each input position, and when a pressing force exceeding the reference is detected only at one input position, and the pressing force exceeding the reference is detected at a plurality of input positions. This is effective when the recognized operation differs depending on the detected case. According to the third modification, even when the pressing force applied to a plurality of input positions is shifted and a pressing force exceeding a reference is temporarily detected only at a part of the input positions, an operation unintended by the user is mistakenly performed. Can be reduced.

  In the description of the first embodiment and the modifications thereof described above, the expression “above” or “below” is used in the determination process using the threshold value, but the determination target distance matches each threshold value. Which determination result is included may be set as appropriate together with the threshold value. For example, with regard to the first threshold value processing for determining the touch state in step S506, a state where the distance from the operation surface to the input position is at least below the first threshold value may be recognized as the touch state. The same applies to the second threshold value in step S406. Further, for example, with regard to the fourth threshold value processing for determining the release of the touch state in step S503, it may be recognized that the touch state is released when the distance from the operation surface to the input position exceeds at least the fourth threshold value. The same applies to the third threshold value in step S403.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

210 Three-dimensional information acquisition unit 211 Operation body acquisition unit 212 Input position acquisition unit 213 Identification unit 214 Recognition unit 215 Display control unit

Claims (18)

Obtaining means for obtaining information corresponding to a distance from the predetermined operation surface with respect to an input position input to a space on the predetermined operation surface;
Based on the information acquired by the acquisition means, the first operation input using one input position whose distance from the predetermined operation surface is less than the first threshold, and the distance to the predetermined operation surface is Recognition means for recognizing a second operation input using a plurality of input positions below a first threshold;
An output control means for controlling an output for responding to the operation recognized by the recognition means;
The output control means includes a plurality of input positions whose distance from the predetermined operation surface is less than a second threshold value that is greater than the first threshold value, and the distance from the predetermined operation surface is the first threshold value. An information processing apparatus characterized by not responding to the first operation while there is one input position below.   The recognizing unit is configured such that the number of input positions whose distance from the predetermined operation surface is less than the second threshold is not plural, and the input position where the distance from the predetermined operation surface is less than the first threshold. The information processing apparatus according to claim 1, wherein in one case, the first operation is recognized, and the output control unit performs output for responding to the recognized first operation. A specifying unit that specifies a plurality of input positions whose distance from the predetermined operation surface is lower than a second threshold value that is greater than the first threshold value as effective input positions for inputting the second operation;
The recognizing means is based on the fact that the distance from the operation surface of all the input positions specified as effective input positions for inputting the second operation by the specifying means is less than the first threshold value. The information processing apparatus according to claim 1, wherein the information processing apparatus recognizes a second operation, and the output control unit performs output for responding to the recognized first operation. Among a plurality of input positions whose distance from the predetermined operation surface is less than a second threshold value that is greater than the first threshold value, an input position whose distance between the plurality of input positions is smaller than a reference is selected as the second operation. Is further provided with a specifying means for specifying as an effective input position for inputting
The recognizing means is based on the fact that the distance from the operation surface of all the input positions specified as effective input positions for inputting the second operation by the specifying means is less than the first threshold value. The information processing apparatus according to claim 1, wherein the information processing apparatus recognizes a second operation, and the output control unit performs output for responding to the recognized second operation.   The recognition means recognizes that the distance from the predetermined operation surface of the input position is at least below the first threshold value as a touch input using the input position, and the first operation is a predetermined single touch. The information processing apparatus according to claim 1, wherein the second operation is a predetermined multi-touch operation.   The recognition means recognizes that the touch input using the input position has been released based on a state where the distance of the input position from the predetermined operation surface exceeds a third threshold value that is greater than the first threshold value. The information processing apparatus according to claim 5, characterized in that:   The specifying means has a fourth distance from the predetermined operation surface of at least one input position among a plurality of input positions specified as an effective input position for inputting the second operation by the specifying means. 5. The information processing apparatus according to claim 3, wherein when the threshold value is exceeded, the specification of all input positions specified as effective input positions for inputting the second operation is canceled.   In order to represent a specific result by the specifying means in a portion corresponding to a plurality of input positions specified as effective input positions for inputting the second operation by the specifying means on the predetermined operation surface The information processing apparatus according to claim 3, further comprising display control means for performing control.   The acquisition unit acquires three-dimensional position information of an end portion of a predetermined operation body from a distance image captured by a distance image sensor installed so as to look down on the predetermined operation surface from above. The information processing apparatus according to any one of claims 1 to 8.   The acquisition means includes, from the proximity sensor installed on the predetermined operation surface, two-dimensional position information parallel to the operation surface of the operation body existing in a range detectable by the proximity sensor, and from the operation surface. The information processing apparatus according to claim 1, wherein information corresponding to a distance is acquired.   The information processing apparatus according to claim 9 or 10, wherein the predetermined operation body is a human hand, and the input position is a position corresponding to a fingertip in the human hand. An acquisition step of acquiring information corresponding to a distance from the predetermined operation surface with respect to an input position input to a space on the predetermined operation surface by an acquisition unit;
Based on the information acquired by the recognition means, a first operation input using one input position whose distance from the predetermined operation surface is lower than a first threshold, and a distance to the predetermined operation surface is Recognizing a second operation that is input using a plurality of input positions below the first threshold,
In the recognition step, there are a plurality of input positions whose distance from the predetermined operation surface is less than a second threshold value that is greater than the first threshold value, and the distance from the predetermined operation surface is equal to the first threshold value. The information processing apparatus control method according to claim 1, wherein the first operation is not recognized when there is one input position below.   A program that causes a computer to operate as the information processing apparatus according to claim 1 by being read and executed by the computer.   A computer-readable storage medium storing the program according to claim 13. Obtaining means for obtaining information corresponding to the magnitude of pressure applied to the predetermined operation surface with respect to an input position input by contacting on the predetermined operation surface;
Using the information acquired by the acquisition means, the first operation input by one input position where the magnitude of the pressure applied to the predetermined operation surface exceeds a first threshold, and the distance to the predetermined operation surface is Recognition means for recognizing a second operation input by a plurality of input positions exceeding the first threshold;
An output control means for controlling an output for responding to the operation recognized by the recognition means;
The recognition means includes a plurality of input positions whose distance from the predetermined operation surface is less than a second threshold value that is smaller than the first threshold value, and the distance from the predetermined operation surface satisfies the first threshold value. An information processing apparatus characterized by not responding to the first operation while there is one more input position. An acquisition step of acquiring information corresponding to the magnitude of pressure applied to the predetermined operation surface with respect to an input position input by contacting the predetermined operation surface by an acquisition unit;
Using the information acquired by the recognition means, the first operation input by one input position where the magnitude of the pressure applied to the predetermined operation surface exceeds the first threshold, and the distance to the predetermined operation surface Recognizing a second operation input by a plurality of input positions exceeding the first threshold;
An output control step of controlling an output for responding to the operation recognized in the recognition step;
In the output control step, there are a plurality of input positions whose distance from the predetermined operation surface is less than a second threshold value that is smaller than the first threshold value, and the distance from the predetermined operation surface is the first threshold value. If the number of input positions exceeds 1, the information processing apparatus control method does not respond to the first operation.   A program that causes a computer to operate as the information processing apparatus according to claim 15 by being read and executed by the computer.   A computer-readable storage medium storing the program according to claim 17.

Images