JP2005078291A - Image projecting and displaying device, pointing position detecting method, program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image projecting and displaying device for, when a user indicates his or her desired position, detecting the position as a pointing position without using any special pointer, and for making a computer perform accurate processing. <P>SOLUTION: This image projecting and displaying device is provided with: a projecting device for projecting a desired image on a projection plane; an imaging means arranged at the side opposite to a user with respect to the projection plane and used for photographing, from the back side of the projection plane, the image projected on the projection plane by using an imaging part where pixels are two-dimensionally arrayed; and a processing means for detecting a pointing position from the shadow information of the finger of the user included in the image photographed by using the imaging means. This processing means detects that the shadow of the finger photographed by using the imaging means as that of a pointing purpose when a density value which is not less than a predetermined density value continues for a predetermined time or more. A cursor position on the projection plane is detected on the basis of the detected pointing position, and when the cursor points the image porjected on the projection plane, a function corresponding to the image is executed. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image projection display device, a pointing position detection method, a program, and a recording medium.

In recent years, presentation systems using computers have been widely used. In such a system, it is common practice to project and display the created material on a large screen using a projector, and directly input coordinates on the display screen to operate the cursor.
A touch sensor is widely used as a means for inputting directly to the display screen. When a finger or other pointing tool touches the screen, the touch sensor detects this, detects the touched position as a pointing position, and moves the cursor.
However, at present, the responsiveness of the touch sensor and the followability when the finger or pointing tool touched on the screen is moved are not good, and it is difficult to say that a comfortable screen operation is possible.

  As an alternative to the pointing position detection method using this touch sensor, the projected image is indicated by a laser pointer or the like, and the position indicated by the laser pointer is detected to execute, edit, enlarge or reduce the command of the computer main body. Pointing devices that can be used are attracting attention.

  For example, the technique disclosed in Patent Document 1 is an instruction unit that can indicate an arbitrary position on a plane within a predetermined range, a camera that captures the plane and outputs a signal corresponding to the captured image, and is output from the camera. Position detecting means for detecting the indicated position on the plane by the indicating means based on the received signal, and the indicated position on the large screen can be detected.

  However, when detecting the laser bright spot designated by the pointing means on the screen every predetermined time by the camera to detect the pointing position, when the image projected on the screen is a high brightness image There is a problem that it becomes difficult to detect the bright spot of the laser beam.

  Moreover, the technique of patent document 2 is an indicator which has a condensing optical system which condenses the light from the single or several light emitting element provided in the display screen or its vicinity on a photoelectric conversion element, and a photoelectric conversion element. An electronic device intended for a plurality of viewers, comprising a computer that calculates the direction of the axis of the indicator from the output signal and displays at least a point mark by the display device at a position corresponding to the axial direction. In presentation systems and electronic conferencing systems, the instructor can indicate the desired position on the display screen from an arbitrary position on the screen, large screen CRT, or other display screen. A pointing system can be used.

  However, this method lacks versatility because a light emitting element must be provided on the display screen in order to detect the position, and in order to calculate the detected position by detecting the directivity of the light intensity and its intensity. In addition, the light receiving element structure needs to be devised, and the problem is that the detected position accuracy on the screen is not high.

In order to solve these problems, the technique disclosed in Patent Document 3 is based on a projector for displaying an image, a CCD camera that captures an image of a display area, and a presenter included in the display area based on an imaging signal of the imaging unit. The present invention has a configuration including a shadow area of a (bar-shaped) pointing image or a processing unit that detects the tip of a bar-shaped shadow as a pointing position from a real image area, and does not require a special pointing tool. Indicates a desired position in the display area, this position can be accurately detected as a pointing position.
Japanese Patent Laid-Open No. 02-306294 Japanese Patent Laid-Open No. 06-308879 Japanese Patent Laid-Open No. 11-345087

  However, since the technique of Patent Document 3 recognizes a bar-shaped shadow, it may be a pointing operation for the purpose of operating a computer window or a pointing operation for the purpose of prompting the viewer to refer to a presentation material. Difficult to distinguish.

  The present invention has been made in consideration of the above-described circumstances. When a user points to a desired position in the display area without requiring a special pointing tool, the position is accurately set as a pointing position. It is an object of the present invention to provide an image projection display device, a pointing position detection method, a program, and a recording medium that can detect the above and cause a computer to perform accurate processing according to the purpose of the user.

  In order to solve the above-described problems, the invention of claim 1 is a projection device that projects a predetermined image on a projection surface, and an imaging unit in which pixels are two-dimensionally arranged on the image projected on the projection surface. An image pickup means for photographing and a processing means for detecting a pointing position from shadow information of a user's finger included in an image photographed by the image pickup means.

  According to a second aspect of the present invention, in the image projection display device according to the first aspect, the imaging unit is installed on the opposite side of the projection plane from the user, and images are taken from behind the projection plane. It is characterized by that.

  According to a third aspect of the present invention, in the image projection display device according to the first or second aspect, prior to using the device, the shadow of the user's finger within a predetermined range projected on the projection plane. A threshold value initial setting means for storing the shadow density information as a threshold value, and the processing means has a shadow density value of a user's finger photographed by the imaging means as the threshold value. When the above is true, the shadow is detected as a pointing purpose.

  According to a fourth aspect of the present invention, in the image projection display apparatus according to the first, second, or third aspect, the processing unit has a predetermined density value equal to or higher than a predetermined density value of a finger shadow photographed by the imaging unit. If it continues for more than a certain time, the shadow is detected as a pointing purpose.

  A fifth aspect of the present invention is the image projection display apparatus according to any one of the first to fourth aspects, further comprising cursor control means for controlling a position of a cursor included in an image projected by the projection apparatus, and the cursor The control means determines a cursor position on the projection plane based on the pointing position detected by the processing means.

  According to a sixth aspect of the present invention, in the image projection display device according to the fifth aspect, the function input means for projecting an image corresponding to a predetermined function on the projection surface, and the cursor controlled by the cursor control means are When pointing to an image corresponding to the function projected on the projection surface, the image processing apparatus has a function execution unit that executes a function corresponding to the image.

  According to a seventh aspect of the present invention, there is provided a projection device that projects a predetermined image onto a projection plane, and an imaging unit that captures an image projected on the projection plane with an imaging unit in which pixels are two-dimensionally arranged. When the position is detected, the projection plane is imaged by the imaging means, and a pointing position is detected from shadow information of a user's finger included in the captured image.

The invention of claim 8 is a program for causing a computer to operate as the image projection display device according to any one of claims 1 to 6.
A ninth aspect of the present invention is a computer-readable recording medium on which the program according to the eighth aspect is recorded.

According to the present invention, it is not necessary to use a special pointing tool or the like, and it is possible to accurately detect the pointing position by the shadow of the user's finger without providing a special sensor on the apparatus side. Can be made to perform accurate processing according to the purpose of the user.
Further, by detecting the detection target other than the shadow of the finger as noise, erroneous detection due to environmental conditions or the like can be prevented.
Further, when the shadow density of the user's finger is stored as an initial setting, a threshold value can be set in accordance with the installation location of the apparatus, so that a more accurate pointing position can be detected.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments according to an image projection display device of the invention will be described with reference to the drawings.
<Embodiment 1>
FIG. 1A shows a configuration using a rear projection device as an image projection display device.
This image projection display device is provided in a display area 1 on which an image projected by a projector (projection apparatus) 2 is displayed, substantially in front of the back of the display area 1, and on a display area 1 which is a projection surface, a predetermined area. An imaging means comprising a projector 2 that projects an image for presentation and a CCD camera or the like that is provided almost in front of the back surface of the display area 1 and that images the display area 1 from the back surface with an image pickup unit in which pixels are two-dimensionally arranged. 3 and a processing device 4 for detecting a position pointed to by the user 5 with the finger 6, selecting an image to be displayed on the projector 2, and executing various applications.

  Alternatively, as shown in FIG. 1B, the projector 2 may be provided almost in front of the front surface of the display area 1, and a predetermined presentation image may be projected from the front surface of the display area 1.

When the user 5 makes a presentation using this image projection display device, the user 5 instructs the processing device 4 to select a desired image and supply it to the projector 2. The image is projected.
The user 5 performs the screen operation while pointing the screen displayed on the display area 1 with the finger 6.
The processing device 4 displays an image, and at the same time, the imaging unit 3 captures an image from the back of the display area 1 and detects the pointing position from the shadow information of the finger 6 included in the captured image.

  At this time, as shown in FIG. 2, the shadow 8 of the user 5 is reflected in the image 7 photographed by the imaging means 3. The shadow 9 of the finger 6 that is pointed is shown in the image as the darkest shadow. Accordingly, when a black spot having a predetermined size appears at a density equal to or higher than a predetermined threshold in the image obtained by photographing the display area 1, it is determined that the black spot is a shadow of the finger, and the user's finger is recognized. Detect the pointing position.

  Predetermined data processing is performed based on the detection result. One example of such data processing is cursor control. For example, as shown in FIG. 1, when the user 5 uses the finger 6 to indicate a desired position of the display area 1, the cursor included in the display image of the display area 1 moves following the pointing position of the finger 6. It will be.

  In addition, it is conceivable that the user 5 touches the display area 1 for a purpose other than the pointing purpose, but this is the same in a certain area 10 around the recognized black spot as shown in FIG. If there is a shadow of level density, it is not considered as a pointing purpose. As a result, a shadow other than a finger or a shadow with a low density on the display area 1 such as the arm or body of the user 5 is determined as noise, and only the shadow of a finger with a high density is recognized as a pointing. The pointing position can be detected.

  When applied to a table-type image projection display device as shown in FIG. 4, it is conceivable that the user puts his hand on the screen or places an object. At this time, if pointing is detected only by the presence or absence of a shadow, a malfunction may be caused. However, by excluding a shadow other than one finger, an appropriate operation can be performed.

FIG. 5 is a block diagram illustrating a functional configuration of the processing device 4. In the figure, the processing device 4 includes at least an image capturing means 41, a processing means 50, a cursor control means 42, and an image display means 43.
The image capturing unit 41 controls the image capturing unit 3 to capture an image projected on the display area 1 and supply it to the processing unit 50.

  The processing means 50 controls the projector 2 to project an image for presentation on the display area 1, and controls the imaging means 3 to use the image included in the image of the display area 1 taken from the back of the display area 1. The pointing position of the finger 6 of the person 5 is automatically recognized.

The cursor control means 42 supplies a cursor position control signal to the image display means 43 based on the detected pointing position, and the image display means 43 includes an image representing the cursor at the indicated position so as to include the display area 1. Project to.
For example, as shown in FIG. 6, the cursor 12 indicated by an arrow is displayed so as to move following the shadow 11 of the finger indicated by a black dot. In this way, the cursor can be moved following the position pointed by the user's finger, so that the cursor position can be easily recognized.

  The image display means 43 generates a presentation document or image instructed by the user as an image signal, supplies the generated image signal to the projector 2, and the projector 2 modulates light by the image signal. , Projected onto the display area 1.

The pointing position detection process in the processing unit 50 will be described with reference to the flowchart of FIG.
First, the aperture and gain setting values optimal for extracting the shadow region of the finger are set as reference values for position detection. Next, the image capture means 41 controls the image capture means 3 based on this reference value, captures an image of the display area 1 and captures it in the memory.

Extracts the shadow area of the finger from the captured image data stored in the memory, and the shadow size, the center coordinates (or the coordinate value of the center of gravity of the density, the coordinate value with the highest density, etc.), the density value of the shadow Is taken out (step S10).
If the size of the shadow is out of the predetermined size range (size <A1 or size> A2) (NO in step S11), the process ends as it is not a finger shadow. In this range, the minimum value (A1) and the maximum value (A2) are determined from the average size of the thickness of human fingers.

  If the shadow size is within a predetermined range (A1 <size <A2)) (YES in step S11) and the shadow density is smaller than the predetermined density value (B) (NO in step S12). Then, the process is ended as not being the shadow of the finger. This predetermined density value (B) is determined by experiment.

If the shadow density is greater than the predetermined density value (B) (YES in step S12), it is regarded as a finger shadow, and there is a shadow with the same level of density in a certain region around the shadow. (YES in step S13), it is determined that it is not pointing with a finger, and the process is terminated.
On the other hand, if there is no shadow of the same level in a certain area around the black spot regarded as a shadow (NO in step S13), it is assumed that the finger is pointing and the coordinates of the center of the finger shadow (or The coordinate value of the density centroid, the coordinate value with the highest density, etc.) may be detected as the pointing position.

  The detected data is output to the cursor control means 42. By repeatedly performing from this image capture to pointing position detection, the cursor can be moved in accordance with the movement of the shadow of the finger.

The predetermined density value (B) described above can also be determined by the threshold initial setting means as follows. Prior to using the image projection display apparatus, for example, when the user turns on the apparatus, a frame 14 indicating a predetermined area is displayed on the display area 13 as shown in FIG. When the user points in the frame with his / her finger and the display area 13 is photographed by the imaging means from the back, a finger shadow 16 is photographed in the photographed image 15 as shown in FIG.
An average value of the density of the shadow of the finger is obtained from this image, and this value is set as the above-mentioned predetermined density value (B) and stored in the memory as an initial setting of the threshold value. The density value B is set to an appropriate value such as the minimum density value or the maximum density value of the finger shadow.
By performing the initial setting in this way, a threshold corresponding to the situation can be set, so that pointing detection becomes more accurate.

  When the pointing position is detected by the method as described above, by placing the imaging means 3 on the back side of the display area 1 (on the opposite side of the display area from the user), the shadow of the fingertip to be pointed is displayed in the display area 1 However, the projector 2 can accurately detect the pointing position regardless of the front projection or the rear projection. In addition, by fixing the imaging unit 3 on the back side, the coordinate axis of the finger shadow on the display area 1 does not change, so that it becomes easy to reflect the position information of the pointing on the projected image.

By using the pointing by the shadow of the finger instead of the mouse, a more comfortable operation can be performed without the need for a mouse.
For example, consider a case where a computer screen 17 as shown in FIG. 10 is projected on the display area 1 and a dialog 18 having a predetermined function is displayed thereon (function input means). The dialog 18 in FIG. 10 includes buttons for scrolling up and down the window (19, 20), left and right scrolling (21, 22), and maximizing and minimizing the window (23, 24). Each window operation is set to be performed when each button is clicked.

If the coordinates on the screen for displaying the dialog 18 are fixed, the coordinates on the screen of each button are also fixed.
Therefore, when it is determined that the shadow of the finger has been pointed and the coordinates of the pointing position are passed to the cursor control means 42, an event assigned to the button corresponding to the area where the coordinates exist is generated, and the computer is Processing corresponding to that is performed (function execution means).

Not only the dialog for window operation as described above, but an icon corresponding to the application software is displayed, and when the shadow of the finger is pointed at the position of the icon, the application software corresponding to the icon is activated. Can also be applied.
Further, the above dialog can be provided with more necessary functions as a pop-up structure without being limited to buttons.
Furthermore, instead of placing the dialog display location at a fixed location, buttons can be provided along the four sides of the screen, so that it is possible to prevent the dialog from taking up more space than necessary.

<Embodiment 2>
As in the first embodiment, simply detecting the pointing position by determining the presence or absence of a shadow area appearing in the display area may cause a malfunction that is not intended by the user.
For example, there may be a case where the pointing position is detected before the finger touches the display area.

  In the second embodiment, accurate pointing is recognized by making use of a change in shadow temporal density. Since the second embodiment has the same configuration as that of the first embodiment, only differences will be described.

As the finger approaches the display area, the shadow density increases over time, and the density level reaches its peak while pointing, and when pointing ends, the finger moves away from the display area, and again the density level. (See FIG. 11).
As shown in FIG. 11, when the peak state is maintained until the density level of the finger shadow exceeds a predetermined time T, it is regarded as a pointing operation, and the processing can be performed more accurately by performing the processing. The predetermined time T is set to an appropriate value such as a time that is generally considered that the user is pressing the button with a mouse or the like, for example, 0.5 seconds to 1 second.
This also improves the performance of screen operations such as accurate button press operations.
Here, when the threshold value initial setting means initially sets the threshold value of the density value of the finger shadow, all density values exceeding the initial set value may be handled as peak values.

Another example of the pointing position detection process in the processing unit 50 will be described with reference to the flowchart of FIG.
First, the aperture and gain setting values optimal for extracting the shadow region of the finger are set as reference values for position detection. Next, the image capture means 41 controls the image capture means 3 based on this reference value, captures an image of the display area 1 and captures it in the memory.

Extract the shadow area of the finger from the captured image data stored in the memory, extract the shadow density, and if it is not equal to the shadow density previously stored in the state memory, the size and center of the extracted shadow The coordinates (or the coordinate value of the center of gravity of the density, the coordinate value with the highest density, etc.) and the date and time when this data was taken in are stored in the state memory. However, if the extracted shadow density is the same as the density value stored in the state memory, the state memory is not updated and is left as it is (step S20).
Here, when the initial setting value (B) of the density of the finger shadow described in the first embodiment is stored in the memory, the density of the extracted shadow of the finger is larger than the value of the initial setting value (B). Sometimes they may all be treated as the same density value.

  If the size of the shadow is out of the predetermined size range (size <A1 or size> A2) (NO in step S21), the process ends as it is not a finger shadow. In this range, the minimum value (A1) and the maximum value (A2) are determined from the average size of the thickness of human fingers.

  If the shadow size is within a predetermined size range (A1 <size <A2) (YES in step S21), and the shadow density is smaller than the predetermined density value (B) (NO in step S22), The process is terminated assuming that it is not a shadow of a finger. The predetermined density value (B) may be determined by experiment, or an initial set value obtained prior to use as described in the first embodiment may be used.

If the shadow density is greater than the predetermined density value (B) (YES in step S22), it is regarded as a finger shadow, and there is a shadow with the same level of density in a certain area around the shadow. (YES in step S23), it is determined that the pointing is not performed with a finger, and the process is terminated.
On the other hand, if there is no shadow with the same level of density within a certain area around the black spot regarded as a shadow (NO in step S23), if the shadow density time (t) is equal to or shorter than the predetermined time (T). (NO in step S24), it is determined that it is not a pointing operation, and the process is terminated.

Here, the shadow density time (t) is calculated by the following equation.
t = (date and time when image data being processed is taken out) − (date and time when image data previously stored in the state memory is taken out)

  On the other hand, if the shadow density time (t) is equal to or longer than the predetermined time (T) (YES in step S24), the state memory is cleared and the coordinates of the center of the shadow of the finger (or The coordinate value of the density centroid, the coordinate value with the highest density, etc.) may be detected as the pointing position (step S25).

  The detected data is output to the cursor control means 42. By repeatedly performing from this image capture to pointing position detection, the cursor can be moved in accordance with the movement of the shadow of the finger.

<Other embodiments>
The present invention is not limited only to the above-described embodiments. Each function constituting the image projection display device of the above-described embodiment is programmed, written in advance in a recording medium such as a ROM, the ROM is mounted on the image projection display device, and these programs are executed by a microprocessor. Needless to say, the object of the present invention is achieved.
In this case, the state read and executed from the recording medium realizes the functions of the above-described embodiment, and the program and the recording medium on which the program is recorded also constitute the present invention.

Note that a program that realizes such a function includes a semiconductor medium (eg, ROM, nonvolatile memory, etc.), an optical medium (eg, DVD, MO, MD, CD, etc.), a magnetic medium (eg, magnetic tape, flexible disk, etc.). Etc.) may be provided in any form of recording medium.
Alternatively, the program stored in the storage device may be directly supplied from the server computer via a communication network such as a network. In this case, the storage device of this server computer is also included in the recording medium of the present invention.

  When provided by such a recording medium, the function of the above-described embodiment is obtained by reading the program from the recording medium and installing the program in the internal storage device or the external storage device, and executing the installed program by the microprocessor. Is realized. Alternatively, the program recorded on the recording medium may be directly executed.

It is a block diagram which shows the image projection display apparatus of this invention. It is an example of the image image | photographed with the imaging means. It is an example which shows the range when determining whether the shadow of a finger | toe is for the purpose of pointing. It is an example of a table-type image projection display device. It is a block diagram which shows the function structure of a processing apparatus. It is a display example of a cursor when the shadow of a finger is used for cursor control. It is a flowchart which shows the process sequence of the pointing position detection process in a process means. It is explanatory drawing when the density of the shadow of a finger is initialized. FIG. 8 shows a photographed image when the user places his / her finger. It is explanatory drawing when operating a function (execution of application software, window operation control, etc.) using the shadow of a finger for cursor control. It is a figure which shows the time passage of the density | concentration of the shadow of a finger | toe until a finger approaches a display area and leaves | separates. It is a flowchart which shows the other process sequence of the pointing position detection process in a process means.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Display area, 2 ... Projector, 3 ... Imaging means, 4 ... Processing apparatus, 5 ... User, 6 ... Finger, 41 ... Image capture means, 42 ... Cursor control means, 43 ... Image display means, 50 ... Processing means.

Claims (9)

  Included in a projection device that projects a predetermined image on a projection plane, an imaging unit that captures an image projected on the projection plane with an imaging unit in which pixels are two-dimensionally arranged, and an image captured by the imaging unit An image projection display device comprising processing means for detecting a pointing position from shadow information of a user's finger.   The image projection display device according to claim 1, wherein the imaging unit is installed on a side opposite to a user with respect to the projection plane, and images are taken from behind the projection plane. Projection display device.   3. The image projection display device according to claim 1, wherein a shadow of a user's finger within a predetermined range projected on the projection plane is photographed by the imaging unit prior to using the device. A threshold initial setting means for storing the shadow density information as a threshold, and the processing means, when the shadow density value of the user's finger photographed by the imaging means is equal to or greater than the threshold, An image projection display device, wherein a shadow is detected as a pointing purpose.   4. The image projection display device according to claim 1, wherein when the shadow of the finger imaged by the imaging unit continues a density value equal to or higher than a predetermined density value for a predetermined time or more, An image projection display device, wherein a shadow is detected as a pointing purpose.   5. The image projection display device according to claim 1, further comprising cursor control means for controlling a position of a cursor included in an image projected by the projection apparatus, wherein the cursor control means is the processing means. An image projection display apparatus, wherein a cursor position on the projection plane is determined based on the pointing position detected in step (b).   6. The image projection display device according to claim 5, wherein a function input means for projecting an image corresponding to a predetermined function onto the projection surface, and a function in which a cursor controlled by the cursor control means is projected onto the projection surface. An image projection display device comprising function execution means for executing a function corresponding to an image when pointing to the image corresponding to the image.   A projection apparatus that projects a predetermined image on a projection plane; and an imaging unit that captures an image projected on the projection plane with an imaging unit in which pixels are two-dimensionally arranged. A pointing position detection method, wherein the projection plane is imaged by an imaging means, and a pointing position is detected from shadow information of a user's finger included in the captured image.   The program for operating a computer as an image projection display apparatus in any one of Claims 1 thru | or 6.   A computer-readable recording medium on which the program according to claim 8 is recorded.

Images