JP2010224666A - Document display control device and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To reproduce the state of a face, to which a medium is attached, so that it is possible to understand an electronic document printed on the medium. <P>SOLUTION: In a document server 30, a reception part 31 receives an imaged image obtained by imaging a whiteboard, to which a medium is attached or an image is projected, from a projection imaging device, a difference information generation part 33 generates a difference image by calculating a difference between imaged images before and after one meaningful operation, a document retrieval part 35 compares the layout of the difference image with the layout of an electronic document to specify the electronic document, which is printed on the medium or which is the original of an image projected, and to generate display information obtained by arranging an element object, in which a link to the electronic document is embedded, at a position corresponding to the position of the difference image in the imaged image on a screen object showing the whiteboard, and a transmission part 39 transmits the display information to a terminal device. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a document display control device and a program.

A technique that enables reflection of a conference using a whiteboard is known (see, for example, Patent Documents 1 and 2).
In the technique of Patent Document 1, each operation performed on the whiteboard is stored together with the elapsed time, and each operation is set to a time difference between the elapsed time corresponding to the operation and the elapsed time corresponding to the previous operation. If it is longer, it is reproduced when the set time has elapsed since the time when the previous operation was reproduced. Otherwise, it is reproduced when the time difference has elapsed since the time when the previous operation was reproduced. ing.
In the technique of Patent Document 2, a captured image obtained by capturing a whiteboard with a video camera, a drawing command of an annotation image to be added to the captured image, a time, and the like are stored, and a captured image and a drawing command designated by the time or the like by a restoration command The projected image created from is transferred to the projector and projected onto the whiteboard.

JP 2007-066631 A JP 2008-258732 A

  An object of the present invention is to reproduce the state of a surface on which a medium is pasted so that an electronic document printed on the medium can be seen.

The invention according to claim 1 is an acquisition unit that acquires a captured image obtained by imaging a surface on which a medium on which an electronic document is printed is pasted, and the captured image acquired by the acquisition unit. Extracting means for extracting a partial image of a portion to which the medium is pasted, specifying means for specifying the electronic document based on the partial image extracted by the extracting means, and the imaging acquired by the acquiring means An instruction means for instructing display of a screen including a display element associated with the electronic document specified by the specifying means at a position of the partial image extracted by the extracting means. A document display control device characterized by being provided.
The invention according to claim 2 is characterized in that the acquisition means obtains a plurality of captured images obtained by capturing the surface each time a medium is attached and a plurality of times when the plurality of captured images are obtained. A plurality of time information shown, and the instruction means is a plurality of screens respectively representing the plurality of captured images acquired by the acquisition means, and each screen is acquired by the acquisition means. 2. The document display control apparatus according to claim 1, wherein a display of a plurality of screens associated with time information indicating a time at which a captured image represented by each screen is obtained is instructed to be displayed. .
The invention according to claim 3 is characterized in that the instruction means instructs display of a screen associated with time information indicating a specified time among the plurality of screens. This is a document display control device.
According to a fourth aspect of the present invention, the acquisition unit acquires a captured image obtained by capturing the surface on which an image of an electronic document is projected by a projection device, and the extraction unit includes the projection device. 4. A partial image of a portion onto which an image of the electronic document is projected is extracted from the captured image acquired by the acquisition unit based on the projected image of the electronic document. The document display control device according to any one of the above.
According to a fifth aspect of the present invention, the specifying means includes: a shape of a region in which the document element of the electronic document in the partial image is disposed; and a shape of a region in which the document element of the electronic document is disposed in the electronic document. The document display control apparatus according to claim 1, wherein the electronic document is specified based on
The invention described in claim 6 is characterized in that the instruction means instructs display of the screen including the display element in which information for referring to the electronic document is embedded. The document display control apparatus according to any one of the above.
The invention according to claim 7 is characterized in that the instruction means instructs display of the screen including the display element on which the image of the electronic document is superimposed. This is a document display control device.
According to an eighth aspect of the present invention, the computer has a function of acquiring a captured image obtained by capturing an image of a surface on which a medium on which an electronic document is printed is pasted on a computer, and the medium is pasted from the captured image. A function of extracting a partial image of a part, a function of specifying the electronic document based on the partial image, and a screen representing the captured image, the position of the partial image being associated with the electronic document And a function for instructing display of a screen including the displayed elements.

According to the first aspect of the present invention, the state of the surface on which the medium is affixed can be reproduced so that the electronic document printed on the medium can be seen.
According to invention of Claim 2, the state of the surface where the medium was stuck can be reproduced so that the time when the state was maintained can be known.
According to invention of Claim 3, the state in the designated time of the surface where the medium was affixed can be reproduced.
According to the invention of claim 4, compared with the case where the present configuration is not provided, the process of reproducing the state of the surface on which the image is projected so that the electronic document that is the source of the image can be understood efficiently. It can be carried out.
According to the invention of claim 5, even when the surface is imaged from such a distance that the surface on which the medium is affixed can be imaged, the electronic document printed on the medium can be specified.
According to the invention of claim 6, it is possible to refer to the electronic document printed on the medium without complicating the screen for reproducing the state of the surface on which the medium is pasted.
According to the seventh aspect of the present invention, it is possible to refer to the electronic document printed on the medium without forcing unnecessary operations on the screen that reproduces the state of the surface on which the medium is pasted.
According to invention of Claim 8, the state of the surface where the medium was stuck can be reproduced so that the electronic document printed on the medium can be understood.

It is the figure which showed the system configuration | structure with which embodiment of this invention is applied. It is the figure shown about the projection imaging device in embodiment of this invention. It is the block diagram which showed the function structure of the document server in embodiment of this invention. It is the figure which showed the state on a white board used by operation | movement description of embodiment of this invention. It is the flowchart which showed operation | movement of the projection imaging device in embodiment of this invention. It is the figure which showed the example of the imaging information recorded with the projection imaging device in embodiment of this invention. It is the flowchart which showed operation | movement of the difference information generation part of the document server in embodiment of this invention. It is the figure which showed the example of the difference information produced | generated by the difference information production | generation part of the document server in embodiment of this invention. It is the flowchart which showed operation | movement of the document search part of the document server in embodiment of this invention. It is the figure which showed the example of the display information produced | generated by the document search part of the document server in embodiment of this invention. It is the figure which showed the example of the screen displayed on a terminal device based on the display information in embodiment of this invention. It is a hardware block diagram of the computer which can implement | achieve embodiment of this invention.

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.
First, the scene where application of the embodiment of the present invention is assumed will be described.
For example, a meeting may be held by projecting an image on a wall or pasting a paper document. In such a conference environment, as a method for recording the content of the conference, there is a method of photographing a wall surface and leaving it with a photograph.
However, when a wall surface on which an image is projected or a paper document is pasted is photographed, only a photograph in which the projected image, the pasted paper document, and the wall surface are all contained in one image is obtained.
Therefore, there is a limit even if it is attempted to return to his / her seat and perform work related to the conference after the conference. In other words, if you want to view or edit any page of a paper document that has been pasted on the wall, you can visually inspect the part of the target document from the photo showing the entire wall and guess the file name and storage location. Must get. Moreover, there is no guarantee that it is the target document file.
Therefore, in the present embodiment, when a meeting is performed by projecting an image on a wall surface or pasting a paper document, an electronic document corresponding to the projected image or paper document is reliably identified, and the personal computer (Personal Computer) etc. can handle this electronic document.

First, the overall configuration of the computer system in the present embodiment will be described.
FIG. 1 shows a configuration example of a computer system according to the present embodiment.
As shown in the figure, this computer system is configured by connecting a terminal device 10, an image forming device 20, a document server 30, a projection imaging device 40, and a terminal device 50 to a network 80.

The terminal device 10 is a computer device that instructs the image forming apparatus 20 to print an electronic document stored in the document server 30. Here, for example, a personal computer, a workstation, or another computer may be used as the terminal device 10.
The image forming apparatus 20 is an apparatus that prints an image on a medium and outputs it as a print document. The image forming apparatus 20 may be a single printer or printing machine, or may be a so-called multifunction machine having a scanner or a communication function. Here, as an image forming method in the image forming apparatus 20, for example, an electrophotographic method may be used, but other methods may be used.
The document server 30 is a computer device that stores and manages electronic documents. In the present embodiment, control is performed to display on the terminal device 50 an electronic document printed on a medium pasted on a wall surface or an electronic document that is the basis of an image projected on the wall surface. That is, the document server 30 is provided as an example of the document display control device. Here, as the document server 30, for example, a personal computer, a workstation, or another computer may be used.

The projection imaging device 40 is a device having both a projector function for projecting an image on a wall surface and a camera function for imaging a subject. The projector function part is an example of a projection apparatus, and the camera function part is an example of an imaging apparatus. In this apparatus, the image projection destination by the projector function may be not only a wall surface, but also a whiteboard, an electronic blackboard, a surface with imitation paper or the like attached to the wall surface, and the like. In addition, since the projection destination of the image by the projector function is a subject imaged by the camera function, this subject may also be a wall surface, a whiteboard, an electronic blackboard, a surface with imitation paper or the like attached to the wall surface, and the like. Here, the whiteboard is a white board used to write characters and figures with a dedicated marker and does not have a function to digitize written letters and diagrams. A white board similar to a board, which has the function of digitizing written characters and drawings. Here, although the projection imaging device 40 having both the projector function and the camera function is used, a separate projection device and imaging device may be used.
The terminal device 50 is a computer device that displays a screen generated based on a wall surface image captured by the projection imaging device 40. Here, as the terminal device 50, for example, a personal computer, a workstation, or another computer may be used.

In this specification, electronic data that is the basis of an image to be recorded on a medium is referred to as “electronic document”. However, this does not mean only data obtained by digitizing a “document” including text. For example, "electronic document" including image data such as pictures, photos, figures (regardless of raster data or vector data), data recorded by database management software or spreadsheet software, and other printable electronic data It is said.
In the present specification, the “medium” may be any material as long as it can print an image. A typical example is paper, but an OHP sheet or a metal plate may be used.

Next, among the devices constituting such a computer system, the configurations of the projection imaging device 40 and the document server 30 that operate particularly in the present embodiment will be described. In the following description, it is assumed that the projection destination of the image by the projection imaging apparatus 40 and the subject are white boards.
FIG. 2 is a diagram illustrating the projection imaging apparatus 40.
First, (a) shows the appearance of the projection imaging apparatus 40.
As illustrated, the projection imaging apparatus 40 is an apparatus in which a projector and a camera are integrated, and includes a projection unit 46 corresponding to a projector and an imaging unit 47 corresponding to a camera.

Further, (b) shows a functional configuration example of the projection imaging apparatus 40.
As illustrated, the projection imaging apparatus 40 includes a control unit 41, a memory 42, an operation unit 43, a liquid crystal monitor 44, and a communication interface (I / F) 45. Further, as described above, the projector 46 and the imaging unit 47 are provided.

The control unit 41 controls various operations in the projector function and the camera function based on the control program. The control program is stored in a ROM (not shown).
The memory 42 is a non-volatile memory that stores data processed by the control unit 41. In the present embodiment, in particular, an image (captured image) captured when an image on the whiteboard changes due to pasting of a medium on the whiteboard, projection of an image, handwriting, and the time (captured) Information associated with (time) is stored (hereinafter referred to as “imaging information”).
The operation unit 43 sends an operation signal corresponding to the operated button or the like to the control unit 41.
The liquid crystal monitor 44 displays an image, text or the like according to an instruction from the control unit 41.
The communication I / F 45 transmits / receives data to / from the document server 30 via the network 80 (see FIG. 1) according to an instruction from the control unit 41.

  The projection unit 46 includes a projection control circuit 460, an LED light source 461, a liquid crystal panel 462, and a projection optical system 463. The projection control circuit 460 sends control signals to the LED light source 461 and the liquid crystal panel 462 according to instructions from the control unit 41. The LED light source 461 illuminates the liquid crystal panel 462 with brightness according to the supply current. The liquid crystal panel 462 constitutes a modulation element, and generates a light image (modulates illumination light) in accordance with a drive signal from the projection control circuit 460. The projection optical system 463 projects the modulated light emitted from the liquid crystal panel 462.

  The imaging unit 47 includes an imaging control circuit 470, an imaging optical system 471, and an imaging element 472. The imaging control circuit 470 controls driving of the imaging device 472 according to an instruction from the control unit 41 and performs signal processing on an image signal output from the imaging device 472. The image after signal processing is stored in the memory 42 as an image file. The imaging optical system 471 forms a subject image on the imaging surface of the imaging element 472. As the image sensor 472, a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS) image sensor, or the like is used.

FIG. 3 is a diagram illustrating a functional configuration example of the document server 30.
As illustrated, the document server 30 includes a receiving unit 31, an imaging information storage unit 32, a difference information generation unit 33, a difference information storage unit 34, a document search unit 35, a display information storage unit 36, and a transmission. Part 39.

The receiving unit 31 receives imaging information stored in the memory 42 of the projection imaging apparatus 40 at an arbitrary time. In the present embodiment, the receiving unit 31 is provided as an example of an acquisition unit that acquires a captured image.
The imaging information storage unit 32 stores imaging information received by the receiving unit 31.
The difference information generation unit 33 performs the process of subtracting the previous imaging information from the subsequent imaging information for the imaging information stored in the imaging information storage unit 32, thereby adding an image added to the previous captured image. Difference information including (difference image) and elapsed time (difference time) from the recording time of the previous image is generated. In the present embodiment, a difference image is used as an example of a partial image, and a difference information generation unit 33 is provided as an example of an extraction unit that extracts a partial image.
The difference information storage unit 34 stores the difference information generated by the difference information generation unit 33.

The document search unit 35 searches for an electronic document that is a source of the difference image included in the difference information. In the present embodiment, a document search unit 35 is provided as an example of a specifying unit that specifies an electronic document.
The display information storage unit 36 stores display information for displaying the object corresponding to the electronic document searched by the document search unit 35 in a state in which the position of the difference image in the captured image is reflected.
The transmission unit 39 reads the display information stored in the display information storage unit 36 and transmits it to the terminal device 50. In the present embodiment, a transmission unit 39 is provided as an example of an instruction unit that instructs display of a screen.

Next, the operation of the present embodiment will be described.
In this embodiment, a medium on which an electronic document is printed is pasted on a whiteboard. An operation at the time of printing an electronic document on such a medium will be described.
First, the terminal device 10 designates an electronic document to be printed, and transmits an electronic document print command to the image forming apparatus 20. As a result, the image forming apparatus 20 prints the image of the electronic document on the medium based on the print command. At this time, in the present embodiment, the terminal device 10 creates an electronic document representing a print image in the image forming apparatus 20 and uploads the electronic document to a pasted document folder of the document server 30. .
In this embodiment, an image of an electronic document is projected onto a whiteboard. Such an electronic document to be projected is also uploaded to a projected document folder in the document server 30 in advance.

Here, a specific example of pasting a medium on a whiteboard, projecting an image, and handwriting used in the description of the operation of the present embodiment will be described.
FIG. 4 is a diagram showing such a specific example in time order. In each of the drawings (a) to (e), a rectangle surrounded by an outer frame indicates a whiteboard, and a rectangle surrounded by a thin solid line frame in the whiteboard is a pasted medium. A rectangle surrounded by a thick solid line frame in the whiteboard indicates the range where the image is projected, and a thick solid line that does not constitute a frame in the whiteboard indicates a handwritten locus .

In (a), medium A is affixed to the whiteboard.
In (b), an image is projected on the whiteboard.
In (c), medium B is affixed to the whiteboard.
In (d), medium C is pasted over medium A on the whiteboard.
In (e), handwriting is performed from medium B to medium C on the whiteboard.
In this example, it is assumed that the electronic documents are printed on the media A, B, and C and stored in the pasted document folder of the document server 30 as described above. The projected image is an electronic document image as described above, and this electronic document is stored in the projected document folder of the document server 30.

First, the operation of the projection imaging apparatus 40 in the present embodiment will be described.
FIG. 5 is a flowchart showing an operation example of the projection imaging apparatus 40.
In the projection imaging apparatus 40, first, the control unit 41 assigns “0” to a variable N indicating an index of imaging information (step 401). Next, the imaging unit 47 is controlled to image the initial state of the whiteboard (step 402). Then, the captured time is recorded as T (0) and the captured image is recorded as P (0) in the memory 42 (step 403).

Thereafter, the control unit 41 repeats the process of imaging the whiteboard and recording if there is a change until an end instruction is issued.
That is, first, the control unit 41 determines whether or not a predetermined time T1 has passed as an imaging interval (step 404), and if the time T1 has not elapsed, the determination in step 404 is repeated and the time T1 is repeated. If elapses, it is determined whether or not there is an end instruction (step 405).
If there is no end instruction, the imaging unit 47 is controlled to image the current state of the whiteboard (step 406). Then, the most recently recorded captured image is compared with the captured image obtained by the current imaging to determine whether there has been a change (step 407).

As a result, if there is no change, the process proceeds to step 404.
On the other hand, even if there is a change, it is not necessarily the change due to the medium being pasted on the whiteboard, the image being projected, or the handwriting being performed. For example, the movement of a presenter or the like in front of the whiteboard may be recognized as a change in the captured image. Therefore, the control unit 41 determines whether the change recognized in step 407 is on the whiteboard (step 408). Specifically, the determination may be made by setting the focus of the imaging unit 47 so as to match the whiteboard and analyzing the sharpness of the changed portion. That is, when the sharpness is high to some extent, it is determined that the change is on the whiteboard, and when the sharpness is lower than that, it is determined that the change is not on the whiteboard.

If the change is not on the whiteboard, go to step 404.
If the change is on the whiteboard, the control unit 41 performs necessary image processing prior to recording the captured image. That is, if there is a presenter in front of the whiteboard, the image of the presenter is removed (step 409). The removed portion is supplemented by using, for example, the latest recorded image of the captured image including the portion on the whiteboard (step 410). Thereafter, “1” is added to the variable N (step 411), the process returns to step 403, and the imaging time T (N) and the captured image P (N) are recorded.
Such a process is continued until an end instruction is issued, and if it is determined in step 405 that an end instruction has been issued, the process ends.

FIG. 6 shows an example of imaging information recorded by the operation of FIG. 5 when the operations shown in FIGS. 4A to 4E are performed on the whiteboard.
The imaging time T (N) is the time when the captured image P (N) was actually recorded.
As for the captured image P (N), regarding the pasting of the medium, the pasting of one medium is regarded as one change, and one captured image P (N) is recorded every time the medium is pasted. (N = 1, 3, 4). Further, regarding the projection of an image, one projection or movement of the projection destination is regarded as one change, and one captured image P (N) is recorded every time the projection is performed or the projection destination is changed. (N = 2). On the other hand, handwriting requires a certain amount of time from the start to the end, and this time may reach several times the imaging interval T1. Therefore, here, handwriting is regarded as a plurality of changes, and the captured image P (N) is recorded for each change (N = 5 to 7).

  As for the pasting of the medium and the projection of the image, a plurality of captured images P (N) may be recorded for the pasting of one medium and the projection of one image. For example, when a medium is pasted, a part of the medium is hidden by a presenter or the like. In this case, the hidden portion may gradually appear according to the movement of the presenter or the like, so that it is recognized as a change each time and is recorded as the captured image P (N).

  The imaging information recorded as described above is sent to the document server 30 at an arbitrary timing. For example, it may be sent when a predetermined time elapses after the recording of imaging information ends, or it may be sent when there is an explicit instruction from the outside after the recording of imaging information ends. Also good.

Next, the operation of the document server 30 in the present embodiment will be described.
First, in the document server 30, the reception unit 31 receives the imaging information sent in this way and stores it in the imaging information storage unit 32.
Thereafter, the difference information generation unit 33 generates difference information based on the imaging information stored in the imaging information storage unit 32.
FIG. 7 is a flowchart showing an operation example of the difference information generation unit 33.
First, the difference information generation unit 33 assigns “0” to the variable N used in FIG. 5 and assigns “0” to the variable M for counting the number of one meaningful operations on the whiteboard ( Step 301). Here, one meaningful operation means, for example, an operation of pasting one medium, an operation of performing projection once, an operation of performing continuous handwriting, and the like. Then, the imaging time T (0) and the captured image P (0) are read from the imaging information storage unit 32 (step 302).

Next, the difference information generation unit 33 generates difference information based on the imaging time T (N) and the captured image P (N) as long as the imaging time T (N) and the captured image P (N) are present. Process.
That is, first, the difference information generation unit 33 adds “1” to the variable N (step 303), and determines whether or not there is a correspondence between the imaging time T (N) and the captured image P (N) (step). 304).

  If there is a correspondence between the imaging time T (N) and the captured image P (N), the imaging time T (N) and the captured image P (N) are read from the imaging information storage unit 32 (step 305). Then, by subtracting the previous imaging time T (N−1) from the current imaging time T (N), the first difference time DT (N) is obtained and stored in the difference information storage unit 34 (step 306). ). Further, by performing image processing for subtracting the previous captured image P (N−1) from the current captured image P (N), the first differential image DP (N) is obtained and stored in the differential information storage unit 34. (Step 307). Here, the first difference time and the first difference image are obtained by simply subtracting the previous imaging information from the current imaging information without considering whether the operation on the whiteboard is one meaningful operation. Information.

Next, the difference information generation unit 33 determines whether or not the first difference time DT (N) exceeds a predetermined time T2 as a time interval for distinguishing meaningful operations on the whiteboard ( Step 308). Here, as the time T2, a value larger than the imaging interval T1 used in FIG. 5 is generally set.
As a result, if the time T2 is exceeded, first, “1” is added to the variable M (step 309). Then, the second differential time CT (M−1) that is the time to the previous meaningful operation is substituted into the second differential time CT (M) that is the time to the current meaningful operation (step 310). The second difference image CP (M) is initialized (step 311).
On the other hand, if the time T2 is not exceeded, the variable M, the second difference time CT (M), and the second difference image CP (M) are left as they are.

Thereafter, the difference information generation unit 33 sequentially obtains the second difference time CT (M), which is the time from the start of the operation to the end of the current meaningful operation, and the first difference time DT (N) obtained in step 306. It is obtained by adding and stored in the difference information storage unit 34 (step 312). When the current operation is an operation in the middle of one meaningful operation, the second differential time CT (M) still does not represent the time until the end of the meaningful operation at this point. Absent. However, by repeating the calculation while adding the variable M by “1”, finally, the second difference time CT (M) represents the time until the end of the meaningful operation.
Further, the difference information generation unit 33 obtains the second difference image CP (M) that is an image of the meaningful operation of this time by sequentially adding the first difference image DP (N) obtained in step 307. And stored in the difference information storage unit 34 (step 313). When the current operation is an operation in the middle of one meaningful operation, the second difference image CP (M) does not yet represent an image of the entire meaningful operation at this point. However, by repeating the calculation while adding the variable M by “1”, the second difference image CP (M) finally represents an image of the entire meaningful operation.
Such processing is continued until the correspondence between the imaging time T (N) and the captured image P (N) is lost, and it is determined in step 304 that there is no correspondence between the imaging time T (N) and the captured image P (N). Then, the process ends.

FIG. 8 shows an example of difference information recorded in the difference information storage unit 34 by the operation of FIG. 7 when the imaging information as shown in FIG. 6 is received.
The first difference time DT (N) is the time from the recording of the (N−1) th captured image to the recording of the Nth captured image.
The first difference image DP (N) is an image added from the (N−1) th captured image recording to the Nth captured image recording. Here, the whiteboard frame is indicated by a dotted line. This is because the first difference image DP (N) does not manage only the content of the added image, but also manages the position information in the entire captured image and the size information for the entire captured image. means. Further, for example, when the medium is attached in a tilted state, information on the angle is also managed.

The second differential time CT (M) is the time from recording of the captured image in the initial state of the whiteboard to recording of the captured image of the Mth meaningful operation on the whiteboard.
The second difference image CP (M) is an image added from the recording of the captured image of the (M-1) th meaningful operation to the whiteboard to the recording of the captured image of the Mth meaningful operation to the whiteboard. It is. That is, the image represents an M-th meaningful operation on the whiteboard.
In the drawing, as an example of combining a plurality of first difference images into a second difference image representing one meaningful operation by such processing, the case of handwriting is shown (N = 5 to 7). However, as described above, a plurality of picked-up images may be recorded for the pasting of one medium and the projection of a single image for the pasting of a medium and the projection of an image. In such a case, the plurality of first difference images may be combined into a second difference image representing one meaningful operation.

Next, the document search unit 35 generates display information based on the difference information stored in the difference information storage unit 34.
FIG. 9 is a flowchart illustrating an operation example of the document search unit 35.
First, the document search unit 35 substitutes “0” for the variable M used in FIG. 7 (step 321), and initializes the screen object V (0) (step 322). Here, the screen object V (M) is a screen object that represents the state of the whiteboard when the Mth meaningful operation is performed. When M = 0, the screen object represents the initial state of the whiteboard. is there.

Next, as long as there is the second difference time CT (M) and the second difference image CP (M), the document search unit 35 is based on the second difference time CT (M) and the second difference image CP (M). A process of generating display information is performed.
That is, first, the document search unit 35 adds “1” to the variable M (step 323), and determines whether or not there is a correspondence between the second difference time CT (M) and the second difference image CP (M). (Step 324).

  Here, if there is a correspondence between the second difference time CT (M) and the second difference image CP (M), the second difference time CT (M) and the second difference image CP (M) are stored in the difference information storage unit 34. (Step 325). Then, by adding the second difference time CT (M) that is the time from the start time of the operation to the whiteboard to the Mth meaningful operation to T (0) that is the start time of the operation to the whiteboard, An operation time VT (M) at which the Mth meaningful operation is performed is obtained and stored in the display information storage unit 36 (step 326). In addition, the second differential image CP (M), which is an image representing the Mth meaningful operation, is converted into an object to generate an element object VO (M) that is an example of a display element, and is stored in the display information storage unit 36. Store (step 327).

Thereafter, the document search unit 35 determines whether or not the shape of the element object VO (M) is a rectangle (step 328). This is because the medium to be pasted on the whiteboard and the shape of the projected image are generally rectangular. However, when a shape other than the rectangle is assumed, the determination condition in step 328 may be flexibly changed. .
As a result, if the shape of the element object VO (M) is not rectangular, the element object VO (M) is an object representing handwriting and does not need to be associated with the electronic document.

On the other hand, if the shape of the element object VO (M) is a rectangle, the electronic document corresponding to the element object VO (M) is specified.
That is, first, the document search unit 35 generates an image by erecting the element object VO (M) with one vertex as the upper left point (step 329). Then, an electronic document having an aspect ratio similar to that of the image and a similar layout is searched from electronic documents stored in a specific folder (step 330).

  Here, if the receiving unit 31 obtains the time projected from the projection imaging device 40 and the image projected at that time, the difference information generating unit 33 obtains the obtained time and the second difference time CT (M). And comparing the obtained time with the second difference image CP (M) so that the second difference image CP (M) is an image projected by the projection imaging device 40. I can guess. That is, if the two times are close to some extent and the two images are somewhat similar, it can be estimated that the second difference image CP (M) is a projected image. On the other hand, the second difference image CP (M) that could not be estimated as a projected image can be considered to represent a pasted medium. Accordingly, when the document search unit 35 receives such an estimation result from the difference information generation unit 33 and is considered to represent an image on which the element object VO (M) is projected in step 330, the document search folder 35 If the electronic document stored in the folder is considered to represent an image of the medium on which the element object VO (M) is pasted, the electronic document stored in the pasted document folder is targeted. You can search for it.

  The search for electronic documents having similar layouts in step 330 is specifically performed by searching for document elements (characters, numbers, symbols, figures, pictures, etc. of the electronic document in the image generated in step 329. ) Is preferably compared with the shape of the area where the document element is placed in the electronic document. Alternatively, not only the shape of the document element but also the color of the document element may be considered.

Next, the document search unit 35 determines whether the electronic document search is successful (step 331).
If the electronic document search is successful, a link to the searched electronic document is embedded in the element object VO (M) (step 332), and the process proceeds to step 334.
On the other hand, if the electronic document search is not successful, it is determined whether there is another vertex (step 333). If there are other vertices, paying attention to the vertices, the processing of steps 329 to 333 is repeated. If there is no other vertex, the process proceeds to step 334 as it is.
Finally, the document search unit 35 applies the M-th meaningful operation to the screen object V (M-1) representing the state of the whiteboard at the time when the (M-1) -th meaningful operation was performed. By adding the element object VO (M) to be represented, the screen object V (M) representing the state of the whiteboard at the time when the Mth meaningful operation is performed is obtained and stored in the display information storage unit 36 (step S40). 334).
Such processing is continued until the correspondence between the second difference time CT (M) and the second difference image CP (M) is lost, and in step 324, the second difference time CT (M) and the second difference image CP (M). If it is determined that there is no correspondence with, the process is terminated.

FIG. 10 shows an example of display information recorded in the display information storage unit 36 by the operation of FIG. 9 when difference information as shown in FIG. 8 is stored.
VT (M) is an operation time when one meaningful operation is performed on the whiteboard.
VO (M) is an element object representing an image added by one meaningful operation on the whiteboard. For example, an element object representing a pasted medium (when M = 1, 3, 4), an element object representing a projected image (when M = 2), an element object representing handwriting (M = 5) In the case of). Among these, a link to the electronic document printed on the medium is embedded in the element object representing the pasted medium, and a link to this electronic document is embedded in the element object representing the image of the projected electronic document. It is. In the figure, the fact that the link is embedded is indicated by shading. Again, the whiteboard frame is indicated by a dotted line. This is because the element object VO (M) is an object corresponding only to an operation on the whiteboard, so that the position information on the screen object representing the entire whiteboard and the size information on the screen object are also managed. means. Further, for example, when the medium is attached in a tilted state, information on the angle is also managed. Therefore, the screen object V (M) is generated by adding the element objects VO (M) in time order to the screen object representing the whiteboard using the information on the position, size, and angle.

Finally, in the document server 30, the transmission unit 39 reads the screen object V (M) stored in the display information storage unit 36 and transmits it to the terminal device 50. Thereby, a screen object is displayed on the terminal device 50.
FIG. 11 shows an example of a screen object displayed on the terminal device 50.
Here, an example of a screen object based on V (5) in FIG. 10 is shown. That is, the element objects 51, 52, and 53 are element objects that represent the media A, B, and C in FIG. 4, respectively, and the element object 54 is an element object that represents a projected image.
Here, as described above, a link to the electronic document is embedded in each element object. Therefore, for example, when the element object 54 is clicked, the editor 55 is activated, and the electronic document that is the basis of the projected image can be edited.
In this example, the link to the electronic document (information for referring to the electronic document) is embedded in the element object, but the image of the electronic document may be displayed superimposed on the element object.

Further, the screen object shown in FIG. 11 is associated with the operation time when the operation that is the basis of each screen object is performed. Then, when the operation time is designated, the terminal device 50 displays a screen object associated with the operation time. Or you may make it display a screen object in order of time from the start of operation with respect to a whiteboard to completion | finish of operation.
This is the end of the description of the operation of the present embodiment.

  In the present embodiment, the electronic document printed on the medium or the electronic document that is the basis of the projected image is specified based on the layout of the electronic document. However, the present invention is not limited to this. For example, information for identifying an electronic document may be embedded in a medium on which the electronic document is printed or an electronic document to be projected, and the electronic document may be specified based on this information. However, the information to be embedded in this case is preferably large enough to be extracted from the captured image without error.

By the way, in the present embodiment, the document server 30 that holds an electronic document performs processing such as acquisition of imaging information, generation of difference information, identification of an electronic document, instruction to display a screen, and the like. Processing may be performed by any apparatus. For example, it may be performed by the projection imaging device 40 or the terminal device 50, or may be performed by a computer that does not hold any other electronic document.
Therefore, the hardware configuration of the computer 90 will be described assuming that these processes are performed by the computer 90.

FIG. 12 is a diagram illustrating a hardware configuration of the computer 90.
As shown in the figure, the computer 90 includes a CPU (Central Processing Unit) 91 as a calculation means, a main memory 92 as a storage means, and a magnetic disk device (HDD: Hard Disk Drive) 93. Here, the CPU 91 executes various types of software such as an OS (Operating System) and applications to realize the above-described functions. The main memory 92 is a storage area for storing various software and data used for execution thereof, and the magnetic disk device 93 is a storage area for storing input data for various software, output data from various software, and the like. .
Further, the computer 90 includes a communication I / F 94 for performing communication with the outside, a display mechanism 95 including a video memory and a display, and an input device 96 such as a keyboard and a mouse.

  The program for realizing the present embodiment can be provided not only by communication means but also by storing it in a recording medium such as a CD-ROM.

DESCRIPTION OF SYMBOLS 10,50 ... Terminal device, 20 ... Image forming device, 30 ... Document server, 40 ... Projection imaging device, 80 ... Network

Claims (8)

An acquisition means for acquiring a captured image obtained by imaging a surface on which a medium on which an electronic document is printed is pasted;
Extraction means for extracting a partial image of a part to which the medium is pasted from the captured image acquired by the acquisition means;
Specifying means for specifying the electronic document based on the partial image extracted by the extracting means;
A screen that represents the captured image acquired by the acquisition unit, and includes a display element associated with the electronic document specified by the specifying unit at the position of the partial image extracted by the extraction unit. A document display control apparatus comprising: instruction means for instructing display of the document. The acquisition unit acquires a plurality of captured images obtained by capturing the surface each time a medium is attached, and a plurality of time information respectively indicating a plurality of times when the plurality of captured images are obtained. ,
The instruction means is a plurality of screens respectively representing the plurality of captured images acquired by the acquisition means, and each screen represents each of the plurality of time information acquired by the acquisition means. The document display control apparatus according to claim 1, wherein a display of a plurality of screens associated with time information indicating a time when a captured image is obtained is instructed.   The document display control apparatus according to claim 2, wherein the instruction unit instructs display of a screen associated with time information indicating a specified time among the plurality of screens. The acquisition unit acquires a captured image obtained by capturing the surface on which an image of an electronic document is projected by a projection device;
The extraction unit extracts a partial image of a portion onto which the image of the electronic document is projected from the captured image acquired by the acquisition unit based on the image of the electronic document projected by the projection device. The document display control device according to claim 1, wherein   The specifying unit is configured to determine the electronic document based on a shape of a region in which the document element of the electronic document is arranged in the partial image and a shape of a region in which the document element of the electronic document is arranged in the electronic document. The document display control apparatus according to claim 1, wherein the document display control apparatus is specified.   The document display control device according to claim 1, wherein the instruction unit instructs display of the screen including the display element in which information for referring to the electronic document is embedded. .   The document display control apparatus according to claim 1, wherein the instruction unit instructs display of the screen including the display element on which an image of the electronic document is superimposed. On the computer,
A function of acquiring a captured image obtained by imaging a surface on which a medium on which an electronic document is printed is pasted;
A function of extracting a partial image of a portion to which the medium is pasted from the captured image;
A function of identifying the electronic document based on the partial image;
A program for realizing a function of instructing display of a screen including the display element associated with the electronic document at a position of the partial image, the screen representing the captured image.

Images