JP2015211363A - Display control device, display control method and display control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately determine a superposed display position of information even if a marker cannot be recognized.SOLUTION: First information 5a, relating to a captured image, to be used for calculating similarity between the captured image and another image in the case where a marker is successfully recognized from the captured image, and second information 5b based on a result of the marker recognition from the captured image are recorded in a storage device 4 by a storage processing section 2. When the recognition of the marker from a first captured image is failed, on the basis of the first information 5a recorded in the storage device 4, the second information 5b corresponding to a second captured image similar to the first captured image is extracted from the storage device 4 by a display position determination section 3. On the basis of the extracted second information 5b, an imaging position and an imaging posture corresponding to the first captured image are calculated and on the basis of the calculated position and posture, a superposed display position of work support information 22 is determined.

Description

  The present invention relates to a display control device, a display control method, and a display control program.

  Augmented Reality (AR) technology is known in which a virtual image is superimposed and displayed at a predetermined position on an image obtained by capturing a real space. This AR technology is currently spreading in applications that support the work of workers by superimposing and displaying work support information indicating work contents and work target locations on a captured image in a work space such as a factory. . For example, it has been proposed to be used for supporting the dismantling work of equipment in a nuclear power plant.

  The method of superimposing and displaying the work support information based on the captured image includes a “marker-based method” that uses a marker of a known shape arranged at a known position in the real space, and a “markerless” method that does not use such a marker. It is roughly divided into “method”. In the marker-based method, for example, when a region including a marker is imaged by an operator, the position and orientation (imaging direction) of the camera are determined based on the coordinate information of the marker recognized from the captured image. Then, the work support information associated with the marker is superimposed and displayed at an appropriate position on the image corresponding to the determined position and orientation. On the other hand, in the markerless method, for example, unknown feature point information is extracted from a captured image, and the position and orientation of the camera are determined based on the extracted feature point information.

  Further, as an example of a technique related to image superposition by the AR technique, the camera position and orientation are estimated based on the measurement result of the angular velocity in the immediately preceding period, and the relationship between the marker detection result from the captured image and the estimated camera position and orientation Based on the above, a technique for correcting the estimated camera position and orientation has been proposed.

  In addition, as an example of image recognition technology, feature quantities extracted from captured images are extracted, and the importance of each feature quantity is determined from the physical properties of each feature section and the illumination conditions at the time of imaging. There has been proposed a technique for recognizing the position and orientation of an object based on a value obtained by weighting an amount with importance.

JP 2005-33319 A JP 2011-175600 A

Yugo Ishii and five others, “Proposal and Evaluation of Nuclear Power Plant Disassembly Support Method Using Augmented Reality”, Transactions of the Virtual Reality Society of Japan Vol. 13, no. 2, June 30, 2008, p. 290-291

  By the way, when the vicinity of the marker arranged in the work space is imaged by the camera, the marker may not be recognized from the captured image. In the marker-based superimposed display process, when the marker cannot be recognized from the captured image, the superimposed position of the work support information cannot be determined, and as a result, the work support information cannot be superimposed and displayed.

  As a countermeasure when the marker cannot be recognized, for example, a method of switching to the markerless method can be considered. However, in the markerless method, the estimation accuracy of the imaging position and the imaging posture in the initial state affects the subsequent superimposition position accuracy. In a situation where the marker cannot be recognized, the estimation accuracy of the imaging position and imaging orientation in the initial state may be reduced, and as a result, the superimposed position accuracy may be reduced.

  In one aspect, an object of the present invention is to provide a display control device, a display control method, and a display control program capable of accurately determining a superimposed display position of information even when a marker cannot be recognized.

  In one proposal, an imaging position and an imaging posture based on the marker are calculated based on the marker position information recognized from the captured image, and a predetermined operation is performed at a position on the captured image based on the calculated imaging position and imaging orientation. Provided is a display control device that displays support information in a superimposed manner. The display control device includes an accumulation processing unit and a display position determination unit. When the accumulation processing unit succeeds in recognizing the marker from the captured image, the accumulation processing unit uses the first information related to the captured image, which is used for calculating the similarity between the captured image and another image, and from the captured image. The second information based on the marker recognition result is recorded in the storage device. The display position determination unit, when the recognition of the marker from the first captured image fails, is based on the first information for each captured image recorded in the storage device and is similar to the first captured image. The second information corresponding to the captured image is extracted from the storage device, the imaging position and the imaging orientation corresponding to the first captured image are calculated based on the extracted second information, and the calculated imaging position and imaging orientation are calculated. The superimposed display position of the work support information is determined based on the above.

In one proposal, a display control method is provided in which processing similar to that of the display control device is performed.
Furthermore, in one proposal, there is provided a display control program that causes a computer to execute the same processing as that of the display control device.

  In one aspect, the superimposed display position of information can be accurately determined even when the marker cannot be recognized.

It is a figure which shows the structural example and process example of a display control apparatus which concern on 1st Embodiment. It is a figure which shows the hardware structural example of the terminal device which concerns on 2nd Embodiment. It is a figure which shows the example of the utilization scene of a terminal device. It is a figure which shows the example of the superimposition display of the work assistance information with respect to a captured image. It is a figure which shows the example of a marker coordinate system, a camera coordinate system, and an image coordinate system. It is a figure shown about a rotation matrix. It is a figure shown about a translation vector. It is a block diagram which shows the structural example of the processing function with which a terminal device is provided. It is a figure which shows the structural example of the information recorded on a comparison image information database. It is a flowchart (the 1) which shows the example of the process sequence of a terminal device. It is a flowchart (the 2) which shows the example of the process sequence of a terminal device. It is a flowchart (the 3) which shows the example of the process sequence of a terminal device. It is a block diagram which shows the structural example of the processing function with which the terminal device which concerns on 3rd Embodiment is provided. It is a figure which shows the structural example of the information recorded on a comparison image information database. It is a flowchart which shows the example of the procedure of the classification process by an information storage control part. It is a flowchart which shows the example of a part of process by the comparison image information selection part of 3rd Embodiment. It is a figure which shows the example of the screen accompanying the display of a guide frame. It is a flowchart which shows a part of example of the process by the comparison image information selection part of 4th Embodiment. It is a flowchart which shows the example of a part of process by the comparison image information selection part of 5th Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram illustrating a configuration example and a processing example of the display control apparatus according to the first embodiment. The display control apparatus 1 illustrated in FIG. 1 calculates an imaging position and an imaging posture with reference to the marker based on the marker position information recognized from the captured image. Then, predetermined work support information is superimposed and displayed at a position on the captured image based on the calculated imaging position and imaging orientation.

  For example, the display control device 1 is equipped with a camera and a display device, and the display control device 1 is carried by an operator. The worker images the work space including the work object on which the marker is arranged. At this time, the display control apparatus 1 recognizes the marker from the captured image, and superimposes and displays the work support information on the captured image according to the above procedure based on the recognition result.

  The work support information is information that presents information related to work such as work contents and work precautions to the worker using virtual image information. An operator can perform work accurately by browsing a screen on which such work support information is superimposed and displayed. In addition, the working efficiency is improved.

  Further, in the process of superimposing and displaying the work support information according to the above procedure, the display control device 1 cannot calculate the imaging position and the imaging posture when the marker cannot be recognized from the captured image, and as a result, the work support information is superimposed. It cannot be displayed. The display control device 1 includes a storage processing unit 2 and a display position determination unit 3 as processing functions for enabling the work support information to be superimposed and displayed at an accurate position on the captured image even when the marker cannot be recognized from the captured image. Have.

  When the accumulation processing unit 2 succeeds in recognizing the marker from the captured image during the process of superimposing and displaying the work support information on the captured image as described above, the accumulation processing unit 2 displays the first information 5a and the second information 5b. Records in the storage device 4. For example, the storage device 4 may be mounted inside the display control device 1, or may be provided outside the display control device 1.

  The 1st information 5a is information about this picked-up image used for calculation of the similarity of the picked-up image which succeeded in marker recognition, and other images. For example, the first information 5a includes the position and feature amount of a feature point obtained from the captured image, or image data of the captured image. The second information 5b is information based on the marker recognition result from the captured image. For example, the second information 5b includes the marker position information in the captured image or the imaging position and the imaging posture calculated from the marker position information.

  The display position determination unit 3 performs the following process when the marker recognition from the captured image fails. Here, it is assumed that marker recognition has failed when the captured image 11a shown in FIG. 1 is captured.

  The display position determination unit 3 extracts, from the storage device 4, second information 5 b corresponding to a captured image similar to the captured image 11 a based on the first information 5 a for each captured image recorded in the storage device 4. To do. In the example of FIG. 1, it is assumed that the image 12 is specified as a past captured image similar to the imaging device 11a. In this case, the display position determination unit 3 extracts the second information 5 b corresponding to the image 12 from the storage device 4.

  It is assumed that the image 12 includes the same marker 21 as the marker 21 that should be originally included in the imaging range of the imaging device 11a, and the marker 21 has been recognized from the image 12 in the past. In this case, the extracted second information 5 b is information based on the recognition result of the marker 21 from the image 12.

  The display position determination unit 3 calculates an imaging position and an imaging attitude corresponding to the captured image 11a based on the extracted second information 5b. The calculated imaging position and imaging attitude indicate the imaging position and imaging attitude based on the marker 21 that should be originally reflected in the captured image 11a. Then, the display position determination unit 3 determines the superimposed display position of the work support information 22 based on the calculated imaging position and imaging attitude.

  In this way, the display control apparatus 1 extracts a captured image similar to the captured image for which the marker could not be recognized from the captured images for which the marker could be recognized in the past. Then, the superimposed display position of the work support information 22 is determined using the second information corresponding to the extracted captured image, that is, the information based on the marker recognition result. As a result, even when the marker cannot be recognized from the captured image 11a, the work support information 22 can be superimposed and displayed at an accurate position on the captured image 11a.

  In particular, the above processing of the display control device 1 when the marker cannot be recognized does not directly calculate the imaging position and the imaging posture based on the natural feature extracted from the image when compared with the method of switching to the markerless method. The processing load is low. Furthermore, unlike the markerless method, it is not necessary to estimate the imaging position and imaging posture in the initial state. For this reason, according to the display control apparatus 1, compared with the method switched to a markerless system, a processing load can be reduced and the precision of the superimposition display position of work assistance information can also be improved.

[Second Embodiment]
FIG. 2 is a diagram illustrating a hardware configuration example of the terminal device according to the second embodiment. The terminal device 100 according to the second embodiment is realized as a portable computer as shown in FIG.

  2 is controlled by a processor 101 as a whole. The processor 101 may be a multiprocessor. The processor 101 is, for example, a central processing unit (CPU), a micro processing unit (MPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), or a programmable logic device (PLD). The processor 101 may be a combination of two or more elements among CPU, MPU, DSP, ASIC, and PLD.

A RAM (Random Access Memory) 102 and a plurality of peripheral devices are connected to the processor 101 via a bus 109.
The RAM 102 is used as a main storage device of the terminal device server 100. The RAM 102 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the processor 101. The RAM 102 stores various data necessary for processing by the processor 101.

  Peripheral devices connected to the bus 109 include an HDD (Hard Disk Drive) 103, a display device 104, an input device 105, a reading device 106, a wireless communication interface 107, and a camera 108.

  The HDD 103 is used as an auxiliary storage device of the terminal device 100. The HDD 103 stores an OS program, application programs, and various data. As the auxiliary storage device, other types of nonvolatile storage devices such as SSD (Solid State Drive) can be used.

  The display device 104 displays an image on the screen in accordance with an instruction from the processor 101. Examples of the display device 104 include a liquid crystal display and an organic EL (ElectroLuminescence) display.

  The input device 105 transmits a signal corresponding to the input operation by the user to the processor 101. Examples of the input device 105 include a touch panel arranged on the display surface of the display device 104, a touch pad, a mouse, a trackball, and operation keys.

  A portable recording medium 106 a is detached from the reading device 106. The reading device 106 reads the data recorded on the portable recording medium 106 a and transmits it to the processor 101. Examples of the portable recording medium 106a include an optical disk, a magneto-optical disk, and a semiconductor memory.

The wireless communication interface 107 transmits and receives data to and from other devices by wireless communication.
The camera 108 digitizes the image signal obtained by the image sensor and transmits it to the processor 101.

With the hardware configuration described above, the processing functions of the terminal device 100 can be realized.
FIG. 3 is a diagram illustrating an example of a usage scene of the terminal device. The worker 200 carries the terminal device 100 in a work space where the work object 201 to be worked exists. A marker is affixed to the work object 201 at a predetermined position for each stage of work. In the example of FIG. 3, two markers 211 and 212 are shown. The shape and size of the outer shape of each marker are the same. Here, it is assumed that each marker has a rectangular shape with the same long side and short side. Moreover, the outer edge part of each marker shall be bordered by the same thickness and the same color line. Furthermore, the information (internal pattern) displayed inside the marker is different for each marker.

  In the terminal device 100, for example, a camera 108 is attached to the back side of the display surface of the display device 104. The worker 200 holds the terminal device 100 over the work target 201 and images the work target area including the marker with the camera 108. Then, the terminal device 100 recognizes the marker from the captured image. Since the internal pattern of the marker is different for each marker, the terminal device 100 can specify the work stage from the recognition result of the internal pattern of the marker. The terminal device 100 reads the work support information associated with the identified work stage, and displays the read work support information superimposed on an appropriate position on the captured image based on the marker recognition result.

In FIG. 3, only one work object 201 is shown as an example, but it goes without saying that a plurality of work objects may exist.
FIG. 4 is a diagram illustrating an example of superimposed display of work support information on a captured image. As described above, the terminal device 100 basically superimposes and displays the work support information associated with the marker on an appropriate position on the captured image based on the recognition result of the marker from the captured image. That is, the terminal device 100 normally performs the superimposing display process of the work support information using the marker-based AR technology.

  An image 221 shown on the upper side of FIG. 4 is an example of an image obtained by imaging the work object 202 to which the marker 213 is attached. The terminal device 100 recognizes the marker 213 from the image 221. At this time, for example, the terminal device 100 detects the coordinates on the image 221 of the four vertices in the outer shape of the marker 213, and calculates information indicating the imaging position and the imaging attitude of the camera 108 based on the coordinates of these vertices. . Hereinafter, the information indicating the imaging position and the imaging posture by the camera 108 may be referred to as “imaging position and orientation information”.

  The terminal device 100 determines a work stage based on display information inside the marker 213, and specifies work support information corresponding to the determined work stage. The terminal device 100 superimposes and displays the specified work support information at an appropriate position on the image 221 calculated from the imaging position and orientation information. An image 222 shown on the lower side of FIG. 4 shows an example in which work support information 222 a to 222 d corresponding to the marker 213 is superimposed on the image 221. For example, the work support information 222a guides the direction in which a predetermined part (handle) of the work target 202 is rotated by the worker using an arrow image. Further, the work support information 222b to 222d presents the work content and information related to the work to the worker using character information.

  Here, a three-dimensional coordinate system with the center of the marker as the origin is referred to as a “marker coordinate system”. The imaging position / orientation information calculated by the terminal device 100 is represented as information in the marker coordinate system. That is, the imaging position and orientation information is information indicating from which position and in what direction the marker is imaged with reference to the center of the marker. In addition, information on the position where the work support information is superimposed and displayed is stored in the terminal device 100 as position information in the marker coordinate system.

  The shape and size of the marker appearing in the captured image changes according to the imaging position and orientation information. From this, the terminal device 100 can calculate imaging position and orientation information based on the position of the vertex of the marker in the captured image. Then, the terminal device 100 assumes that the work support information exists at a specified position in the space indicated by the marker coordinate system, and captures the work support information from the position indicated by the calculated imaging position and orientation information. Then, it is calculated in what position and in what shape the work support information appears on the captured image. Thereby, according to the position of the camera 108 with respect to the marker and the imaging direction, the work support information can be superimposed and displayed at an appropriate position on the captured image.

Here, with reference to FIGS. 5 to 7, a method for calculating the position when the work support information is superimposed and displayed on the captured image will be described.
FIG. 5 is a diagram illustrating an example of a marker coordinate system, a camera coordinate system, and an image coordinate system. In FIG. 5, a marker coordinate system in which the center of the marker 214 is the origin and the surface of the marker 214 is the XY plane is defined as a matrix [X _m Y _m Z _m 1] ^T. Note that “T” in the upper right indicates a transposed matrix. A three-dimensional “camera coordinate system” having the camera focus as the origin and the center in the imaging direction as the Z axis is defined as a matrix [X _c Y _c Z _c 1] ^T. Further, a two-dimensional “image coordinate system” having the origin at the upper left of the captured image is defined as a matrix [x _c y _c 1] ^T. The Z axis in the camera coordinate system is orthogonal to the image plane 231 in the image coordinate system at the center point 232 of the captured image.

  The marker position information in the captured image is expressed as position information in the image coordinate system. Further, as described above, the camera position / orientation information determined based on the marker position information can be expressed as the focus position and orientation (imaging direction) of the camera in the marker coordinate system.

  The coordinate conversion between the marker coordinate system and the camera coordinate system is defined by the following equation (1), for example.

In Equation (1), R represents a 3 × 3 rotation matrix. T represents a translation vector and is represented as a matrix of 3 rows and 1 column.
Next, the projective transformation from the camera coordinate system to the image coordinate system is defined by the following equation (2). Moreover, the matrix P in Formula (2) is represented as Formula (3).

  In Expression (2), h is a scalar. The matrix P indicates internal parameters calculated from the focal length and the angle of view obtained from camera calibration. The matrix P is obtained in advance from, for example, a captured image obtained by imaging a marker with a marker of a known size set at a known distance.

FIG. 6 is a diagram illustrating the rotation matrix. From the coordinates of the four vertices of the marker 214 in the captured image, straight lines l _i (where i = 1, 2, 3, 4) indicating the four sides of the marker 214 are obtained.

Here, for example, it is assumed that the sides l ₁ and l ₂ of the marker 214 are given by the following equations (4-1) and (4-2).
l ₁ : a ₁ x _c + b ₁ y _c + c ₁ = 0 (4-1)
l ₂ : a ₂ x _c + b ₂ y _c + c ₂ = 0 (4-2)
From the equations (2), (3), (4-1), and (4-2), the following equations (5-1) and (5-2) are obtained.
_{a 1 P 11 X c + (} a 1 P 12 + b 1 P 22) Y c + (a 1 P 13 + b 1 P 23 + c 1) Z c = 0 ··· (5-1)
a ₂ P ₁₁ X _c + (a ₂ P ₁₂ + b ₂ P ₂₂ ) Y _c + (a ₂ P ₁₃ + b ₂ P ₂₃ + c ₂ ) Z _c = 0 (5-2)
Expression (5-1) represents a plane S ₁ passing through the side of the marker 214 corresponding to the straight line l ₁ and the focal point 234 of the camera 108. Expression (5-2) represents a surface S ₂ passing through the side of the marker 214 corresponding to the straight line l ₂ and the focal point 234 of the camera 108. From the equations (5-1) and (5-2), equations indicating the normal vectors n ₁ and n ₂ of the surfaces S ₁ and S ₂ are obtained. If the direction vector of the side of the marker 214 in the space corresponding to the straight line l ₁ is V ₁ , the direction vector is given from the outer product of the normal vectors n ₁ and n ₂ .

For the straight lines l ₃ and l ₄ , the direction vector V ₂ is obtained by performing the same calculation as described above. Furthermore, a direction vector V ₃ orthogonal to the plane including the direction vectors V ₁ and V ₂ is obtained. The rotation matrix R of the above equation (1) is obtained as R = [V ₁ V ₂ V ₃ ].

  FIG. 7 is a diagram showing the translation vector. In FIG. 7, the marker 214 is arranged in the camera coordinate system so that the center point thereof coincides with the origin of the camera coordinate system (step S1), and the marker 214 is rotated by the rotation matrix R from this state, and further the translation vector T Let us consider that the image is projected onto the captured image on the image plane 231 by the matrix P (step S3).

The three-dimensional coordinates of the marker 214 in the marker coordinate system are M _i (where i = 1, 2, 3, 4), and the two-dimensional coordinates of the vertex of the marker 214 on the captured image are m _i (where i = 1, 2). , 3, 4). The vertex of the marker 214 in the marker coordinate system is represented as a matrix [M _ix M _iy M _iz ] ^T (where i = 1, 2, 3, 4), and the vertex of the marker 214 on the captured image is represented by a matrix [m _ix m _iy ] ^T (where i = 1, 2, 3, 4), the following equation (6) is obtained using the above equations (1) and (2).

Expression (6) is expanded using the length of one side of the marker 214 in the camera coordinate system, and by performing such processing for each side, (t _x , t _y , t _z ) is set as an unknown. Simultaneous linear equations are obtained. By solving the simultaneous linear equations translation vector _{T [t x t y t z} ] T is obtained.

6 and 7, the rotation matrix R and translation vector T in Expression (1) can be obtained from the coordinates of the four vertices of the marker 214 on the captured image.
Next, based on the above equation (2), the coordinates of the points in the image coordinate system are expressed as the following equations (7-1) and (7-2) using the coordinates of the points in the camera coordinate system. The

  The position information of the image data indicating the work support information is defined by the coordinates of the marker coordinate system and stored in the terminal device 100. Therefore, after the coordinate information of the stored work support information is converted into the coordinate information of the camera coordinate system using Expression (1), the image coordinate system is further converted using Expressions (7-1) and (7-2). The display position of the work support information in the captured image can be calculated by converting to the two-dimensional coordinate information.

  The data indicating the work support information stored in advance in the terminal apparatus 100 is, for example, work support based on the image data (for example, RGB data) for each coordinate inside the displayed shape and the center of the marker 214. Information display position information.

  Among these, the display position information of the work support information may be stored as, for example, three-dimensional coordinates in the marker coordinate system. Further, the display position information of the work support information may be stored as each value of the translation component and the rotation axis component. Hereinafter, the latter case will be described.

When the display position of the work support information in the marker coordinate system is represented as (X _m , Y _m , Z _m , 1) and the image data is represented as (x, y, z, 1), the following equation (8) is obtained. It holds.
_{[X m Y m Z m 1} ] T = M [x y z 1] T ··· (8)
When M is a transformation matrix shown in the following equation (9), the translation component p (t _x , t _y , t _z ) and the rotation axis component (rotation angle) (l, m, n, θ) of the work support information are Are represented by the following equations (10) and (11), respectively.

  By the way, when the terminal device 100 performs the superimposed display processing of information based on the marker recognition result as described above, if the marker cannot be recognized from the captured image, the imaging position / orientation information cannot be calculated, and as a result, The information cannot be displayed in a superimposed manner.

  As a case where the marker cannot be recognized, for example, a case where dirt generated during the operation adheres to the marker, or a case where the marker is arranged at a shadow position where the illumination light is difficult to hit can be considered. In these cases, it may be possible to solve the problem by replacing the marker with dirt attached or by preparing the environment around the marker so that there is no difference in brightness around the marker. There is a problem that cannot be dealt with.

  A method of switching to the markerless method when the marker cannot be recognized from the captured image is also conceivable. However, the markerless method has a larger processing load than the marker base AR. In the markerless method, it is necessary to prepare a large amount of template information in order to cope with various shooting positions and postures and to improve robustness to environmental changes. For this reason, maintaining the same overlay position accuracy as the marker-based method increases the development and manufacturing costs of the device. Therefore, when switching to the markerless method, the same overlay position accuracy as the marker-based method is maintained. It is actually difficult to do.

  Furthermore, in the markerless system, the imaging position and imaging posture in the initial state are estimated. The estimation of the imaging position and the imaging posture in the initial state needs to be performed with high accuracy because the estimation accuracy affects the subsequent superposition position accuracy. Thus, there is a method of performing the imaging position and orientation in the initial state based on the marker recognition result, but this method cannot be used in a situation where the marker cannot be recognized from the current captured image. Although a method using the recognition result of the most recently recognized marker is also conceivable, the recognition result may be erroneous depending on the environment and conditions at the time of marker recognition, and the estimation accuracy of the imaging position and imaging orientation may be reduced. .

  Therefore, in the present embodiment, the terminal device 100 accumulates in the storage device the marker recognition result and the feature amount information from the captured image that has been successfully recognized by the marker. If the terminal device 100 fails to recognize the marker, the terminal device 100 compares the feature amount information stored in the storage device with the feature amount information extracted from the current image, and the past imaging that has succeeded in marker recognition is performed. An image similar to the current image is specified among the images. Then, the terminal device 100 acquires a marker recognition result from the identified image from the storage device, estimates the imaging position / orientation information corresponding to the current image using the acquired marker recognition result, and superimposes and displays the work support information. To decide. As described above, the terminal device 100 accurately determines the superimposed display position of the work support information by using the marker recognition result from the image similar to the current image among the past captured images that have succeeded in marker recognition. Can do.

FIG. 8 is a block diagram illustrating a configuration example of processing functions included in the terminal device.
The terminal device 100 includes a captured image acquisition unit 111, a marker recognition unit 112, an imaging position / orientation estimation unit 113, a superimposed display control unit 114, an information accumulation control unit 115, a comparative image information selection unit 116, and an imaging position / orientation estimation unit 117. . The processing of each block is realized by the processor 101 executing a predetermined program, for example.

  In addition, a marker image database (DB) 120, a work support information database 130, and a comparative image information database 140 are stored in a storage device (for example, HDD 103) in the terminal device 100.

The captured image acquisition unit 111 acquires image data of the captured image from the camera 108.
The marker image database 120 stores marker template images used in marker recognition processing. The marker image database 120 stores template images of a plurality of markers having different internal patterns, and each template image corresponds to one work stage. Hereinafter, it is assumed that each work stage is identified by a work ID, and in the marker image database 120, each template image is associated with an individual work ID.

  The marker recognition unit 112 recognizes the marker from the captured image acquired by the captured image acquisition unit 111 using the template image in the marker image database 120. For example, the marker recognition unit 112 recognizes the marker by extracting the outline of the marker from the captured image, and further recognizes the internal pattern of the marker by matching with the template image. As a result, the marker recognizing unit 112 outputs the coordinate information of the four vertices of the marker in the captured image and the work ID corresponding to the recognized internal pattern.

  The imaging position / orientation estimation unit 113 calculates imaging position / orientation information based on the coordinate information of the four vertices of the marker recognized by the marker recognition unit 112. As described above, the imaging position / orientation information is information indicating the position and orientation (imaging direction) of the camera 108 with respect to the center of the marker in the marker coordinate system.

  The work support information database 130 stores data indicating work support information to be superimposed on the captured image. As described above, the data indicating the work support information includes, for example, image data for each coordinate in the displayed shape and display position information of the work support information. The display position information of the work support information is defined as a value in the marker coordinate system with reference to the center of the corresponding marker.

Data indicating the work support information is associated with the work ID. Further, data indicating a plurality of work support information may be associated with one work ID.
The superimposed display control unit 114 acquires the work ID recognized by the marker recognizing unit 112 via the imaging position / orientation estimation unit 113 and specifies the work support information corresponding to the acquired work ID from the work support information database 130. To do. The superimposed display control unit 114 calculates the display position in the captured image of the identified work support information based on the imaging position / orientation information calculated by the imaging position / orientation estimation unit 113, and sets the imaging position / orientation information at the calculated display position. Display.

  Note that, in a period in which the marker cannot be recognized from the captured image, the superimposed display control unit 114 acquires the imaging position / orientation information from the imaging position / orientation estimation unit 117 instead of the imaging position / orientation estimation unit 113.

  When the marker recognizing unit 112 succeeds in recognizing the marker from the captured image, the information accumulation control unit 115 records information regarding the captured image in the comparison image information database 140. The information recorded in the comparison image information database 140 includes, for example, a recognized work ID, the position and feature amount of a feature point extracted from the captured image, and the position information of the recognized marker.

Thereafter, the captured image in which the related information is recorded in the comparison image information database 140 and has been successfully recognized by the marker may be referred to as a “comparison image”.
The comparison image information selection unit 116, when the marker recognition unit 112 fails to recognize the marker from the captured image, stores information on the comparison image appropriate for use in the estimation of the imaging position / orientation information from the comparison image information database 140. Select. Hereinafter, a captured image in which marker recognition has failed may be referred to as a “recognition failure image”.

  Specifically, the comparison image information selection unit 116 performs the following processing. First, the comparison image information selection unit 116 calculates the similarity between the recognition failure image and the comparison image based on the position and feature amount of the feature point recorded in the comparison image database 140. The comparison image information selection unit 116 specifies a comparison image whose similarity with the recognition failure image satisfies a predetermined condition.

  Next, the comparison image information selection unit 116 evaluates the correspondence between the feature points in the comparison image and the feature points extracted from the recognition failure image from the identified comparison images, and the evaluation result is a predetermined condition. Select a comparison image that satisfies For this evaluation, a transformation matrix that converts the coordinates of the feature points in the comparison image to the coordinates of the feature points in the recognition failure image is used.

  The comparison image information selection unit 116 calculates a transformation matrix from the coordinates of the feature points of the identified comparison image and recognition failure image. As the transformation matrix, a homography transformation matrix described later can be used. The comparison image information selection unit 116 compares the conversion result obtained by converting the coordinates of the feature points in the comparison image by the conversion matrix with the coordinates of the corresponding feature points of the recognition failure image, and based on the distance between these feature points. Evaluate conversion accuracy.

  The comparison image information selection unit 116 selects a comparison image whose conversion accuracy evaluation value satisfies a predetermined condition from the comparison images specified using the similarity. The comparison image information selection unit 116 reads the marker position information corresponding to the selected comparison image from the comparison image information database 140 and outputs the information to the imaging position / orientation estimation unit 117.

  The imaging position / orientation estimation unit 117 calculates an imaging position and an imaging orientation based on the marker position information output from the comparative image information selection unit 116 in the same procedure as the imaging position / orientation estimation unit 113, and the superimposed display control unit. To 114.

FIG. 9 is a diagram illustrating a configuration example of information recorded in the comparative image information database.
The comparison image information database 140 is provided with a record 141 for each work ID. In each record 141, one or more pieces of comparison image information are registered in association with the work ID. The comparison image information is information corresponding to each comparison image (that is, a captured image that has been successfully recognized by the marker).

  In each comparison image information, an image ID, marker position information, and a plurality of feature point information are registered. The image ID is information for identifying the comparison image. The marker position information includes the coordinates of the four vertices for the marker recognized from the comparison image. Information recognized by the marker recognition unit 112 is registered as the marker position information. The feature point information includes the coordinates of feature points and feature amounts extracted from the comparison image. The coordinates of the feature points and the feature amounts are detected from the comparison image by the information accumulation control unit 115.

Next, processing of the terminal device 100 will be described using a flowchart. 10 to 12 are flowcharts illustrating examples of processing procedures of the terminal device.
[Step S <b> 11] The captured image acquisition unit 111 acquires image data of the captured image from the camera 108. The marker recognizing unit 112 performs marker recognition processing from the captured image acquired by the captured image acquiring unit 111 using the template image in the marker image database 120.

  [Step S12] It is determined whether the captured image acquisition unit 111 has successfully recognized the marker. If the marker recognition is successful, the process of step S13 is executed. If marker recognition fails, the process of step S21 is executed.

  [Step S <b> 13] The imaging position / orientation estimation unit 113 acquires coordinate information of four vertices of the marker from the marker recognition unit 112. The imaging position / orientation estimation unit 113 calculates imaging position / orientation information using the acquired coordinate information. The calculated imaging position / orientation information corresponds to the rotation matrix R and the translation vector T described above.

  [Step S14] The superimposed display control unit 114 acquires the work ID recognized by the marker recognition unit 112 via the imaging position / orientation estimation unit 113, and the work corresponding to the acquired work ID from the work support information database 130. Identify support information. The superimposed display control unit 114 determines the display position of the identified work support information in the captured image based on the imaging position / orientation information calculated by the imaging position / orientation estimation unit 113 using the above-described equations (1) and (7-1). , (7-2). The superimposed display control unit 114 displays the imaging position / orientation information at the calculated display position in the captured image.

  [Step S15] The information accumulation control unit 115 acquires data of a comparison image (that is, a captured image that has been successfully recognized by the marker), marker position information, and the recognized work ID from the marker recognition unit 112. The information accumulation control unit 115 registers the acquired marker position information in the record 141 corresponding to the acquired work ID in the comparison image information database 140. In addition, the information accumulation control unit 115 extracts feature points from the acquired comparison image, and registers the extracted feature point position information and feature information in the same record 141.

  In addition, the process of step S15 may be performed before the process of step S14 or step S13. Further, the process of step S15 may be executed in parallel with the processes of steps S13 and S14. Furthermore, the process of step S15 may be performed as follows, for example. The information accumulation control unit 115 executes the registration process of the work ID and the marker position information in the process of step S15 when triggered by the recognition success in step S12. In addition, the information accumulation control unit 115 performs a feature point extraction process from the comparison image, and the extracted feature point position information and feature information registration process at an arbitrary timing thereafter.

  Next, processing when the marker cannot be recognized will be described with reference to FIGS. 11 and 12. When the marker cannot be recognized, the process of extracting a comparison image similar to the recognition failure image (FIG. 11), and the coordinate conversion accuracy between the extracted comparison image and the recognition failure image are evaluated. And processing for selecting an optimal comparison image for estimating the imaging position and orientation information (steps S31 to S34 in FIG. 12).

  [Step S21] When the marker recognizing unit 112 fails to recognize the marker, the comparison image information selecting unit 116 displays information notifying that the marker recognition has failed and information for requesting an operation on the confirmation button. 104 is displayed. For example, when the display device 104 is a touch panel type, a confirmation button is displayed on the screen of the display device 104. When the pressing operation of the confirmation button by the worker is detected, the process of step S22 is executed.

  In the processing after step S22, the captured image acquired in step S11 may be a processing target as a “recognition failure image”. However, since a certain amount of time has elapsed since step S11 by waiting for an operation on the confirmation button, the recognition failure image may be set as follows. The comparison image information selection unit 116 acquires a new captured image from the captured image acquisition unit 111 when the pressing operation of the confirmation button is detected. In the processing after step S22, the captured image acquired at this time is a processing target as a recognition failure image.

[Step S22] The comparative image information selection unit 116 identifies a work ID indicating the current work stage.
For example, when the terminal device 100 has a function of managing the progress of work, in step S22, the work ID indicating the work stage next to the work stage where the work is completed is specified. Further, the work ID may be input by an operator's input operation.

  [Step S23] The comparison image information selection unit 118 extracts a plurality of feature points from the recognition failure image. The comparison image information selection unit 118 temporarily records the extracted feature point positions and feature amounts in, for example, the RAM 102.

  [Step S24] The comparison image information selection unit 118 refers to the comparison image information database 140 and selects one of the comparison images associated with the work ID specified in step S22. Actually, one image ID indicating a comparison image is selected from the record 141 corresponding to the specified work ID in the comparison image information database 140.

[Step S25] The comparison image information selection unit 118 performs matching between the comparison image selected in step S24 and the recognition failure image.
Specifically, the comparison image information selection unit 118 reads feature point information associated with the image ID selected in step S23 from the comparison image information database 140. The comparison image information selection unit 118 performs a matching process between the feature points of the comparison image indicated by the read feature point information and the feature points extracted from the recognition failure image in step S23. In the matching process, for example, a pair of feature points corresponding to each other is identified from the positional relationship of the feature points in each image, and whether the feature amount difference of each feature point in the identified pair is within the threshold Ta Is determined (where Ta is greater than 0). When a pair of feature points is specified from the positional relationship and the difference between the feature amounts of the feature points in the pair is within the threshold Ta, it is determined that one set of matching has been successful.

  [Step S26] The comparison image information selection unit 118 compares the matching result with, for example, the following two determination conditions. As a first determination condition, it is determined whether the ratio of the number of successful pairs to the total number of feature points extracted from the recognition failure image is larger than a threshold value Tb (where Tb is greater than 0 and equal to or less than 1). . As a second determination condition, it is determined whether the number of pairs that have been successfully matched is greater than a threshold value Tc (where Tc is greater than 0).

The comparison image information selection unit 118 determines whether or not the matching result using the comparison image selected in step S24 satisfies both of the two conditions.
[Step S27] The comparison image information selection unit 118 determines whether all the comparison images corresponding to the corresponding work IDs whose information is registered in the comparison image information database 140 have been selected. If all the comparison images have been selected, the process of step S28 is executed. On the other hand, if there is an unselected comparison image, the process of step S24 is executed.

  [Step S28] The comparison image information selection unit 118 determines whether one or more comparison images satisfying the determination condition shown in step S26 have been extracted. If one or more comparison images satisfying the determination condition can be extracted, the process of step S31 is executed. On the other hand, if no comparison image satisfying the determination condition has been extracted, the process ends. In the latter case, for example, the comparison image information selection unit 118 may cause the display device 104 to display instruction information such as “please shoot around the marker”.

  [Step S31] The comparison image information selection unit 118 selects one of the comparison images that satisfy the determination condition shown in step S26. Actually, an image ID indicating a comparative image is selected.

  [Step S32] The comparison image information selection unit 118 calculates a transformation matrix H using a pair of feature points that have been successfully matched between the selected comparison image and the recognition failure image. This conversion matrix H is a homography matrix for converting the coordinates on the comparison image into the coordinates on the recognition failure image.

  The transformation matrix H is defined as the following equation (12).

Assuming that the transformation matrix H transforms the coordinate value P _r (x _r , y _r ) on the comparison image into the coordinate value P _w (x _w , y _w ) on the recognition failure image, P _r and P _w Can be expressed by the following formula (13-1) using a general formula of projective transformation.
sP _r = HP _w (13-1)
The parameter s in Expression (13-1) represents a scaling factor based on scaling indefiniteness. When this equation (13-1) is arranged for x _w and y _w , the following equations (13-2) and (13-3) are obtained.
_{x w = (H 11 x r} + H 12 y r + H 13) / (H 31 x r + H 32 y r + H 33) ··· (13-2)
_{y w = (H 21 x r} + H 22 y r + H 23) / (H 31 x r + H 32 y r + H 33) ··· (13-3)
Here, the matrix H has 8 degrees of freedom because of the degree of freedom of magnification by scalars, and therefore H ₃₃ = 1. Since there are eight unknowns and an equation can be generated for each pair of corresponding points, the unknowns are uniquely determined if the coordinates of at least four feature point pairs are obtained.

Assume that n feature point pairs that have succeeded in matching are obtained, and the coordinates of the i-th feature point on the comparison image are P _ri (x _ri , y _ri ), and the corresponding feature points on the recognition failure image If P _{wi is} (x _wi , y _wi ), the following equations (14-1) and (14-2) are obtained between P _ri and P _wi .
_{H 11 x ri + H 12 y} ri + H 13 -H 31 x ri x wi -H 32 y ri x wi = x wi ··· (14-1)
_{H 21 x ri + H 22 y} ri + H 23 -H 31 x ri y wi -H 32 y ri y wi = y wi ··· (14-2)
Equations (14-1) and (14-2) obtained above for four feature point pairs (i = 1, 2, 3, 4) are expressed in matrix form as the following equation (15): Can express.

The values of H ₁₁ ,..., H ₃₂ can be obtained by substituting the coordinate values of P _ri and P _wi and calculating the inverse matrix. Further, when five or more feature point pairs are obtained, the unknown can be obtained by using the least square method.

  [Step S33] For each feature point pair, the comparison image information selection unit 116 substitutes the coordinates of the feature points on the comparison image into the calculated transformation matrix H, and the corresponding coordinates on the recognition failure image. A distance D from the feature point coordinates is calculated.

  [Step S34] The comparative image information selection unit 116 determines whether the calculated distance D satisfies the following two determination conditions. As a first determination condition, it is determined whether the calculated minimum value Dmin of the distance D is equal to or less than a threshold value Tdmin. As a second determination condition, it is determined whether the number of pairs of feature point pairs for which the calculated distance D is equal to or greater than the threshold value Tdlim is equal to or less than the threshold value Tdnum. Tdmin, Tdlim, and Tdnum are all greater than 0 and satisfy the relationship of Tdmin <Tdlim.

  When both the first determination condition and the second determination condition are satisfied, it is determined that the comparison image selected in step S31 is suitable for the calculation of the imaging position and orientation information. In this case, the process of step S35 is performed. If at least one of the first determination condition and the second determination condition is not satisfied, the process of step S31 is executed.

  Although not shown, if steps S31 to S34 are repeated and there is no comparison image that satisfies both the first determination condition and the second determination condition, it is determined as “No” in step S28. The processing is terminated in the same manner as in the case where it has been done.

  [Step S35] The comparison image information selection unit 116 reads marker position information corresponding to the comparison image determined to be suitable for the calculation of the imaging position and orientation information in step S34 from the comparison image information database 140. The comparison image information selection unit 116 converts the read marker position information, that is, the coordinates of the four vertices, into the conversion matrix H calculated in step S32 using the corresponding comparison image. The coordinates of each vertex after conversion indicate an estimated position of the coordinates of each vertex of the marker in the recognition failure image.

  [Step S36] The marker position information converted in step S35 is input to the imaging position / orientation estimation unit 117. The imaging position / orientation estimation unit 117 calculates imaging position / orientation information using the input marker position information after conversion. Thereby, the estimated value of the imaging position and orientation information corresponding to the recognition failure image is calculated.

  [Step S37] The imaging position and orientation information calculated in Step S36 and the work ID specified in Step S22 are input to the superimposed display control unit 114. The superimposed display control unit 114 specifies work support information corresponding to the input work ID from the work support information database 130. The superimposed display control unit 114 determines the display position of the identified work support information in the captured image based on the input imaging position / orientation information by using the above-described equations (1), (7-1), and (7-2). Use to calculate. The superimposed display control unit 114 displays the imaging position / orientation information at the calculated display position in the captured image.

  In the processes of FIGS. 10 to 12 described above, when the terminal device 100 fails in the marker recognition, the terminal device 100 performs a recognition failure by matching feature points from the comparison images registered in the comparison image information database 140. A comparative image similar to the image is extracted. Thereafter, the imaging position / orientation information is calculated using information of similar comparative images, so that the display position of the work support information can be accurately determined.

  Further, the terminal device 100 estimates the imaging position / orientation information by evaluating the coordinate conversion accuracy using the conversion matrix H for converting the coordinates from the comparison image determined to be similar to the recognition failure image. An appropriate comparison image is selected. By calculating the imaging position / orientation information using the comparison images thus selected, the accuracy of the display position of the work support information is improved.

In addition, the terminal device 100 of the above 2nd Embodiment can also be deform | transformed as follows.
Instead of registering the recognized marker position information in the comparative image information database 140 in step S15, the information accumulation control unit 115 uses the imaging position and orientation information calculated based on the marker position information as the comparative image information database 140. Register with. As the registered imaging position / orientation information, a value calculated by the imaging position / orientation estimation unit 113 is used.

  Further, the following processing is performed instead of steps S35 and S36 in FIG. The imaging position / orientation estimation unit 117 reads out the imaging position / orientation information corresponding to the comparison image determined to be suitable for the calculation of the imaging position / orientation information from the comparison image information database 140. The imaging position / orientation estimation unit 117 converts the read imaging position / orientation information using the conversion matrix H calculated in step S32. The imaging position / orientation estimation unit 117 outputs the converted imaging position / orientation information to the superimposed display control unit 114.

  Even in such a process, the work support information can be superimposed and displayed at an accurate position. However, according to the processing described in FIGS. 10 to 12, it is much more processing load to convert the marker position information using the conversion matrix H than to convert the imaging position / orientation information using the conversion matrix H. Low. Furthermore, when the marker position information is converted using the conversion matrix H and the imaging position / orientation information is calculated using the converted marker position information, the error at the time of conversion by the conversion matrix H becomes smaller, and the work support information Superposition position accuracy can be increased.

[Third Embodiment]
Next, a terminal device according to the third embodiment will be described. The terminal device according to the third embodiment is obtained by modifying a part of the terminal device 100 according to the second embodiment. Hereinafter, differences from the terminal device 100 according to the second embodiment will be described.

  FIG. 13 is a block diagram illustrating a configuration example of processing functions included in the terminal device according to the third embodiment. In FIG. 13, the same components as those in FIG. 8 are denoted by the same reference numerals, and the description thereof is omitted.

  The terminal device 100a shown in FIG. 13 includes the information accumulation control unit 115, the comparison image information selection unit 116, and the comparison image information database 140 of the terminal device 100 shown in FIG. 116a and the comparison image information database 140a.

  Similar to the information accumulation control unit 115 in FIG. 8, the information accumulation control unit 115 a registers information regarding the captured image in the comparative image information database 140 a when the marker recognition unit 112 succeeds in recognizing a marker. However, in addition to the work ID, the position and feature amount of the feature point, and the marker position information, the information accumulation control unit 115a also stores the imaging position / orientation information calculated by the imaging position / orientation estimation unit 113 in the comparison image information database 140a. sign up.

  In addition, the information accumulation control unit 115a classifies information for each comparison image corresponding to the same work ID registered in the comparison image information database 140a into a plurality of groups based on the imaging position and orientation information. Furthermore, the information accumulation control unit 115a selects a representative image from the comparison images belonging to each group.

  When the marker recognizing unit 112 fails to recognize the marker from the captured image, the comparative image information selecting unit 116a first detects the recognition failure image that has failed to recognize the marker, and the representative image of each group in the comparative image database 140a. Perform matching. Then, the comparison image information selection unit 116a excludes the group to which the representative image that has failed in matching belongs from the processing target, and is similar to the comparison image information selection unit 116 in FIG. 8 from the comparison targets belonging to the remaining groups. By processing, information of a comparison image appropriate for use in estimation of imaging position and orientation information is selected.

  Here, depending on the surrounding environment of the location where the marker is installed, such as the width of the passage in front of the facility where the marker is installed and the positional relationship with the adjacent equipment, and the display position of the work support information set for the marker Some restrictions are imposed on the imaging position and imaging angle. Accordingly, the information accumulation control unit 115 classifies past captured images (comparison images) that have been successfully recognized with the same marker into groups based on the imaging position and orientation information calculated based on the marker recognition result. It is possible.

  Further, in the comparison process between the recognition failure image and the comparison image by the comparison image information selection unit 116, the feature point matching between the images may fail if there is a large difference in the imaging position and orientation information between the images to be compared. High nature. For this reason, it is not necessary to set a comparison image having a large difference in imaging position / orientation information from the recognition failure image as a matching processing target.

  Accordingly, the comparison image information selection unit 116a performs matching between the recognition failure image and the representative image of each classified group, and excludes the group to which the representative image that failed to match belongs from the target of subsequent processing. . As a result, the number of matching processes is reduced, and the processing load can be reduced.

FIG. 14 is a diagram illustrating a configuration example of information recorded in the comparative image information database.
The comparison image information database 140a is provided with a record 141a for each work ID, as in the second embodiment. Further, one or more pieces of comparison image information are registered in each record 141a, and the comparison image information is classified and registered for each group. A group is identified by a group ID. Further, a representative image ID is registered for each group. The representative image ID indicates an image ID given to information corresponding to the representative image among the comparison image information belonging to the group.

  In addition to the image ID, the marker position information, and the plurality of feature point information, the imaging position / orientation information is registered in the comparison image information. As the imaging position / orientation information, the rotation matrix R and the translation vector T calculated by the imaging position / orientation estimation unit 113 are registered as they are. Further, for example, in order to facilitate the grouping process, the rotation matrix R and the translation vector T include a camera position (x, y, z) with reference to the marker center in the marker coordinate system, and the marker coordinate system. It may be converted into a rotation angle (rx, ry, rz) with reference to the three axes and registered in the comparative image information database 140a. Further, as the rotation angle (rx, ry, rz), the rotation angle θ calculated by the equation (18) described later may be used.

Next, portions of the processing of the terminal device 100a that are different from the processing illustrated in FIGS. 10 to 12 will be described. First, the processing of the information accumulation control unit 115a will be described.
When the marker recognizing unit 112 succeeds in recognizing the marker, in step S15 in FIG. 10, the information accumulation control unit 115a performs imaging position / posture estimation in addition to the work ID, the position and feature amount of the feature point, and the marker position information. The imaging position / orientation information calculated by the unit 113 is registered in the comparison image information database 140a. Furthermore, the information accumulation control unit 115a performs the classification process shown in FIG. 15 on the information registered in the comparative image information database 140a, for example, at regular intervals.

FIG. 15 is a flowchart illustrating an example of the procedure of classification processing by the information accumulation control unit. The process of FIG. 15 is performed on the comparison image information associated with the same work ID.
[Step S51] The information accumulation control unit 115a classifies the comparison images into groups using the imaging position and orientation information of each comparison image registered in the comparison image information database 140a. As a classification method, the k-means method can be used. Here, when the k-means method is used, as the imaging position / orientation information used for the classification calculation, from the rotation matrix R and the translation vector T, the camera position (x, y, z) and the rotation angle (rx, ry, rz) can reduce the processing load.

The information accumulation control unit 115a updates the comparative image information database 140 so that the comparative images are classified for each group.
[Step S52] The information accumulation control unit 115a identifies a representative image for each classified group. For example, the information accumulation control unit 115a determines that the camera position (x, y, z) or the rotation angle θ of the comparison images belonging to the group is the camera position (x, y, z) or the rotation angle θ of the entire group. A comparative image closest to the center of gravity is specified as a representative image. When specifying the representative image, the information accumulation control unit 115 a registers the image ID of the representative image in the field of the representative image ID provided in the corresponding group in the comparison image information database 140.

  Next, the process of the comparison image information selection unit 116a is a process obtained by modifying the processes of steps S21 to S28 and S31 to S35 of FIGS. 11 and 12 by the comparison image information selection unit 116 as follows.

  FIG. 16 is a flowchart illustrating an example of a part of the process performed by the comparison image information selection unit according to the third embodiment. In the process of FIG. 16, steps S61 and S62 are added between step S23 and step S24 of FIG.

  [Step S61] The comparison image information selection unit 116a uses the feature points in each representative image for each group corresponding to the corresponding work ID registered in the comparison image information database 140 and the feature points extracted from the recognition failure image. Match. The matching method and the method for determining whether the matching is successful are the same as in step S25.

  The comparison image information selection unit 116a determines whether the following two determination conditions are satisfied as a result of matching with each representative image. As a first determination condition, it is determined whether the ratio of the number of successful combinations to the total number of feature points extracted from the recognition failure image is larger than a threshold value Tb1 (however, Tb1 is greater than 0 and less than or equal to 1). Yes, Tb1 <Tb). As a second determination condition, it is determined whether or not the number of pairs successfully matched is greater than a threshold Tc1 (where Tc1 is greater than 0 and Tc1 <Tc).

  [Step S62] When there is a representative image that does not satisfy even one of the above two conditions, the comparative image information selection unit 116a selects a comparative image belonging to the group corresponding to the representative image for the processing from step S24 onward. Exclude from the target.

  As described above, in the third embodiment, when the terminal device 100a fails to recognize the marker, the terminal device 100a performs matching between the recognition failure image and the representative image of each group classified based on the imaging position and orientation information. Do. Then, the group to which the representative image that has failed in matching belongs is excluded from the target of subsequent processing. As a result, the number of matching processes is reduced, and the processing load can be reduced.

[Fourth Embodiment]
Next, a terminal device according to the fourth embodiment will be described. The terminal device according to the fourth embodiment is obtained by modifying a part of the terminal device 100a according to the third embodiment. Hereinafter, differences from the terminal device 100a according to the third embodiment will be described.

  The basic configuration of the processing functions of the terminal device according to the fourth embodiment is the same as that of the terminal device 100a shown in FIG. Therefore, the terminal device according to the fourth embodiment will be described using the reference numerals shown in FIG.

FIG. 17 is a diagram illustrating an example of a screen accompanying display of a guide frame.
The terminal device 100a according to the fourth embodiment displays a guide frame 252 in the captured image 251a on the display device 104 as shown in the upper side of FIG. Then, the operator is prompted to match the outer shape of the marker shown in the captured image 251a with the guide frame 252. For example, a guide image 253 for prompting the operator to perform such an operation is displayed on the captured image 251a. Further, a completion button 254 for the operator to input that such an operation has been completed is displayed on the captured image 251a.

  If the marker is not recognized by the terminal device 100a, but the operator can see at least a part of the marker through the captured image 251a, the operator changes the imaging position and direction so that the outer shape of the marker matches the guide frame 252. Then, when the outer shape of the marker matches the guide frame 252, the operator presses the completion button 254 in that state. Then, a captured image in a state where the outer shape of the marker matches the guide frame 252 is captured by the terminal device 100a as in a captured image 251b illustrated on the lower side of FIG. The terminal device 100a uses the captured image that has been taken in as a recognition failure image, and performs a process for determining the superimposed position of the work support information.

  Here, in the comparison image information database 140a of the terminal device 100a according to the fourth embodiment, as in the third embodiment, the comparison image information classified for each work ID is further added to the imaging position and orientation information. Based on classification and registration. The comparison images belonging to each group classified based on the imaging position / orientation information are captured at similar imaging positions and imaging directions for each group.

  The terminal device 100a makes the shape, size, and display position of the guide frame 252 to be displayed the same as the shape, size, and display position of the marker recognized from the representative image of a certain group. In this case, the imaging position and direction when the marker on the captured image is aligned with the guide frame 252 by the operation of the operator is similar to the imaging position and direction when the representative image belonging to the group is captured. Become. For this reason, in the process for determining the superimposed position of the work support information in the terminal device 100a, sufficient superimposed position accuracy can be obtained only by using the comparison image information belonging to the group.

  Furthermore, it is desirable that the group to be used has the largest number of registered comparison image information. Thereby, a large number of pieces of comparative image information captured at an imaging position and direction similar to the current imaging position and direction can be used for the superposition position determination process. As a result, there is a higher possibility that the superimposition position accuracy is improved than in the case of using comparative image information belonging to another group.

  FIG. 18 is a flowchart illustrating an example of a part of processing performed by the comparison image information selection unit according to the fourth embodiment. In the process of FIG. 18, steps S71 and S72 are added between step S23 and step S24 of FIG.

  [Step S71] The comparison image information selection unit 116a selects a group having the largest number of comparison image information registrations from the record 141a corresponding to the corresponding work ID in the comparison image information database 140a. The comparative image information selection unit 116a reads marker position information corresponding to the representative image of the selected group from the record 141a.

  The comparative image information selection unit 116a causes the display device 104 to display a guide frame having the same shape as the outer shape of the marker indicated by the read marker position information. Specifically, the coordinates of the four vertices of the guide frame are made the same as the coordinates of the four vertices included in the read marker position information. Also, as shown in FIG. 17, an image for notifying the worker to align the marker position with the position of the guide frame and an operation button for the operator to perform the operation completion operation are also displayed.

  [Step S72] When the operation completion operation is performed by the operator, the comparison image information selection unit 116a captures the captured image at that time through the captured image acquisition unit 111, and the recognition failure image that uses the captured image thereafter. And Further, the comparison image information selection unit 116a sets only the comparison image information belonging to the group selected in step S71 as a processing target after step S24, and excludes other comparison image information from the processing target.

In the above fourth embodiment, it is possible to reduce the processing load by narrowing down the comparison image information used for the determination process of the superimposition position while maintaining the superimposition position accuracy of the work support information.
[Fifth Embodiment]
Next, a terminal device according to a fifth embodiment will be described. The terminal device according to the fifth embodiment is obtained by modifying a part of the terminal device 100a according to the third embodiment. Hereinafter, differences from the terminal device 100a according to the fifth embodiment will be described.

  The basic configuration of the processing functions of the terminal device according to the fifth embodiment is the same as that of the terminal device 100a shown in FIG. Therefore, the terminal apparatus according to the fifth embodiment will be described using the reference numerals shown in FIG.

  When the terminal device 100a according to the fifth embodiment fails to recognize the marker, the terminal device 100a prompts the operator to input the outer shape of the marker on the captured image. The operator can input the outer shape of the marker on the captured image when at least part of the marker can be viewed through the captured image. As an input method at this time, for example, a method of inputting the positions of four vertices on the screen of the marker outline by a touch operation, a mouse click operation, or the like can be considered. Thereby, the terminal device 100a acquires information on the shape and position of the marker on the captured image.

  The terminal device 100a extracts a comparison image in which a marker similar to the marker based on the input information (that is, highly relevant) is recognized from the comparison image corresponding to the corresponding work ID. In this process, the terminal device 100a performs matching between information regarding the marker indicated by the input shape and position and information regarding the representative image of each group corresponding to the corresponding work ID. In this matching, for example, in addition to the four vertices of the marker, for example, imaging position and orientation information based on the marker may be used.

  As a result of the matching, the terminal device 100a selects a representative image whose matching accuracy is a certain value or more. Then, the terminal device 100a uses only the comparison image information belonging to the group corresponding to the selected representative image for the process of determining the superposition position of the work support information.

By the way, in the fifth embodiment, the rotation matrix R and the translation vector T calculated by the imaging position / orientation estimation unit 113 may be registered as they are in the comparison image information database 140a as the imaging position / orientation information. However, for example, to convert a rotation matrix R rotation axis vector [r _x r _y r _z] and the rotation angle theta, the transformed rotational axis vector [r _x r _y r _z] and the rotation angle theta, the rotation matrix R Instead, it may be registered in the comparative image information database 140a. Thereby, the value optimized for matching is registered.

Relationship between the rotation matrix R calculated from the recognition result of the marker, and the rotation axis vector [r _x r _y r _z] and the rotation angle θ, using the formula Rodriguez expressed as the following equation (16) The

From the relation of formula (16), the rotation axis vector [r _x r _y r _z] and the rotation angle theta, each of the following equation (17), can be expressed as (18).
_{[R x r y r z]} = || (r 32 -r 23) (r 13 -r 31) (r 21 -r 12) || ··· (17)
θ = cos ⁻¹ ((trace (R) −1) / 2) (18)
Equation (17) is a normalized expression of the rotation axis vector. Also, θ takes a value between 0 and π. Also, trace (R) in equation (18) represents the sum of the diagonal elements of R.

Such rotation axis vector [r _x r _y r _z] and the rotation angle θ matching processing using a will be described later.
FIG. 19 is a flowchart illustrating an example of part of processing performed by the comparison image information selection unit according to the fifth embodiment. In the process of FIG. 19, steps S81 to S83 are added between step S23 and step S24 of FIG.

  [Step S81] The comparison image information selection unit 116a prompts the operator to input the shape of the marker on the captured image. For example, the comparison image information selection unit 116a causes the display device 104 to display image information for prompting such an input operation. An operation button for notifying that the operator has completed such an input operation is also displayed.

  [Step S82] When the operator inputs a marker shape and the operation button is pressed, the comparative image information selection unit 116a captures the captured image from the captured image acquisition unit 111, and uses the captured image thereafter. Recognition failure image. Further, the comparison image information selection unit 116a recognizes the coordinates of the four vertices of the marker on the captured image based on the information input by the operator.

The comparison image information selection unit 116a calculates imaging position / orientation information corresponding to the input marker shape based on the recognized coordinates. For example, the comparison image information selection unit 116a calculates the rotation matrix R and the translation vector T in the same procedure as the imaging position / orientation estimation units 113 and 117. Furthermore, based on the calculated rotation matrix R, the above formula (17), calculates a rotation axis vector [r _x r _y r _z] and the rotation angle θ according to (18).

  [Step S83] The comparison image information selection unit 116a matches the information related to the input marker shape with the information related to the representative image of each group corresponding to the corresponding work ID. In this matching, for example, matching of imaging position and orientation information and matching of coordinates of four vertices are performed.

As an example, here, the comparison image information selecting section 116a, first, between the input marker shape and each representative image, the rotation axis vector [r _x r _y r _z] and translation component (component of the translation vector T) Each of the images is matched, and a representative image having a matching accuracy equal to or higher than a certain threshold is selected. Next, the comparison image information selection unit 116a performs matching for each of the rotation angle θ and the coordinates of the four vertices, and further selects representative images that have a matching accuracy equal to or higher than a threshold value.

First, the matching of the rotation axis vector [r _x r _y r _z], the rotation axis vector based on the respective input marker shape and representative images [r _x r _y r _z] angle ω of is calculated. The angle omega, regarded as the difference of the rotation axis vector _{[r x r y r z]} . In translation component matching, a difference d in distance between translation components is calculated.

The angle ω formed by the rotation axis vector r ₁ = [r _x1 r _y1 r _z1 ] and r ₂ = [r _x2 r _y2 r _z2 ] is calculated by the following equation (19). Further, the distance difference d between the translation components [t _x1 t _y1 t _z1 ] and [t _x2 t _y2 t _z2 ] is calculated by the following equation (20).

Comparative image information selecting section 116a is rotated axis vector [r _x r _y r _z] angle ω of becomes equal to or less than a predetermined threshold value, and selecting a representative image differences d of the distance of the translation component is equal to or lower than a predetermined threshold value To do.

  Next, the comparison image information selection unit 116a calculates the difference between the rotation angle θ and the total difference between the coordinates of the four vertices between the input marker shape and the selected representative image. . The comparison image information selection unit 116a selects a representative image in which the difference in the rotation angle θ is equal to or smaller than a predetermined threshold value and the total value of the coordinate differences between the vertices is equal to or smaller than the predetermined threshold value.

  The comparison image information selection unit 116a extracts a representative image that satisfies all the conditions for the above matching, and sets a comparison image belonging to the group corresponding to the extracted representative image as a processing target after step S24. In addition, the comparison image information selection unit 116a excludes the comparison images belonging to the group corresponding to the representative image that does not satisfy even one of the above-described conditions from the targets of the processing after step S24.

  By such processing, the terminal device 100a can narrow down the comparison image information used for the determination processing of the superimposition position while maintaining the determination accuracy of the superimposition position of the work support information, and can reduce the processing load.

In the process of step S83, the rotation axis vector _{[r x r y r z]} , translational components, rotational angle theta, and among the parameters of the four vertices of the coordinates, at least one, or by two or more combinations, A matching process may be performed.

  Further, in step S83, the matching target with the input marker shape may be all comparison images associated with the corresponding work ID instead of the representative image. In this case, the comparison image that satisfies the matching accuracy condition is the target of processing after step S24, and the other comparison images are excluded from the processing target.

  The processing functions of the devices (display control device 1 and terminal devices 100 and 100a) described in the above embodiments can be realized by a computer. In that case, a program describing the processing contents of the functions that each device should have is provided, and the processing functions are realized on the computer by executing the program on the computer. The program describing the processing contents can be recorded on a computer-readable recording medium. Examples of the computer-readable recording medium include a magnetic storage device, an optical disk, a magneto-optical recording medium, and a semiconductor memory. Examples of the magnetic storage device include a hard disk device (HDD), a flexible disk (FD), and a magnetic tape. Optical disks include DVD (Digital Versatile Disc), DVD-RAM, CD-ROM (Compact Disc-Read Only Memory), CD-R (Recordable) / RW (ReWritable), and the like. Magneto-optical recording media include MO (Magneto-Optical disk).

  When distributing the program, for example, a portable recording medium such as a DVD or a CD-ROM in which the program is recorded is sold. It is also possible to store the program in a storage device of a server computer and transfer the program from the server computer to another computer via a network.

  The computer that executes the program stores, for example, the program recorded on the portable recording medium or the program transferred from the server computer in its own storage device. Then, the computer reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. In addition, each time a program is transferred from a server computer connected via a network, the computer can sequentially execute processing according to the received program.

Regarding the above embodiments, the following supplementary notes are further disclosed.
(Supplementary note 1) An imaging position and an imaging posture based on the marker are calculated based on the marker position information recognized from the captured image, and predetermined work support is provided at a position on the captured image based on the calculated imaging position and imaging posture. In a display control device that superimposes and displays information,
When the recognition of the marker from the captured image is successful, the first information about the captured image used for calculating the similarity between the captured image and another image, and the recognition result of the marker from the captured image A storage processing unit that records the second information based on the storage device;
When recognition of the marker from the first captured image fails, a second captured image similar to the first captured image is obtained based on the first information for each captured image recorded in the storage device. The corresponding second information is extracted from the storage device, the imaging position and the imaging posture corresponding to the first captured image are calculated based on the extracted second information, and the calculated imaging position and imaging A display position determining unit that determines the superimposed display position of the work support information based on the posture;
A display control device comprising:

(Supplementary Note 2) The display position determination unit
Extracting the first information corresponding to the second captured image similar to the first captured image from the storage device;
Based on the extracted first information, a conversion formula for converting the coordinates of the plurality of feature points in the second captured image into the coordinates of the plurality of feature points in the first captured image is calculated, and the conversion Calculate the evaluation value of coordinate transformation by formula,
When the evaluation value satisfies a predetermined condition, an imaging position and an imaging posture are calculated based on the second information corresponding to the second captured image.
The display control apparatus according to Supplementary Note 1, wherein

(Supplementary Note 3) The second information includes marker position information in the captured image.
The display position determination unit converts the position information of the marker included in the second information corresponding to the second captured image using the conversion formula when the evaluation value satisfies the predetermined condition. Calculating the imaging position and orientation based on the converted marker position information,
The display control apparatus according to Supplementary Note 2, wherein

(Supplementary Note 4) The second information includes marker position information in the captured image.
The display position determining unit
Extracting the first information corresponding to the second captured image similar to the first captured image from the storage device;
Based on the extracted first information, a conversion formula for converting the coordinates of the plurality of feature points in the second captured image into the coordinates of the plurality of feature points in the first captured image is calculated.
The marker position information included in the second information corresponding to the second captured image is converted using the conversion formula, and the imaging position and the imaging posture are calculated based on the converted marker position information. ,
The display control apparatus according to Supplementary Note 1, wherein

(Supplementary Note 5) In the storage device, the first information and the second information are classified and recorded into a plurality of groups based on an imaging position and an imaging attitude calculated from the corresponding captured image,
The display position determination unit uses the first information selected one by one from the plurality of groups, and uses the first imaging from the captured images respectively corresponding to the selected first information. Extract a captured image similar to the image, select a group to which the extracted captured image belongs, and use the plurality of first information included in the selected group to correspond to the first information included in the group Extracting the second captured image similar to the first captured image from the captured images
The display control apparatus according to Supplementary Note 1, wherein

(Supplementary Note 6) The display position determination unit
When recognition of the marker from the captured image fails, a group having the largest number of the first information and the second information is selected from the plurality of groups, and the second group is selected from the selected group. Select one piece of information, and display a guide frame of the same shape and size as the marker based on the selected second information at the same position as the marker on the captured image,
Using the captured image captured after the guide frame is displayed, using the plurality of first information included in the selected group, from among captured images corresponding to the first information included in the group. Extracting the second captured image similar to the captured image captured after display of the guide frame;
The display control apparatus according to appendix 5, characterized in that:

(Appendix 7) The display position determination unit
When recognition of the marker from the first captured image fails, an operation for inputting information on the shape and position of the marker in the first captured image is received,
The second information selected one by one from the plurality of groups and the information on the marker based on the input position information are compared, and the captured image corresponding to the selected second information is compared. From the inside, the captured image in which the marker indicating the relevance with the marker based on the position information satisfies the predetermined condition is determined,
Using the plurality of first information included in the group to which the determined captured image belongs, the first image similar to the first captured image among the captured images corresponding to the first information included in the group. 2 captured images are extracted,
The display control apparatus according to appendix 5, characterized in that:

(Supplementary Note 8) An imaging position and an imaging attitude based on the marker are calculated based on the marker position information recognized from the captured image, and a predetermined work support is provided at a position on the captured image based on the calculated imaging position and imaging attitude. In a display control method for displaying information superimposed,
When the recognition of the marker from the captured image is successful, the first information about the captured image used for calculating the similarity between the captured image and another image, and the recognition result of the marker from the captured image And recording the second information based on the storage device,
When recognition of the marker from the first captured image fails, a second captured image similar to the first captured image is obtained based on the first information for each captured image recorded in the storage device. The corresponding second information is extracted from the storage device, the imaging position and the imaging posture corresponding to the first captured image are calculated based on the extracted second information, and the calculated imaging position and imaging Determine the superimposed display position of the work support information based on the posture,
A display control method characterized by the above.

(Supplementary Note 9) In the calculation of the imaging position and the imaging posture corresponding to the first captured image,
Extracting the first information corresponding to the second captured image similar to the first captured image from the storage device;
Based on the extracted first information, a conversion formula for converting the coordinates of the plurality of feature points in the second captured image into the coordinates of the plurality of feature points in the first captured image is calculated, and the conversion Calculate the evaluation value of coordinate transformation by formula,
When the evaluation value satisfies a predetermined condition, an imaging position and an imaging posture are calculated based on the second information corresponding to the second captured image.
The display control method according to appendix 8, which is characterized by the above.

(Supplementary Note 10) The second information includes marker position information in the captured image.
In the calculation of the imaging position and the imaging posture corresponding to the first captured image,
The display position determination unit converts the position information of the marker included in the second information corresponding to the second captured image using the conversion formula when the evaluation value satisfies the predetermined condition. Calculating the imaging position and orientation based on the converted marker position information,
The display control method according to appendix 9, wherein

(Supplementary Note 11) The second information includes marker position information in the captured image.
In the calculation of the imaging position and the imaging posture corresponding to the first captured image,
Extracting the first information corresponding to the second captured image similar to the first captured image from the storage device;
Based on the extracted first information, a conversion formula for converting the coordinates of the plurality of feature points in the second captured image into the coordinates of the plurality of feature points in the first captured image is calculated.
The marker position information included in the second information corresponding to the second captured image is converted using the conversion formula, and the imaging position and the imaging posture are calculated based on the converted marker position information. ,
The display control method according to appendix 8, which is characterized by the above.

(Supplementary Note 12) In the storage device, the first information and the second information are classified and recorded in a plurality of groups based on an imaging position and an imaging posture calculated from a corresponding captured image,
In the extraction of the second information corresponding to the second captured image, the first information selected from the plurality of groups is used, and the selected first information is used. A captured image similar to the first captured image is extracted from the captured images to be selected, a group to which the extracted captured image belongs is selected, and a plurality of the first information included in the selected group is used to Extracting the second captured image similar to the first captured image from the captured images corresponding to the first information included in the group;
The display control method according to appendix 8, which is characterized by the above.

(Additional remark 13) When recognition of the marker from a captured image fails, the group with the largest number in which the said 1st information and the said 2nd information are contained is selected from among the said several groups, From the selected group One of the second information is selected, and a guide frame having the same shape and the same size as the marker based on the selected second information is displayed at the same position as the marker on the captured image.
In the extraction of the second information corresponding to the second captured image, using the captured information captured after the guide frame is displayed, using the plurality of first information included in the selected group, Extracting the second captured image similar to the captured image captured after the display of the guide frame from the captured images corresponding to the first information included in the group;
The display control method according to supplementary note 12, characterized by:

(Additional remark 14) When recognition of the marker from the said 1st captured image fails, the operation which inputs the information of the shape and position of the marker in the said 1st captured image is received,
In the extraction of the second information corresponding to the second captured image,
The second information selected one by one from the plurality of groups and the information on the marker based on the input position information are compared, and the captured image corresponding to the selected second information is compared. From the inside, the captured image in which the marker indicating the relevance with the marker based on the position information satisfies the predetermined condition is determined,
Using the plurality of first information included in the group to which the determined captured image belongs, the first image similar to the first captured image among the captured images corresponding to the first information included in the group. 2 captured images are extracted,
The display control method according to supplementary note 12, characterized by:

(Supplementary Note 15) An imaging position and an imaging attitude based on the marker are calculated based on the marker position information recognized from the captured image, and predetermined work support is provided at a position on the captured image based on the calculated imaging position and imaging attitude. In a display control program for causing a computer to execute processing for displaying information in a superimposed manner,
In the computer,
When the recognition of the marker from the captured image is successful, the first information about the captured image used for calculating the similarity between the captured image and another image, and the recognition result of the marker from the captured image And recording the second information based on the storage device,
When recognition of the marker from the first captured image fails, a second captured image similar to the first captured image is obtained based on the first information for each captured image recorded in the storage device. The corresponding second information is extracted from the storage device, the imaging position and the imaging posture corresponding to the first captured image are calculated based on the extracted second information, and the calculated imaging position and imaging Determine the superimposed display position of the work support information based on the posture,
A display control program for executing processing.

DESCRIPTION OF SYMBOLS 1 Display control apparatus 2 Accumulation process part 3 Display position determination part 4 Memory | storage device 5a 1st information 5b 2nd information 11a, 11b Captured image 12 Image 21 Marker 22 Work support information

Claims (9)

Based on the marker position information recognized from the captured image, the imaging position and orientation are calculated based on the marker, and predetermined work support information is superimposed on the position on the captured image based on the calculated imaging position and orientation. In the display control device to be
When the recognition of the marker from the captured image is successful, the first information about the captured image used for calculating the similarity between the captured image and another image, and the recognition result of the marker from the captured image A storage processing unit that records the second information based on the storage device;
When recognition of the marker from the first captured image fails, a second captured image similar to the first captured image is obtained based on the first information for each captured image recorded in the storage device. The corresponding second information is extracted from the storage device, the imaging position and the imaging posture corresponding to the first captured image are calculated based on the extracted second information, and the calculated imaging position and imaging A display position determining unit that determines the superimposed display position of the work support information based on the posture;
A display control device comprising: The display position determining unit
Extracting the first information corresponding to the second captured image similar to the first captured image from the storage device;
Based on the extracted first information, a conversion formula for converting the coordinates of the plurality of feature points in the second captured image into the coordinates of the plurality of feature points in the first captured image is calculated, and the conversion Calculate the evaluation value of coordinate transformation by formula,
When the evaluation value satisfies a predetermined condition, an imaging position and an imaging posture are calculated based on the second information corresponding to the second captured image.
The display control apparatus according to claim 1. The second information includes marker position information in the captured image,
The display position determination unit converts the position information of the marker included in the second information corresponding to the second captured image using the conversion formula when the evaluation value satisfies the predetermined condition. Calculating the imaging position and orientation based on the converted marker position information,
The display control apparatus according to claim 2. The second information includes marker position information in the captured image,
The display position determining unit
Extracting the first information corresponding to the second captured image similar to the first captured image from the storage device;
Based on the extracted first information, a conversion formula for converting the coordinates of the plurality of feature points in the second captured image into the coordinates of the plurality of feature points in the first captured image is calculated.
The marker position information included in the second information corresponding to the second captured image is converted using the conversion formula, and the imaging position and the imaging posture are calculated based on the converted marker position information. ,
The display control apparatus according to claim 1. In the storage device, the first information and the second information are classified and recorded into a plurality of groups based on an imaging position and an imaging posture calculated from a corresponding captured image,
The display position determination unit uses the first information selected one by one from the plurality of groups, and uses the first imaging from the captured images respectively corresponding to the selected first information. Extract a captured image similar to the image, select a group to which the extracted captured image belongs, and use the plurality of first information included in the selected group to correspond to the first information included in the group Extracting the second captured image similar to the first captured image from the captured images
The display control apparatus according to claim 1. The display position determining unit
When recognition of the marker from the captured image fails, a group having the largest number of the first information and the second information is selected from the plurality of groups, and the second group is selected from the selected group. Select one piece of information, and display a guide frame of the same shape and size as the marker based on the selected second information at the same position as the marker on the captured image,
Using the captured image captured after the guide frame is displayed, using the plurality of first information included in the selected group, from among captured images corresponding to the first information included in the group. Extracting the second captured image similar to the captured image captured after display of the guide frame;
The display control apparatus according to claim 5. The display position determining unit
When recognition of the marker from the first captured image fails, an operation for inputting information on the shape and position of the marker in the first captured image is received,
The second information selected one by one from the plurality of groups and the information on the marker based on the input position information are compared, and the captured image corresponding to the selected second information is compared. From the inside, the captured image in which the marker indicating the relevance with the marker based on the position information satisfies the predetermined condition is determined,
Using the plurality of first information included in the group to which the determined captured image belongs, the first image similar to the first captured image among the captured images corresponding to the first information included in the group. 2 captured images are extracted,
The display control apparatus according to claim 5. Based on the marker position information recognized from the captured image, the imaging position and orientation are calculated based on the marker, and predetermined work support information is superimposed on the position on the captured image based on the calculated imaging position and orientation. In the display control method to
When the recognition of the marker from the captured image is successful, the first information about the captured image used for calculating the similarity between the captured image and another image, and the recognition result of the marker from the captured image And recording the second information based on the storage device,
When recognition of the marker from the first captured image fails, a second captured image similar to the first captured image is obtained based on the first information for each captured image recorded in the storage device. The corresponding second information is extracted from the storage device, the imaging position and the imaging posture corresponding to the first captured image are calculated based on the extracted second information, and the calculated imaging position and imaging Determine the superimposed display position of the work support information based on the posture,
A display control method characterized by the above. Based on the marker position information recognized from the captured image, the imaging position and orientation are calculated based on the marker, and predetermined work support information is superimposed on the position on the captured image based on the calculated imaging position and orientation. In a display control program for causing a computer to execute
In the computer,
When the recognition of the marker from the captured image is successful, the first information about the captured image used for calculating the similarity between the captured image and another image, and the recognition result of the marker from the captured image And recording the second information based on the storage device,
When recognition of the marker from the first captured image fails, a second captured image similar to the first captured image is obtained based on the first information for each captured image recorded in the storage device. The corresponding second information is extracted from the storage device, the imaging position and the imaging posture corresponding to the first captured image are calculated based on the extracted second information, and the calculated imaging position and imaging Determine the superimposed display position of the work support information based on the posture,
A display control program for executing processing.

Images