JP2020194482A - Display control method, display control program, and information processor - Google Patents

Abstract

To reduce workload when projection images of three-dimensional models are displayed in a superimposed manner.SOLUTION: A processing section 12 acquires a captured image 40 including a structure captured by an imaging device and acquires projection images 31,32,33, etc. of the structure from a storage section 11. The processing section 12 determines the projection image 32 (a first projection image) corresponding to a shape of the structure included in the captured image 40 among the projection images 31,32,33, etc., and displays ridge lines (a ridge line group 51) for a three-dimensional model included in the identified projection image 32 and edge lines (an edge line group 52) extracted from the captured image 40. The processing section 12 generates a projection image (a second projection image) of the three-dimensional model in which positions of a predetermined number of ridge lines are respectively associated with positions of the predetermined number of edge lines associated with the predetermined number of ridge lines when receiving an operation for respectively associating the predetermined number of ridge lines included in the displayed ridge lines with the predetermined number of edge lines included in the displayed edge lines. The processing section 12 superimposes the generated projection image on the captured image.SELECTED DRAWING: Figure 1

Description

The present invention relates to a display control method, a display control program, and an information processing device.

Three-dimensional CAD (Computer Aided Design) is used for the design of various structures. In addition, a user such as a manufacturer may confirm whether or not the manufactured structure is manufactured as designed in operations such as inspection. Therefore, it is considered to support the work such as inspection of the user by using the information processing technology. For example, there is a proposal of a display control device that superimposes and displays an captured image of a manufactured structure and a model of a three-dimensional CAD structure.

It should be noted that a template in which feature information representing a plurality of positions included in the contour when the three-dimensional model is mapped to a virtual plane based on a specific viewpoint and positions in three dimensions corresponding to the plurality of positions are associated with each other. There is also a proposal for an information processing device that creates a template for each specific viewpoint. The proposed information processing apparatus derives the position and orientation of the object in three dimensions based on the feature information representing the edge of the captured image of the object and the feature information of the template.

In addition, the shape, dimensions, placement angle, etc. of the product material for plant construction are measured in a non-contact manner, and the object boundary information of the product material obtained from the measured information is compared with the three-dimensional model information created in advance at the time of design. There is also a proposal for a plant construction support device to be used. The proposed plant construction support device acquires the individual identification number of the product material by collation and determines the arrival of the product material for plant construction at the plant construction site.

Japanese Unexamined Patent Publication No. 2018-142109 JP-A-2017-182274 Japanese Unexamined Patent Publication No. 2012-180191

The information processing device displays the captured image of the structure and the 3D model corresponding to the structure on the screen, and changes the posture of the 3D model. The posture of the displayed 3D model is changed according to the user's operation. May be changed to make it possible to compare the captured image of the structure with the 3D model. In this case, the posture of the 3D model when the 3D model is displayed for the first time becomes a problem.

For example, it is conceivable to display the three-dimensional model in a predetermined specific initial posture. However, the greater the degree of deviation between the posture of the structure in the captured image and the initial posture of the three-dimensional model, the more difficult it is for the user to change the posture of the three-dimensional model in order to appropriately compare the two.

In one aspect, it is an object of the present invention to provide a display control method, a display control program, and an information processing device that can reduce the workload when superimposing and displaying a projected image of a three-dimensional model.

In one aspect, a display control method is provided. In this display control method, a computer acquires a photographed image including a structure imaged by an imaging device, acquires a plurality of projection images generated from a three-dimensional model of the structure, and obtains a plurality of projected images. Among them, the first projected image according to the shape of the structure included in the acquired captured image is specified, and the ridgeline of the three-dimensional model included in the identified first projected image and the edge line extracted from the captured image. Is displayed, and when an operation of associating a predetermined number of ridges included in the displayed ridges with a predetermined number of edge lines included in the displayed edge lines is accepted, the positions of the predetermined number of ridges become the predetermined number of ridges. A process is executed in which a second projected image of the three-dimensional model corresponding to each of the positions of a predetermined number of associated edge lines is generated, and the generated second projected image is superimposed and displayed on the captured image.

Also, in one aspect, a display control program is provided.
Also, in one aspect, an information processing device is provided.

On one aspect, the workload of superimposing and displaying the projected image of the three-dimensional model can be reduced.

It is a figure which shows the information processing apparatus of 1st Embodiment. It is a figure which shows the hardware example of the information processing apparatus of the 2nd Embodiment. It is a figure which shows the functional example of an information processing apparatus. It is a figure which shows the example of the posture candidate table. It is a figure which shows the example of the posture candidate image. It is a figure which shows the example of the inverted index table. It is a figure which shows the example of the photographed image. It is a figure which shows the example of the outline included in the photographed image. It is a figure which shows the example of the edge included in the posture candidate image. It is a figure which shows the example of the point on the outline of a posture candidate image. It is a figure which shows the example of the line segment included in each posture candidate image. It is a figure which shows the example of the change of the outline according to the line-of-sight direction. It is a figure which shows the example of narrowing down a posture candidate. It is a figure which shows the example (continuation) of narrowing down a posture candidate. It is a figure which shows the example of the inverted index with respect to PostScript. It is a figure which shows the example of the hashing of an inverted index. It is a figure which shows the example of the hash value of an inverted index. It is a figure which shows the restoration example of an inverted index. It is a figure which shows the narrowing-down example of an inverted index. It is a figure which shows the example of false detection of an outline. It is a figure which shows the correction example at the time of false detection of an outline. It is a flowchart which shows the example of outline extraction. It is a flowchart which shows the narrowing-down example of a posture candidate. It is a flowchart which shows the example of 3D model superimposition control. It is a figure which shows the initial display example of a posture candidate image. It is a figure which shows the comparative example of the initial display of a posture candidate image.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
The first embodiment will be described.

FIG. 1 is a diagram showing an information processing apparatus according to the first embodiment.
The information processing device 10 supports the comparison between the captured image of the structure and the three-dimensional model. The information processing device 10 is connected to the display device 20. The information processing device 10 has a storage unit 11 and a processing unit 12.

The storage unit 11 may be a volatile storage device such as a RAM (Random Access Memory) or a non-volatile storage device such as an HDD (Hard Disk Drive) or a flash memory. The processing unit 12 may include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and the like. The processing unit 12 may be a processor that executes a program. A "processor" can include a set of multiple processors (multiprocessors).

The storage unit 11 stores a plurality of projected images generated from the three-dimensional model of the structure. For example, the storage unit 11 stores the projection images 31, 32, 33, ... Generated from the three-dimensional model of the target structure. The projected images 31, 32, 33, ... Are two-dimensional images generated from the three-dimensional model when the posture of the three-dimensional model is changed in steps of the rotation angle A1 and the elevation angle A2. The coordinate systems of the projected images 31, 32, 33, ... (For example, the horizontal direction is the x-axis and the vertical direction is the y-axis) are normalized in advance. The origin positions of the projected images 31, 32, and 33 are predetermined positions (for example, the center of each image).

The processing unit 12 acquires a photographed image including a structure imaged by the imaging device. In FIG. 1, the illustration of the imaging device is omitted. For example, the processing unit 12 acquires the captured image 40. The captured image 40 includes the structure image 41. The structure image 41 is an image of the structure captured by the imaging device. It can be said that the structure image 41 is an image showing the shape of the structure when viewed from the imaged direction.

The processing unit 12 acquires a plurality of projected images generated from the three-dimensional model of the structure. For example, the processing unit 12 acquires projection images 31, 32, 33, ... Generated from the three-dimensional model of the structure corresponding to the structure image 41 from the storage unit 11.

The processing unit 12 identifies the first projected image according to the shape of the structure included in the acquired captured image from the acquired plurality of projected images. For example, the processing unit 12 identifies the projected image 32 according to the shape of the structure included in the captured image 40 among the projected images 31, 32, 33, ....

More specifically, the processing unit 12 normalizes the coordinate system of the captured image 40 according to the coordinate system of the projected images 31, 32, 33, .... The origin position of the captured image 40 is a position corresponding to each of the projection images 31, 32, 33, .... The processing unit 12 extracts the position of the edge included in the structure image 41, the direction in which the edge extends, the position of the curve, and the like from the normalized captured image 40. The processing unit 12 determines the position of the edge extracted from the structure image 41, the direction in which the edge extends, the position of the curve, and the like, the position of the ridge line included in the projected images 31, 32, 33, ..., The direction in which the ridge line extends. , And match the position of the curve. Among the projected images 31, 32, 33, ..., The processing unit 12 identifies the projected image 32 that best matches the structure image 41 by collation.

Various methods can be considered as the collation method by the processing unit 12. For example, the processing unit 12 may perform collation using data (PDL data) represented in a page description language (PDL). PDL is a data format that expresses an image as text information. An example of PDL is PostScript®. The processing unit 12 collates the PDL data representing the shape of the structure based on the structure image 41 with the PDL data representing the shape of the three-dimensional model based on the projected images 31, 32, 33, ..., By collating the structure. The projected image 32 that best matches the object image 41 may be specified. By performing collation using PDL data, collation can be executed at a higher speed than performing collation using image data.

Alternatively, the processing unit 12 may use the inverted index of the PDL data for collation. For example, it is conceivable that the processing unit 12 creates an inverted index for each of the PDL data of the projected images 31, 32, 33, .... Based on the inverted index, the processing unit 12 searches the PDL data corresponding to the projected image, which has similar description contents to the PDL data of the structure image 41, among the PDL data of the projected images 31, 32, 33, ... You may. Collation can be executed faster by using the inverted index.

Further, when the processing unit 12 uses the PDL data for comparing the structure image 41 with the projected images 31, 32, 33, ..., The processing unit 12 outlines the shape of the structure in the structure image 41 and the projected images 31, 32, 33. , ... PDL data corresponding to the outline of the shape of the three-dimensional model based on each may be created. By using the PDL data corresponding to the outline, the collation can be performed with higher accuracy than using the edge other than the outline. In addition, when PDL data is not used, the processing unit 12 may compare the outlines of both images.

The processing unit 12 displays the ridge line of the three-dimensional model included in the specified projection image and the edge line extracted from the captured image. For example, the processing unit 12 causes the display device 20 to display the display image 50. The display image 50 includes a ridge line group 51 and an edge line group 52. The ridge line group 51 is a set of ridge lines of the three-dimensional model included in the projection image 32. The edge line group 52 is a set of edge lines of the structure image 41 extracted from the captured image 40.

The processing unit 12 accepts an operation of associating a predetermined number of ridge lines included in the displayed ridge lines with a predetermined number of edge lines included in the displayed edge lines. For example, the information processing device 10 is connected to an input device (not shown). The user can use the input device to perform an operation of associating the displayed ridge line with the displayed edge line. For example, the user can perform an operation of associating a predetermined number of ridge lines included in the ridge line group 51 with a predetermined number of edge lines included in the edge line group 52 while looking at the display image 50. The processing unit 12 accepts a user's operation of associating a ridge line with an edge line via an input device.

The processing unit 12 generates a second projection image of the three-dimensional model in which the positions of the predetermined number of ridge lines correspond to the positions of the predetermined number of edge lines associated with the predetermined number of ridge lines. The processing unit 12 superimposes and displays the generated second projected image on the captured image. In FIG. 1, the illustration of the second projected image is omitted. As a result, the user can compare the image of the actually manufactured structure with the image of the three-dimensional model, and can confirm whether or not the manufactured structure is manufactured as designed.

According to the information processing device 10, a photographed image including a structure captured by the image pickup device is acquired. Multiple projections generated from the 3D model of the structure are acquired. Among the plurality of acquired projected images, the first projected image corresponding to the shape of the structure included in the acquired captured image is specified. The ridgeline of the three-dimensional model included in the identified first projected image and the edgeline extracted from the captured image are displayed. An operation of associating a predetermined number of ridgelines included in the displayed ridgeline with a predetermined number of edge lines included in the displayed edge line is accepted. A second projection image of the three-dimensional model is generated in which the positions of the predetermined number of ridges correspond to the positions of the predetermined number of edge lines associated with the predetermined number of ridges. The generated second projected image is superimposed and displayed on the captured image.

As a result, it is possible to reduce the workload when superimposing and displaying the projected image of the three-dimensional model.
For example, it is conceivable to display the three-dimensional model in a predetermined specific initial posture. However, the greater the degree of deviation between the posture of the structure in the captured image and the initial posture of the three-dimensional model, the more difficult it is for the user to change the posture of the three-dimensional model in order to appropriately compare the two.

Therefore, the information processing device 10 identifies a projected image (first projected image) of the three-dimensional model having an initial posture close to the posture of the captured structure, and displays the specified projected image in the captured image. Reduce the discrepancy between the orientation of the structure and the initial orientation of the 3D model. Therefore, for example, in the display image 50, the user performs an operation of associating a predetermined number of ridge lines of the ridge line group 51 corresponding to the initial posture of the three-dimensional model with a predetermined number of edge lines included in the edge line group 52. Is also possible. That is, the user does not have to change the initial posture of the three-dimensional model by the user's own operation. By making it possible to omit the operation by the user in this way, the workload of the user can be reduced.

In addition, since the user can appropriately perform the association operation between the ridgeline of the three-dimensional model and the edge line extracted from the captured image, the posture of the structure reflected in the captured image can be changed to the relevant posture. The corresponding second projected image can be appropriately generated and superposed. In this way, the information processing device 10 can support the user to easily confirm the difference between the three-dimensional model and the manufactured structure.

[Second Embodiment]
Next, a second embodiment will be described.
FIG. 2 is a diagram showing a hardware example of the information processing apparatus according to the second embodiment.

The information processing device 100 includes a CPU 101, a RAM 102, an HDD 103, an output IF (InterFace) 104, an input IF 105, 106, a medium reader 107, and a NIC (Network Interface Card) 108. The CPU 101 corresponds to the processing unit 12 of the first embodiment. The RAM 102 or the HDD 103 corresponds to the storage unit 11 of the first embodiment.

The CPU 101 is a processor that executes a program instruction. The CPU 101 loads at least a part of the programs and data stored in the HDD 103 into the RAM 102 and executes the program. The CPU 101 may include a plurality of processor cores. Further, the information processing device 100 may have a plurality of processors. The processes described below may be performed in parallel using multiple processors or processor cores. Also, a set of multiple processors may be referred to as a "multiprocessor" or simply a "processor".

The RAM 102 is a volatile semiconductor memory that temporarily stores a program executed by the CPU 101 and data used by the CPU 101 for calculation. The information processing device 100 may be provided with a type of memory other than RAM, or may be provided with a plurality of memories.

The HDD 103 is a non-volatile storage device that stores software programs such as an OS (Operating System), middleware, and application software, and data. The information processing device 100 may be provided with other types of storage devices such as a flash memory and an SSD (Solid State Drive), or may be provided with a plurality of non-volatile storage devices.

The output IF 104 outputs an image to the display 111 connected to the information processing apparatus 100 in accordance with a command from the CPU 101. As the display 111, any kind of display such as a CRT (Cathode Ray Tube) display, a liquid crystal display (LCD: Liquid Crystal Display), a plasma display, and an organic EL (OEL: Organic Electro-Luminescence) display can be used.

The input IF 105 acquires an input signal from the input device 112 connected to the information processing device 100 and outputs the input signal to the CPU 101. As the input device 112, a pointing device such as a mouse, a touch panel, a touch pad, or a trackball, a keyboard, a remote controller, a button switch, or the like can be used. Further, a plurality of types of input devices may be connected to the information processing device 100.

The input IF 106 acquires image data captured by the camera 113 from the camera 113 connected to the information processing device 100 in accordance with a command from the CPU 101, and stores the image data in the RAM 102 or the HDD 103. The camera 113 is an imaging device that images a manufactured object (structure). The camera 113 may be a stereo camera.

The medium reader 107 is a reading device that reads programs and data recorded on the recording medium 114. As the recording medium 114, for example, a magnetic disk, an optical disk, a magneto-optical disk (MO: Magneto-Optical disk), a semiconductor memory, or the like can be used. The magnetic disk includes a flexible disk (FD) and an HDD. Optical discs include CDs (Compact Discs) and DVDs (Digital Versatile Discs).

The medium reader 107 copies, for example, a program or data read from the recording medium 114 to another recording medium such as the RAM 102 or the HDD 103. The read program is executed by, for example, the CPU 101. The recording medium 114 may be a portable recording medium and may be used for distribution of programs and data. Further, the recording medium 114 and the HDD 103 may be referred to as a computer-readable recording medium.

The NIC 108 is an interface that is connected to the network 115 and communicates with another computer via the network 115. The NIC 108 is connected to a communication device such as a switch or a router belonging to the network 115 by a cable, for example.

FIG. 3 is a diagram showing a functional example of the information processing device.
The information processing device 100 includes a CAD data storage unit 120, a posture candidate data storage unit 130, an inverted index storage unit 140, a captured image storage unit 150, a posture candidate management unit 160, and a display control unit 170.

The CAD data storage unit 120, the posture candidate data storage unit 130, the inverted index storage unit 140, and the captured image storage unit 150 are realized by using the storage areas of the RAM 102 and the HDD 103. The posture candidate management unit 160 and the display control unit 170 are realized by executing the program stored in the RAM 102 by the CPU 101.

The CAD data storage unit 120 stores the CAD data of the three-dimensional model. In the following, the three-dimensional model may be abbreviated as "3D (Dimensions) model". The CAD data is generated by CAD software in the design process of the structure (product) and stored in advance in the CAD data storage unit 120.

The posture candidate data storage unit 130 stores the posture candidate data of the 3D model. The posture candidate data includes a plurality of posture candidate images. The posture candidate image is a two-dimensional image (2D image) of the 3D model viewed from various directions.

The inverted index storage unit 140 stores an inverted index created based on each of the plurality of posture candidate images. The inverted index is created based on the PDL data created from the posture candidate image. In the second embodiment, PostScript is assumed as an example of PDL. However, PDL other than PostScript may be used.

The captured image storage unit 150 stores the captured image captured by the camera 113.
The posture candidate management unit 160 generates a plurality of posture candidate images and an inverted index for each posture candidate image based on the CAD data stored in the CAD data storage unit 120. The posture candidate management unit 160 includes a posture candidate data generation unit 161, a first PDL data generation unit 162, and an inverted index generation unit 163.

The posture candidate data generation unit 161 generates a plurality of posture candidate images when the posture of the 3D model is changed at a predetermined rotation angle and a predetermined elevation angle step based on the CAD data stored in the CAD data storage unit 120. To do. The posture candidate data generation unit 161 stores the posture candidate image in the posture candidate data storage unit 130 in association with the rotation angle and the elevation angle of the 3D model.

The first PDL data generation unit 162 generates PDL data (PostScript data in this example) representing the shape of the 3D model included in the posture candidate image based on the posture candidate image stored in the posture candidate data storage unit 130. The first PDL data generation unit 162 extracts an outline of the shape of the 3D model from the posture candidate image and generates PDL data corresponding to the outline. The first PDL data generation unit 162 stores the generated PDL data in the inverted index storage unit 140.

Here, the reason for paying attention to the outline is that the edge existing inside the structure may or may not be acquired depending on the imaging position, but it is the boundary line between the structure to be inspected and other objects. This is because the outline always exists and can be acquired with relatively high accuracy. Therefore, by using the PDL data corresponding to the outline, it is possible to improve the accuracy of narrowing down the posture candidates described later.

The inverted index generation unit 163 generates an inverted index based on the PDL data generated by the first PDL data generation unit 162. The inverted index generation unit 163 stores the generated inverted index in the inverted index storage unit 140 in association with the PDL data of the generation source or the posture candidate image.

The display control unit 170 causes the user to display the structure image included in the captured image captured by the camera 113 and the 3D model of the structure on the display 111 when the inspection work or the like is performed by the user. To support the work of. For example, a technique for displaying an image such as a 3D model on top of a captured image is called AR (Augmented Reality). The display control unit 170 includes a second PDL data generation unit 171, a search unit 172, and a display processing unit 173.

The second PDL data generation unit 171 generates PDL data (PostScript data in this example) representing the shape of the structure image included in the captured image. The second PDL data generation unit 171 extracts an outline of the shape of the structure from the structure image and generates PDL data corresponding to the outline. The second PDL data generation unit 171 provides the generated PDL data to the search unit 172. The second PDL data generation unit 171 may store the generated PDL data in the captured image storage unit 150.

The search unit 172 is based on the PDL data or the inverted index stored in the inverted index storage unit 140, and the PDL data or the inverted index of the posture of the 3D model that best matches the PDL data generated by the second PDL data generation unit 171. To search for. The search unit 172 provides the display processing unit 173 with the identification information of the 3D model corresponding to the searched PDL data or the inverted index.

The display processing unit 173 acquires a posture candidate image (posture candidate image of the initial posture) corresponding to the identification information acquired from the search unit 172 from the posture candidate data storage unit 130. The display processing unit 173 causes the display 111 to display the structure image included in the captured image stored in the captured image storage unit 150 and the acquired posture candidate image.

The display processing unit 173 accepts a user's operation of associating the edge of the structure image (sometimes referred to as an edge line) included in the captured image with the edge included in the posture candidate image. The display processing unit 173 is a 3D model in which the position of the edge in the posture candidate image corresponds to the position of the edge in the captured image associated with the edge based on the CAD data stored in the CAD data storage unit 120. Generate a projected image of. The display processing unit 173 superimposes and displays the generated projected image on the captured image.

FIG. 4 is a diagram showing an example of a posture candidate table.
The posture candidate table 131 is an example of posture candidate data stored in the posture candidate data storage unit 130. The posture candidate table 131 is generated by the posture candidate data generation unit 161. The posture candidate table 131 includes items of a posture candidate ID (IDentifier), an angle of rotation, an elevation angle, and a posture candidate image.

In the item of posture candidate ID, the posture candidate ID which is the identification information of the posture candidate is registered. In the rotation angle item, information on the rotation angle of the 3D model corresponding to the posture candidate is registered. The angle of rotation may be given, for example, as an angle around the z-axis in the vertical direction orthogonal to the x-axis and the y-axis when the x-axis and the y-axis orthogonal to each other are defined on the horizontal plane on which the 3D model is installed. .. Alternatively, the angle of rotation may be given by an angle around each of the x-axis, y-axis, and z-axis. In the item of elevation angle, information on the elevation angle (angle between the line of sight looking at the 3D model and the horizontal plane) when the 3D model is viewed is registered, which corresponds to the posture candidate. In the item of the posture candidate image, the data of the 2D image (posture candidate image) of the posture candidate of the 3D model is registered.

For example, in the posture candidate table 131, a record including a posture candidate ID "1", a rotation angle "Z11", an elevation angle "Z21", and a posture candidate image corresponding to the rotation angle and the elevation angle is registered.

In one example, the posture candidate data generation unit 161 generates posture candidate images when the rotation angle is changed by 30 ° and the elevation angle is changed by 15 ° for one 3D model, and the posture candidate table 131. Register with. However, the posture candidate data generation unit 161 may generate a posture candidate image in finer angle increments or coarser angle increments and register it in the posture candidate table 131.

Next, an example of the posture candidate image generated by the posture candidate data generation unit 161 will be described.
FIG. 5 is a diagram showing an example of a posture candidate image.
The posture candidate images P1 to P12 are 12 posture candidate images generated for a certain 3D model in increments of a rotation angle of 30 ° corresponding to an elevation angle of 45 °. The posture candidate image P1 is a case where the rotation angle is 0 °. The posture candidate image P2 is a case where the rotation angle is 30 °. The posture candidate image P3 is a case where the rotation angle is 60 °. Hereinafter, similarly, the posture candidate images P4 to P12 are also posture candidate images when the rotation angle is changed in increments of 30 °. Here, the case of an elevation angle of 45 ° is illustrated, but 12 posture candidate images are generated for each of the elevation angles of 0 °, 15 °, 45 °, 60 °, and so on, and are registered in the posture candidate table 131. ..

FIG. 6 is a diagram showing an example of an inverted index table.
The inverted index table 141 is stored in the inverted index storage unit 140. The information registered in the inverted index table 141 is generated by the first PDL data generation unit 162 and the inverted index generation unit 163. The inverted index table 141 includes items of posture candidate ID, PDL data, and inverted index.

The posture candidate ID is registered in the posture candidate ID item. PDL data (PostScript data in this example) is registered in the PDL data item. The PDL data is generated by the first PDL data generation unit 162 based on the posture candidate image. An inverted index is generated for the item of the inverted index. The inverted index is generated by the inverted index generation unit 163 based on the PDL data.

For example, in the inverted index table 141, a record including a posture candidate ID “1”, PDL data “PDL-D1”, and an inverted index “INDEX-T1” is registered. PDL data and inverted index records for other posture candidate images are also registered in the inverted index table 141.

Next, an example of a photographed image of a structure manufactured with respect to the illustrated 3D model will be described.
FIG. 7 is a diagram showing an example of a captured image.

The captured image 151 is generated by capturing the manufactured structure with the camera 113. The information processing device 100 acquires the captured image 151 from the camera 113 and stores it in the captured image storage unit 150. The captured image 151 includes the structure image 151a. The structure image 151a is a 2D image of the structure.

FIG. 8 is a diagram showing an example of an outline included in a captured image.
The outline image 152 is an image including the outline 152a. The outline 152a is a contour line connecting the outer peripheral edges of the structure image 151a. The outline 152a is specified by the second PDL data generation unit 171 based on the captured image 151. For example, the second PDL data generation unit 171 includes the captured image 151 by specifying a boundary line between regions where the pixel values are relatively significantly different (for example, there is a difference of the pixel values or more) in the captured image 151. Detect the edge to be. The second PDL data generation unit 171 detects a region surrounded by some edges among the plurality of detected edges, and extracts a closed line connecting the edges surrounding the region as an outline 152a.

Alternatively, the camera 113 may be a stereo camera (binocular camera) that simultaneously captures an object from two different directions, or a monocular camera that can move left and right. In this case, the second PDL data generation unit 171 can also identify the outline 152a of the structure by parallax based on the two captured images captured by the camera 113 by the existing technique.

Next, an example of the edge extracted from the posture candidate image by the first PDL data generation unit 162 will be described. As an example, an example of an edge with respect to the above-mentioned posture candidate image P2 will be described.
FIG. 9 is a diagram showing an example of an edge included in the posture candidate image.

The posture candidate image P2a emphasizes the outline P21a of the shape of the 3D model corresponding to the posture candidate among the edges included in the posture candidate image P2. Of the posture candidate images P2a, the line represented by the thick line corresponds to the outline P21a. The first PDL data generation unit 162 extracts the outline P21a from the posture candidate image P2a by the same processing as the second PDL data generation unit 171.

The posture candidate image P2b emphasizes the edges P21b, P22b, P24b, and P23b included in the posture candidate image P2, which are a part of the edges that do not belong to the outline P21a. Of the posture candidate images P2b, the four line segments represented by thick lines correspond to the edges P21b, P22b, P24b, and P23b. Here, it can be said that the edge is a ridge line formed by two surfaces in the 3D model.

FIG. 10 is a diagram showing an example of points on the outline of the posture candidate image.
The shape of the outline P21a is represented by the coordinates of the points on the outline P21a and the lines (line segments or curves) connecting the points. As a point on the outline P21a, a point where the two edges connect (a node on the outline P21a) is used.

For example, normalized coordinates are defined for the posture candidate image P2. The lateral coordinates of the posture candidate image P2 are defined as the X-axis, and the direction from the left side to the right side is defined as the positive direction of the X-axis. Further, the vertical coordinates of the posture candidate image P2 are defined as the Y axis, and the direction from the lower side to the upper side is defined as the positive direction of the Y axis. The origin O of the coordinates is a point at the center of the 3D model in the posture candidate image P2 (for example, the center of gravity of the shape represented by the outline P21a). Then, the size in the X-axis direction and the size in the Y-axis direction of the posture candidate image P2 are normalized so that the size in the X-axis direction and the size in the Y-axis direction in each posture candidate image are constant.

Similarly, for the outline 152a illustrated in FIG. 8, a coordinate system normalized so that the size matches the posture candidate image is defined, and information on the coordinates of each point and the line (line segment or curve) connecting each point. Can be used to represent the shape of the outline 152a. For example, in the case of a line segment, information on a vector including the position of the line segment and the direction in which the line segment extends can be obtained, and in the case of a curve, information on the position of the curve and the control point that defines the curve can be obtained. The shape of the outline 152a is represented by the combination of the curve information.

Here, with respect to the outline and outline 152a of each posture candidate image, the extraction order of the lines connecting each point and the two points is predetermined in the first PDL data generation unit 162 and the second PDL data generation unit 171. The first PDL data generation unit 162 and the second PDL data generation unit 171 generate PDL data corresponding to the outline according to a predetermined extraction order, respectively. For example, the first PDL data generation unit 162 and the second PDL data generation unit 171 are the points farthest from the origin in the image normalized with the center of the object as the reference point (origin) in which both XY are in the positive region. PDL data corresponding to the outline is generated so as to follow each point clockwise from (start point). Here, the "center of the object" is, for example, the center of gravity of the shape represented by the outline of the 3D model or the structure image.

In the example of FIG. 10, 12 points A to L and two points α and β are shown as points on the outline P21a. Control points B, α, β, and C represent Bezier curves with points B and C as end points. In this case, the coordinates of the points A to L and the points α and β and the line (line segment or curve) connecting the points A to L can be used as information representing the shape of the outline P21a. For example, in the example of FIG. 10, the point farthest from the origin where both XY are in the positive region is the point A. Therefore, the first PDL data generation unit 162 generates PDL data in which points and lines connecting the points are described in a clockwise order such as points A, B, α, β, C, D, ..., L. To do.

FIG. 11 is a diagram showing an example of a line segment included in each posture candidate image.
Point A is the starting point of the outline of the 3D model in each of the posture candidate images P1 to P12. The point B is the next point after the start point in each of the posture candidate images P1 to P12. The line segment connecting the points A and B in each of the posture candidate images P1 to P12 is the line segment (edge) first described in the PDL data for the outline.

As illustrated in FIG. 11, even in the same 3D model, the shape represented by the outline differs for each posture candidate image. This is because the visible edges of the 3D model differ from the line-of-sight direction when the 3D model is observed from various angles.

FIG. 12 is a diagram showing an example of a change in the outline according to the line-of-sight direction.
For example, consider a 3D model V1 with a non-transparent triangular prism. The 3D model V1 has height edges V11, V12, V13. The observation point a faces the surface including the edges V11 and V13. Therefore, when the 3D model V1 is observed from the observation point a, the edges V11 and V13 are observed, but the edge V12 is not observed. Further, the observation point b faces the surface including the edges V12 and V13. Therefore, when the 3D model V1 is observed from the observation point b, the edges V12 and V13 are observed, but the edge V11 is not observed.

The example of FIG. 12 illustrates a simple shape, but in a 3D model having a complicated shape, the observable edge is different for each posture candidate image, so that the shape represented by the outline for the posture candidate image is also different. Become.

The search unit 172 identifies the outline of the posture candidate image similar to the outline included in the captured image based on the line segment included in the outline of each posture candidate image. Next, an example of narrowing down posture candidates by the search unit 172 will be described.

FIG. 13 is a diagram showing an example of narrowing down posture candidates.
Here, as an example, the narrowing down of the posture candidate images P1 to P12 with respect to the outline image 152 will be described. The outline image 152 includes points A and B in a region where both XY are positive in the normalized XY coordinate system with the center of the shape represented by the outline 152a as the origin. Among the posture candidate images P1 to P12, the posture candidate images in which the points A and B are included in the region where both XY are positive in the normalized XY coordinate system are the posture candidate images P2 and P6 to P9. ..

Therefore, the search unit 172 narrows down the posture candidate images P1 to P12 to the posture candidate images P2 and P6 to P9. The search unit 172 excludes posture candidate images P1, P3 to P5, and P10 to P12 from the candidates.

FIG. 14 is a diagram showing an example (continued) of narrowing down posture candidates.
In the normalized XY coordinate system, the outline image 152 includes points B and C as end points in a region where both XY are positive, and includes a curve having points α and β as control points in addition to points B and C. .. Of the posture candidate images P2, P6 to P9, in the normalized XY coordinate system, points B and C are included as end points in the region where both XY are positive, and points α and β are controlled in addition to points B and C. The posture candidate image including the curve as a point is the posture candidate image P2.

Therefore, the search unit 172 narrows down the posture candidate images P2, P6 to P9 to the posture candidate images P2. The search unit 172 excludes posture candidate images P6 to P9 from the candidates.
In this way, the search unit 172 identifies the posture candidate image P2 corresponding to the outline of the 3D model most similar to the outline 152a from the posture candidate images P1 to P12.

The search unit 172 can perform the collation illustrated in FIGS. 13 and 14 by using the PDL data corresponding to the outline or the inverted index of the PDL data. Next, an example of an inverted index when PostScript data is used as PDL data will be described. However, the data is not limited to PostScript, and PDL data in a format other than PostScript may be used.

FIG. 15 is a diagram showing an example of an inverted index for PostScript.
In the example shown in FIG. 15, a case where PostScript data of outline 200 is generated will be described. The outline 200 is formed by a line segment 201, a curve 202, a line segment 203, and a line segment 204. The line segment 201 is a line segment having points A and B as end points. The curve 202 is a curve having points B and C as end points, and is a Bezier curve whose shape is determined by control points B, C, α and β.

The first PDL data generation unit 162 and the second PDL data generation unit 171 generate PostScript data 210 of outline 200 based on points A, B, C, D, α, and β. The PostScript data 210 is an example of PDL data.

The PostScript data 210 includes the parameters "Xa, Ya", "Xb, Yb", "Xα, Yα", "Xβ, Yβ", "Xc, Yc". “Xa, Ya” indicates the coordinates of the point A. “Xb, Yb” indicates the coordinates of the point B. “Xα, Yα” indicates the coordinates of the point α. “Xβ, Yβ” indicates the coordinates of the point β. “Xc, Yc” indicates the coordinates of the point C.

The PostScript data 210 includes a plurality of types of commands. Specifically, the commands are "newpath", "moveto", "lineto", "curveto", "stroke", and "showpage". The command may be called a control statement.

The inverted index generation unit 163 reads the character strings included in the PostScript data 210 in order from the upper row, and arranges the read character strings to generate the PostScript conversion data 220.

The inverted index generation unit 163 generates an inverted index 230 based on the command or coordinates included in the PostScript conversion data 220 and the appearance position of the command or coordinates. The horizontal axis of the inverted index 230 is a value corresponding to the offset. The vertical axis of the inverted index 230 is the axis corresponding to the command or coordinates.

The offset indicates the position from the beginning of the PostScript conversion data 220. The offset of the start position of the PostScript conversion data 220 is set to "0", and thereafter, 1 is added to the offset in units of commands or coordinates. For example, in the example of PostScript conversion data 220, the offsets of "newpath", "Xa, Ya", "movetto", ... Are "0", "1", "2", ..., Respectively.

The inverted index generation unit 163 scans the PostScript conversion data 220 to identify the correspondence between the “offset” and the “command or coordinate”. The inverted index generation unit 163 sets the flag “1” in the inverted index 230 based on the correspondence between the specified “offset” and the “command or coordinates”. The parts of the inverted index 230 other than the flag "1" are "0".

For example, the command "newpath" appears at the offset "0" of the PostScript conversion data 220. Therefore, the inverted index generation unit 163 sets the flag “1” at the position where the offset “0” of the inverted index and the command “newpath” intersect. The inverted index generation unit 163 generates the inverted index 230 by tracing the commands or coordinates included in the PostScript conversion data 220 in order from the offset "0" and repeatedly executing the above processing.

The inverted index generation unit 163 may reduce the data size of the inverted index by hashing the inverted index. For example, the inverted index generation unit 163 can hash the inverted index with a prime number (base) by using a bitmap wrapping technique.

FIG. 16 is a diagram showing an example of hashing of an inverted index.
In the example described with reference to FIG. 16, a 32-bit register is assumed. For example, the inverted index generation unit 163 can hash the inverted index by using the 32-bit register of the information processing apparatus 100. Further, as an example, the inverted index generation unit 163 is assumed to hash the bitmap of each row of the inverted index based on the prime numbers (bases) of "29" and "31". Here, a case where the hashed bitmap h11 and the hashed bitmap h12 are generated from the bitmap b1 will be described.

Bitmap b1 is assumed to indicate a bitmap obtained by extracting a row having an inverted index (for example, an inverted index 230 shown in FIG. 15). The hashed bitmap h11 is a bitmap hashed by the base "29". The hashed bitmap h12 is a bitmap hashed by the base "31".

The inverted index generation unit 163 associates the position of each bit of the bitmap b1 with the value of the remainder obtained by dividing by one base with the position of the hashed bitmap. When the bit position of the corresponding bitmap b1 is set to "1", the inverted index generation unit 163 performs a process of setting "1" to the position of the associated hashed bitmap.

An example of the process of generating the hashed bitmap h11 of the base “29” from the bitmap b1 will be described. First, the inverted index generation unit 163 copies the information of the positions "0 to 28" of the bitmap b1 to the hashed bitmap h11.

Since the remainder obtained by dividing the bit position "35" of the bitmap b1 by the base "29" is "6", the position "35" of the bitmap b1 corresponds to the position "6" of the hashed bitmap h11. Can be attached. Since the inverted index generation unit 163 has "1" set at the position "35" of the bitmap b1, "1" is set at the position "6" of the hashed bitmap h11.

Since the remainder obtained by dividing the bit position "43" of the bitmap b1 by the base "29" is "14", the position "43" of the bitmap b1 corresponds to the position "14" of the hashed bitmap h11. Can be attached. Since the inverted index generation unit 163 has "1" set at the position "43" of the bitmap b1, "1" is set at the position "14" of the hashed bitmap h11.

The inverted index generation unit 163 generates the hashed bitmap h11 by repeatedly executing the above processing for the position where “1” is set with respect to the position “29” or more of the bitmap b1. The inverted index generation unit 163 skips the position where "0" is set with respect to the position "29" or higher of the bitmap b1 and processes the next position.

Next, an example of the process of generating the hashed bitmap h12 of the base "31" from the bitmap b1 will be described. First, the inverted index generation unit 163 copies the information of the positions "0 to 30" of the bitmap b1 to the hashed bitmap h12.

Since the remainder obtained by dividing the bit position "35" of the bitmap b1 by the base "31" is "4", the position "35" of the bitmap b1 corresponds to the position "4" of the hashed bitmap h12. Can be attached. Since the inverted index generation unit 163 has "1" set at the position "35" of the bitmap b1, "1" is set at the position "4" of the hashed bitmap h12.

Since the remainder obtained by dividing the bit position "43" of the bitmap b1 by the base "31" is "12", the position "43" of the bitmap b1 corresponds to the position "12" of the hashed bitmap h12. Can be attached. Since the inverted index generation unit 163 has "1" set at the position "43" of the bitmap b1, "1" is set at the position "12" of the hashed bitmap h12.

The inverted index generation unit 163 generates the hashed bitmap h12 by repeatedly executing the above processing for the position where "1" is set with respect to the position "31" or more of the bitmap b1. The inverted index generation unit 163 skips the position where "0" is set with respect to the position "31" or higher of the bitmap b1 and processes the next position.

The inverted index generation unit 163 generates a hashed inverted index by compressing each row of the inverted index by the above-mentioned folding technique. The hashed bitmaps at the bases "29" and "31" are given information on the rows (commands or coordinates) of the bitmap of the generating source.

FIG. 17 is a diagram showing an example of a hash value of an inverted index.
The hashed bitmap h21 is an example of a hashed bitmap corresponding to the command “movetto”. The hashed bitmap h22 is an example of a hashed bitmap corresponding to the command “lineto”. The hashed bitmap h23 is an example of a hashed bitmap corresponding to the coordinates “Xa, Ya”. In this way, a hashed bitmap is generated for each command and coordinates included in the PostScript data. The inverted index generation unit 163 stores the hashed bitmap in the inverted index storage unit 140 instead of the inverted index, so that when the size of the inverted index is relatively large (larger than 64 bits in this example), it is saved. Data size can be compressed.

When the hashed bitmap is used, the search unit 172 restores the inverted index from the hashed bitmap and uses it for narrowing down the posture candidates as follows.
FIG. 18 is a diagram showing an example of restoration of an inverted index.

Here, as an example, a case where the search unit 172 restores the bitmap b1 based on the hashed bitmap h11 and the hashed bitmap h12 will be described.
FIG. 18A shows a step of expanding from a hashed bitmap to an intermediate bitmap in the procedure of restoring the inverted index.

The search unit 172 generates an intermediate bitmap h11a from the hashed bitmap h11 at the base “29”. The search unit 172 copies the values at positions 0 to 28 of the hashed bitmap h11 to positions 0 to 28 of the intermediate bitmap h11a, respectively.

The search unit 172 repeatedly executes a process of copying the values at positions 0 to 28 of the hashed bitmap h11 for each “29” for the values after the position 29 of the intermediate bitmap h11a. FIG. 18 shows an example in which the values of positions 0 to 14 of the hashed bitmap h11 are copied to the positions 29 to 43 of the intermediate bitmap h11a.

The search unit 172 generates an intermediate bitmap h12a from the hashed bitmap h12 having the base "31". The search unit 172 copies the values at positions 0 to 30 of the hashed bitmap h12 to positions 0 to 30 of the intermediate bitmap h12a, respectively.

The search unit 172 repeatedly executes a process of copying the values at positions 0 to 30 of the hashed bitmap h12 for each “31” for the values after the position 31 of the intermediate bitmap h12a. FIG. 18 shows an example in which the values of the hashed bitmap h12 positions 0 to 12 are copied to the positions 31 to 43 of the intermediate bitmap h12a.

In this way, an intermediate bitmap is generated from the hashed bitmap.
FIG. 18B shows a step of performing an AND operation between intermediate bitmaps in the procedure of restoring the inverted index.

The search unit 172 restores the bitmap b1 before hashing by performing an AND operation on the intermediate bitmap h11a and the intermediate bitmap h12a. The search unit 172 can generate a bitmap corresponding to a command or coordinates by repeatedly executing the same process for other hashed bitmaps. In this way, the inverted index is restored.

FIG. 19 is a diagram showing an example of narrowing down the inverted index.
The search unit 172 searches for the posture candidate image corresponding to the outline 152a extracted from the captured image 151 by narrowing down the inverted index. For example, the search unit 172 acquires PostScript data corresponding to the outline 152a from the second PDL data generation unit 171. Then, the search unit 172 collates whether the command or coordinates included in the PostScript data is included in the PostScript data corresponding to each posture candidate image by using the inverted index.

For example, it is assumed that the character string "newpath Xa Ya moveo ..." is included in the position (or may be another position) corresponding to the offsets "6" to "9" of the PostScript data corresponding to the outline 152a. In the figure, "newpath" may be abbreviated as "np" and "moveto" may be abbreviated as "mt".

In this case, the search unit 172 restores the bitmap corresponding to the inverted index command "newpath" (step ST1). In FIG. 19, a bitmap b11 is shown as an example of a bitmap in which the offset “6” is the flag “1”.

The search unit 172 shifts the bitmap b11 to the left to obtain the bitmap b12 (step ST2). The search unit 172 restores the bitmap corresponding to the coordinates “Xa Ya” of the corresponding inverted index (step ST3).

The search unit 172 performs an AND operation on the bitmaps b12 and b13 to obtain the bitmap b14 (step ST4). In the bitmap b14, when the offset "7" is the flag "1", the PostScript data corresponding to the corresponding inverted index has the character string "newpath Xa Ya" at the position corresponding to the offsets "6" and "7". It turns out to include. At this point, the posture candidate corresponding to the inverted index that does not include the character string "newpath Xa Ya" is either lowered in priority as a candidate (similarity to outline 152a) or excluded from the candidates.

Subsequently, the search unit 172 shifts the bitmap b14 to the left to obtain the bitmap b15 (step ST5). The search unit 172 restores the bitmap b16 corresponding to the command "movetto" of the corresponding inverted index (step ST6).

The search unit 172 performs an AND operation on the bitmaps b15 and b16 to obtain the bitmap b17 (step ST7). In the bitmap b17, when the offset "8" is the flag "1", the PostScript data corresponding to the corresponding inverted index is a character string of "newpath Xa Ya moveo" at the position corresponding to the offsets "6" to "8". It can be seen that it contains. At this point, the posture candidate corresponding to the inverted index that does not include the character string "newpath Xa Ya moveo" is either lowered in priority as a candidate or excluded from the candidates.

After that, similarly, the search unit 172 shifts the bitmap b17 to the left to obtain the bitmap b18 (step ST8). Then, the search unit 172 can specify the subsequent command or coordinate (in the PostScript data corresponding to the posture candidate image) based on the bitmap b18, and can collate it with the command or coordinate of the PostScript data of the outline 152a. For example, the search unit 172 has a degree of matching (similarity) with a series of character strings included in the PostScript data of the outline 152a among the PostScript data of the outline generated from each posture candidate image by the collation using the inverted index. Search for the one with the highest.

By the way, when the manufactured structure is imaged by the camera 113, other structures (other objects) around the structure are reflected in the photographed image, so that the outline of the structure is appropriate from the photographed image. It may not be possible to extract to.

FIG. 20 is a diagram showing an example of false detection of an outline.
The captured image 153 includes a structure image 153a of the target structure and a structure image 153b of another structure. For example, when another structure is placed in front of the target structure at the time of imaging by the camera 113, as shown in the captured image 153, the structure image 153b causes a part of the target structure to be displayed. Hidden structure images 153a may be acquired.

In this case, the second PDL data generation unit 171 may generate an outline image 154 based on the captured image 153. The outline image 154 includes an outline 154a detected based on the structure images 153a and 153b. However, the outline 154a also includes the outline of a structure other than the target structure, which is inappropriate as an outline of the target structure.

Therefore, the display processing unit 173 provides a correction function when such an outline is erroneously detected.
FIG. 21 is a diagram showing a correction example when an outline is erroneously detected.

FIG. 21 (A) shows an example of designating the outline extraction range. For example, the display processing unit 173 displays the outline 154a on the display 111 and prompts the user to confirm whether or not the outline 154a is appropriate. When the user determines that the outline 154a is inappropriate, the display processing unit 173 accepts the user's designation of the outline extraction range in the captured image 153. For example, the user can operate the pointer K1 by the input device 112 to specify the extraction range R1 of the outline of the target structure in the captured image 153. The display processing unit 173 provides the information of the designated extraction range R1 to the second PDL data generation unit 171. For the second PDL data, the outline is extracted again within the acquired extraction range R1.

FIG. 21B shows an example of complementing the outline. For example, the display processing unit 173 displays the outline 154a on the display 111 and prompts the user to confirm whether or not the outline 154a is appropriate. When the user determines that the outline 154a is inappropriate, the display processing unit 173 accepts the user's completion of the missing line segment 154b in the outline 154a. For example, the user can operate the pointer K1 with the input device 112 to input an additional line segment 154b in the outline image 154. In FIG. 21B, circles are added to both ends of the line segment 154b so that it can be easily identified as the added line segment. The circles at both ends of the line segment 154b correspond to both ends of the line segment 154b, as well as both ends of the line 154c and both ends of the line 154d (represented by dotted lines in the figure).

In this case, the display processing unit 173 may further accept the operation of the user who selects the combination of the line 154c which is a part of the outline 154a and the added line segment 154b as the modified outline. Then, the second PDL data generation unit 171 removes the line 154d which is a part of the outline 154a based on the information acquired by the display processing unit 173, and corrects the line which is a combination of the line segment 154b and the line 154c. Detect as a later outline.

The second PDL data generation unit 171 detects the edges of the captured image 151 and the captured image 153 in small pieces because the boundary of the object cannot be properly determined based on the pixel value or the like, and extracts the combination of edges forming the outline. Sometimes you can't. Also in this case, the second PDL data generation unit 171 can complement the outline by accepting the addition of the line segment by the user as illustrated in FIG. 21 (B).

Next, the processing procedure by the information processing apparatus 100 will be described. The PDL data (for example, PostScript data) and the inverted index for each posture candidate image and each posture candidate image are created before the following procedure is started, and are stored in the posture candidate data storage unit 130 and the inverted index storage unit 140. Stored in advance.

FIG. 22 is a flowchart showing an example of outline extraction.
(S10) The second PDL data generation unit 171 analyzes the captured image 151 stored in the captured image storage unit 150, and identifies a boundary line of a region having a difference of a predetermined value or more in pixel values.

(S11) The second PDL data generation unit 171 extracts the specified boundary line as an edge. The second PDL data generation unit 171 generates an outline image including an outline formed by the extracted edges. The display processing unit 173 displays the captured image 151 and the generated outline image on the display 111, and prompts the user to confirm. When the outline extraction range is specified by the user, the second PDL data generation unit 171 detects the outline based on the edges included in the specified range and generates an outline image.

(S12) The second PDL data generation unit 171 determines whether or not there is an input for changing the outline extraction range by the user. If there is a change input, the process proceeds to step S13. If there is no change input, the process proceeds to step S14.

(S13) The second PDL data generation unit 171 changes the outline extraction range to the range specified by the user. Then, the process proceeds to step S11.
(S14) The second PDL data generation unit 171 determines whether or not there is a complementary input of the outline by the user. If there is a completion input, the process proceeds to step S15. If there is no completion input, the process proceeds to step S16.

(S15) The second PDL data generation unit 171 complements the outline with the input contents of the completion of the outline by the user.
(S16) The second PDL data generation unit 171 generates PDL data (for example, PostScript data) of the extracted outline and provides it to the search unit 172. Then, the outline extraction process is completed.

In this way, when the second PDL data generation unit 171 acquires the captured image 151, the second PDL data generation unit 171 displays an outline candidate line representing the shape of the structure included in the captured image 151, and the extraction range of the outline candidate line in the captured image 151. Or accepts the input of a line that complements the outline candidate line. As a result, the outline of the structure shown in the captured image 151 can be appropriately acquired. As a result, the accuracy of identifying the posture candidate image in the subsequent processing can be improved.

FIG. 23 is a flowchart showing an example of narrowing down posture candidates.
(S20) The search unit 172 compares the PDL data of all the outlines of the posture candidate images with the PDL data of the outlines acquired from the captured image 151. Here, the search unit 172 can use the inverted index for the comparison. The search unit 172 can narrow down the posture candidates at high speed by using the inverted index. However, the search unit 172 may compare PDL data with each other without using the inverted index. Further, the search unit 172 may also generate an inverted index for the outline PDL data acquired from the captured image 151, and collate the inverted indexes with each other.

(S21) The search unit 172 is based on the coordinate information and vector information included in the two PDL data (the outline PDL data of the posture candidate image and the outline PDL data acquired from the captured image 151), and the two PDL data. Calculate the similarity of. For example, the higher the degree to which the search unit 172 matches a series of character strings representing commands and coordinates included in the outline PDL data acquired from the captured image 151, the higher the similarity of the outline PDL data of the corresponding posture candidate image. To increase. For example, the search unit 172 may consider that the shorter the distance between the coordinates, the greater the degree of agreement between the coordinates.

(S22) The search unit 172 provides the display processing unit 173 with the posture candidate ID of the posture candidate of the 3D model having the highest degree of similarity. The display processing unit 173 outputs the posture candidate image P2 corresponding to the corresponding posture candidate ID based on the posture candidate table 131. Specifically, the display processing unit 173 displays the posture candidate image P2 corresponding to the corresponding posture candidate ID together with the captured image 151 on the display 111. Then, the process of narrowing down the posture candidates is completed.

In step S22, the display processing unit 173 may output a plurality of edges extracted from the captured image 151 instead of the captured image 151 or together with the captured image 151. Further, the display processing unit 173 may output a plurality of edges extracted from the posture candidate image P2 in place of the posture candidate image P2 or together with the posture candidate image P2.

The user can confirm the posture candidate image P2 displayed together with the captured image 151 and perform an operation of associating the edge included in the posture candidate image P2 with the edge included in the captured image 151.

As described above, when the search unit 172 identifies the posture candidate image (first projected image), the information indicating the first outline of each of the plurality of candidate images and the second structure included in the captured image 151 are used. By comparing with the information showing the outline, the posture candidate image corresponding to the first outline having the highest degree of coincidence with the second outline is identified.

For example, the information indicating the first outline is the first text information (PDL data) including the drawing command of the first outline. Further, the information indicating the second outline is the second text information (PDL data) including the drawing command of the second outline. In the comparison of the information showing the first and second outlines, the search unit 172 identifies the posture candidate image corresponding to the first text information having the highest degree of matching between the second text information and the description content. As a result, the posture candidate image corresponding to the captured image can be efficiently identified. For example, the accuracy of identifying the posture candidate image is improved. In addition, posture candidate images can be searched at high speed.

In addition, the search unit 172 compares the description contents of the first and second text information based on the inverted index indicating the appearance position in the first text information of each of the plurality of drawing commands, as illustrated in FIG. Then, the drawing commands included in the first and second text information and the appearance order of the drawing commands are compared. As a result, the posture candidate image can be searched at a higher speed.

FIG. 24 is a flowchart showing an example of 3D model superimposition control.
(S30) The display processing unit 173 accepts a user operation of associating the edge (ridge line) of the posture candidate image P2 of the 3D model with the edge of the captured image 151.

(S31) The display processing unit 173 generates a projection image of the 3D model in which the edge of the captured image 151 is associated with the edge of the 3D model based on the CAD data of the 3D model stored in the CAD data storage unit 120. .. The display processing unit 173 can generate a projection image of the 3D model in which the edge of the captured image 151 is associated with the edge of the 3D model by the existing technology.

(S32) The display processing unit 173 superimposes and displays the projected image of the generated 3D model on the captured image 151. Then, the process of 3D model superimposition control is completed.
By comparing the projected image of the 3D model superimposed on the captured image 151 with the structure image 151a in the captured image 151, the user can easily perform operations such as inspection of the manufactured structure.

FIG. 25 is a diagram showing an initial display example of the posture candidate image.
FIG. 25 shows an example of the screen 300 displayed on the display 111 by the process of step S22 of FIG. 23. The screen 300 includes a captured image 151 and a posture candidate image P2.

The posture of the 3D model in the posture candidate image P2 approximates the posture of the structure in the captured image 151. Therefore, the user associates the edge of the 3D model in the posture candidate image P2 with the edge of the structure in the captured image 151 without performing an operation of changing the posture of the 3D model projected on the posture candidate image P2. The operation can be easily performed. For example, the user operates the pointer K1 by the input device 112 to easily associate the edge connecting the points Q11 and Q12 included in the captured image 151 with the edge connecting the points Q21 and Q22 included in the posture candidate image P2. be able to.

FIG. 26 is a diagram showing a comparative example of initial display of posture candidate images.
In FIG. 26, as a comparative example, an example of the screen 400 including the captured image 151 and the posture candidate image P3 of the predetermined initial posture is shown. In this case, the initial posture of the 3D model in the posture candidate image P3 is the posture when the 3D model is viewed from the direction of the elevation angle of 90 °, and the posture of the structure in the captured image 151 (predetermined posture of the structure at an elevation angle of about 45 °). It is different from the attitude when viewed from the direction). Therefore, unless the user performs an operation of changing (rotating) the posture of the 3D model displayed on the posture candidate image P3, the edge of the structure in the captured image 151 and the edge of the 3D model in the posture candidate image P3. It is difficult to confirm the correspondence with. Therefore, the user is forced to operate the pointer K1 by the input device 112 to change the posture of the 3D model in the posture candidate image P3. The operation of changing the posture of the 3D model to match the posture of the structure in the captured image 151 may be difficult for a relatively inexperienced user.

Therefore, as illustrated in FIG. 25, the information processing apparatus 100 makes it possible to omit the operation of changing the posture of the 3D model by the user, and can improve the efficiency of the work by the user (for example, shorten the work time).

In particular, by using the PDL data corresponding to the outline (for example, PostScript data), the posture candidates of the 3D model can be narrowed down with high accuracy.
Further, by using the inverted index for narrowing down the posture candidates, the narrowing down process can be speeded up.

The information processing of the first embodiment can be realized by causing the processing unit 12 to execute the program. Further, the information processing of the second embodiment can be realized by causing the CPU 101 to execute the program. The program can be recorded on a computer-readable recording medium 114.

For example, the program can be distributed by distributing the recording medium 114 on which the program is recorded. Alternatively, the program may be stored in another computer and distributed via the network. For example, the computer may store (install) a program recorded on the recording medium 114 or a program received from another computer in a storage device such as RAM 102 or HDD 103, read the program from the storage device, and execute the program. Good.

10 Information processing device 11 Storage unit 12 Processing unit 20 Display device 31, 32, 33 Projection image 40 Captured image 41 Structure image 50 Display image 51 Ridge line group 52 Edge line group

Claims (7)

The computer
Acquire a photographed image including the structure imaged by the image pickup device,
Obtain multiple projection images generated from the three-dimensional model of the structure,
Among the plurality of acquired projected images, the first projected image corresponding to the shape of the structure included in the acquired captured image is specified.
The ridge line of the three-dimensional model included in the specified first projection image and the edge line extracted from the captured image are displayed.
When an operation of associating a predetermined number of ridge lines included in the displayed ridge line with a predetermined number of edge lines included in the displayed edge line is accepted, the positions of the predetermined number of ridge lines correspond to the predetermined number of ridge lines. A second projection image of the three-dimensional model corresponding to each of the positions of the predetermined number of attached edge lines is generated.
The generated second projection image is superimposed and displayed on the captured image.
A display control method characterized by that. In the identification of the first projected image, the information indicating the first outline of each of the plurality of projected images is compared with the information indicating the second outline of the structure included in the captured image, and the second outline is compared. The display control method according to claim 1, wherein the first projection image corresponding to the first outline having the highest degree of coincidence with the outline of the above is specified. The information indicating the first outline is the first text information including the drawing instruction of the first outline, and the information indicating the second outline is the second text information including the drawing instruction of the second outline. It is the text information of
In the comparison of the information showing the first and the second outline, the first projection image corresponding to the first text information having the highest degree of matching between the second text information and the description content is specified. ,
The display control method according to claim 2. In the comparison of the description contents of the first and second text information, the first and second text information are based on the inverted index indicating the appearance position in the first text information of each of the plurality of drawing commands. The display control method according to claim 3, wherein the drawing command included in each is compared with the appearance order of the drawing command. When the computer further acquires the captured image, it displays an outline candidate line representing the shape of the structure included in the captured image, and specifies the extraction range of the outline candidate line in the captured image, or The display control method according to any one of claims 1 to 4, which accepts an input of a line that complements the outline candidate line. On the computer
Acquire a photographed image including the structure imaged by the image pickup device,
Obtain multiple projection images generated from the three-dimensional model of the structure,
Among the plurality of acquired projected images, the first projected image corresponding to the shape of the structure included in the acquired captured image is specified.
The ridge line of the three-dimensional model included in the specified first projection image and the edge line extracted from the captured image are displayed.
When an operation of associating a predetermined number of ridge lines included in the displayed ridge line with a predetermined number of edge lines included in the displayed edge line is accepted, the positions of the predetermined number of ridge lines correspond to the predetermined number of ridge lines. A second projection image of the three-dimensional model corresponding to each of the positions of the predetermined number of attached edge lines is generated.
The generated second projection image is superimposed and displayed on the captured image.
A display control program characterized by executing processing. A storage unit that stores multiple projected images generated from a three-dimensional model of a structure,
A photographed image including the structure captured by the imaging device is acquired, the plurality of projected images of the structure are acquired from the storage unit, and among the acquired plurality of projected images, the acquired image is used. A first projected image corresponding to the shape of the included structure is specified, and the ridgeline of the three-dimensional model included in the specified first projected image and the edge line extracted from the captured image are displayed. Then, when an operation of associating a predetermined number of ridge lines included in the displayed ridge line with a predetermined number of edge lines included in the displayed edge line is received, the position of the predetermined number of ridge lines is changed to the predetermined number of ridge lines. A processing unit that generates a second projected image of the three-dimensional model corresponding to each of the positions of the predetermined number of edge lines associated with, and superimposes and displays the generated second projected image on the captured image. ,
An information processing device characterized by having.

Images