JP2010219825A - Photographing device for three-dimensional measurement - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a photographing device for three-dimensional measurement, which exactly guides a photographer to the next predetermined photographing position from the present position. <P>SOLUTION: The photographing device 1 for three-dimensional measurement includes a photographing section 2 that acquires a live image 11 by photographing a measurement object 8 while a photographer is moving and timely acquires a photographs image as a still image in a photographing device for three-dimensional measurement in which the measurement object 8 is overlapped and photographed in sequence by a single camera, a display section 3 that displays in the same screen the live image 11 at the present position of the photographing section 2 and the sample image 12 of a simulation object as a still image that is photographed at nearly identical relative positional relation with the next predetermined photographing position, a photographing position determination section 4 that compares the live image 11 at the present position with the sample image 12 in the photographing section 2 and determines the relative position of the photographing section 2 to the next predetermined photographing position, and a removing information noticing section 5 that prepares guide information and notices it to the display section so that the photographer can move to the next predetermined photographing position from the present position. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image photographing apparatus for three-dimensional measurement. More specifically, the present invention relates to a three-dimensional measurement image photographing apparatus that guides a photographer from a current position to a next photographing scheduled position.

  In order to grasp the whole image of the measurement object and reproduce it as a three-dimensional model image, it is necessary to link the photographed images taken from a plurality of photographing positions. In order to measure the three-dimensional coordinates of the photographing apparatus or the object from the plurality of photographed images taken by the photographer in this way, the feature points (on the object) corresponding to each other on each of the two or more photographed images. Represents the same point) and needs to be tracked. In this case, since feature points inappropriate for three-dimensional measurement can be mixed in the captured image, it is possible to accurately determine the capturing position, orientation, or position coordinates of the target object of the capturing apparatus while determining the suitability of the feature points in the captured image. An image processing apparatus capable of measurement has been proposed. (See Patent Document 1)

JP 2007-183256 A (FIGS. 1 to 11, paragraphs 0021 to 0065)

  In order to accurately and efficiently reproduce the three-dimensional model image of the measurement target, it is necessary to appropriately arrange a plurality of shooting positions for shooting the measurement target in advance. Then, it is necessary to reliably move to a predetermined shooting scheduled position for shooting. However, there is a problem that it is difficult for a moving photographer to actually determine whether or not the current position is a planned photographing position by looking at the measurement object and its surroundings. In particular, inexperienced people, inexperienced people, and even more accustomed to shooting for creating a 3D model image, when measuring objects such as automobiles, works of art, buildings, terrain are large, and the surface shape is complicated When a similar repetitive pattern appears, it is difficult to take a picture while appropriately taking an overlapping range between adjacent images.

  The present invention has been made in view of the above problems, and is for three-dimensional measurement that guides a photographer accurately from the current position to the next scheduled shooting position so that the photographer can accurately move to the next scheduled shooting position. An object is to provide an image photographing apparatus.

  In order to achieve the above object, the three-dimensional measurement image photographing apparatus 1 according to the first aspect of the present invention sequentially photographs, for example, as shown in FIGS. In the three-dimensional measurement image photographing apparatus, a photographing unit 2 that obtains a live image 11 by photographing a measurement object 8 while a photographer moves, and obtains a photographed image that is a still image in a timely manner, and a photographing unit A display unit 3 for displaying a live image 11 at the current position 2 and a sample image 12 of a simulated object, which is a still image photographed with substantially the same relative positional relationship as the next scheduled photographing position, on the same screen; The live image 11 at the current position of the unit 2 and the sample image 12 are compared, and the shooting position determination unit 4 that determines the relative position of the shooting unit 2 with respect to the next shooting scheduled position, from the current position to the next shooting scheduled position Photographer moves And a movement notification unit 5 which creates the guide information is notified to the display unit as.

Here, the live image refers to an image acquired by a video camera or a digital camera at the current position, for example. Usually, a measurement object is displayed as a live image on a finder, a display or the like while a photographer moves using a video camera or a digital camera. A captured image refers to a still image captured by a shutter operation or the like. In addition, the sample image of the simulated target is typically the same relative positional relationship as the planned shooting position for other vehicles (preferably similar in shape) when the measurement target is a vehicle. An image selected from captured images (still images) already captured at (direction, distance). However, in the same type of car or the same car, there may be cases where you want to correct blurred parts or foreign-matter-contaminated parts, obtain higher-quality photographic images, or shoot under different conditions. It shall include model objects and objects to be measured. The same relative positional relationship means that the direction and distance between the photographing unit and the measurement object are equal to the direction and distance between the photographing apparatus that photographed the sample image and the simulated object. Considering the difference between the simulation object and the simulated object, the direction and the distance are not limited to the same, and the difference may be within ± 10%, for example. In addition, the photographed image that has already been photographed with the same relative positional relationship includes a photographed image obtained by photographing the simulated object or the measurement object installed at the photographing site from the planned photographing position. Further, the next scheduled shooting position may be determined according to a predetermined order from a predetermined shooting schedule position, or the shortest scheduled shooting position may be selected from the current position. In addition, the guide information may be information that guides the direction and distance, but the guide related to the distance can also be expressed in directions such as “forward”, “back”, “near”, and “far”, so only the direction. It may be a guide.
When configured in this manner, it is possible to provide a three-dimensional measurement image photographing apparatus that accurately guides the photographer from the current position to the next photographing scheduled position.

  The three-dimensional measurement image photographing apparatus 1A according to the second aspect of the present invention is a three-dimensional measurement image photographing apparatus according to the first aspect, for example, in a photographed image acquired by the photographing unit 2, as shown in FIG. A feature point extracting unit 21 that extracts a feature point, and a shooting position measurement that obtains a two-dimensional coordinate or a three-dimensional coordinate of the shooting position at which the shot image is shot from the screen position of the shot image of the feature point extracted by the feature point extraction unit 21 The display unit 3 includes a unit 23 and a planned next shooting position selection unit 24 that selects a planned next shooting position from a plurality of planned shooting positions based on the obtained position coordinates of the shooting position. The image of the simulated object photographed with the same relative positional relationship as the next photographing scheduled position selected as is displayed.

Here, in order to obtain the two-dimensional coordinates or the three-dimensional coordinates of the photographing position, for example, a DLT (Direct Linear Transformation) method or a relative orientation is used. In addition, as the number of captured images and the number of feature points are used, the accuracy of position coordinates can be increased.
According to this configuration, since the two-dimensional coordinates or three-dimensional coordinates of the photographing position of the photographer are obtained, the photographer is accurately guided to the next photographing scheduled position based on the quantitative positional relationship with the planned photographing position. it can.

  The three-dimensional measurement image photographing apparatus 1 according to the third aspect of the present invention is the three-dimensional measurement image photographing apparatus according to the first or second aspect. For example, as shown in FIG. It is a display which shows the relative moving direction with respect to the measurement object 8, or the next imaging | photography scheduled position.

Here, the display indicating the relative movement direction is, for example, displaying an arrow on the display screen, displaying "Please move to the right" in text, and announcing "Please move to the right" in voice. Etc. In addition, for example, if the next shooting scheduled position number is 2, the number 2 displayed near the scheduled shooting position on the display screen is displayed blinking. Please display ", announce with voice" Please move to number 2 ", etc.
If comprised like this aspect, it can alert | report an imaging | photography next position to a photographer intelligibly using the display of guide information.

  The three-dimensional measurement image photographing apparatus 1 according to the fourth aspect of the present invention is the three-dimensional measurement image photographing apparatus according to any one of the first to fourth aspects, as shown in FIG. 8 or FIG. A code identification unit 22 for identifying an identification code of a mark CT having another identification code is provided, and the imaging unit 2 attaches a captured image together with the measurement object 8 to the measurement object 8 or an object existing around it. The feature point extraction unit 21 extracts the mark CT as a feature point from the photographed image, and the code identification unit 22 extracts the mark CT extracted by the feature point extraction unit 21. Identify the identification code.

Here, the mark having an identification code that can be identified by itself and others is arranged in a pattern such as a code, a barcode, a two-dimensional barcode, and a color code using an identification code such as a numeric string, a character string, and a symbol. Contains a code with distinctiveness. Of these, in the embodiment, an example in which a color code is mainly used will be described.
If comprised in this way, an imaging | photography position can be correctly calculated | required from a picked-up image by sticking the mark which has the identification code which can identify self-others to the measurement object or the thing which exists in the circumference | surroundings.

  According to the present invention, it is possible to provide an image photographing apparatus for three-dimensional measurement that accurately guides a photographer from a current position to a next photographing scheduled position.

1 is a block diagram illustrating a configuration of a three-dimensional measurement image capturing apparatus according to Embodiment 1. FIG. FIG. 6 is a first diagram illustrating an example of a display screen according to the first embodiment. FIG. 6 is a second diagram illustrating an example of a display screen according to the first embodiment. FIG. 10 is a third diagram illustrating an exemplary display screen according to the first embodiment. FIG. 6 is a diagram illustrating an example of a processing flow of image shooting in the first embodiment. FIG. 6 is a block diagram illustrating a configuration of a three-dimensional measurement image capturing apparatus according to a second embodiment. It is a figure which shows the example of a color code target. FIG. 10 is a first diagram illustrating an example of a display screen according to the second embodiment. FIG. 12 is a second diagram illustrating an example of a display screen according to the second embodiment. It is explanatory drawing of the gravity center position detection using a retro target. It is a figure for demonstrating corresponding point search. It is a figure for demonstrating mutual orientation. FIG. 10 is a diagram illustrating an example of a processing flow of image capturing in the second embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals, and redundant description is omitted.

[Configuration of image photographing device for three-dimensional measurement]
FIG. 1 is a block diagram showing a configuration of a three-dimensional measurement image photographing apparatus 1 according to the first embodiment of the present invention. The three-dimensional measurement image photographing apparatus 1 includes a photographing unit 2, a display unit 3, a photographing position determination unit 4, a movement information notification unit 5, an image storage unit 6, and a control unit 7. The photographing position determination unit 4, the movement information notification unit 5, and the control unit 7 are provided in a personal computer (PC) 9. The photographing unit 2 is composed of, for example, a video camera or a digital camera, and captures a live image by photographing the measurement object 8 at the current position while the photographer moves, displays it on a viewfinder, a display, etc., and operates a shutter. For example, a captured image that is a still image is captured and acquired. In this embodiment, it is assumed that a live image is taken and a still image is taken with a video camera or a digital camera. In a video camera, a photographed image is usually stored in an external memory by a shutter operation, but if there is an internal memory, it is stored there. In a digital camera, a captured image is stored in an internal memory by a shutter operation. The image storage unit 6 is composed of, for example, a hard disk and used as a database. The image storage unit 6 stores a photographed image photographed by the photographing unit 2, a sample image of a simulated object photographed in advance with the same relative positional relationship as the planned photographing position, and the like. Although the internal memory of the camera may be used as the image storage unit 6, it is preferable to use a PC hard disk because it is suitable for various processing at high speed. In this embodiment, a camera using an external memory stores a captured image directly in the image storage unit 6, and a camera using an internal memory transfers the captured image to the image storage unit 6. The same relative positional relationship means that the direction and the distance between the imaging unit and the measurement object are equal to the direction and the distance between the imaging device that captured the sample image and the simulation object. Each sample image is stored in association with the scheduled shooting position. In addition, a live image for determining the relative position of the photographing unit 2 with respect to the next photographing scheduled position is also acquired by a shutter operation or the like and stored in the image storage unit 6. For example, the photographer inputs 1 for a photographed image and 2 for a live image from a stored live image for determining a relative position. In addition, a live image taken by a video camera or a digital camera is always overwritten in the temporary memory of the image storage unit 6 and transferred to the display unit 3. The display unit 3 includes a display such as a liquid crystal display, for example. At least a live image at the current position of the photographing unit 2 and a sample image of a simulated object that is a still image photographed at the next scheduled photographing position are displayed on the same screen. It is preferable to display both images side by side or side by side because they are easy to compare. However, if they are displayed on the same screen, they can be compared. In addition, the display unit 3 has a speaker when performing voice guidance.

  2 to 4 show examples of display screens displayed on the display unit 3. The live image 11 is displayed on the left side and the sample image 12 is displayed on the right side. The sample image 12 of the simulated object is, for example, when the imaging object 8 is an automobile, the relative positional relationship (direction, distance) that is substantially the same as the scheduled imaging position for other automobiles (the more similar the shape is). ) Refers to an image selected from a photographed image (still image) of a car that has already been photographed. The display unit 3 selects and displays the sample image stored in the image storage unit 6 from the images associated with the next shooting scheduled position. In addition, a shooting position image 13 that two-dimensionally represents the positional relationship between the shooting target and the planned shooting position is displayed. In the shooting position image 13, an installation area 14 of the shooting target object 8 is displayed, and the shooting scheduled positions are numbered (indicated by numbers 1 to 8 in circles), and each shooting planned position (◯ The direction in which the object 8 is observed and photographed from 8) within 1 to ○ is indicated by arrows. The photographer may move to the planned shooting position in the order of the numbers assigned to the planned shooting position and sequentially shoot the shooting target 8. When a still image is shot and the photographer inputs 1 from the input key, the sample image 12 changes to an image at the next scheduled shooting position. In order to know the current position during movement, the photographer inputs 2 from the input key when acquiring a live image.

  The display screen of FIG. 2 shows the case where the photographing unit 2 is at the planned photographing position (1 in the circle). In the live image 11 and the sample image 12, a car photographed from the front is displayed, and the directions of the cars are substantially coincident. In the shooting position image 13, the next shooting scheduled position (1 in ○) is displayed in red, for example, and the other shooting planned positions (2 to 8 in ○) are displayed in black, for example. The display screen of FIG. 3 shows the case where the photographing unit 2 is at the planned photographing position (2 in the circle). In the live image 11 and the sample image 12, a car photographed from the right front is displayed, and the directions of the cars are substantially the same. In the shooting position image 13, the next scheduled shooting position (2 in circles) is displayed in red, for example, and the other scheduled shooting positions (1, 3 to 8 in circles) are displayed in black, for example. The display screen of FIG. 4 shows the case where the photographing unit 2 is in the middle of the planned photographing position (1 in circle) and the planned photographing position (8 in circle). For example, it is a case where the photographer has overtaken when moving to the scheduled shooting position (8 in the circle) after shooting at the planned shooting position (1 to 7 in the circle). The live image 11 shows a car taken from the front and slightly left, the sample image 12 shows the car taken from the front left, and the shooting position image 13 shows the next shooting position (8 in circle). Is displayed in red, for example, and other scheduled shooting positions (1 to 7 in a circle) are displayed in black, for example. In addition, the live image 11 is displayed with the text “Please move to the right”, and guide information is provided so that the photographer moves in the direction of the scheduled shooting position (8 in the circle) that is the next scheduled shooting position. Is informed. Note that an arrow toward the scheduled shooting position (8 in the circle) may be displayed and guided between the planned shooting position (1 in the circle) and the planned shooting position (8 in the circle) of the shooting position image 13. good. If the photographer moves clockwise with respect to the car 8 according to this guide information, the next shooting scheduled position (8 within ○) can be reached, and if the next shooting scheduled position (8 within ○) is reached, the live image 11 is reached. Since the direction of the vehicle in the sample image 12 is substantially the same, it can be seen that the next shooting position (8 in ○) has been reached. At this time, a character display indicating that “OK” has been reached may be displayed on the live image 11 or the shooting position image 13, or “OK” may be announced by voice. When a still image is shot, the sample image 12 changes to an image associated with the next scheduled shooting position. If shooting is performed at all scheduled shooting positions (1 to 8 in a circle), the image associated with the next scheduled shooting position disappears and the sample image 12 becomes blank. In this case, the sample image 12 may be displayed with a letter “shooting complete” or may be announced with a voice “shooting complete”.

  The photographing position determination unit 4 compares the live image 11 at the current position of the photographing unit 2, that is, the photographer, with the sample image 12, and determines the relative position of the photographing unit 2 with respect to the next photographing scheduled position. The shooting position determination unit 4 analyzes the direction of the car in the live image 11. For example, a large number of sample images 12 are stored in the database in association with planned shooting positions (1 to 8 in the circles), and a pattern relatively similar to the live image 11 is extracted from the pattern 7 of these sample images. Compare with image 11. For example, if the pattern is similar to both the planned shooting position (1 in circle) and the planned shooting position (2 in circle), the shooting unit 2 determines the planned shooting position (1 in circle) and the planned shooting position. If it is determined that it is in the middle of (2 in ○) and it is more similar to the sample image at the planned shooting position (1 in ○), it is closer to the planned shooting position (1 in ○). If it is more similar to the sample image at the scheduled shooting position (2 in the circle), it is determined that it is closer to the planned shooting position (2 in the circle). The similarity between the pattern of the live image 11 and the pattern of the sample image 12 is obtained by, for example, extracting feature points from the live image 11 and the sample image 12 and statistically processing the distance between the corresponding feature points. Also good. In addition, sample images associated with positions other than the planned shooting positions (1 to 8 in the circle) (for example, sample images shot with the same relative positional relationship as the middle of the planned shooting positions, samples shot from diagonally above) Image etc.) may also be used as a comparison target.

  The relative position (direction, distance) of the photographing unit 2 with respect to the planned photographing position is determined from the direction and size of the measurement object in the photographed image. For example, sample images taken from a number of directions and distances are stored in the image storage unit 6 in association with directions and distances, and the obtained live image 11 is compared with these sample images and has high similarity. An image is extracted and the relative position of the photographing unit 2 with respect to the planned photographing position is determined. When the car is reflected in the acquired live image, the photographing unit 2 is near the car. When the car is small, the photographing part 2 is determined to be far from the car. Further, when the live image 11 is acquired from the direction of the planned shooting position (1 in circle), it is the front of the car, and when it is acquired from the direction of the planned shooting position (2 in circle), the right side of the car is scheduled to be shot. If acquired from the direction of the position (3 within ○), the right side of the car, if acquired from the direction of the planned shooting position (4 within ○), the right rear of the car, the planned shooting position (5 within ○) When acquired from the direction of the rear of the car if acquired from the direction, from the direction of the left side of the car when acquired from the direction of the planned shooting position (6 in circle), to the left of the car if acquired from the direction of the planned position of shooting (7 within ○) On the other hand, when it is acquired from the direction of the planned shooting position (8 in ○), it is determined as the left front of the automobile. In addition, when it is acquired from these intermediate positions, for example, between the planned shooting position (1 in ◯) and the planned shooting position (2 in ◯), it is determined to be slightly in front of the automobile.

  The movement information notification unit 5 creates guide information so that the photographer moves from the current position of the photographing unit 2 to the next photographing scheduled position and causes the display unit 3 to inform the guide information. The movement information notification unit 5 notifies the guide information based on the determination of the relative position of the photographing unit 2 with respect to the next photographing scheduled position by the photographing position determination unit 4. For example, when the photographing unit 2 is at the planned photographing position (1 in the circle) or the planned photographing position (2 in the circle), the guide information need not be created, and “OK” may be created. . Also, for example, when the planned shooting position (1 in ○) and the planned shooting position (8 in ○) are in the middle, and the next shooting planned position is the planned shooting position (8 in ○), “to the right “Please move” guide information is generated and displayed on the display, which is the display unit 3, or announced on the speaker. Also, for example, when the planned shooting position (1 in circle) and the planned shooting position (2 in circle) are in the middle, and the next shooting scheduled position is the planned shooting position (2 in circle), “to the left “Please move” guide information is generated and displayed on the display, which is the display unit 3, or announced on the speaker. Further, the guide information may be information that guides the direction and the distance, but the guide related to the distance can also be expressed by the direction, so that only the direction may be guided. Further, when the distance between the measurement object 8 and each planned shooting position (1 to 8 in the circle) is predetermined (for example, 5 m), such as when the measurement object 8 is photographed from the surroundings, photographing from a distant place is possible. When the influence of the distance can be ignored, a lateral guide such as “to the right” or “to the left” may be used.

  The control unit 7 has a control program in the built-in memory, controls each part of the three-dimensional measurement image photographing apparatus 1, controls the flow of signals and data, and executes the function as the three-dimensional measurement image photographing apparatus. . In particular, a live image is always acquired from the imaging unit 2 and displayed on the display unit 3, and the next scheduled shooting position is determined in order of the number assigned to the planned shooting position for each still image shooting, from the image storage unit 6. An image of a simulated object similar to the measurement object 8 is selected from the images associated with the next shooting scheduled position and displayed on the display unit 3 as a sample image 12. Further, the shooting position determination unit 4 compares the live image 11 with the sample image 12 to determine the relative position of the shooting unit 2 with respect to the next shooting scheduled position, and the movement information notification unit 5 takes a shot at the next shooting scheduled position. Guide information is created so that a person moves.

  FIG. 5 shows an example of a processing flow of image shooting in the present embodiment. First, a scheduled shooting position (1 to 8 in a circle) and a shooting order are set in the control unit 6 (S101). Here, shooting is performed in the order of the number of the scheduled shooting position. Next, a sample image associated with the first next scheduled shooting position is displayed on the display unit 3 together with the live image (S102). The photographer compares the live image 11 with the sample image 12 and determines that the first next shooting scheduled position has been reached, and acquires the shot image (S103). When shooting is performed and, for example, the photographer inputs 1, the sample image changes to an image corresponding to the next scheduled shooting position (S104). While the photographer compares the live image 11 with the sample image 12 again, it is determined that the photographer has reached the next scheduled shooting position, and acquires a captured image (S105). In this way, the image display and shooting are repeated in the order of the number of the scheduled shooting position, and the photographer determines that the last scheduled shooting position has been reached and acquires the shot image (S106). When shooting at the scheduled shooting position is completed, the photographer displays the captured image on the display unit 3 and analyzes it, and analyzes whether the image was shot at an appropriate scheduled shooting position or whether the image is free from defects such as blurring and contamination. If there is an inappropriate image (NO in S108), the planned shooting position is set again (returning to S101), and a shortage of captured images is acquired. If there is no inappropriate image (YES in S108), the shooting is terminated. At the time of analysis, the shooting position determination unit 4 analyzes the direction of the car in the shot image, and displays an arrow indicating the direction (front) of the car in the installation area 14 of the shooting object in the shot position image 13. Also good.

  As described above, according to the present embodiment, it is possible to provide a three-dimensional measurement image photographing apparatus that accurately guides the photographer from the current position to the next photographing scheduled position.

  In Example 2, a mark having an identification code that can be identified by itself is attached to an object to be measured or a surrounding object, a position coordinate of the shooting position is obtained using the position coordinates of the mark, and the obtained shooting position An example in which the next shooting scheduled position is automatically selected based on the position coordinates will be described.

  FIG. 6 is a block diagram illustrating the configuration of the three-dimensional measurement image capturing apparatus 1 according to the second embodiment. A feature point extraction unit 21, a code identification unit 22, a shooting position measurement unit 23, and a next shooting scheduled position selection unit 24 are added to the configuration of FIG. These are provided in a personal computer (PC) 9. The feature point extraction unit 21 extracts feature points in the captured image acquired by the imaging unit 2. The code identification unit 22 identifies the identification code of the mark. The shooting position measurement unit 23 obtains two-dimensional coordinates or three-dimensional coordinates of the shooting position at which the shot image is shot from the screen position in the shot image of the feature point extracted by the feature point extraction unit 21. The next scheduled shooting position selection unit 24 selects the next scheduled shooting position from a plurality of planned shooting positions based on the obtained position coordinates of the shooting position. Further, the photographing position determination unit 4 determines the relative position of the photographing unit with respect to the next photographing scheduled position based on the obtained position coordinates of the photographing position.

  The feature point extraction unit 21 extracts feature points in a plurality of photographed images acquired by the photographing unit 2. In this embodiment, a color code target is affixed around an automobile that is a measurement object, and the color code target can be used as a feature point.

[Color code target]
FIG. 7 shows an example of the color code target CT. 7A shows three color code target areas, FIG. 7B shows six color code targets, and FIG. 7C shows nine color code targets. The color code target CT (CT1 to CT3) in FIGS. 7A to 7C includes a position detection pattern (retro target portion) P1, a reference color pattern (reference color portion) P2, and a color code pattern (color code portion). P3 and an empty pattern (white portion) P4.

  The retro target portion P1 is used for detecting the target itself, detecting its center of gravity, detecting the direction of the target, and detecting the target area.

  The reference color portion P2 is used for reference at the time of relative comparison and for color calibration for correcting color misregistration in order to cope with color misregistration due to shooting conditions such as illumination and a camera. Furthermore, the reference color portion P2 can be used for color correction of the color code target CT created by a simple method. For example, when using a color code target CT printed by a color printer (inkjet, laser, sublimation type printer, etc.) where color management is not performed, individual differences appear in the color of the printer used, but the reference color portion P2 By comparing and correcting the color of the color code portion P3 relative to each other, the influence of individual differences can be suppressed.

  The color code part P3 expresses a code by a combination of color schemes for each unit area. The number of codes that can be expressed varies depending on the number of code colors used for the code. For example, when the number of code colors is n, n × n × n codes can be represented. In order to increase the reliability, even when the condition that the colors used in other unit areas are not used redundantly, n × (n−1) × (n−2) codes can be expressed. If the number of code colors is increased, the number of codes can be increased. Furthermore, if the condition that the number of unit areas of the color code portion P3 is equal to the number of code colors is imposed, all the code colors are used for the color code portion P3. Therefore, not only the comparison with the reference color portion P2, By relatively comparing the colors between the unit areas of the color code part P3, the color of each unit area can be confirmed to determine the identification code, and the reliability can be improved. Furthermore, if a condition for making the area of each unit region all the same is added, the color code target CT can also be used for detection from the image. This is because the area occupied by each color is the same even between the color code targets CT having different identification codes, and thus almost the same dispersion value is obtained from the detection light from the entire color code portion. In addition, since the boundary between the unit areas is repeated at equal intervals and a clear color difference is detected, the target CT can be detected from the image from such a repeated pattern of detection light.

  The white portion P4 is used for detecting the orientation of the color code target CT and for color misregistration calibration. Among the four corners of the target CT, there is a place where no retro target is arranged, and this can be used for detecting the direction of the target CT. Thus, the white part P4 should just be a pattern different from a retro target. Therefore, a character string such as a number for visually confirming the code may be printed on the white portion, or may be used as a code area such as a barcode. Further, in order to increase the detection accuracy, it can be used as a template pattern for template matching.

  The code identification unit 22 identifies the identification code of the color code target CT. Using the color code target correspondence table that records the correspondence between the pattern arrangement and the code number, the identification code is determined from the color arrangement in the color code portion P3 of the color code target CT, and a number is assigned to the color code target CT.

  FIG. 8 shows an example of a display screen displayed on the display unit. In the live image 11, the color code target CT is pasted around the automobile that is the measurement object 8. This is because the color code target CT arranged around or on the imaging object 8 is automatically extracted, and the imaging position measurement unit 23 captures the image from the position of the color code target CT where the identification code is identified. This is because the two-dimensional coordinates or the three-dimensional coordinates of the current position of the unit 2 are obtained, and a sample image corresponding to the next scheduled shooting position is automatically displayed. In the shooting position image 13, it is displayed that the current position of the shooting unit 2 is between the planned shooting position (6 in the circle) and the planned shooting position (7 in the circle). In addition, the sample image 12 displays an image of the next shooting scheduled position (6 in the circle) taken from the rear left side of the car, and the movement information notification unit 5 “moves to the right” below the sample image 12. Guide information is displayed.

  FIG. 9 shows another example of the display screen displayed on the display unit. FIG. 9 shows a display example of a live image, in which a color code target CT and a retro target (a white circle framed in black) are pasted on the surface of the earthenware. In this case as well, as in the case of the automobile in FIG. 8, the color code target CT is automatically extracted, the camera position is obtained from the position of the color code target CT where the identification code is identified, and the next sample image is automatically obtained. This is because of the display. The retro target itself does not have discrimination, but discriminability arises from the positional relationship with the color code target CT pasted around it, so that the two-dimensional coordinates or the three-dimensional coordinates of the current position of the imaging unit 2 are obtained. Can be used for

  The position of the color code target CT is obtained by detecting the retro target P1. When a retro target is affixed in addition to the color code target CT, the mark position can be obtained by detecting the retro target even when a retro target is affixed instead of the color code target CT.

  FIG. 10 is an explanatory diagram of center-of-gravity position detection using a retro target. In this embodiment, the retro target is formed by two concentric circles. In FIG. 10A1, the lightness of the inner circle portion 204 that is the inner side of the small circle among the concentric circles is bright, and the lightness of the outer circle portion 206 that is an annular portion formed between the small circle and the large circle is dark. The retro target 200, FIG. 10 (A2) is a brightness distribution diagram in the diameter direction of the retro target 200 of (A1), and FIG. 10 (B1) is a retro target in which the brightness of the inner circle portion 204 is dark and the brightness of the outer circle portion 206 is bright. 200 and FIG. 10 (B2) show the brightness distribution diagram in the diameter direction of the retro target 200 of (B1). When the brightness of the inner circle portion 204 is bright as shown in FIG. 10 (A1), the retro target has a bright portion with a large amount of reflected light at the center of gravity in the captured image of the measurement object 1, and thus the light amount distribution of the image. As shown in FIG. 10 (A2), the inner circle portion 204 and the center position of the retro target can be obtained from the threshold value To of the light amount distribution. In the color code target CT of the present embodiment, in the case of (A1), it is sufficient that one circle corresponding to a small circle is formed at the center of the square position detection pattern P1, and the outside of the great circle is bright. May be dark, but it should be dark. When it is dark, substantially only one circle (small circle) is formed. In the case of (B1), the outside of the great circle may be bright or dark, but when the outside of the great circle is bright, only one dark circle (small circle) is formed on the substantially bright base. It is.

When the target existence range is determined, the barycentric position is calculated by, for example, the moment method. For example, the plane coordinates of the retro target 200 shown in FIG. 10A1 are assumed to be (x, y). Then, (Expression 1) and (Expression 2) are calculated for points in the x and y directions where the brightness of the retro target 200 is greater than or equal to the threshold value To.
xg = {Σx * f (x, y)} / Σf (x, y) ---- (Equation 1)
yg = {Σy * f (x, y)} / Σf (x, y) ---- (Formula 2)
(Xg, yg): coordinates of the center of gravity position, f (x, y): light quantity on (x, y) coordinates In the case of the retro target 200 shown in FIG. (Expression 1) and (Expression 2) are calculated for the points in the x and y directions. Thereby, the position of the center of gravity of the retro target 200 is obtained.

  When a color code target or a retro target is not used, the feature point extraction unit 21 extracts feature points from a plurality of captured images. The feature points include, for example, a center position, a gravity center position, a corner position of the measurement object 8, a position having a characteristic different from the others, a sign attached to or projected on the measurement object 8. A feature extraction operator is used for feature point extraction. Here, an example in which a Moravec operator is used will be described.

  The MORAVEC operator has long been used as a general-purpose feature extractor. The MORAVEC operator uses, for example, a 3 × 3 pixel around a certain target pixel as a mask, and sets the minimum value of the density difference (direction density difference) when the mask moves one pixel in each of the four directions around the target pixel. The feature value. It is characterized by simple and high-speed processing and relatively easy hardware. In order to perform high-speed processing, a memory several times as large as an image is required. Although the feature point extraction by the Moravec operator has been described here, other operators, for example, the Harris operator and others, and anything that can detect feature points may be used.

The shooting position measurement unit 23 obtains two-dimensional coordinates or three-dimensional coordinates of the shooting position at which the shot image is shot from the screen position in the shot image of the feature point extracted by the feature point extraction unit 21. When the position of the extracted target is known, the two-dimensional coordinates or the three-dimensional coordinates of the feature point of the measurement object 8 and the imaging position can be obtained using the DLT method.
[DLT method]
The DLT method approximates the relationship between the photographic coordinates and the three-dimensional coordinates (target point coordinates) of the subject using a cubic projective transformation equation.
The basic expression of the DLT method is (Expression 3-1).
If the denominator is deleted from (Equation 3-1), the following linear form can be derived.

Furthermore, when (Formula 3-2) is modified, the following formula is obtained.
If (Expression 3-3) is solved directly using the least square method, 11 unknown variables L _{1 to} L ₁₁ that determine the relationship between the photographic coordinates and the target point coordinates can be acquired. Therefore, by solving (Equation 3-3) for six points whose position coordinates are known, the photographic coordinates (x, y) of the subject and the three-dimensional coordinates (target point coordinates) (X, Y, Z) of the subject are obtained. The relationship is obtained, and the position coordinates (0, 0, 0) of the photographing position corresponding to the convergence point of the straight line connecting the photograph coordinates (x, y) and the target point coordinates (X, Y, Z) can be obtained. In this case, the shooting position is a position (−X, −Y, −Z) with respect to the target point coordinates.

When the position of the target is not fixed, or when the position coordinates of the feature point are unknown, the feature point of the measurement object 8 and the image are captured by mutual orientation using the feature points of the overlapping portions in two or more captured images. Find the 2D or 3D coordinates of the position.
First, two captured images are set as stereo images (one is a reference image and the other is a search image), and feature points are associated (corresponding points are searched). Corresponding point search is performed by cross-correlation processing.

[Correlation processing]
Cross correlation coefficient method

FIG. 11 is a diagram for explaining the corresponding point search. As shown in FIG. 11, the template image of N ₁ × N ₁ pixel is moved over the search range (M ₁ −N ₁ +1) ² in the input image of M ₁ × M ₁ pixel larger than that, and C The upper left position of the template image in which (a, b) is maximized is obtained, and it is considered that the template image has been searched. In the case of left and right images, for example, a template image of N ₁ × N ₁ pixel is set on the left image, a search region of M ₁ × M ₁ pixel is set on the right image, and this operation is performed on each image. What is necessary is to do about the position.
Here, the normalized cross-correlation process has been described with respect to the corresponding point search, but other methods such as a residual sequential test method (SSDA), a direction code matching method (OCM), and the like may be used.

Next, the photographing position in the photographing unit 2 and the three-dimensional coordinates of the feature points are obtained from the points associated with the left and right images. First, a method for obtaining the camera position and tilt by the relative orientation method will be described.
[Calculation of external orientation elements: mutual orientation]
A model image refers to a stereoscopic image obtained when two or more stereoscopic photographs are reproduced as they were taken. Forming relatively similar model images is called relative orientation. That is, the relative orientation determines the positions and inclinations of the left and right projection centers so that two corresponding light beams of the stereoscopic photograph meet.

FIG. 12 is a diagram for explaining relative orientation. Next, the details of the orientation calculation of each model image will be described. By this calculation, the positions of the left and right cameras (three-dimensional coordinates and three-axis tilt) are obtained.
Those parameters are obtained by the following coplanar conditional expression.

The origin of the model coordinate system is taken as the left projection center, and the line connecting the right projection centers is taken as the X axis. For the scale, the base line length is taken as the unit length. The parameters to be obtained at this time are the rotation angle κ ₁ of the left camera, the rotation angle φ _{1 of} the Y axis, the rotation angle κ _{2 of the} right camera, the rotation angle φ ₂ of the Y axis, and the rotation of the X axis. the five of the rotation angle of the corner ω _2. In this case, since the rotation angle ω ₁ of the X axis of the left camera is 0, it is not necessary to consider.

Under such conditions, the coplanar conditional expression of (Equation 4) becomes (Equation 5), and each parameter can be obtained by solving this expression.

Here, between the model coordinate system XYZ and the camera coordinate system xyz, the following coordinate transformation relational expressions (formula 6) and (formula 7) hold.

Using these equations, unknown parameters are obtained by the following procedure.
(A) The initial approximate value of the unknown parameter is normally 0.
(B) The coplanar conditional expression (Expression 5) is Taylor-expanded around the approximate value, and the value of the differential coefficient when linearized is obtained by (Expression 6) and (Expression 7), and the observation equation is established.
(C) A least square method is applied to obtain a correction amount for the approximate value.
(D) Correct the approximate value.
(E) Using the corrected approximate value, the operations (b) to (e) are repeated until convergence.
By obtaining unknown parameters (κ ₁ , φ ₁ , κ ₂ , φ ₂ , ω ₂ ), the position and tilt of the camera can be obtained.

  The next scheduled shooting position selection unit 24 selects the next scheduled shooting position from a plurality of planned shooting positions based on the obtained position coordinates of the shooting position. The next shooting scheduled position selection unit 24 has a position coordinate of a plurality of shooting scheduled positions and a shooting scheduled position table that stores whether shooting has been performed or not shot at the shooting scheduled position. Referring to the scheduled shooting position table, for example, compare the distance between the obtained shooting position and each scheduled shooting position (the distance that bypasses the measured object if it is in between), and the unscheduled scheduled shooting position Of these, the scheduled shooting position that is the shortest distance from the obtained shooting position is selected as the next scheduled shooting position. For example, in the shooting position image 13 of FIG. 8, the current position of the shooting unit 2 is between the planned shooting position (6 within ○) and the planned shooting position (7 within ○), and shooting is performed among unscheduled shooting planned positions. When the planned position (6 in ○) is at the shortest distance from the shooting position for which the position coordinates are obtained, an image shot on the sample image 12 with the same relative positional relationship as the planned shooting position (6 in ○). Is displayed as a sample image of the next shooting position. If there are two planned shooting positions at the shortest distance, the selection is made with priority given to either clockwise or counterclockwise.

FIG. 13 shows an example of a processing flow of image capturing.
First, a measurement object is selected (S10), and a color code target is pasted on the measurement object or an object existing around it (S20). A scheduled shooting position is determined in advance, and shooting is started (S30). That is, the live image 11 is displayed on the display unit 3, and the photographer operates the shutter at an appropriate position during movement, for example, and inputs 2 from the input key to acquire the live image. Next, the feature point extraction unit 21 detects a color code target in the acquired live image (S35), and the code identification unit 22 identifies the identification code of the color code target (S40). The photographing position measurement unit 23 calculates and obtains the two-dimensional coordinates or the three-dimensional coordinates of the current position of the photographing unit 2 from the position of the color code target in the obtained live image and the position coordinates to which the color code target is pasted ( S50). The next shooting scheduled position selection unit 24 obtains the next shooting scheduled position based on the obtained position coordinates of the current position of the shooting unit 2 (S55), and the shooting position determination unit 4 determines the obtained current position of the shooting unit 2. Based on the position coordinates, the relative position of the photographing unit 2 with respect to the next photographing scheduled position is determined. The movement information notification unit 5 creates guide information in accordance with the determination of the shooting position determination unit 4. The display unit 3 displays the sample image 12 photographed with the same relative positional relationship as the next photographing scheduled position (S60). At this time, the current position is displayed on the shooting position image 13 of the display unit 3 and the guide information created by the movement information notification unit 5 is displayed. The photographer follows the guide information and arrives at the next shooting scheduled position while comparing the live image 11 and the sample image 12, and for example, operates the shutter and inputs 1 from the input key to acquire the shot image ( S70). Next, for the acquired captured image, the acquisition of the captured image (S70) is sequentially repeated from the detection of the color code target (S35). If the photographer wants to know the current position after obtaining the photographed image (S70) or wants to obtain guide information, the user operates the shutter at that position and inputs 2 from the input key to obtain the live image. . In this case, although the relative position of the photographing unit 2 with respect to the next photographing scheduled position is different from the case of photographing image acquisition, the photographing image acquisition (S70) is performed from the detection of the color code target (S35). When shooting has been performed at all scheduled shooting positions (YES in S80), shooting is terminated.

  As described above, according to the present embodiment, it is possible to provide a three-dimensional measurement image photographing apparatus that accurately guides the photographer from the current position to the next photographing scheduled position.

  The present invention can also be realized as an invention of a three-dimensional measurement image photographing method described in the flowcharts of the above-described embodiments, and as a program for causing a three-dimensional measurement image photographing apparatus to execute the method invention. is there. The program may be stored and used in a built-in storage unit of the three-dimensional measurement image capturing apparatus, may be stored and used in an external storage device, or may be downloaded from the Internet and used. Moreover, it is realizable also as a recording medium which recorded the said program.

  Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and it is obvious that various modifications can be made to the embodiments without departing from the spirit of the present invention. .

  For example, in the above embodiment, an example in which a live image and a still image are acquired by the same video camera or digital camera has been described. For example, a position slightly shifted left and right at each planned shooting position (for example, a position moved by one step) Then, a single set of stereo images may be taken. In this way, it is close to shooting with a stereo camera, and camera orientation and shooting position measurement are facilitated. Further, in the second embodiment, an example is described in which the shooting position measurement unit obtains the shooting position by capturing the live image and the shot image and the photographer's key input, and the next shooting position of the sample image is changed. When the position selection unit 24 selects the next shooting scheduled position, the next shooting scheduled position of the sample image may be automatically changed. In addition, the number of planned shooting positions, the arrangement of targets, the text of guide information, and the like can be changed as appropriate.

  The present invention can be used for photographing for three-dimensional measurement and three-dimensional model image creation. In particular, when the inexperienced person or an unfamiliar person takes a picture, and even if an accustomed person takes a picture, the measurement object is large, the surface shape is complicated, the same repetitive pattern appears, etc. It is possible to shoot images with adjacent images in an appropriate overlapping range.

DESCRIPTION OF SYMBOLS 1,1A Three-dimensional measurement image imaging device 2 Imaging unit 3 Display unit 4 Imaging position determination unit 5 Movement information notification unit 6 Image storage unit 7 Control unit 8 Measurement object 9 Personal computer (PC)
DESCRIPTION OF SYMBOLS 11 Live image 12 Sample image 13 Shooting position image 14 Installation area | region 21 of a to-be-photographed object Feature point extraction part 22 Code identification part 23 Shooting position measurement part 24 Next shooting position selection part 200 Retro target 204 Inner circle part 206 Outer circle part CT Color Code target CT
P1 Retro target part P2 Reference color part P3 Color code part P4 White part To Threshold

Claims (4)

In a three-dimensional measurement image capturing apparatus that sequentially captures an object to be measured while overlapping with a single camera;
An imaging unit that acquires a live image by capturing an object to be measured while the photographer moves and acquires a captured image that is a still image in a timely manner;
A display unit for displaying a live image at the current position of the shooting unit and a sample image of a simulated target object, which is a still image shot at substantially the same relative positional relationship as the next shooting scheduled position, on the same screen;
A shooting position determination unit that compares a live image at the current position of the shooting unit with the sample image and determines a relative position of the shooting unit with respect to the next shooting scheduled position;
A movement information notifying unit that creates guide information so that a photographer moves from the current position to the next scheduled shooting position and causes the display unit to notify the guide information;
An image capturing device for three-dimensional measurement. A feature point extraction unit that extracts feature points in a plurality of captured images acquired by the imaging unit;
A shooting position measuring unit for obtaining a two-dimensional coordinate or a three-dimensional coordinate of a shooting position at which the shot image is shot from a screen position in the shot image of the feature point extracted by the feature point extraction unit;
A planned next shooting position selection unit that selects a planned next shooting position from a plurality of planned shooting positions based on the obtained position coordinates of the shooting position;
The display unit displays, as the sample image, an image of a simulated object photographed with the same relative positional relationship as the selected next scheduled photographing position;
The three-dimensional measurement image capturing apparatus according to claim 1. The guide information is a display indicating a relative movement direction of the imaging unit with respect to the measurement object or the next imaging scheduled position;
The image photographing device for three-dimensional measurement according to claim 1 or 2. A code identification unit for identifying the identification code of a mark having an identification code that can identify itself and others;
The imaging unit captures and acquires the captured image so as to include a plurality of the marks attached to the measurement object or an object existing therearound together with the measurement object;
The feature point extraction unit extracts the mark as a feature point from the captured image;
The code identification unit identifies the identification code of the mark extracted by the feature point extraction unit;
The image photographing device for three-dimensional measurement according to any one of claims 1 to 3.