JP2006059165A - Three-dimensional modeling device, geometric pattern, three-dimensional modeling data generating method, three-dimensional modeling program and recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a three-dimensional modeling device capable of easily effecting three-dimensional modeling to an object. <P>SOLUTION: The invention comprises a calibration part 10 (calibration means) that calculates calibration data from image data obtained from an image in which the object is photographed together with a check pattern board (geometric pattern), a silhouette extraction part 11 (silhouette extraction means) that calculates silhouette data of the object using standard check pattern data obtained from a standard image in which only a check pattern board is photographed, the calibration data, and the image data, and a voxel data generation part 12 (three-dimensional data generation means) that generates voxel data (three-dimensional data) based on the silhouette data and the calibration data. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a three-dimensional modeling apparatus that generates three-dimensional data of an object from an image of the object, and a geometric pattern, a three-dimensional modeling data generation method, a three-dimensional modeling program, and a recording medium related thereto.

  Conventionally, a method for converting a three-dimensional shape of an object existing in the real world into data has been proposed. For example, there are a method for obtaining a corresponding point between multi-viewpoint images and measuring a distance by triangulation or a method for measuring a distance by irradiating pattern light such as a laser and using it as a clue.

  However, these methods have a problem that many errors occur in searching for corresponding points, or input is possible only at a specific place where the device is installed because a large-scale device is required.

  Therefore, a method for estimating the shape of an object using a silhouette image has been proposed. This method is called visual volume intersection method and is disclosed in “Wu et al.,“ Active real-time photography and interactive editing and display of 3D video images ”, IEICE PRMU2001-187, Feb.2001”. . The prior art using this is described in Patent Documents 1-5.

  In the prior art described in Patent Document 1, three-dimensional modeling is performed by the following steps. First, a plurality of images are generated by capturing computer graphics images of an object represented in the virtual space from the three-dimensional shape data of the object at a plurality of camera positions in the virtual space. Next, the camera data at the camera position when captured is generated. Next, object restoration data is generated by individually processing the shape and texture of the object based on the image data and camera data.

  The prior art described in Patent Document 2 is a three-dimensional modeling device that generates three-dimensional shape data of a subject based on a plurality of camera images obtained by photographing a subject having a three-dimensional shape with a plurality of cameras arranged around the subject. 3D modeling is performed by the following steps. First, a rough shape measurement is performed to measure a rough shape of a subject circumscribing the subject from a plurality of camera images. Next, a reference camera image taken by sequentially switching a reference camera serving as a reference among the plurality of cameras, using the vicinity of the approximate shape measured by the approximate shape measuring means as a search range, and the reference camera Corresponding points corresponding to the subject image in the reference camera image are searched from the adjacent camera image based on the adjacent camera image taken by the adjacent camera adjacent to the image. Next, the distance from the reference camera to the corresponding point is calculated based on the distance between the reference camera and the adjacent camera of the corresponding point of the subject searched by the corresponding point searching means. Next, the three-dimensional shape data of the subject is generated based on the distances from the plurality of cameras to the corresponding points of the subject calculated by the distance information calculating means.

  The prior art described in Patent Document 3 performs three-dimensional modeling by the following steps. First, pattern light is irradiated to the three-dimensional object. Next, an image of the three-dimensional object is input. Next, the approximate shape is calculated from the input image. Next, a detailed shape is calculated from the input image. Next, three-dimensional shape data of the three-dimensional object is generated based on the calculated schematic shape and detailed shape.

The prior art described in Patent Document 4 performs three-dimensional modeling by the following steps. First,
First volume data is generated by converting a plurality of images of objects with different viewpoints into volume data using a silhouette method. Next, second volume data is generated by converting the three-dimensional data obtained by the three-dimensional measurement for the object into volume data. Next, the first volume data and the second volume data are integrated into one volume data.

The prior art described in Patent Document 5 receives a plurality of captured images obtained by capturing the same subject from a number of viewpoint directions and viewpoint information thereof, restores the approximate shape of the subject, and further sets the viewpoint on the shape surface. 3D image information generation method for adding color information of a corresponding captured image (more specifically, a plurality of captured images obtained by capturing the same subject from a plurality of viewpoint directions and the viewpoint information thereof are input, and 3D image information generating method) that restores the schematic shape and further adds color information of the captured image corresponding to the viewpoint to the surface of the shape, and generates 3D image information by the following steps. First, among the captured images, a plurality of images in which the entire shape of the subject is captured are selected as silhouette images, and the approximate shape of the subject is restored as a set of minute regions using the silhouette information. Subsequently, the size and shape of the hexagonal image for adding color information of the photographed image to the restored schematic shape, and the correspondence between the hexagonal pixels constituting the hexagonal image and the photographed image are determined. Next, the hexagonal image obtained in the step of determining the correspondence is assigned to all the minute regions on the shape surface, and the projection line connecting the viewpoint of the photographed image and the minute region is intersected with the photographed image for each minute region. Are assigned to the corresponding hexagonal pixels. Next, for the hexagonal pixel to which no color is assigned in the assigning step, the step of determining the color using the color of the adjacent hexagonal pixel, and a two-dimensional array of all hexagonal images composed of the hexagonal pixel The image data is merged into one or a plurality of image data.
JP 2003-141565 A (publication date: May 16, 2003) JP 2003-271928 A (publication date: September 26, 2003) JP 2001-243468 A (publication date: September 7, 2001) JP 2002-366934 A (publication date: December 20, 2002) JP 2003-99800 A (publication date: April 4, 2003)

  However, in the conventional technique described in Patent Document 1, an object in a virtual space that has already been displayed by computer graphics is targeted, and in order to model an object in the real world, a system that inputs this into computer graphics is required. In the prior art described in Patent Document 2, since a subject having a three-dimensional shape has to be photographed by a plurality of cameras arranged around it, a large camera system is required at the time of photographing. Moreover, in the prior art described in Patent Document 3, an apparatus for irradiating pattern light is required. Further, in the prior art described in Patent Document 4, when generating the second volume data, the three-dimensional data obtained by the three-dimensional measurement for the object is converted into the volume data, so that the three-dimensional measurement for the object is required. Become. Further, in the prior art described in Patent Document 5, it is necessary to photograph the same subject from many viewpoint directions, for example, a plurality of images in which the entire shape of the subject is photographed are necessary.

  As described above, in the above-described conventional technique, a large-scale photographing apparatus, complicated photographing, or a large amount of data processing is required, and three-dimensional data cannot be easily generated.

  The three-dimensional modeling apparatus of the present invention has been made to solve the above-described problems, and an object thereof is to provide a three-dimensional modeling apparatus that can easily three-dimensionally model an object.

  In order to solve the above-described problem, the three-dimensional modeling apparatus of the present invention includes a calibration unit that calculates calibration data from image data obtained from each image in which an object is photographed together with a geometric pattern, and only the geometric pattern. A silhouette extraction means for calculating silhouette data of the object using the reference geometric pattern data obtained from the reference image taken of the image, the image data and the calibration data corresponding to the image data, and corresponding to each of a plurality of images And a three-dimensional data generating means for generating three-dimensional data using silhouette data and calibration data.

  In the above-described configuration, the calibration means includes calibration data (for example, a projection matrix or a camera that associates the coordinate system of the image with the coordinate system of the camera from image data obtained from an image obtained by photographing the object together with the geometric pattern. Parameter).

  The silhouette extracting means calculates the silhouette data of the object using the reference geometric pattern data obtained from the reference image obtained by photographing only the geometric pattern, the calibration data, and the image data.

  The three-dimensional data generation means generates three-dimensional data based on the silhouette data and the calibration data.

  As described above, the three-dimensional modeling apparatus according to the present invention is an image obtained by photographing an object together with a geometric pattern and a reference image obtained by photographing only the geometric pattern (or reference geometric pattern data itself). Dimension modeling is performed.

  Therefore, it is possible to easily (simplely) convert various objects into three-dimensional data without requiring dedicated photographing equipment and processing devices as in the prior art. Accordingly, if the three-dimensional modeling apparatus of the present invention is used for entertainment, distribution (for example, product images of a WEB shop), medical treatment (for example, three-dimensional data of a living body), etc., it is very useful and convenient.

  In the three-dimensional modeling apparatus of the present invention, it is preferable that the geometric pattern is a planar geometric pattern, and the calibration means calculates calibration data by performing planar calibration.

  According to the above configuration, since the geometric pattern is a plane, for example, a portable geometric pattern board can be used as the geometric pattern. In addition, since the calibration means only needs to perform plane calibration, the amount of data to be processed is also reduced. As a result, various objects can be converted into three-dimensional data more easily (simplely).

  In the three-dimensional modeling apparatus of the present invention, the geometric pattern is preferably a check pattern. When a geometric pattern is used as a check pattern, calibration and silhouette extraction are easy.

  In the three-dimensional modeling apparatus of the present invention, it is preferable that the three-dimensional data generation unit is a voxel data generation unit that generates voxel data, and facilitates the generation of the three-dimensional data.

  In the three-dimensional modeling apparatus of the present invention, it is preferable that an input unit that can input a plurality of points surrounding the object is provided for each image to be processed by the calibration unit. According to this configuration, the calibration process can be performed efficiently.

  In the three-dimensional modeling apparatus of the present invention, the calibration data includes projection matrix data that associates each image and geometric pattern, and photographing viewpoint data (for example, camera parameters). The silhouette extracting means preferably calculates the silhouette data by a difference process between the conversion data of the reference geometric pattern data based on the projection matrix data and the corresponding image data. According to this configuration, the silhouette extraction process can be performed efficiently.

  In the three-dimensional modeling apparatus of the present invention, the calibration data includes projection matrix data that associates each image and geometric pattern, and photographing viewpoint data (for example, camera parameters). Preferably, the voxel data generation means generates voxel data by an intersection determination process between the projection data in the voxel space based on the projection matrix data and the silhouette data. According to this configuration, the voxel data generation process can be performed efficiently.

  In the three-dimensional modeling apparatus of the present invention, it is preferable that voxel data correction means for deleting data related to internal voxels from the voxel data is provided. According to this configuration, it is possible to improve data processing efficiency by reducing unnecessary voxel data.

  The three-dimensional modeling apparatus of the present invention preferably includes polygon data generation means for generating polygon data as surface information from the voxel data. According to the said structure, data processing efficiency can be improved by processing only the surface information (data) of a target object.

  The three-dimensional modeling apparatus of the present invention preferably includes color data generation means for generating color data associated with the polygon data using the image data, calibration data, and polygon data. According to the said structure, a target object can be expressed with a color and the expressive power of a target object can be improved. In this case, the color data generating means selects an image suitable for color generation from the image using the photographing viewpoint data and polygon data, and the projection is applied to the image data of the selected image and the polygon data. It is preferable to generate color data by associating with projection data using matrix data. In this way, color data generation processing can be performed efficiently.

  The geometric pattern of the present invention is characterized in that it can be used for taking an input image of the three-dimensional modeling apparatus.

  The three-dimensional modeling data generation method of the present invention includes a calibration process for calculating calibration data from image data obtained from each image obtained by photographing an object together with a planar geometric pattern, and only the geometric pattern. A silhouette extraction step of calculating silhouette data of the object using reference geometric pattern data obtained from a reference image taken of the image, the image data, and calibration data corresponding thereto, and each of the plurality of images And a voxel data generation step of generating voxel data using silhouette data and calibration data.

  The three-dimensional modeling program of the present invention is characterized by causing a computer to realize each means of the three-dimensional modeling apparatus.

  The recording medium of the present invention is characterized in that the above three-dimensional modeling program is stored in a computer so as to be readable.

  As described above, the three-dimensional modeling apparatus of the present invention can capture an object from only an image obtained by photographing an object together with a geometric pattern and a reference image obtained by photographing only the geometric pattern (or reference geometric pattern data itself). 3D modeling is performed. Therefore, it is possible to easily (simplely) convert various objects into three-dimensional data without requiring dedicated photographing equipment and processing devices as in the prior art. Accordingly, if the three-dimensional modeling apparatus of the present invention is used for entertainment, distribution (for example, product images of a WEB shop), medical treatment (for example, three-dimensional data of a living body), etc., it is very useful and convenient.

  An embodiment of the present invention will be described below with reference to FIGS.

  Here, FIG. 1 is a block diagram showing a configuration of the three-dimensional modeling apparatus 1 according to the present embodiment.

  As shown in FIG. 1, the three-dimensional modeling apparatus 1 of the present invention includes a control unit 2, a storage unit 3, a user interface 23 (input unit), and a display unit 9.

  The control unit 2 includes a data input unit 6, a calibration unit 10 (calibration unit), a silhouette extraction unit 11 (silhouette extraction unit), and a voxel data generation unit 12 (three-dimensional data generation unit, voxel data generation unit). A voxel data correcting unit 13 (voxel data correcting unit), a polygon data generating unit 15 (polygon data generating unit), a color data generating unit 16 (color data generating unit), a color data adding unit 17, and a display control. Part 21.

  The control unit 2 generates voxel data of the object from the input image data of the object, converts it into polygon data, adds color data, and causes the display unit 9 to display the data. Note that the function of each unit of the control unit 2 is realized, for example, by executing a predetermined program stored in a memory such as the storage unit 3 by a microprocessor (not shown) or the like (described later).

  In the data input unit 6, an image photographed by the user (an image of the object photographed together with the check pattern board and an image of only the check pattern board) is input as image data. Here, image data obtained from an image of only the check pattern board is used as reference check pattern data. The data input unit 6 stores these image data in the storage unit 3.

  Based on the input image data, the calibration unit 10 calculates calibration data, that is, projection matrix data that associates the image with the check pattern board, and camera parameters (photographing viewpoint data).

  The silhouette extraction unit 11 converts the input reference check pattern data by the projection matrix data obtained by the calibration unit 10, performs a difference process using the converted data and the corresponding image data, Calculate silhouette data.

  The voxel data generation unit 12 first obtains projection data in the voxel space using the projection matrix data obtained by the calibration unit 10. Then, the voxel data generation unit 12 performs intersection determination processing using the projection data of the voxel space and the silhouette data obtained by the silhouette extraction unit 11 to generate voxel data.

  The voxel data correction unit 13 deletes data related to internal voxels from the voxel data obtained by the voxel data generation unit 12. Specifically, the presence of voxels adjacent in the six directions around the voxel is checked, and if there are other voxels in all six directions, the voxel is deleted as being located inside the object.

  The polygon data generation unit 15 generates polygon data as surface information from the voxel data based on the voxel data from the voxel data generation unit 12 and the data obtained by the voxel data correction unit 13.

  The color data generation unit 16 generates color data based on the polygon data obtained by the polygon data generation unit 15 and the camera parameters obtained by the calibration unit 10.

  The color data adding unit 17 associates the color data generated by the color data generating unit 16 with the polygon data generated by the polygon data generating unit 15 and adds the color data to the polygon data. In addition, the display control unit 21 causes the display unit 9 to display the final data obtained by the color data adding unit 17.

  The storage unit 3 stores the image data input from the data input unit 6 or temporarily stores data from each unit in cooperation with each unit of the control unit 2. Each unit of the control unit 2 reads necessary data from the storage unit 3 as needed.

  From the user interface 23, when the calibration unit 10 generates calibration data, points (for example, four points) on the check pattern board 22 that surround the object are input by the user.

  The function of each unit of the control unit 2 supplies a recording medium in which a program code (execution format program, intermediate code program, source program) of a processed document search program is recorded so as to be readable by a computer to a system or apparatus. This can also be achieved by the computer (or CPU, MPU, DSP) of the system or apparatus reading and executing the program code recorded on the recording medium.

  In this case, the program code itself read from the recording medium realizes the above-described function, and the recording medium recording the program code constitutes the present invention.

  Specifically, the control unit 2 is realized by a predetermined program stored in a memory such as the storage unit 3 being executed by a microprocessor (not shown).

  The recording medium for supplying the program code can be configured to be separable from the system or apparatus. The recording medium may be a medium that is fixedly supported so that the program code can be supplied. Even if the recording medium is attached to the system or apparatus so that the recorded program code can be directly read by the computer, the recording medium can be connected via the program reading apparatus connected to the system or apparatus as an external storage device. It may be mounted so that it can be read.

  For example, as the recording medium, a disk including a tape system such as a magnetic tape or a cassette tape, a magnetic disk such as a floppy (registered trademark) disk / hard disk, and an optical disk such as a CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

  The program code may be recorded so that the computer can read out from the recording medium and directly execute it, or after being transferred from the recording medium to the program storage area of the main memory, the computer can read out from the main memory and execute it. It may be recorded as follows.

  Furthermore, the 3D modeling apparatus 1 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. Specifically, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication A network, a satellite communication network, etc. can be used. In addition, the transmission medium constituting the communication network is not particularly limited, and specifically, it is an infrared ray such as IrDA or a remote control even in a wired manner such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line or the like. , Bluetooth (registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like. The present invention can also be realized in the form of a carrier wave or a data signal sequence in which the program code is embodied by electronic transmission.

  The program for reading the program code from the recording medium and storing it in the main memory, and the program for downloading the program code from the communication network are stored in advance in a system or apparatus so as to be executable by a computer. To do.

  The functions of the control unit 2 described above are not only realized by executing the program code read out by the computer, but the OS or the like operating on the computer based on the instruction of the program code performs the actual processing. It is also realized by performing part or all.

  Furthermore, the function described above is obtained by writing the program code read from the recording medium into a memory provided in a function expansion board attached to the computer or a function expansion unit connected to the computer, and then the program code. Based on the instruction, the CPU or the like provided in the function expansion board or function expansion unit also implements part or all of the actual processing.

  Next, processing steps performed by the above-described three-dimensional modeling apparatus 1 will be described in detail along the flowchart of FIG. Here, FIGS. 3A and 3B are images of an object (banana) photographed together with the check pattern board, and FIG. 3C is a reference image obtained by photographing only the check pattern board. .

  First, an image input process (S1 and S2 in FIG. 2) is performed. In this step, the user places an object on the check pattern board 22 and photographs the object together with the check pattern board 22 from an arbitrary viewpoint (S1, see FIGS. 3A and 3B). At this time, the outer periphery of the check pattern board 22 is taken. A single digital camera or the like may be prepared. Further, the photographing viewpoint is arbitrary, and it is not necessary to record from which position and angle the image is taken. It is also possible to take a picture with the check pattern board 22 behind the object.

  Image data obtained by photographing is input to the data input unit 6 of the three-dimensional modeling apparatus 1. Here, a single reference image of only the check pattern board 22 with the object removed is taken (S2, see FIG. 3C), and the reference check pattern data obtained from this reference image is also input to the data input unit 6. deep. Note that the number of viewpoints at the time of shooting is preferably 12 viewpoints or more (shooting 360 degrees around the object at 12 viewpoints every 30 degrees). In addition, the check pattern board 22 may be a board in which a predetermined check pattern is printed on paper with a printer or the like and is attached to an appropriate flat object.

Next, a calibration process is performed in the calibration unit 10 (S3). In this step, camera parameters (photographing viewpoint data) and projection matrix data that associates the coordinate system of the input image with the coordinate system of the check pattern board 22 are calculated. In this step, as shown in FIG. 4, the user designates four vertices (P _{1 to} P ₄ ) of the check pattern board 22 in the image via the user interface 10. Of course, the check pattern board 22 may be provided with marks (colors etc.) that can designate these four points. The upper four vertices (P _{1 to} P ₄ ) are designated so as to surround the object (banana). The specific process of the calibration process is shown below.

  In the present embodiment, the Zhang method is used as the calibration method. Details of this method are disclosed in, for example, “Z. Zhang,“ A Flexible New Technique for Camera Calibration ”, Technical Report MSR-TR-98-71, Microsoft Research, 1998”.

  In this Zhang method, a planar pattern (for example, check pattern board 22) depicting a reference point whose position on a two-dimensional plane is known is used as a reference object, not a solid having a three-dimensional extent. The reference plane or the camera is moved to a plurality of different positions to capture a plane pattern, and the camera parameters are calculated by calculating the homography between the reference plane and the image plane. At this time, the reference plane or the camera can be moved freely, and it is not necessary to know the movement. Thereby, a regular window of a building or a regular pattern output on paper by a printer can be used for calibration, and calibration in various situations can be realized. The principle will be described below.

First, the three-dimensional point M = [X, Y, Z] ^T and the projected point m = [U, V] ^T
Projection formula between

Exists.
Here, the matrix A is an internal parameter matrix and can be written as follows.

Note that α _u and α _v are scale coefficients formed by the product of the image size and the focal length. Further, (u ₀ , v ₀ ) is the image center, and λ is a parameter representing the distortion of the image coordinate axes.

Considering a reference plane with Z = 0 in coordinates in a three-dimensional space, and assuming that the i-th column vector of the rotation matrix R is r _i (i = 1, 2, 3), equation (1) is

It becomes.

  From this, the point on the reference plane and the point on the image plane are

Will be related by the homography H (projection matrix)

It can be expressed as.

The homography H can be calculated if there are four or more corresponding points on the three-dimensional plane and the image plane.
Here, if H = [h ₁ , h ₂ , h ₃ ],

It becomes.

Homography H has 8 degrees of freedom, of which external parameters are degrees of freedom, and therefore can only have two constraints on internal parameters.
Here, since the property of the rotation matrix R and the unit vector in which r ₁ and r ₂ are orthogonal, the following two expressions hold.

  As a result, a constraint equation related to the two internal camera variables is obtained from one image. That is, when obtaining five camera internal variables, at least three images are required.

Here, A ^−r A ⁻¹ has objectivity,

A vector in which the elements of B

It is defined as

The i-th column vector of homography H is
If h _i = [h ₁₁ h ₁₂ h ₁₃ ] ^T , (i = 1, 2, 3),


Is

Can be expressed as

  Thus, equations (6) and (7) can be written as:

If n images are obtained, by stacking the above n equations,
Vb = 0... Equation (12) is obtained.
Here, V is a 2n × 6 matrix.

Thus, b is obtained as an eigenvector corresponding to the minimum eigenvalue of V ^T V.
At this time, if n ≧ 3,
A solution for b can be obtained directly. When n = 2, by setting s = 0 in the internal parameters,

Is added to equation (12) to obtain a solution.

Further, if n = 1, only two internal parameters can be obtained. Therefore, for example, it is possible to obtain a solution by making only α _u and α _v unknown and making the remaining internal parameters known.

  If B is obtained by obtaining b,

Therefore, the internal parameters of the camera are calculated as follows.

  In addition, if A is obtained from this, the external parameter is also obtained from equation (5),

Can be obtained as

  As described above, when all the internal parameters are unknown, calibration can be performed by using three or more images taken with the internal parameters fixed from different viewpoints. At this time, generally, the larger the number of images, the higher the parameter estimation accuracy. It is also shown that the error increases when the rotation between images used for calibration is small.

  Following this calibration process, the silhouette extraction unit 11 performs a silhouette extraction process (S4 in FIG. 2). In the silhouette extraction step, an object region (silhouette) is extracted (silhouette data is calculated). Specific processing is as follows.

  First, each input image is converted to gray scale (256 gradations). Next, data is generated by deforming the reference check pattern data input in the multi-viewpoint image capturing process with the homography matrix H obtained in the calibration process. Next, a difference process is performed between this data and the image data input in the multi-viewpoint image capturing process. Thereby, silhouette data is calculated, and a silhouette image (white portion in the center) as shown in FIG. 5 can be obtained.

  Subsequent to the silhouette extraction step, the voxel data generation unit 12 performs a three-dimensional shape restoration step (S5, voxel data generation step). In this three-dimensional shape restoration step, the silhouette image obtained by the silhouette extraction step is projected onto a virtual voxel space, and shape information is restored by the view volume intersection method. The voxel space is a set of uniform cubic elements, and each cubic element is called a voxel.

  In the view volume intersection method, first, voxels are projected onto a silhouette image using the projection matrix obtained in the calibration process. The principle diagram is shown in FIG. Here, when the voxel is projected onto the object region of the silhouette image, the voxel is left as it is, and when the voxel is projected onto the background region, the deletion parameter of the voxel is increased by one. These processes are performed for all voxels. Further, these processes are performed on each silhouette image. Finally, the voxel whose deletion parameter is larger than a certain value is deleted. The above process will be described more specifically as follows. Here, Pj is a projection matrix obtained from the jth image among n input images, and Sj is a silhouette image.

  First, as shown by the arrow in FIG. 6, one cube is projected onto the silhouette image Sj (see FIG. 6). For the projection, the projection matrix Pj obtained in the calibration process is used. The projected area is a polygon surrounded by at most six vertices.

  Next, an intersection determination process is performed to determine the intersection state between the silhouette projected from the cube and the silhouette of the object. If the silhouette of the cube is completely contained in the silhouette of the object, “BLACK” is set, “WHITE” is set if it is not included at all, and “GRAY” is set if a portion is included.

  The process ends when all cubes become “BLACK” or “WHITE” or reach any level. This algorithm is summarized as shown in FIG.

  Following this silhouette extraction step, the voxel data correction unit 13 performs a voxel deletion step (S6). In this voxel deletion step, voxels located inside the object that are not necessary for display are deleted, leaving only the voxels located on the surface of the object. The specific procedure is as follows.

  First, the presence of voxels adjacent in the six directions around the voxel is checked. If there are other voxels in all six directions, the voxel is deleted as being located inside the object. The other voxels are voxels located on the surface, and a flag (surface flag) indicating the surface is set for the direction in which the voxel does not exist among the six adjacent directions.

  Subsequent to the voxel deletion process, the polygon data generation unit 15 performs a conversion process to polygon data (S7).

  In the conversion from voxel data to polygon data, the polygon located on the surface is known among the six surfaces (polygons) constituting the voxel in the process of deleting the voxel described above. Coordinates are calculated from the center coordinates of the voxel and the size of one side.

  Following the conversion process to polygon data, the color data generation unit 16 performs a color data generation process (S8).

  Here, first, an optimal image for determining the color of Voxel is selected from a plurality of images. Specifically, among the six faces of Voxel, the normal line n is obtained for each external face, and the normal line and the optical axis l of the camera at the time of shooting are

Select an image that satisfies

  Then, Voxel is projected onto each image by the projection matrix obtained by calibration, and color data is determined from the corresponding region.

  Following this color data generation step, the color data addition unit 17 performs a color data addition step (S9).

  In the color data adding step, the color data obtained in the color data generating step is associated with the polygon data, and the color data is added to the polygon data. Since the size of each polygon is the same as the voxel surface, it can be made sufficiently fine depending on the specification at the time of restoration, or it can be made rough enough to recognize the saddle shape.

  Finally, the display control unit 21 displays the data obtained in the color data adding step on the display unit (S10). Polygon data is described as three-dimensional coordinates of extension points, normal vectors, and color data. A dedicated viewer can be used for the display. However, polygon data can be converted into a general data format such as VRML and DXF by a simple data conversion program and displayed.

  As described above, the three-dimensional modeling apparatus 1 includes an image obtained by photographing an object (banana in FIG. 3A) together with the check pattern board and a reference image obtained by photographing only the check pattern board (or the reference check pattern data itself). The three-dimensional modeling of the object is performed only from the above.

  Therefore, it is possible to easily (simplely) convert various objects into three-dimensional data (voxel data) without requiring dedicated photographing equipment and processing devices as in the prior art. Thus, if the present three-dimensional modeling apparatus 1 is used for entertainment, distribution (for example, product images of a WEB shop), medical treatment (for example, three-dimensional data of a living body), etc., it is very useful and convenient.

  In the three-dimensional modeling apparatus 1, the portable check pattern board 22 is used as a geometric pattern, and the calibration unit 10 performs planar calibration. As a result, various objects can be converted into three-dimensional data more easily (simplely).

  In addition, since the three-dimensional modeling apparatus 1 uses a check pattern as a geometric pattern, calibration and silhouette extraction are easy.

  Further, since the three-dimensional modeling apparatus 1 uses voxel data as three-dimensional data, it is easy to generate three-dimensional data.

  In addition, since the three-dimensional modeling apparatus 1 includes the user interface 23 that can input a plurality of points surrounding the object with respect to the image to be processed by the calibration unit, the calibration process is efficiently performed. It can be carried out.

  In the three-dimensional modeling apparatus 1, since the silhouette extraction unit 11 calculates silhouette data by performing a difference process between the conversion data of the reference geometric pattern data based on the projection matrix data and the corresponding image data, a silhouette extraction process is performed. Can be performed efficiently.

  Further, in the three-dimensional modeling apparatus 1, the voxel data generation unit 12 generates voxel data by the intersection determination process between the projection data in the voxel space based on the projection matrix data and the silhouette data, so that the voxel data generation process is efficient. Can be done automatically.

  Further, since the three-dimensional modeling apparatus 1 includes the voxel data correction unit 13 and can reduce unnecessary voxel data, data processing can be performed efficiently.

  Further, since the three-dimensional modeling apparatus 1 includes the polygon data generation unit 15 and can process only the surface information (data) of the object, data processing can be performed efficiently.

  The three-dimensional modeling apparatus 1 includes color data generation means for generating color data associated with the polygon data using the image data, calibration data, and polygon data, and can express the object in color. The ability to express the object is high. The color data generation unit 16 selects an image suitable for color generation from the image using the camera parameters and polygon data, and uses the projection matrix data for the image data of the selected image and the polygon data. Since the color data is generated by associating with the projection-processed data, the color data generation process can be performed efficiently.

  It should be noted that the present invention is not limited to the above-described embodiment, and various modifications are possible within the scope shown in the claims, and an implementation obtained by appropriately combining technical means disclosed in the embodiment. The form is also included in the technical scope of the present invention.

  The three-dimensional modeling apparatus of the present invention can easily perform three-dimensional modeling from an image obtained from a digital camera, a mobile phone camera, or the like. Therefore, it can be used in various fields such as entertainment, distribution (for example, product samples at WEB shops), medical care (modeling of living bodies), architecture (modeling of structures), apparel or incidents, accidents, etc. .

It is a block diagram which shows the structure of the three-dimensional modeling apparatus in this Embodiment. It is a flowchart which shows the process process in the said three-dimensional modeling apparatus. FIGS. 3A and 3B are images of an object (banana) photographed together with the check pattern board, and FIG. 3C is a reference image obtained by photographing only the check pattern board. It is a figure which shows each point input into an image from a user interface at a calibration process. It is a figure which shows the silhouette image obtained in a silhouette extraction part. It is a figure which shows the principle of the voxel projection in a voxel data generation part. It is a figure which shows the algorithm of the intersection determination process in a voxel data generation part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 3D modeling apparatus 2 Control part 3 Memory | storage part 6 Data input part 10 Calibration part (calibration means)
11 Silhouette extraction unit (silhouette extraction means)
12 voxel data generation unit (three-dimensional data generation means, voxel data generation means)
13 Voxel data correction unit (Voxel data correction means)
15 Polygon data generator (polygon data generator)
16 color data generation unit (color data generation means)
17 Color data adding unit (color data adding means)
21 Display control unit 22 Check pattern board (geometric pattern)
23 User interface (input unit)

Claims (15)

Calibration means for calculating calibration data from image data obtained from each image in which the object is photographed together with a geometric pattern;
Silhouette extraction means for calculating silhouette data of the object using reference geometric pattern data obtained from a reference image in which only a geometric pattern is photographed, and the image data and the calibration data corresponding thereto,
A three-dimensional modeling apparatus comprising: three-dimensional data generating means for generating three-dimensional data using silhouette data and calibration data corresponding to each of a plurality of images.   2. The three-dimensional modeling apparatus according to claim 1, wherein the geometric pattern is a planar geometric pattern, and the calibration means calculates calibration data by performing planar calibration.   3. The three-dimensional modeling apparatus according to claim 2, wherein the geometric pattern is a check pattern.   3. The three-dimensional modeling apparatus according to claim 2, wherein the three-dimensional data generating means is voxel data generating means for generating voxel data.   5. The three-dimensional modeling apparatus according to claim 4, further comprising an input unit capable of inputting a plurality of points surrounding the object with respect to each image to be processed by the calibration means. The calibration data includes projection matrix data that associates each image and geometric pattern, and photographing viewpoint data.
5. The three-dimensional modeling apparatus according to claim 4, wherein the silhouette extraction means calculates silhouette data by a difference process between conversion data of reference geometric pattern data based on the projection matrix data and the corresponding image data. The calibration data includes projection matrix data that associates each image and geometric pattern, and photographing viewpoint data.
5. The three-dimensional modeling apparatus according to claim 4, wherein the voxel data generation means generates voxel data by an intersection determination process between projection data in the voxel space based on the projection matrix data and the silhouette data.   8. The three-dimensional modeling apparatus according to claim 7, further comprising voxel data correcting means for deleting data related to internal voxels from the voxel data.   8. The three-dimensional modeling apparatus according to claim 7, further comprising polygon data generating means for generating polygon data as surface information from the voxel data.   8. The three-dimensional modeling apparatus according to claim 7, further comprising color data generation means for generating color data associated with the polygon data using the image data, calibration data, and polygon data.   The color data generation means selects an image suitable for color generation from the image using the photographing viewpoint data and polygon data, and uses the projection matrix data for the image data of the selected image and the polygon data. The three-dimensional modeling apparatus according to claim 10, wherein the color data is generated by associating with the projection-processed data.   The geometric pattern used when imaging | photography the input image of the three-dimensional modeling apparatus of any one of Claims 1-3. A calibration process for calculating calibration data from image data obtained from each image in which the object is photographed with a planar geometric pattern;
A silhouette extraction step of calculating silhouette data of the object using reference geometric pattern data obtained from a reference image in which only a geometric pattern is photographed, and the image data and the calibration data corresponding thereto;
A method for generating three-dimensional modeling data, comprising: a voxel data generation step for generating voxel data based on silhouette data and calibration data corresponding to each of a plurality of images.   A three-dimensional modeling program for causing a computer to realize each means according to claim 1.   15. A recording medium in which the three-dimensional modeling program according to claim 14 is stored in a computer readable manner.

Images