JP6867645B2 - Image processing equipment, methods, and programs - Google Patents

Description

The present invention relates to image processing devices, methods, and programs.

Conventionally, a technique for restoring the three-dimensional shape of a subject by using a plurality of images (multi-viewpoint images) taken from two or more different viewpoints by an image capturing device such as a camera has been proposed (for example, Non-Patent Document 1).

In addition, from the position of each corresponding point on the multi-viewpoint image and geometric information such as the position, orientation, and angle of view of the image capturing device used for shooting the multi-viewpoint image, the corresponding corresponding point can be obtained by using the principle of triangulation. A technique for calculating position information in a three-dimensional space has been proposed (for example, Non-Patent Document 2). In this technique, the positions of the points constituting the surface of the subject are obtained by applying the process of calculating the position information of the corresponding points to all the detected corresponding points. Further, by determining the combination of the three points from which the position information is obtained, the surface (mesh) constituting the surface of the subject can be obtained. Also, when the combination of all points is determined, a three-dimensional mesh model is generated. Then, the image of the area corresponding to each mesh is extracted from the multi-viewpoint image to obtain a texture image, the texture image is pasted on all the meshes, and then the shadow surface processing is performed according to the viewpoint of the image to be reproduced. As a result, an observation image at an arbitrary viewpoint is generated.

Yamao, Sakai, Ito, Aoki, "Study on restoration of high-precision and high-density 3D point cloud from moving images", IPSJ Research Report, 2013-CVIM-187 (17), pp.1-8, 2013. Yaguchi, Kimura, Saito, Kanade, "Arbitrary viewpoint image generation using uncalibrated multi-view camera system", IPSJ Journal of Computer Vision and Image Media (CVIM) Vol.42, No, SIG 6 (CVIM). pp.9-21, 2001.

If an appropriate captured image cannot be selected when extracting the texture image to be pasted on each mesh from the captured image (multi-viewpoint image), the color, brightness, and space of the texture image may be affected by shading and geometric constraints. In some cases, quality such as resolution was insufficient. Then, in this case, the fidelity of the image from an arbitrary viewpoint that is finally output to the real thing also decreases.

The present invention has been made in view of the above points, and an object of the present invention is to provide an image processing device, a method, and a program capable of generating an appropriate texture image for a three-dimensional mesh model.

In order to achieve the above object, the image processing apparatus according to the present invention includes an image acquisition unit that acquires a plurality of images obtained by photographing a subject from a plurality of different viewpoints by the image capturing apparatus, and the subject using the plurality of images. A mesh model generation unit that generates a three-dimensional mesh model in which the surface of the image is represented by a mesh, and the image capturing device and illumination at the time of photographing the mesh and the plurality of images for each mesh of the three-dimensional mesh model. The weight coefficient calculation unit that calculates the weight coefficient of each of the plurality of images according to the positional relationship and the orientation with at least one of the light sources, and the texture image corresponding to each mesh of the three-dimensional mesh model are the plurality of images. In the case of generating from, the mesh is configured to include a texture image generation unit that generates the texture image by weighting the mesh according to the weighting coefficient of each of the plurality of images.

The image processing method according to the present invention is a three-dimensional mesh model in which a plurality of images obtained by photographing a subject from a plurality of different viewpoints are acquired by an image capturing apparatus, and the surface of the subject is represented by a mesh using the plurality of images. For each mesh of the three-dimensional mesh model, the plurality of images are generated according to the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. When each weight coefficient of the image is calculated and a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, the weight coefficient of each of the plurality of images with respect to the mesh is corresponding to the weight coefficient. Weighting is performed to generate the texture image.

Further, the program of the present invention is a program for making a computer function as each part of the above-mentioned image processing apparatus.

As described above, according to the image processing apparatus, method, and program of the present invention, the weighting coefficient according to the positional relationship and orientation between each mesh of the three-dimensional mesh model and at least one of the image capturing apparatus and the illumination light source is determined. By calculating and weighting according to the weighting coefficient to generate a texture image, an effect that an appropriate texture image can be generated for the three-dimensional mesh model can be obtained.

It is a block diagram of the image processing system which concerns on embodiment. It is a figure which shows an example of the method of photographing a subject from a plurality of different viewpoints. It is a figure which shows an example of the method of photographing a subject from a plurality of different viewpoints. It is a figure which shows an example of the method of photographing a subject from a plurality of different viewpoints. It is a figure for demonstrating the process of calculating the position and the depth of the corresponding point which concerns on embodiment. It is a figure for demonstrating the mesh which concerns on embodiment. It is a figure for demonstrating the process of calculating the mesh geometric information which concerns on embodiment. It is a figure for demonstrating an example of the process of calculating the weighting coefficient which concerns on embodiment. It is a figure for demonstrating an example of the process of calculating the weighting coefficient which concerns on embodiment. It is a figure for demonstrating the process which generates the output image which concerns on embodiment. It is a flowchart which shows the processing routine of the image processing of the image processing apparatus which concerns on embodiment. It is a block diagram of the image processing system which concerns on the modification.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

<Configuration of image processing system>
The configuration of the image processing system according to the present embodiment will be described. As shown in FIG. 1, the image processing system 10 according to the present embodiment includes a plurality of cameras 201 to 20N as an example of an image capturing device, and an image processing device 30.

The cameras 201 to 20N are arranged three-dimensionally (for example, in the horizontal direction) so that the subject can be photographed from a plurality of different viewpoints. N is an integer of 2 or more. For example, as shown in FIG. 2, three cameras are arranged side by side in the horizontal direction.

If the subject can be photographed from a plurality of different viewpoints, one camera may be used. For example, as shown in FIG. 3, one camera may be fixed and the subject may be photographed a plurality of times while rotating the subject in one direction (horizontal direction in the example of FIG. 3). Further, for example, as shown in FIG. 4, one camera may be moved in one direction (horizontal direction in the example of FIG. 4) to shoot a subject at a plurality of different positions.

The image processing device 30 according to the present embodiment can be configured by a computer including a CPU, a RAM, and a ROM that stores a program for executing each processing routine described later and various data. Functionally, as shown in FIG. 1, the image processing device 30 includes an image acquisition unit 31, a shooting geometric information storage unit 32, a three-dimensional shape estimation unit 33, a mesh model generation unit 34, a mesh geometric information calculation unit 35, and a weighting coefficient. It includes a calculation unit 36, a texture image generation unit 37, an output image generation unit 38, and a color reproduction processing unit 39.

The image acquisition unit 31 captures a plurality of captured images (hereinafter, simply "captured images") obtained by photographing a subject whose three-dimensional shape is to be estimated by the cameras 201 to 20N while being irradiated with the illumination light from the illumination light source. To get). In addition, the image acquisition unit 31 acquires a plurality of images obtained by photographing a subject whose three-dimensional shape and texture pattern are known in advance, such as a checker pattern, with cameras 201 to 20N. Although the details will be described later, since the image obtained by taking a picture in advance is used for calculating the taking geometric information, it will be referred to as "an image for calculating taking geometric information" below. If the shape of a part of the subject whose three-dimensional shape is to be estimated is known, the captured image may also serve as an image for calculating the captured geometric information.

The shooting geometric information storage unit 32 is based on each of the images for calculating the shooting geometric information acquired by the image acquisition unit 31, the positional relationship between the cameras 201 to 20N, the illumination light source, and the subject, and the optical parameters of the cameras 201 to 20N. (Number of pixels, image angle, focal length, etc.) is calculated. When the image for calculating the shooting geometry information is an image of a subject having a checker pattern, the length of each rectangle of the checker pattern is known, so that it is based on each rectangle in the image and the length of the known side. , The positional relationship between the cameras 201 to 20N and the subject is calculated. Further, when the image for calculating the shooting geometric information is an image obtained by shooting a subject whose three-dimensional shape is known, it is based on the three-dimensional shape of the subject, the brightness value of the subject in the image, and the positional relationship between the cameras 201 to 20N and the subject. Therefore, the positional relationship between the illumination light source and the cameras 201 to 20N is calculated. Then, the photographing geometric information storage unit 32 captures the photographed geometric information calculation image using the calculated positional relationship between the cameras 201 to 20N, the illumination light source, and the subject, and the optical parameters of the cameras 2001 to 20N. It is output to the memory as shooting geometric information data in association with the shot image shot by 2001 to 20N. The photographing geometric information storage unit 32 may output the photographing geometric information data as an electronic file to the non-volatile storage unit. Further, the captured geometric information data may be stored in the storage unit in advance and read out when the captured image is captured.

As shown in FIG. 5, the three-dimensional shape estimation unit 33 tracks the corresponding points between each pair of the plurality of captured images, and uses the captured geometric information data to determine the position of the point on the surface of the subject and the position on the surface of the subject according to the principle of triangulation. Calculate the depth. The three-dimensional shape estimation unit 33 generates three-dimensional point cloud data by performing the process of calculating the position and the depth for all the detected corresponding points. In the present embodiment, the three-dimensional shape estimation unit 33 calculates the feature amount in the image when detecting the corresponding point between the images. In the calculation of the feature amount, the amplitude information and the phase information of the image are calculated. Use at least one of.

As shown in FIG. 6, one surface (triangular mesh) can be defined by selecting three points from the three-dimensional point cloud data generated by the three-dimensional shape estimation unit 33. The mesh model generation unit 34 deletes and integrates points so that the surface of the subject can be expressed correctly, and calculates the optimum combination of three points to generate a three-dimensional point cloud generated by the three-dimensional shape estimation unit 33. From the data, a three-dimensional mesh model that represents the surface of the subject with a mesh is generated.

As shown in FIG. 7, once the three points constituting each mesh are determined, the normal of the mesh can be calculated based on the positions and depths of the three points. The mesh geometry information calculation unit 35 calculates the positions and orientations of the cameras 201 to 20N when each photographed image is photographed with respect to the mesh, using the position of the mesh, the direction of the normal line, and the photographed geometry information data. Further, the mesh geometric information calculation unit 35 calculates the positional relationship between the mesh and the illumination light source and the incident angle of the illumination light from the illumination light source by using the position of the mesh, the direction of the normal line, and the photographing geometric information data. The angle of incidence is the angle formed by the normal of the mesh and the optical axis of the illumination light from the illumination light source. Further, in the following, the position and orientation of the camera when each captured image is taken with respect to the mesh calculated by the mesh geometric information calculation unit 35, the positional relationship between the mesh and the illumination light source, and the incident angle of the illumination light from the illumination light source. , Is called "mesh geometric information data".

The weight coefficient calculation unit 36 calculates the weight coefficient of each of the captured images including the image information corresponding to the mesh based on the mesh geometric information data of the mesh for each mesh of the three-dimensional mesh model.

As shown in FIG. 8, in the captured image in which the normal direction of the mesh and the optical axis (line-of-sight direction) of the camera coincide with each other, the spatial resolution in the mesh region (region used as the texture) is maximized. Therefore, when the weight resolution of the texture is given the highest priority, the weighting coefficient calculation unit 36 sets the weighting coefficient of the captured image in which the normal direction of the mesh and the optical axis of the camera match to 1, and the weight of the captured image that does not match. Let the coefficient be 0. If there is no captured image in which the normal direction of the mesh and the optical axis of the camera match, the weighting coefficient calculation unit 36 calculates the weighting coefficient of the captured image in which the normal direction of the mesh and the optical axis of the camera are closest to each other. It may be set to 1 and the weighting coefficient of the other captured images may be set to 0. Further, for example, the weighting coefficient calculation unit 36 sets the weighting coefficient of the captured image in which the angle formed by the normal direction of the mesh and the optical axis of the camera is within a predetermined range to 1, and sets the normal direction of the mesh and the optical axis of the camera to 1. The weighting coefficient of the captured image whose angle formed with and is outside the predetermined range may be set to 0. Further, the weighting coefficient calculation unit 36 may calculate a larger value as the weighting coefficient as the angle formed by the normal direction of the mesh and the optical axis of the camera is smaller. Examples of the weighting coefficient in this case include the cosine value of the angle formed by the normal direction of the mesh and the optical axis of the camera.

In addition, when the specular reflection direction of the illumination light from the illumination light source in the mesh and the optical axis of the camera coincide with or are relatively close to each other, the region in the captured image corresponding to the mesh includes the specular reflection of the illumination light. There is a high possibility that it will be. Therefore, when the weight coefficient calculation unit 36 gives the highest priority to the color information of the texture, the angle formed by the specular reflection direction of the illumination light and the optical axis of the camera in order to remove the influence of the specular reflection of the illumination light. The weighting coefficient of the captured image in which is within the predetermined range is set to 0, and the weighting coefficient of the captured image in which the angle formed by the specular reflection direction of the illumination light and the optical axis of the camera is outside the predetermined range is set to 1. The weighting coefficient calculation unit 36 may calculate a larger value as the weighting coefficient as the angle formed by the specular reflection direction of the illumination light and the optical axis of the camera increases. Examples of the weighting coefficient in this case include a sine value of an angle formed by the normal direction of the mesh and the optical axis of the camera. Further, as shown in FIG. 9, in order to suppress the influence of the incident angle of the illumination light, the cosine value of the incident angle of the illumination light may be used as a weighting coefficient of the captured image. In this case, the weighting coefficient is a value of 0 or more and 1 or less. If the incident angle of the illumination light is smaller, the larger the value, the weighting coefficient is not limited to the cosine value of the incident angle of the illumination light. For example, the nth power (exponential power) of the cosine value of the incident angle of the illumination light is used. It may be a weighting coefficient. Further, based on Euler's formula, the exponential power of the complex exponential (exp (ai) = cos a + i sin a) may be used as the weighting coefficient instead of the cosine function.

Further, the weighting coefficient calculation unit 36 may calculate the weighting coefficient of the captured image by combining the above-mentioned three methods when balancing the spatial resolution and the color information. For example, the weighting coefficient calculation unit 36 sets the weighting coefficient of the captured image in which the angle formed by the normal direction of the mesh and the optical axis of the camera is within a predetermined range to 1, and calculates this weighting coefficient from the incident angle of the illumination light. The result obtained by multiplying the weighting coefficients to be obtained may be used as the final weighting coefficient of the captured image. Further, for example, the result obtained by multiplying the weighting coefficients calculated by each of the three methods may be used as the final weighting coefficient of the captured image.

In many cases, one shot image contains an area corresponding to a plurality of meshes, so that one shot image has a format in which weighting coefficients for each of the plurality of meshes are associated with the meshes. Become.

Further, for example, the user can specify which of the spatial resolution and the color information is prioritized, or whether the spatial resolution and the color information are balanced.

The texture image generation unit 37 extracts an image of a region corresponding to each mesh from the captured image, and generates a texture image corresponding to each mesh. At this time, the texture image generation unit 37 multiplies the image extracted from each captured image by the weighting coefficient associated with the mesh of each captured image for each mesh, and adds the result of the multiplication. A texture image corresponding to the mesh is generated. The texture image generation unit 37 may exclude the captured image having a weighting coefficient of 0 from the processing in advance.

As shown in FIG. 10, the output image generation unit 38 is an image of the subject viewed from a viewpoint at a position designated by the user or the like, and is an image for output to an image output device such as a display device (hereinafter referred to as an image). Generate an "output image"). Specifically, the output image generation unit 38 attaches the texture image generated by the texture image generation unit 37 to each mesh of the three-dimensional mesh model based on the three-dimensional mesh model and the mesh geometric information data. Further, the output image generation unit 38 determines the positional relationship between each mesh and the viewpoint, and the illumination for each mesh, based on the designated viewpoint, the position and direction of the illumination light source, the three-dimensional mesh model, and the mesh geometric information data. Calculate the incident angle of light. Then, the output image generation unit 38 represents the subject observed from the designated viewpoint by performing the light ray tracing process and the shadow processing after performing the shadow surface processing for removing the mesh that cannot be observed from the designated viewpoint. Generates an output image that corresponds to the image. When the captured image is a multi-band image, the output image generated by the output image generation unit 38 may also be a multi-band image.

The color reproduction processing unit 39 calculates the color corresponding to each pixel of the output image output by the output image generation unit 38 based on the input color reproduction parameter, and outputs the calculated color of each pixel to the image output device. To do. Examples of color reproduction parameters include optical characteristics such as camera sensitivity of cameras 201 to 20N, spectral information on illumination light at the time of image capture, spectral information on illumination light in the environment to be reproduced, and characteristic data of an image output device. And so on.

For example, when the captured image acquired by the image acquisition unit 31 is a color image, that is, when the output image output by the output image generation unit 38 is a color image, the color reproduction processing unit 39 of the image output device Performs image correction processing (color conversion, gamma correction, etc.) according to the characteristics.

When the output image output by the output image generation unit 38 is a multi-band image, the color reproduction processing unit 39 uses the color reproduction parameters to estimate the spectral reflectance, calculate the reflected light spectrum, and obtain a color image ( Conversion to RGB images, Lab images, etc.) and correction processing according to the characteristics of the image output device are performed in sequence. In this case, the color reproduction processing unit 39 may directly calculate the color image from the multi-band image without estimating the spectral reflectance and calculating the reflected light spectrum. Further, the color reproduction processing by the color reproduction processing unit 39 may be performed at the time of generating the texture image by the texture image generation unit 37, or may be performed on the texture image used by the output image generation unit 38 when generating the output image. You may.

<Operation of image processing system>
Next, the operation of the image processing system 10 according to the present embodiment will be described. When a plurality of captured images obtained by photographing a subject whose three-dimensional shape is to be estimated by a plurality of cameras 2001 to 20N are input to the image processing device 30, the image processing device 30 executes the image processing routine shown in FIG. Will be done. Here, a case where the image for calculating the shooting geometric information is shot in advance and stored in each of the cameras 201 to 20N will be described.

In step S10, the image acquisition unit 31 acquires a plurality of captured images obtained by photographing with the cameras 201 to 20N and a plurality of captured geometric information calculation images from the cameras 201 to 20N. Next, in step S12, as described above, the shooting geometric information storage unit 32 positions the cameras 201 to 20N, the illumination light source, and the subject based on each of the shooting geometric information calculation images acquired in step S10. The relationship and the optical parameters of the cameras 201-20N are calculated. Then, the photographing geometric information storage unit 32 captures the photographed geometric information calculation image using the calculated positional relationship between the cameras 201 to 20N, the illumination light source, and the subject, and the optical parameters of the cameras 2001 to 20N. It is output to the memory as shooting geometric information data in association with the shot image shot by 2001 to 20N.

Next, in step S14, the three-dimensional shape estimation unit 33 tracks the corresponding points between each pair of the plurality of captured images acquired in step S10, and uses the captured geometric information data corresponding to the captured images of each pair. Then, the position and depth of the point on the surface of the subject are calculated by the principle of triangulation. The three-dimensional shape estimation unit 33 generates three-dimensional point cloud data by performing the process of calculating the position and the depth for all the detected corresponding points.

Next, in step S16, the mesh model generation unit 34 was generated in step S14 by deleting and integrating the points and calculating the optimum combination of the three points so that the surface of the subject can be expressed correctly. From the 3D point cloud data, a 3D mesh model that represents the surface of the subject with a mesh is generated.

Next, in step S18, the mesh geometric information calculation unit 35 sets the mesh position, the direction of the normal, and the photographed geometric information data calculated in step S12 for each mesh of the three-dimensional mesh model generated in step S16. Is used to calculate the mesh geometric information data. Next, in step S20, as described above, the weighting coefficient calculation unit 36 is based on the mesh geometric information data of the mesh calculated in step S18 for each mesh of the three-dimensional mesh model generated in step S16. Then, the weighting coefficient of each of the captured images including the image information corresponding to the mesh is calculated.

Next, in step S22, the texture image generation unit 37 extracts an image of a region corresponding to each mesh from the captured image, and generates a texture image corresponding to each mesh. At this time, the texture image generation unit 37 multiplies the image extracted from each captured image by the weighting coefficient associated with the mesh of each captured image for each mesh, and adds the result of the multiplication. A texture image corresponding to the mesh is generated.

Next, in step S24, the output image generation unit 38 attaches the texture image generated in step S22 to each mesh of the three-dimensional mesh model based on the three-dimensional mesh model and the mesh geometric information data. Further, the output image generation unit 38 determines the positional relationship between each mesh and the viewpoint, and the illumination for each mesh, based on the designated viewpoint, the position and direction of the illumination light source, the three-dimensional mesh model, and the mesh geometric information data. Calculate the incident angle of light. Then, the output image generation unit 38 represents the subject observed from the designated viewpoint by performing the light ray tracing process and the shadow processing after performing the shadow surface processing for removing the mesh that cannot be observed from the designated viewpoint. Generates an output image that corresponds to the image.

Next, in step S26, as described above, the color reproduction processing unit 39 calculates the color corresponding to each pixel of the output image output by the output image generation unit 38 based on the input color reproduction parameter. The calculated color of each pixel is output to the image output device. When the processing of step S26 is completed, the image processing routine ends.

As described above, according to the image processing apparatus according to the present embodiment, a weighting coefficient is calculated according to the positional relationship and orientation between each mesh of the three-dimensional mesh model and the image capturing apparatus and the illumination light source, and the weighting coefficient is calculated. By generating a texture image by weighting according to the above, it is possible to generate an appropriate texture image for the three-dimensional mesh model.

The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the gist of the present invention.

For example, in the above embodiment, the case where each functional unit of the image processing device 30 is realized by executing a program has been described as an example, but the present invention is not limited to this. For example, each functional unit of the image processing device 30 may be realized by hardware such as FPGA (Field-Programmable Gate Array), or may be realized by a combination of hardware and software.

Further, in the above embodiment, the weight coefficient calculation unit 36 has the angle formed by the normal direction of the mesh and the optical axis of the camera at the time of taking an image, the normal direction of the mesh, and the output image generation unit 38 outputs an image. The weighting coefficient may be calculated according to the difference between the angle formed by the line-of-sight direction from the designated viewpoint used in generating the image (modification example 1). In this case, the smaller the difference is, the larger the weighting coefficient is calculated.

Further, in the above embodiment, the weight coefficient calculation unit 36 calculates the angle formed by the specular reflection direction of the illumination light at the time of shooting the image on the mesh and the optical axis of the camera, and the output image generation unit 38 on the mesh outputs the output image. The weighting coefficient may be calculated according to the difference between the specular reflection direction of the illumination light from the specified illumination light source used at the time of generation and the line-of-sight direction from the specified viewpoint (modification example). 2). In this case, the smaller the difference is, the larger the weighting coefficient is calculated.

Further, in the above embodiment, the weight coefficient calculation unit 36 generates an output image by the incident angle of the illumination light at the time of taking an image with respect to the normal direction of the mesh and the output image generation unit 38 with respect to the normal direction of the mesh. The weighting coefficient may be calculated according to the incident angle of the illumination light from the designated illumination light source used in the case and the difference (modification example 3). In this case, the smaller the difference is, the larger the weighting coefficient is calculated.

In these three variants, images taken from a plurality of different viewpoints are obtained by being taken in a situation closer to the output image (that is, the image to be reproduced) generated according to the specified viewpoint and light source. Since the weighting coefficient of the image is large, it is possible to generate an appropriate texture image for the three-dimensional mesh model. Further, in these three modified examples, as shown in FIG. 12, the reproduction environment geometric information data such as the position and orientation of the designated viewpoint and the illumination light source is input to the weighting coefficient calculation unit 36. Further, the weighting coefficient of the captured image may be calculated by combining two or more of the weighting coefficient calculation method of these three modified examples and the three weighting coefficient calculation method described in the above embodiment. In this case, a form is exemplified in which the result obtained by multiplying the weighting coefficient calculated by each of the combination calculation methods is used as the final weighting coefficient of the captured image.

Further, in the specification of the present application, the program has been described as an embodiment in which the program is pre-installed, but the program can be stored in a computer-readable recording medium and provided, or provided via a network. It is also possible to do.

10 Image processing system 201-20N Camera 30 Image processing device 31 Image acquisition unit 32 Imaging geometric information storage unit 33 Three-dimensional shape estimation unit 34 Mesh model generation unit 35 Mesh geometric information calculation unit 36 Weight coefficient calculation unit 37 Texture image generation unit 38 Output Image generation unit 39 Color reproduction processing unit

Claims (13)

An image acquisition unit that acquires a plurality of images of a subject taken from a plurality of different viewpoints by an image capturing device, and an image acquisition unit.
A mesh model generation unit that generates a three-dimensional mesh model in which the surface of the subject is represented by a mesh using the plurality of images, and a mesh model generation unit.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Weight coefficient calculation unit that calculates the weight coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. A texture image generation unit that generates the texture image by adding an image obtained by multiplying the image extracted from the mesh by the weight coefficient of each of the plurality of images with respect to the mesh.
Only including,
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
The weight coefficient calculation unit determines the weight according to the angle formed by the specular reflection direction of the illumination light from the illumination light source at the time of photographing the image on the mesh and the optical axis of the image capturing apparatus at the time of photographing the image. An image processing device that calculates a coefficient. An image acquisition unit that acquires a plurality of images of a subject taken from a plurality of different viewpoints by an image capturing device, and an image acquisition unit.
A mesh model generation unit that generates a three-dimensional mesh model in which the surface of the subject is represented by a mesh using the plurality of images, and a mesh model generation unit.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Weight coefficient calculation unit that calculates the weight coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. A texture image generation unit that generates the texture image by adding an image obtained by multiplying the image extracted from the mesh by the weight coefficient of each of the plurality of images with respect to the mesh.
Only including,
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
The weighting coefficient calculation unit is an image processing device that calculates the weighting coefficient according to the incident angle of the illumination light from the illumination light source at the time of shooting the image with respect to the normal direction of the mesh. An image acquisition unit that acquires a plurality of images of a subject taken from a plurality of different viewpoints by an image capturing device, and an image acquisition unit.
A mesh model generation unit that generates a three-dimensional mesh model in which the surface of the subject is represented by a mesh using the plurality of images, and a mesh model generation unit.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Weight coefficient calculation unit that calculates the weight coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. A texture image generation unit that generates the texture image by adding an image obtained by multiplying the image extracted from the mesh by the weight coefficient of each of the plurality of images with respect to the mesh.
Only including,
The weight coefficient calculation unit uses the positional relationship and orientation between the mesh and the designated viewpoint used when generating an image viewed from the designated viewpoint, and uses the normal direction of the mesh and the image. The weighting coefficient is calculated according to the difference between the angle formed by the optical axis of the image capturing apparatus at the time of photographing and the angle formed by the normal direction of the mesh and the line-of-sight direction from the designated viewpoint. Image processing device. An image acquisition unit that acquires a plurality of images of a subject taken from a plurality of different viewpoints by an image capturing device, and an image acquisition unit.
A mesh model generation unit that generates a three-dimensional mesh model in which the surface of the subject is represented by a mesh using the plurality of images, and a mesh model generation unit.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Weight coefficient calculation unit that calculates the weight coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. A texture image generation unit that generates the texture image by adding an image obtained by multiplying the image extracted from the mesh by the weight coefficient of each of the plurality of images with respect to the mesh.
Only including,
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
The weight coefficient calculation unit includes the mesh, the designated viewpoint, and the designated viewpoint used when generating an image viewed from a designated viewpoint when the illumination light is irradiated from the designated illumination light source. Using the positional relationship and orientation with the illumination light source, the angle formed by the normal reflection direction of the illumination light from the illumination light source at the time of photographing the image in the mesh and the optical axis of the image capturing apparatus at the time of photographing the image. An image processing device that calculates the weighting coefficient according to the difference between the normal reflection direction of the illumination light from the designated illumination light source in the mesh and the line-of-sight direction from the designated viewpoint. An image acquisition unit that acquires a plurality of images of a subject taken from a plurality of different viewpoints by an image capturing device, and an image acquisition unit.
A mesh model generation unit that generates a three-dimensional mesh model in which the surface of the subject is represented by a mesh using the plurality of images, and a mesh model generation unit.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Weight coefficient calculation unit that calculates the weight coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. A texture image generation unit that generates the texture image by adding an image obtained by multiplying the image extracted from the mesh by the weight coefficient of each of the plurality of images with respect to the mesh.
Only including,
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
The weight coefficient calculation unit has a positional relationship between the mesh and the designated illumination light source, which is used when generating an image viewed from a designated viewpoint when the illumination light is irradiated from the designated illumination light source. And the orientation, the incident angle of the illumination light from the illumination light source at the time of taking the image with respect to the normal direction of the mesh and the incident angle of the illumination light from the designated illumination light source with respect to the normal direction of the mesh. An image processing device that calculates the weighting coefficient according to the difference between. An image acquisition unit that acquires a plurality of images of a subject taken from a plurality of different viewpoints by an image capturing device, and an image acquisition unit.
A mesh model generation unit that generates a three-dimensional mesh model in which the surface of the subject is represented by a mesh using the plurality of images, and a mesh model generation unit.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Weight coefficient calculation unit that calculates the weight coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. A texture image generation unit that generates the texture image by adding an image obtained by multiplying the image extracted from the mesh by the weight coefficient of each of the plurality of images with respect to the mesh.
Only including,
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
The weight coefficient calculation unit calculates a first weight coefficient according to an angle formed by the normal direction of the mesh and the optical axis of the image capturing apparatus at the time of capturing the image, and captures the image on the mesh. A second weighting coefficient is calculated according to the angle formed by the normal reflection direction of the illumination light from the illumination light source at the time and the optical axis of the image capturing apparatus at the time of photographing the image, and the second weighting coefficient is calculated with respect to the normal direction of the mesh. The mesh and the designation used when generating an image viewed from a designated viewpoint by calculating a third weighting coefficient according to the incident angle of the illumination light from the illumination light source at the time of taking the image. Using the positional relationship and orientation with the viewpoint, the angle formed by the normal direction of the mesh and the optical axis of the image capturing device at the time of shooting the image, the normal direction of the mesh, and the designated viewpoint. When the fourth weighting coefficient is calculated according to the difference between the angle formed by the line-of-sight direction and the image viewed from the specified viewpoint when the illumination light is emitted from the specified illumination light source. Using the positional relationship and orientation of the mesh, the designated viewpoint, and the designated illumination light source, the direction of normal reflection of the illumination light from the illumination light source at the time of taking the image on the mesh and the said. The angle formed by the optical axis of the image capturing apparatus at the time of taking an image, the angle formed by the normal reflection direction of the illumination light from the designated illumination light source in the mesh, and the line-of-sight direction from the designated viewpoint. A fifth weighting coefficient is calculated according to the difference between, and the mesh and the designation used when generating an image viewed from a designated viewpoint when the illumination light is irradiated from the designated illumination light source. Using the positional relationship and orientation with the illuminated light source, the incident angle of the illumination light from the illumination light source at the time of taking the image with respect to the normal direction of the mesh and the specified illumination with respect to the normal direction of the mesh. After calculating the sixth weighting coefficient according to the difference between the incident angle of the illumination light from the light source, the mesh is based on two or more of the first to sixth weighting coefficients. An image processing device that calculates the final weighting coefficient of the image. Acquire multiple images of the subject from multiple different viewpoints with an image capturing device,
Using the plurality of images, a three-dimensional mesh model in which the surface of the subject is represented by a mesh is generated.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Calculate the weighting coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. An image processing method for generating the texture image by adding an image obtained by multiplying the image extracted from the image by the weight coefficient of each of the plurality of images with respect to the mesh .
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
In calculating the weighting coefficient, the angle formed by the normal reflection direction of the illumination light from the illumination light source at the time of photographing the image on the mesh and the optical axis of the image capturing apparatus at the time of photographing the image An image processing method for calculating the weighting coefficient. Acquire multiple images of the subject from multiple different viewpoints with an image capturing device,
Using the plurality of images, a three-dimensional mesh model in which the surface of the subject is represented by a mesh is generated.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Calculate the weighting coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. An image processing method for generating the texture image by adding an image obtained by multiplying the image extracted from the mesh by the weighting coefficient of each of the plurality of images to the mesh .
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
The calculation of the weighting coefficient is an image processing method for calculating the weighting coefficient according to the incident angle of the illumination light from the illumination light source at the time of shooting the image with respect to the normal direction of the mesh. Acquire multiple images of the subject from multiple different viewpoints with an image capturing device,
Using the plurality of images, a three-dimensional mesh model in which the surface of the subject is represented by a mesh is generated.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Calculate the weighting coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. An image processing method for generating the texture image by adding an image obtained by multiplying the image extracted from the image by the weight coefficient of each of the plurality of images with respect to the mesh .
In calculating the weighting coefficient, the normal direction of the mesh and the normal direction of the mesh are used by using the positional relationship and orientation between the mesh and the designated viewpoint used when generating an image viewed from the specified viewpoint. The weighting coefficient is determined according to the difference between the angle formed by the optical axis of the image capturing device at the time of capturing the image and the angle formed by the normal direction of the mesh and the line-of-sight direction from the designated viewpoint. Image processing method to calculate. Acquire multiple images of the subject from multiple different viewpoints with an image capturing device,
Using the plurality of images, a three-dimensional mesh model in which the surface of the subject is represented by a mesh is generated.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Calculate the weighting coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. An image processing method for generating the texture image by adding an image obtained by multiplying the image extracted from the image by the weight coefficient of each of the plurality of images with respect to the mesh .
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
In calculating the weighting coefficient, the mesh, the designated viewpoint, and the designation used when generating an image viewed from a designated viewpoint when the illumination light is irradiated from the designated illumination light source. Using the positional relationship and orientation with the illuminated light source, the normal reflection direction of the illumination light from the illumination light source at the time of photographing the image in the mesh and the optical axis of the image capturing apparatus at the time of photographing the image are formed. An image processing method for calculating the weighting coefficient according to the difference between the angle and the angle formed by the normal reflection direction of the illumination light from the designated illumination light source in the mesh and the line-of-sight direction from the designated viewpoint. .. Acquire multiple images of the subject from multiple different viewpoints with an image capturing device,
Using the plurality of images, a three-dimensional mesh model in which the surface of the subject is represented by a mesh is generated.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Calculate the weighting coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. An image processing method for generating the texture image by adding an image obtained by multiplying the image extracted from the image by the weight coefficient of each of the plurality of images with respect to the mesh .
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
In calculating the weighting coefficient, the mesh used to generate an image viewed from a specified viewpoint when the illumination light is emitted from the designated illumination light source and the designated illumination light source are used. Using the positional relationship and orientation, the incident angle of the illumination light from the illumination light source at the time of taking the image with respect to the normal direction of the mesh and the illumination light from the designated illumination light source with respect to the normal direction of the mesh. An image processing method for calculating the weighting coefficient according to the difference between the incident angle and the angle of incidence . Acquire multiple images of the subject from multiple different viewpoints with an image capturing device,
Using the plurality of images, a three-dimensional mesh model in which the surface of the subject is represented by a mesh is generated.
For each mesh of the three-dimensional mesh model, each of the plurality of images depends on the positional relationship and orientation between the mesh and at least one of the image capturing device and the illumination light source at the time of capturing each of the plurality of images. Calculate the weighting coefficient of
When a texture image corresponding to each mesh of the three-dimensional mesh model is generated from the plurality of images, an image of a region corresponding to the mesh is extracted from each of the plurality of images, and each of the plurality of images. An image processing method for generating the texture image by adding an image obtained by multiplying the image extracted from the image by the weight coefficient of each of the plurality of images with respect to the mesh .
The plurality of images are images of a subject photographed by the image capturing device when the illumination light from the illumination light source is irradiated.
In calculating the weight coefficient, the first weight coefficient is calculated according to the angle formed by the normal direction of the mesh and the optical axis of the image capturing apparatus at the time of photographing the image, and the image in the mesh is calculated. The second weighting coefficient is calculated according to the angle formed by the normal reflection direction of the illumination light from the illumination light source at the time of shooting and the optical axis of the image capturing device at the time of shooting the image, and the normal line of the mesh. The mesh and the mesh used when generating an image viewed from a specified viewpoint by calculating a third weighting coefficient according to the incident angle of the illumination light from the illumination light source when the image is taken with respect to the direction. Using the positional relationship and orientation with the designated viewpoint, the angle formed by the normal direction of the mesh and the optical axis of the image capturing apparatus when the image is taken, the normal direction of the mesh, and the designation. The fourth weighting coefficient is calculated according to the difference between the angle formed by the line-of-sight direction from the viewpoint and the image viewed from the specified viewpoint when the illumination light is emitted from the specified illumination light source. The direction of normal reflection of the illumination light from the illumination light source at the time of taking the image on the mesh by using the positional relationship and the orientation of the mesh and the designated viewpoint and the designated illumination light source. The angle formed by the light axis of the image capturing apparatus at the time of capturing the image, the normal reflection direction of the illumination light from the designated illumination light source in the mesh, and the line-of-sight direction from the designated viewpoint. A fifth weighting coefficient is calculated according to the difference between the angle and the mesh, and the mesh used to generate an image viewed from a specified viewpoint when the illumination light is emitted from the specified illumination light source. Using the positional relationship and orientation with the designated illumination light source, the incident angle of the illumination light from the illumination light source at the time of taking the image with respect to the normal direction of the mesh and the designation with respect to the normal direction of the mesh. After calculating the sixth weighting coefficient according to the difference between the incident angle of the illumination light from the illumination light source and the incident angle of the illumination light, the above-mentioned is based on two or more of the first to the sixth weighting coefficients. An image processing method for calculating the final weighting coefficient of the image with respect to the mesh . A program for causing a computer to function as each part of the image processing apparatus according to any one of claims 1 to 6.

Images