CN117237547B - Image reconstruction method, reconstruction model processing method and device - Google Patents
Image reconstruction method, reconstruction model processing method and device Download PDFInfo
- Publication number
- CN117237547B CN117237547B CN202311518981.4A CN202311518981A CN117237547B CN 117237547 B CN117237547 B CN 117237547B CN 202311518981 A CN202311518981 A CN 202311518981A CN 117237547 B CN117237547 B CN 117237547B
- Authority
- CN
- China
- Prior art keywords
- face
- point
- sampling point
- current
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 80
- 238000003672 processing method Methods 0.000 title abstract description 10
- 238000005070 sampling Methods 0.000 claims abstract description 485
- 230000005855 radiation Effects 0.000 claims abstract description 457
- 230000014509 gene expression Effects 0.000 claims abstract description 171
- 210000005036 nerve Anatomy 0.000 claims abstract description 78
- 238000004590 computer program Methods 0.000 claims abstract description 22
- 238000012545 processing Methods 0.000 claims description 118
- 238000009877 rendering Methods 0.000 claims description 106
- 230000001537 neural effect Effects 0.000 claims description 101
- 230000000694 effects Effects 0.000 claims description 61
- 238000000605 extraction Methods 0.000 claims description 35
- 230000002776 aggregation Effects 0.000 claims description 34
- 238000004220 aggregation Methods 0.000 claims description 34
- 238000012549 training Methods 0.000 claims description 27
- 230000003287 optical effect Effects 0.000 claims description 22
- 238000007499 fusion processing Methods 0.000 claims description 19
- 238000002372 labelling Methods 0.000 claims description 14
- 230000001815 facial effect Effects 0.000 claims description 10
- 230000004927 fusion Effects 0.000 claims description 10
- 230000008569 process Effects 0.000 claims description 10
- 230000011218 segmentation Effects 0.000 claims description 8
- 238000010276 construction Methods 0.000 claims description 7
- 238000004364 calculation method Methods 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 9
- 238000005516 engineering process Methods 0.000 description 12
- 239000011159 matrix material Substances 0.000 description 11
- 239000013598 vector Substances 0.000 description 9
- 238000010586 diagram Methods 0.000 description 8
- 238000006243 chemical reaction Methods 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 4
- 238000004891 communication Methods 0.000 description 3
- 238000011161 development Methods 0.000 description 3
- 230000018109 developmental process Effects 0.000 description 3
- 230000009467 reduction Effects 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000006399 behavior Effects 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000013135 deep learning Methods 0.000 description 2
- 238000013136 deep learning model Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- OKTJSMMVPCPJKN-UHFFFAOYSA-N Carbon Chemical compound [C] OKTJSMMVPCPJKN-UHFFFAOYSA-N 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000003190 augmentative effect Effects 0.000 description 1
- 230000033228 biological regulation Effects 0.000 description 1
- WHHGLZMJPXIBIX-UHFFFAOYSA-N decabromodiphenyl ether Chemical compound BrC1=C(Br)C(Br)=C(Br)C(Br)=C1OC1=C(Br)C(Br)=C(Br)C(Br)=C1Br WHHGLZMJPXIBIX-UHFFFAOYSA-N 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000008921 facial expression Effects 0.000 description 1
- 229910021389 graphene Inorganic materials 0.000 description 1
- 210000003128 head Anatomy 0.000 description 1
- 230000006698 induction Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000002787 reinforcement Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
Landscapes
- Image Analysis (AREA)
Abstract
The present application relates to an image reconstruction method, a reconstruction model processing method, an apparatus, a computer device, a storage medium and a computer program product, which are applicable to the field of artificial intelligence, and the method includes: acquiring shooting parameters corresponding to an image to be processed, constructing target radiation rays based on the shooting parameters, and respectively sampling on each target radiation ray to obtain a plurality of sampling points on each target radiation ray; extracting face gestures and expression parameters of a target object in an image to be processed; performing nerve radiation treatment on the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point to obtain the face probability that the current sampling point belongs to the face surface point; for each target radiation ray, face surface points corresponding to the current target radiation ray are determined based on the face probability corresponding to each sampling point on the current target radiation ray, and a three-dimensional face model is constructed based on the face surface points corresponding to each target radiation ray. And the accuracy of the three-dimensional face model is improved.
Description
Technical Field
The present invention relates to the field of computer technology, and in particular, to an image reconstruction method, a reconstruction model processing method, an apparatus, a computer device, a storage medium, and a computer program product.
Background
With the development of computer technology, various applications emerge, such as: shopping application program, social application program and image processing application program, wherein the image processing application program provides various functional special effects, and can promote the interest of playing. The image processing application program needs to finish the reconstruction of the three-dimensional face firstly to finish the addition of the special effect.
In the traditional technology, when a three-dimensional face model is reconstructed, face pose and expression parameters are extracted from an image to be processed, and the face pose and the expression parameters are fused into the prior face model to obtain the three-dimensional model of the face in the image to be processed. However, the three-dimensional face model obtained in this way is not high in accuracy and is not realistic enough.
Disclosure of Invention
In view of the foregoing, it is desirable to provide an image reconstruction method, a processing method for reconstructing a model, an apparatus, a computer device, a computer readable storage medium, and a computer program product that can improve accuracy of a three-dimensional face model.
In a first aspect, the present application provides an image reconstruction method. The method comprises the following steps:
acquiring shooting parameters corresponding to an image to be processed, determining virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray;
Extracting face gestures and expression parameters of a target object in an image to be processed;
aiming at each sampling point, performing nerve radiation processing on the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point to obtain the face probability that the current sampling point belongs to the face surface point;
for each target radiation ray, face surface points corresponding to the current target radiation ray are determined based on the face probability corresponding to each sampling point on the current target radiation ray, and a three-dimensional face model is constructed based on the face surface points corresponding to each target radiation ray.
In a second aspect, the present application provides an image reconstruction apparatus, comprising:
the acquisition module is used for acquiring shooting parameters corresponding to the image to be processed, determining virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray;
the extraction module is used for extracting the facial pose and expression parameters of the target object in the image to be processed;
The nerve radiation processing module is used for carrying out nerve radiation processing on the three-dimensional coordinates, the human face posture and the expression parameters of the current sampling point aiming at each sampling point to obtain the human face probability that the current sampling point belongs to the human face surface point;
the construction module is used for determining face surface points corresponding to the current target radiation rays based on the face probabilities corresponding to the sampling points on the current target radiation rays for each target radiation ray, and constructing a three-dimensional face model based on the face surface points corresponding to the target radiation rays.
In some embodiments, the acquisition module is specifically configured to determine an optical axis ray based on the virtual shooting point and a center point of the virtual image plane; translating the virtual image plane along the optical axis ray to a preset distance in a direction close to the virtual shooting point to obtain a low-beam image plane, and translating the virtual image plane along the optical axis ray to a preset distance in a direction far away from the virtual shooting point to obtain a high-beam image plane; and aiming at each target radiation ray, taking the intersection point of the current target radiation ray and the low beam image plane as a sampling starting point, taking the intersection point of the current target radiation ray and the high beam image plane as a sampling ending point, and sampling between the sampling starting point and the sampling ending point on the current target radiation ray to obtain a plurality of sampling points on the current target radiation ray.
In some embodiments, the extraction module is specifically configured to input the image to be processed into a face feature extraction model, take a face pose output by the face feature extraction model as a face pose of a target object in the image to be processed, and take an expression parameter output by the face feature extraction model as an expression parameter of the target object in the image to be processed; the face feature extraction model is obtained by training based on face sample data, wherein the face sample data comprises a face image sample, and a labeling gesture and a labeling expression parameter of a face in the face image sample.
In some embodiments, the expression parameters include expression deformation parameters and shape deformation parameters, and the image reconstruction device further includes a standardization module, configured to perform fusion processing on the three-dimensional coordinates of the face pose and the current sampling point for each sampling point, so as to obtain a pose standardization result; carrying out fusion processing on the expression deformation parameters and the expression deformation network parameters to obtain an expression deformation standardization result; performing fusion processing on the shape deformation parameters and the shape deformation network parameters to obtain a shape deformation standardization result; carrying out aggregation treatment on the gesture standardization result, the expression deformation standardization result and the shape deformation standardization result to obtain a standardization sampling point corresponding to the current sampling point; the nerve radiation processing module is specifically used for carrying out nerve radiation processing on three-dimensional coordinates, face gestures and expression parameters of the current standardized sampling points aiming at each standardized sampling point to obtain the face probability that the current standardized sampling points belong to face surface points.
In some embodiments, the nerve radiation processing module is specifically configured to input three-dimensional coordinates, a face pose and expression parameters of a current sampling point into the nerve radiation field model, and use a probability output by the nerve radiation field model as a face probability that the current sampling point belongs to a face surface point; the nerve radiation field model is used for predicting whether the sampling points belong to face surface points.
In some embodiments, the construction module is further configured to calculate a normal gradient of the face probability at the current sampling point; inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinate of the current sampling point, the face probability and the high-level characteristic into the trained neural rendering model to obtain the color information of the current sampling point, wherein the high-level characteristic is output after the neural radiation field model processes the three-dimensional coordinate, the face gesture and the expression parameter of the current sampling point input into the neural rendering model; aiming at each target radiation ray, determining target color information of a face surface point corresponding to the current radiation ray based on the color information corresponding to each sampling point on the current radiation ray; and rendering the three-dimensional face model based on the target color information of each face surface point to obtain a rendered three-dimensional face model.
In some embodiments, the construction module is specifically configured to obtain distances between each sampling point and the virtual shooting point on the current target radiation ray; the distances between each sampling point and the virtual shooting point and the face probabilities corresponding to each sampling point are subjected to statistical processing to obtain the surface point distances; and determining the face surface point corresponding to the current target radiation ray based on the virtual shooting point and the surface point distance.
In some embodiments, the obtaining module is further configured to respond to the special effect processing request, and display the image input reminding information; receiving a target image input by an object based on image input reminding information, and taking the target image as an image to be processed; taking shooting parameters corresponding to the target image as shooting parameters corresponding to the image to be processed; and acquiring special effect processing parameters corresponding to the special effect processing request, and rendering the three-dimensional face model based on the special effect processing parameters to obtain a special effect image matched with the special effect processing request.
In a third aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor which when executing the computer program performs the steps of:
Acquiring shooting parameters corresponding to an image to be processed, determining virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray;
extracting face gestures and expression parameters of a target object in an image to be processed;
aiming at each sampling point, performing nerve radiation processing on the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point to obtain the face probability that the current sampling point belongs to the face surface point;
for each target radiation ray, face surface points corresponding to the current target radiation ray are determined based on the face probability corresponding to each sampling point on the current target radiation ray, and a three-dimensional face model is constructed based on the face surface points corresponding to each target radiation ray.
In a fourth aspect, the present application also provides a computer-readable storage medium. A computer readable storage medium having stored thereon a computer program which when executed by a processor performs the steps of:
Acquiring shooting parameters corresponding to an image to be processed, determining virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray;
extracting face gestures and expression parameters of a target object in an image to be processed;
aiming at each sampling point, performing nerve radiation processing on the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point to obtain the face probability that the current sampling point belongs to the face surface point;
for each target radiation ray, face surface points corresponding to the current target radiation ray are determined based on the face probability corresponding to each sampling point on the current target radiation ray, and a three-dimensional face model is constructed based on the face surface points corresponding to each target radiation ray.
In a fifth aspect, the present application also provides a computer program product. Computer program product comprising a computer program which, when executed by a processor, realizes the steps of:
Acquiring shooting parameters corresponding to an image to be processed, determining virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray;
extracting face gestures and expression parameters of a target object in an image to be processed;
aiming at each sampling point, performing nerve radiation processing on the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point to obtain the face probability that the current sampling point belongs to the face surface point;
for each target radiation ray, face surface points corresponding to the current target radiation ray are determined based on the face probability corresponding to each sampling point on the current target radiation ray, and a three-dimensional face model is constructed based on the face surface points corresponding to each target radiation ray.
Firstly, shooting parameters corresponding to an image to be processed are obtained, virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space are determined based on the shooting parameters, a plurality of radiation rays are constructed based on the virtual shooting points and the virtual image planes, target radiation rays are determined based on the radiation rays, and sampling is carried out on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray; extracting face gestures and expression parameters of a target object in an image to be processed; then, aiming at each sampling point, carrying out nerve radiation treatment on the three-dimensional coordinates, the human face posture and the expression parameters of the current sampling point to obtain the human face probability that the current sampling point belongs to the human face surface point; aiming at each target radiation ray, determining a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray; and finally, constructing a three-dimensional face model based on the face surface points corresponding to the target radiation rays. The shooting points and the image plane are simulated by combining the actual shooting parameters, so that a three-dimensional radiation field is constructed, and the face surface point reduction mode is performed by combining means such as nerve radiation field treatment and the like, so that the finally obtained three-dimensional face model is more accurate and lifelike.
Drawings
FIG. 1 is a diagram of an application environment for an image reconstruction method in one embodiment;
FIG. 2 is a flow chart of an image reconstruction method in one embodiment;
FIG. 3 is a schematic diagram of constructing radiation rays in one embodiment;
FIG. 4 is a flow diagram of sample point normalization processing in one embodiment;
FIG. 5 is a flow diagram of a method of processing a reconstructed model in one embodiment;
FIG. 6 is a flow chart of a method of reconstructing a model according to another embodiment;
FIG. 7 is a flow diagram of constructing a three-dimensional face model in one embodiment;
FIG. 8 is a block diagram of an image reconstruction apparatus in one embodiment;
FIG. 9 is a block diagram of a processing device for reconstructing a model in one embodiment;
fig. 10 is an internal structural view of a computer device in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application will be further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the present application.
The image reconstruction method provided by the embodiment of the application can be applied to an application environment shown in fig. 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on the cloud or other servers. The image reconstruction method provided in the embodiment of the present application may be performed by the terminal 102 or the server 104 alone, or performed by the terminal 102 and the server 104 cooperatively. The following description is made with the terminal 102 alone as an example: firstly, shooting parameters corresponding to an image to be processed are obtained, virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space are determined based on the shooting parameters, a plurality of radiation rays are constructed based on the virtual shooting points and the virtual image planes, target radiation rays are determined based on the radiation rays, sampling is carried out on each target radiation ray based on the virtual image planes, and a plurality of sampling points on each target radiation ray are obtained; then, extracting the face gesture and expression parameters of a target object in the image to be processed; aiming at each sampling point, performing nerve radiation processing on the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point to obtain the face probability that the current sampling point belongs to the face surface point; aiming at each target radiation ray, determining a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray; and finally, constructing a three-dimensional face model based on the face surface points corresponding to the target radiation rays.
The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services.
The image reconstruction method and the processing method of the reconstruction model provided by the embodiment of the application can be applied to the field of artificial intelligence (Artificial Intelligence, AI), wherein the artificial intelligence is the theory, method, technology and application system which utilizes a digital computer or a machine controlled by the digital computer to simulate, extend and expand the intelligence of a person, sense the environment, acquire knowledge and acquire the best result by using the knowledge. In other words, artificial intelligence is an integrated technology of computer science that attempts to understand the essence of intelligence and to produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence, i.e. research on design principles and implementation methods of various intelligent machines, enables the machines to have functions of sensing, reasoning and decision.
The image reconstruction method and the processing method of the reconstruction model provided by the embodiment of the application can be used for optimizing the traditional Computer Vision technology (CV), wherein the Computer Vision technology is a science for researching how to enable a machine to 'see', and further means that a camera and a Computer are used for replacing human eyes to perform machine Vision such as identification, detection and measurement on a target, and further perform graphic processing, so that the Computer processing becomes an image which is more suitable for human eyes to observe or transmit to an instrument to detect. As a scientific discipline, computer vision research-related theory and technology has attempted to build artificial intelligence systems that can acquire information from images or multidimensional data. The large model technology brings important innovation for the development of computer vision technology, and a pre-trained model in the vision fields of swin-transformer, viT, V-MOE, MAE and the like can be rapidly and widely applied to downstream specific tasks through fine tuning. Computer vision techniques typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content/behavior recognition, three-dimensional object reconstruction, 3D techniques, virtual reality, augmented reality, synchronous positioning, and map construction, among others, as well as common biometric recognition techniques such as face recognition, fingerprint recognition, and others.
The neural radiation field model and the neural rendering model in the embodiments of the present application are realized through Machine Learning (ML), which is a multi-domain interdisciplinary, and relates to multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and the like. It is specially studied how a computer simulates or implements learning behavior of a human to acquire new knowledge or skills, and reorganizes existing knowledge structures to continuously improve own performance. Machine learning is the core of artificial intelligence, a fundamental approach to letting computers have intelligence, which is applied throughout various areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, confidence networks, reinforcement learning, transfer learning, induction learning, teaching learning, and the like. The pre-training model is the latest development result of deep learning, and integrates the technology.
In some embodiments, as shown in fig. 2, an image reconstruction method is provided, and the method is applied to a computer device for illustration. The computer device may be the terminal or the server in fig. 1, or a system of the terminal and the server, the method comprising the steps of:
Step 202, acquiring shooting parameters corresponding to an image to be processed, determining virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray.
The computer device is provided with an image shooting device, for example, the image shooting device can be a camera, a video camera and the like, and the computer device can shoot an image to be processed through the image shooting device. The computer equipment can acquire shooting parameters used when the image shooting device shoots the image to be processed, and the shooting parameters are taken as shooting parameters corresponding to the image to be processed. By way of example, the photographing parameters may include parameters of camera intrinsic parameters, camera extrinsic parameters, and the like.
After obtaining the shooting parameters corresponding to the image to be processed, the computer equipment can perform coordinate conversion based on the shooting parameters, when obtaining the three-dimensional coordinate point of the image shooting device and the three-dimensional coordinate range of the image plane corresponding to the image to be processed when the image shooting device shoots the image to be processed, can determine the virtual shooting point of the image to be processed in the three-dimensional space based on the three-dimensional coordinate point of the image shooting device, and determine the virtual image plane of the image to be processed in the three-dimensional space based on the three-dimensional coordinate range of the image plane corresponding to the image to be processed.
After the computer equipment obtains the virtual shooting point and the virtual image plane of the image to be processed in the three-dimensional space, the virtual shooting point can be used as a ray end point, a plurality of rays can be put into the virtual image plane, and the put plurality of rays can be used as radiation rays.
In a specific implementation, the computer device may take the virtual shooting point as a ray end point, put rays into the pixel center of each pixel on the virtual image plane, and may obtain a plurality of radiation rays matched with the number of pixels.
For example, referring to fig. 3, the virtual shooting point is point O, the virtual image plane includes 7×5 pixels, where the pixel center of one pixel is point a, and radiation corresponding to the pixel is obtained by throwing radiation from point O to point a, and 7×5 radiation can be obtained by performing this processing on all the pixels.
After obtaining a plurality of radiation rays, the computer device can take all the radiation rays as target radiation rays, or screen a part of all the radiation rays as target radiation rays. The embodiments of the present application are not limited in this regard.
In a specific implementation, after the computer device obtains a plurality of radiation rays, for each row of radiation rays on the virtual image plane, the row of radiation rays can be sampled according to a preset rule, and the radiation rays obtained by sampling are used as target radiation rays.
After determining the target radiation rays, the computer equipment can sample each target radiation ray based on the virtual image plane to obtain a plurality of sampling points on each target radiation ray.
In a specific implementation, a low beam image plane and a high beam image plane may be determined based on the virtual image plane, a sampling start point and a sampling end point may be determined based on the low beam image plane and the high beam image plane for each target radiation ray, and sampling may be performed between the sampling start point and the sampling end point to obtain a plurality of sampling points on the target radiation ray.
After determining the sampling start point and the sampling end point, the computer device may start with the sampling start point, and adopt a point on the target radiation ray in a direction towards the sampling end point at intervals of a fixed distance until the next sampling point is the sampling end point, or until the next sampling point is no longer between the sampling start point and the sampling end point.
Step 204, extracting the face pose and expression parameters of the target object in the image to be processed.
The target object may be a face, the face pose is used for describing an azimuth direction of the face, the face pose may include a pitch angle (pitch), a yaw angle (yaw) and a roll angle (roll), the expression parameter is used for describing a face expression state, the expression parameter may include an expression deformation parameter and a shape deformation parameter, the expression deformation parameter is used for describing an expression deformation state, and the shape deformation parameter is used for describing a shape deformation state.
For example, the expression deformation parameter may be a vector of 1×41 dimensions, where each dimension is used to describe the expression deformation of the face, such as whether to open the mouth, close the eyes, and so on; the shape deformation parameters may be 1 x 100 dimensional vectors, each dimension describing shape deformation of the face, e.g., whether the nose is large, whether the eyes are small, etc.
The computer equipment can utilize the face feature extraction model to extract the face gesture and expression parameters of the target object in the image to be processed. The face feature extraction model is obtained based on training of face sample data; alternatively, the computer device may utilize a microexpressive capturing and modeling (Detailed Expression Capture and Animation, DECA) algorithm to extract facial pose and expression parameters of the target object in the image to be processed; alternatively, the computer device may further extract the facial pose and expression parameters of the target object in the image to be processed in other manners, which is not limited in the embodiment of the present application.
Step 206, for each sampling point, performing nerve radiation processing on the three-dimensional coordinates, the face gesture and the expression parameters of the current sampling point to obtain the face probability that the current sampling point belongs to the face surface point.
The face gesture may include a pitch angle (pitch), a yaw angle (yaw) and a roll angle (roll), the face gesture may be converted into a rotation matrix, and for each sampling point, the computer device may perform neural radiation processing on three-dimensional coordinates, the rotation matrix and expression parameters of the current sampling point, to obtain a face probability that the current sampling point belongs to a face surface point.
In a specific implementation, for each sampling point, the computer device may perform neural radiation processing on the three-dimensional coordinates, the rotation matrix and the expression parameters of the current sampling point by using a neural radiation field model, so as to obtain a face probability that the current sampling point belongs to a face surface point. The face probability that the current sampling point belongs to the face surface point is also called Occupancy (Occ) of the current sampling point.
Step 208, for each target radiation ray, determining a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray, and constructing a three-dimensional face model based on the face surface point corresponding to each target radiation ray.
After obtaining the face probability that each sampling point belongs to a face surface point, the computer equipment can acquire the distance between each sampling point and a virtual shooting point on the target radiation ray according to each target radiation ray, and determine the face surface point corresponding to the target radiation ray based on the distance between each sampling point and the virtual shooting point and the face probability corresponding to each sampling point. And (3) processing each target radiation ray to obtain the face surface point corresponding to each target radiation ray.
After obtaining the face surface points corresponding to each target radiation ray, the computer equipment can obtain a curved surface capable of covering all face surface points, and the curved surface is used as a three-dimensional face model.
The virtual shooting point is a point O, the virtual image plane contains 7×5 pixels, rays are cast from the point O to the pixel center of each pixel, 7×5 radiation rays can be obtained, the 7×5 radiation rays are all used as target radiation rays, face surface points corresponding to each target radiation ray are obtained through the above method, 7×5 face surface points are obtained, and a curved surface capable of covering the 7×5 face surface points is used as a three-dimensional face model.
In the above embodiment, first, shooting parameters corresponding to an image to be processed are obtained, virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space are determined based on the shooting parameters, a plurality of radiation rays are constructed based on the virtual shooting points and the virtual image planes, target radiation rays are determined based on the radiation rays, and sampling is performed on each target radiation ray based on the virtual image planes, so that a plurality of sampling points on each target radiation ray are obtained; extracting face gestures and expression parameters of a target object in an image to be processed; then, aiming at each sampling point, carrying out nerve radiation treatment on the three-dimensional coordinates, the human face posture and the expression parameters of the current sampling point to obtain the human face probability that the current sampling point belongs to the human face surface point; aiming at each target radiation ray, determining a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray; and finally, constructing a three-dimensional face model based on the face surface points corresponding to the target radiation rays. The shooting points and the image plane are simulated by combining the actual shooting parameters, so that a three-dimensional radiation field is constructed, and the face surface point reduction mode is performed by combining means such as nerve radiation field treatment and the like, so that the finally obtained three-dimensional face model is more accurate and lifelike.
In some embodiments, sampling on each target radiation ray based on the virtual image plane respectively, to obtain a plurality of sampling points on each target radiation ray includes: determining an optical axis ray based on the virtual photographing point and a center point of the virtual image plane; translating the virtual image plane along the optical axis ray to a preset distance in a direction close to the virtual shooting point to obtain a low-beam image plane, and translating the virtual image plane along the optical axis ray to a preset distance in a direction far away from the virtual shooting point to obtain a high-beam image plane; and aiming at each target radiation ray, taking the intersection point of the current target radiation ray and the low beam image plane as a sampling starting point, taking the intersection point of the current target radiation ray and the high beam image plane as a sampling ending point, and sampling between the sampling starting point and the sampling ending point on the current target radiation ray to obtain a plurality of sampling points on the current target radiation ray.
The computer equipment can determine the center point of the virtual image plane, and the rays are put into the center point from the virtual shooting point, so that the optical axis rays can be obtained.
The preset distance may be an average thickness of the human head, or may be a distance determined by combining actual experience, which is not limited in the embodiment of the present application.
Under the condition that the optical axis ray is determined, the computer equipment can translate the virtual image plane along the optical axis ray to a preset distance in a direction close to the virtual shooting point to obtain a low-beam image plane, and translate the virtual image plane along the optical axis ray to a preset distance in a direction far away from the virtual shooting point to obtain a high-beam image plane.
And taking the intersection point of the target radiation ray and the low beam image plane as a sampling starting point, taking the intersection point of the target radiation ray and the high beam image plane as a sampling ending point, starting with the sampling starting point, and taking one point at intervals of a fixed distance on the target radiation ray towards the direction of the sampling ending point until the next sampling point is the sampling ending point or the next sampling point is not between the sampling starting point and the sampling ending point, so that a plurality of sampling points on the target radiation ray can be obtained.
In the above embodiment, a specific manner of sampling on a target radiation ray is provided, firstly, an optical axis ray is determined, then, a low beam image plane and a high beam image plane are determined based on a virtual image plane and the optical axis ray, for each target radiation ray, a sampling start point and a sampling end point are defined based on the two planes, and then, sampling is performed between the sampling start point and the sampling end point.
In some embodiments, extracting facial pose and expression parameters of a target object in an image to be processed includes: inputting the image to be processed into a face feature extraction model, taking the face gesture output by the face feature extraction model as the face gesture of a target object in the image to be processed, and taking the expression parameter output by the face feature extraction model as the expression parameter of the target object in the image to be processed; the face feature extraction model is obtained by training based on face sample data, wherein the face sample data comprises a face image sample, and a labeling gesture and a labeling expression parameter of a face in the face image sample.
The face feature extraction model can be trained in the following manner: acquiring face sample data, the face sample data comprising: the method comprises the steps of inputting a face image sample and a labeling gesture and a labeling expression parameter of a face in the face image sample into a face feature extraction model to be trained to obtain a predicted gesture and a face predicted expression parameter, calculating model loss based on the predicted gesture, the predicted expression parameter, the labeling gesture and the labeling expression parameter, carrying out parameter adjustment on the face feature extraction model to be trained based on the model loss, and stopping until a stopping condition of model training is met to obtain a face feature extraction model after training is completed.
The computer equipment can input the image to be processed into the face feature extraction model after training, take the face gesture output by the face feature extraction model as the face gesture of the target object in the image to be processed, and take the expression parameter output by the face feature extraction model as the expression parameter of the target object in the image to be processed.
In the above embodiment, a specific way of extracting the face pose and the expression parameter of the target object in the image to be processed is provided, and the face pose and the expression parameter extracted by using the deep learning model are more accurate, so that the three-dimensional face model obtained later is more lifelike.
In some embodiments, referring to fig. 4, the expression parameters include an expression deformation parameter and a shape deformation parameter, and the image reconstruction method provided in the embodiments of the present application further includes:
step 402, for each sampling point, fusion processing is performed on the three-dimensional coordinates of the face gesture and the current sampling point, so as to obtain a gesture standardization result.
Step 404, fusion processing is carried out on the expression situation variable parameters and the expression deformation network parameters to obtain an expression deformation standardization result;
step 406, fusion processing is carried out on the shape deformation parameters and the shape deformation network parameters, and a shape deformation standardized result is obtained;
And step 408, performing aggregation processing on the gesture standardization result, the expression deformation standardization result and the shape deformation standardization result to obtain a standardization sampling point corresponding to the current sampling point.
In order to obtain the three-dimensional face model without the gestures and the expressions, the computer equipment can perform standardized processing on the sampling points after obtaining the sampling points to obtain standardized sampling points. Each sampling point corresponds to a normalized sampling point. And subsequently, aiming at each standardized sampling point, carrying out nerve radiation treatment on the three-dimensional coordinates, the human face posture and the expression parameters of the current standardized sampling point to obtain the human face probability that the current standardized sampling point belongs to the human face surface point.
In parameterized face model reconstruction techniques, the following conversion formula exists:
the conversion formula can be deduced from the above:
wherein T represents a normalized sampling point,representing the three-dimensional coordinates of the sampling points, R representing the rotation matrix,/->Representing expression deformation parameters->Representing the expression deformation network parameters, < >>Representing shape deformation parameters->Representing shape deformation network parameters. The formula can be adopted to perform standardization processing on the sampling points to obtain standardized sampling points.
In a specific implementation, after the computer equipment obtains the face gesture, the face gesture can be converted into a rotation matrix, and for each sampling point, the transpose of the rotation matrix is calculated firstThen the three-dimensional coordinates of the sampling points and the transpose of the rotation matrix are +.>Multiplying to obtain +.>The method comprises the steps of carrying out a first treatment on the surface of the Multiplying the expression deformation parameters and the expression deformation network parameters to obtain +.>Multiplying the shape deformation parameters and the shape deformation network parameters to obtain +.>Use +.>Minus->And->The normalized sampling point T can be obtained.
Exemplary, the three-dimensional coordinates of the sampling points are1 x 3 vector, transpose of rotation matrixMay be a 3 x 3 matrix, < >>A vector of 1 x 3; vector with expression deformation parameters of 1×41, matrix with expression deformation network parameters of 41×3, +.>A vector of 1 x 3; vector with shape deformation parameters of 1×100, matrix with shape deformation network parameters of 100×3, +.>A vector of 1 x 3; the resulting T is also a 1 x 3 vector.
In the above embodiment, after the target radiation ray is sampled to obtain the sampling point, the sampling point may be subjected to standardization processing, specifically, the posture standardization result, the expression deformation standardization result and the shape deformation standardization result are first obtained, and then the three results are subjected to aggregation processing, so that the corresponding standardization sampling point is obtained, so that the finally obtained three-dimensional face model does not have the posture and the expression, and is more convenient for application of the three-dimensional face model.
In some embodiments, performing neural radiation processing on three-dimensional coordinates, face pose and expression parameters of a current sampling point to obtain a face probability that the current sampling point belongs to a face surface point, including: inputting three-dimensional coordinates, face gestures and expression parameters of the current sampling point into a nerve radiation field model, and taking the probability output by the nerve radiation field model as the face probability that the current sampling point belongs to a face surface point; the nerve radiation field model is used for predicting whether the sampling points belong to face surface points.
The computer equipment can input three-dimensional coordinates, facial gestures and expression parameters of the current sampling point into the nerve radiation field model, the nerve radiation field model predicts the face based on the data input into the nerve radiation field model, and the face probability that the current sampling point belongs to the face surface point is output.
The training process of the nerve radiation field model comprises the following steps: acquiring shooting parameters corresponding to an image sample, determining virtual shooting points and virtual image planes of the image sample in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, screening the plurality of radiation rays to obtain at least one target radiation ray, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray; extracting face gestures and expression parameters of a target object in an image sample; inputting three-dimensional coordinates, face gestures and expression parameters of the current sampling point to a nerve radiation field model to be trained aiming at each sampling point to obtain face probability and high-level characteristics of the current sampling point belonging to the face surface point; inputting the face gesture, the expression parameters, the three-dimensional coordinates of the current sampling point, the face probability and the high-level features into a neural rendering model to be trained to obtain the color information of the current sampling point; for each radiation ray, determining a face surface point corresponding to the current radiation ray based on the face probability corresponding to each sampling point on the current radiation ray, and determining target color information of the face surface point based on the color information corresponding to each sampling point on the current radiation ray; calculating model loss based on target color information and image samples of each face surface point; based on model loss, parameter adjustment is carried out on the nerve radiation field model to be trained and the nerve rendering model until stopping when the model training stopping condition is met, and the trained nerve radiation field model and nerve rendering model are obtained.
It should be noted that: when face prediction is performed on each sampling point by using the neural radiation field model, the sampling points can be standardized to obtain standardized sampling points, three-dimensional coordinates, face gestures and expression parameters of the standardized sampling points are input into the neural radiation field model, and face probability that the standardized sampling points belong to face surface points is obtained.
In the above embodiments, a specific manner of nerve radiation treatment is provided: the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling points are input into the nerve radiation field model, the probability output by the nerve radiation field model is used as the face probability of the current sampling points belonging to the face surface points, and the mode of predicting the face by using the deep learning model is higher in accuracy, so that the three-dimensional face model obtained later is more lifelike.
In some embodiments, the image reconstruction method provided in the embodiments of the present application further includes: solving a normal gradient of the face probability at the current sampling point; inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinate of the current sampling point, the face probability and the high-level characteristic into the trained neural rendering model to obtain the color information of the current sampling point, wherein the high-level characteristic is output after the neural radiation field model processes the three-dimensional coordinate, the face gesture and the expression parameter of the current sampling point input into the neural rendering model; aiming at each target radiation ray, determining target color information of a face surface point corresponding to the current radiation ray based on the color information corresponding to each sampling point on the current radiation ray; and rendering the three-dimensional face model based on the target color information of each face surface point to obtain a rendered three-dimensional face model.
Under the condition that the normalization processing is not added, the computer equipment can calculate the normal gradient of the face probability of the sampling point belonging to the face surface point at the sampling point aiming at each sampling point. Specifically, the computer device may calculate a normal gradient of a face probability of the sampling point belonging to the face surface point at a corresponding normalized sampling point, and determine, based on the normal gradient and the face pose, a normal gradient of the face probability of the sampling point belonging to the face surface point at the sampling point.
Under the condition of adding the standardization processing, after the standardized sampling points corresponding to the sampling points are obtained, the normal gradient of the face probability of the standardized sampling point belonging to the face surface point at the corresponding sampling point can be calculated by the computer equipment aiming at each standardized sampling point. Specifically, the computer device may calculate a normal gradient of a face probability of the normalized sample point belonging to the face surface point at the normalized sample point, and determine, based on the normal gradient and the face pose, a normal gradient of a face probability of the normalized sample point belonging to the face surface point at a corresponding sample point.
In a specific implementation, under the condition of adding the standardized processing, the computer equipment can calculate the normal gradient of the face probability of the standardized sampling point belonging to the face surface point at the current sampling point by adopting the following formula:
Wherein,representing the current sampling point, +.>Representing a standardized sampling point obtained by performing a standardized processing on the current sampling point, and +.>For standardizing the face probability that the sampling point belongs to the face surface point, < +.>Normal gradient of face probability representing normalized sampling point belonging to face surface point at current sampling point, +.>Normal gradient of face probability representing normalized sampling point belonging to face surface point at normalized sampling point, ++>Representing the face pose.
After the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point are input into the nerve radiation field model by the computer equipment, the nerve radiation field model not only outputs the face probability that the current sampling point belongs to the face surface point, but also outputs high-level features.
After the computer equipment calculates the normal gradient of the face probability of the standardized sampling point belonging to the face surface point at the current sampling point, the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinates of the standardized sampling point, the face probability of the standardized sampling point belonging to the face surface point and the high-level characteristics are input into a trained neural rendering model, and the trained neural rendering model outputs the color information of the standardized sampling point.
Wherein the color information of the normalized sample points includes values of red component (R), green component (G) and blue component (B).
After obtaining color information of each sampling point on the target radiation ray, the computer equipment multiplies the color information of each sampling point and the face probability corresponding to the sampling point on the target radiation ray to obtain a multiplication result at the sampling point, adds the multiplication results at all the sampling points on the target radiation ray to obtain an addition result, and divides the addition result by the sum of the face probabilities corresponding to all the sampling points on the target radiation ray to obtain the target color information of the face surface point corresponding to the target radiation ray. And (3) processing all the target radiation rays to obtain the target color information of the face surface points corresponding to the target radiation rays.
After obtaining the target color information of each face surface point, the computer equipment renders each face surface point on the three-dimensional face model into a corresponding color based on the target color information, so as to obtain a rendered three-dimensional face model.
In the above embodiment, a specific manner of rendering the three-dimensional face model is provided: firstly, color information of each sampling point is obtained through a trained neural rendering model, then, for each target radiation ray, based on the color information of each sampling point on the target radiation ray, target color information of a face surface point corresponding to the target radiation ray is determined, and based on the target color information of each face surface point, the three-dimensional face model is rendered, so that a rendered three-dimensional face model is obtained, the finally obtained three-dimensional face model is a model under a shooting scene corresponding to an image to be processed, and special effect processing under the shooting scene is facilitated.
In some embodiments, determining the face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray includes: obtaining the distance between each sampling point and a virtual shooting point on the current target radiation ray; the distances between each sampling point and the virtual shooting point and the face probabilities corresponding to each sampling point are subjected to statistical processing to obtain the surface point distances; and determining the face surface point corresponding to the current target radiation ray based on the virtual shooting point and the surface point distance.
The computer equipment can acquire the distance between each sampling point and the virtual shooting point on the current target radiation ray, multiply the distance between the sampling point and the virtual shooting point by the face probability corresponding to the sampling point for each sampling point on the current target radiation ray to obtain a multiplication result corresponding to the sampling point, add the multiplication results corresponding to all the sampling points on the current target radiation ray to obtain an addition result, and divide the addition result by the sum of the face probabilities corresponding to all the sampling points on the target radiation ray to obtain the surface point distance.
The coordinates of the virtual shooting point and the coordinates of the virtual image plane are known, and the coordinates of the current target radiation ray can be obtained by solving based on the coordinates of the virtual shooting point and the coordinates of the pixel center corresponding to the current target radiation ray. The coordinates of the face surface point corresponding to the current target radiation ray can be calculated based on the coordinates of the virtual shooting point, the coordinates of the current target radiation ray and the surface point distance.
In the above embodiment, for each target radiation ray, after obtaining the face probability corresponding to each sampling point on the target radiation ray, the face surface point corresponding to the target radiation ray may be determined based on the distance between each sampling point and the virtual shooting point on the target radiation ray and the face probability corresponding to each sampling point, and the shooting point and the image plane are simulated by combining the actual shooting parameters, so as to construct a three-dimensional radiation field, and further combine means such as neural radiation field processing to perform face surface point restoration, so that the finally obtained three-dimensional face model is more accurate and lifelike.
In some embodiments, acquiring shooting parameters corresponding to an image to be processed includes: responding to the special effect processing request, and displaying the image input reminding information; receiving a target image input by an object based on image input reminding information, and taking the target image as an image to be processed; taking the shooting parameters corresponding to the target image as the shooting parameters corresponding to the image to be processed.
The method comprises the steps that an application program with a special effect processing function is installed on computer equipment, the application program is provided with a special effect selection interface, an object can select a favorite special effect on the special effect selection interface, after the application program detects the selection operation of the object, special effect processing parameters corresponding to the special effect selected by the object are obtained, a special effect processing request is generated based on the special effect processing parameters, and after the application program detects the event, image input reminding information is displayed.
The image input reminding information can be in a text form or a voice form, and the embodiment of the application is not limited to the text form.
After the object sees the image displayed by the application program and inputs the reminding information, an image can be shot through an image shooting device on the computer equipment to serve as a target image, the target image is input into the application program, or an image is selected from images locally stored in the computer equipment to serve as the target image and is input into the application program.
After receiving the target image, the application program may take the target image as an image to be processed, obtain a shooting parameter used when the image shooting device shoots the target image, and take the shooting parameter as a shooting parameter corresponding to the image to be processed.
The application program can acquire special effect processing parameters corresponding to the special effect processing request, and render the three-dimensional face model based on the special effect processing parameters to obtain a special effect image matched with the special effect processing request.
The special effect processing parameters comprise special effect identification and rendering parameters, and the rendering parameters comprise high light intensity, ambient light intensity, light ray deviation, rendering materials and the like. The embodiments of the present application are not limited in this regard.
In a specific implementation, the application program can analyze the special effect processing request to obtain special effect processing parameters corresponding to the special effect selected by the object, namely special effect processing parameters corresponding to the special effect processing request, and the application program can perform rendering processing on the three-dimensional face model by adopting technologies such as offline rendering, real-time rendering, cloud rendering and the like based on the special effect processing parameters to obtain a special effect image corresponding to the special effect selected by the object, namely a special effect image matched with the special effect processing request.
In the above embodiment, the object may trigger to generate the special effect processing request, the computer device may respond to the special effect processing request, and may remind the object to input an image, the computer device may use the image input by the object as an image to be processed, and after the three-dimensional face model is constructed, render the three-dimensional face model based on the special effect processing parameters carried in the feature processing request, so as to obtain a special effect image matched with the special effect processing request, and increase the interestingness of the image playing method.
In some embodiments, as shown in fig. 5, a method for processing a reconstructed model is provided, and the method is used for computer equipment for illustration. The computer device may be the terminal or the server in fig. 1, or a system of the terminal and the server, the method comprising the steps of:
Step 502, capturing parameters corresponding to an image sample are obtained, virtual capturing points and virtual image planes of the image sample in a three-dimensional space are determined based on the capturing parameters, a plurality of radiation rays are constructed based on the virtual capturing points and the virtual image planes, target radiation rays are determined based on the radiation rays, and sampling is performed on each target radiation ray based on the virtual image planes, so that a plurality of sampling points on each target radiation ray are obtained.
The image sample may be obtained by shooting by an image shooting device on a computer device, or may be obtained by other shooting devices, and for convenience of explanation, the hardware for shooting the image sample is referred to as a target device.
The computer equipment can acquire shooting parameters used when the target equipment shoots the image sample, and the shooting parameters are taken as shooting parameters corresponding to the image sample. By way of example, the photographing parameters may include parameters of camera intrinsic parameters, camera extrinsic parameters, and the like.
After obtaining shooting parameters corresponding to the image sample, the computer device can perform coordinate conversion based on the shooting parameters to obtain a three-dimensional coordinate point of the target device and a three-dimensional coordinate range of an image plane corresponding to the image sample when the target device shoots the image sample, and can determine a virtual shooting point of the image sample in a three-dimensional space based on the three-dimensional coordinate point of the target device and determine a virtual image plane of the image sample in the three-dimensional space based on the three-dimensional coordinate range of the image plane corresponding to the image sample.
After the virtual shooting point and the virtual image plane of the image sample in the three-dimensional space are obtained, the computer equipment can take the virtual shooting point as a ray end point, put a plurality of rays into the virtual image plane, and take the put plurality of rays as radiation rays. The manner in which the target radiation ray is determined from the radiation rays and the manner in which the target radiation ray is sampled can be referred to the description of the foregoing embodiments, and the embodiments of the present application are not repeated herein.
Step 504, extracting face pose and expression parameters of a target object in the image sample.
The computer equipment can input the image sample into a face feature extraction model, take the face gesture output by the face feature extraction model as the face gesture of a target object in the image sample, and take the expression parameter output by the face feature extraction model as the expression parameter of the target object in the image sample.
Step 506, for each sampling point, inputting the three-dimensional coordinates, the face gesture and the expression parameters of the current sampling point into the neural radiation field model to be trained, so as to obtain the face probability and the high-level characteristics of the current sampling point belonging to the face surface point; and inputting the face gesture, the expression parameters, the three-dimensional coordinates of the current sampling point, the face probability and the high-level features into a neural rendering model to be trained, and obtaining the color information of the current sampling point.
The normalized processing may be performed on each sampling point to obtain a normalized sampling point corresponding to the sampling point, and the three-dimensional coordinates of the normalized sampling point, the face pose and the expression parameter extracted in step 504 may be input to a neural radiation field model to be trained, where the neural radiation field model outputs the face probability and the high-level features of the normalized sampling point belonging to the face surface point. The face pose, the expression parameter, the three-dimensional coordinates of the standardized sampling point, the face probability that the standardized sampling point belongs to the face surface point, and the high-level features extracted in the step 504 are input into a neural rendering model to be trained, and the neural rendering model to be trained outputs the color information of the standardized sampling point.
Step 508, for each target radiation ray, determining a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray, and determining target color information of the face surface point based on the color information corresponding to each sampling point on the current target radiation ray.
The computer equipment can acquire the distance between each sampling point and the virtual shooting point on each target radiation ray; the distances between each sampling point and the virtual shooting point and the face probabilities corresponding to each sampling point are subjected to statistical processing to obtain the surface point distances; and determining the face surface point corresponding to the current target radiation ray based on the virtual shooting point and the surface point distance. The detailed process may refer to the description of the foregoing embodiments, and the embodiments of the present application are not repeated herein.
The computer device can determine the target color information of the face surface point corresponding to the target radiation ray based on the color information corresponding to each sampling point on the target radiation ray and the face probability corresponding to each sampling point.
Step 510, calculating model loss based on the target color information and the image samples of each face surface point.
The pixels on the image sample and the pixels on the virtual image plane have a one-to-one correspondence, each radiation ray corresponds to one pixel on the virtual image plane, and all radiation rays can be used as target radiation rays, so that each target radiation ray corresponds to one pixel on the virtual image plane, each target radiation ray corresponds to one face surface point, the corresponding relationship can be obtained, and the corresponding relationship between the pixels on the image sample and the face surface points is one-to-one.
The computer equipment can perform face segmentation processing on the image sample to obtain face pixel points and non-face pixel points; the computer equipment can acquire a first face surface point corresponding to the face pixel point, determine rendering loss according to the pixel value of the face pixel point and the target color information of the first face surface point, and take the rendering loss as model loss. Or the computer equipment acquires a second face surface point corresponding to the non-face pixel point, determines mask loss according to the probability label corresponding to the non-face pixel point and the target color information of the second face surface point, and takes the mask loss as model loss. Alternatively, the computer device obtains a rendering loss and a mask loss, performs weighted summation on the two losses, and takes the weighted summation result as a model loss.
And step 512, based on the model loss, performing parameter adjustment on the neural radiation field model to be trained and the neural rendering model until the model training stopping condition is met, and obtaining the trained neural radiation field model and the neural rendering model. The trained neural radiation field model and the neural rendering model are used for constructing a three-dimensional face model.
The computer equipment can judge the model loss, when the model loss meets the stopping condition of model training, parameter adjustment is not performed any more, and the currently obtained nerve radiation field model and the nerve rendering model are used as the nerve radiation field model and the nerve rendering model after training is completed; when the model loss does not meet the model training stopping condition, determining a parameter adjustment gradient based on the model loss, performing parameter adjustment on the neural radiation field model to be trained and the neural rendering model based on the parameter adjustment gradient, and returning to the step 502 until the calculated model loss meets the model training stopping condition.
In the above embodiment, a training manner of a neural radiation field model and a neural rendering model is provided, shooting parameters corresponding to an image sample are obtained, virtual shooting points and virtual image planes of the image sample in a three-dimensional space are determined based on the shooting parameters, a plurality of radiation rays are constructed based on the virtual shooting points and the virtual image planes, target radiation rays are determined based on the radiation rays, and sampling is performed on each target radiation ray based on the virtual image planes, so as to obtain a plurality of sampling points on each target radiation ray; extracting face gestures and expression parameters of a target object in an image sample; inputting three-dimensional coordinates, face gestures and expression parameters of the current sampling point to a nerve radiation field model to be trained aiming at each sampling point to obtain face probability and high-level characteristics of the current sampling point belonging to the face surface point; inputting the face gesture, the expression parameters, the three-dimensional coordinates of the current sampling point, the face probability and the high-level features into a neural rendering model to be trained to obtain the color information of the current sampling point; for each radiation ray, determining a face surface point corresponding to the current radiation ray based on the face probability corresponding to each sampling point on the current radiation ray, and determining target color information of the face surface point based on the color information corresponding to each sampling point on the current radiation ray; calculating model loss based on target color information and image samples of each face surface point; based on model loss, parameter adjustment is carried out on the nerve radiation field model to be trained and the nerve rendering model until stopping when the model training stopping condition is met, and the trained nerve radiation field model and nerve rendering model are obtained. The trained neural radiation field model and the neural rendering model are used for constructing a three-dimensional face model. The shooting points and the image plane are simulated by combining the actual shooting parameters, so that a three-dimensional radiation field is constructed, and the face surface point reduction mode is performed by combining means such as nerve radiation field treatment and the like, so that the finally obtained three-dimensional face model is more accurate and lifelike.
In some embodiments, the method for processing a reconstructed model provided in the embodiments of the present application further includes: solving a normal gradient of the face probability at the current sampling point; inputting the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level features into a neural rendering model to be trained to obtain the color information of the current sampling point, wherein the method comprises the following steps: and inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinate of the current sampling point, the face probability and the high-level characteristic into a neural rendering model to be trained, and obtaining the color information of the current sampling point.
The method and the device have the advantages that the discontinuity of sampling has a certain influence on the rendering effect, the normal gradient of the face probability at the current sampling point is obtained, and the normal gradient is also used as the input of the neural rendering model to be trained, so that the influence of the discontinuity of sampling on the rendering effect is reduced.
Under the condition that the normalization processing is not added, the computer equipment can calculate the normal gradient of the face probability of the sampling point belonging to the face surface point at the sampling point aiming at each sampling point. The normal gradient is used as input to the neural rendering model to be trained. That is, the computer device may input the normal gradient, the face pose, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability, and the high-level features to the neural rendering model to be trained, and the neural rendering model to be trained may output the color information of the current sampling point.
Under the condition of adding the standardization processing, after obtaining the standardization sampling points corresponding to each sampling point, aiming at each standardization sampling point, the computer equipment can calculate the normal gradient of the face probability of the standardization sampling point belonging to the face surface point at the corresponding sampling point, and the normal gradient is used as the input of the neural rendering model to be trained. That is, the computer device may input the normal gradient, the face pose, the expression parameter, the three-dimensional coordinates of the current standardized sampling point, the face probability, and the high-level features to the neural rendering model to be trained, and the neural rendering model to be trained may output the color information of the current standardized sampling point.
In the above embodiment, after obtaining the face probability that the sampling point belongs to the face surface point, the normal gradient of the face probability at the current sampling point is also obtained; the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level characteristic are input into the neural rendering model to be trained, the color information of the current sampling point is obtained, the normal gradient is added into the input of the neural rendering model to be trained, the influence of the discontinuity of sampling on the rendering effect can be reduced, and the accuracy of the output color information is improved.
In some embodiments, determining the target color information of the face surface point based on the color information corresponding to each sampling point on the current radiation ray includes: aiming at each sampling point on the current radiation ray, carrying out fusion processing on color information and face probability corresponding to the current sampling point to obtain a fusion result corresponding to the current sampling point; performing aggregation treatment on fusion results corresponding to each sampling point on the current radiation ray to obtain a color aggregation result, and performing aggregation treatment on face probabilities corresponding to each sampling point on the current radiation ray to obtain a probability aggregation result; and determining target color information of the face surface points based on the color aggregation result and the probability aggregation result.
For each sampling point on the current radiation ray, the computer equipment can multiply the color information corresponding to the current sampling point with the face probability to obtain a fusion result corresponding to the current sampling point. And adding the fusion results corresponding to the sampling points on the current radiation ray to obtain a color aggregation result. And adding the face probabilities corresponding to the sampling points on the current radiation ray to obtain a probability aggregation result. And dividing the color aggregation result by the probability aggregation result to obtain the target color information of the face surface point.
In the above embodiment, a specific manner of determining the target color information of each face surface point is provided, and the target color information of each face surface point can be used for calculating the subsequent model loss, so that the color information output by the neural rendering model obtained by training is more similar to the original image, and the reasoning accuracy of the neural rendering model obtained by training is improved.
In some embodiments, the computing model loss based on the target color information and the image samples for each face surface point corresponds one-to-one to the pixel points on the image samples, comprising: carrying out face segmentation processing on the image sample to obtain face pixel points and non-face pixel points; acquiring a first face surface point corresponding to a face pixel point, and determining rendering loss according to the pixel value of the face pixel point and target color information of the first face surface point; acquiring a second face surface point corresponding to the non-face pixel point, and determining mask loss according to the probability label corresponding to the non-face pixel point and the face probability of the second face surface point; model loss is calculated based on the rendering loss and the mask loss.
The computer device can input the image sample into a pre-trained face segmentation model, and the face segmentation model can output whether each pixel point on the image sample is a face pixel point or a non-face pixel point. Because each face surface point corresponds to a pixel point on the image sample one by one, the computer device can acquire the face surface points corresponding to the face pixel points on the image sample, so that the face surface points are all called as first face surface points for convenience in distinguishing, and can also acquire the face surface points corresponding to the non-face pixel points on the image sample, so that the face surface points are called as second face surface points for convenience in distinguishing.
In a specific implementation, the computer device may calculate the rendering loss using the following formula:
wherein P is a human face pixel point,representing the number of face pixels on the image sample,/-for>Is the pixel value of the face pixel point, < >>And the target color information of the first face surface point corresponding to the face pixel point is obtained.
In a specific implementation, the computer device may calculate the mask loss using the following formula:
wherein P is a non-face pixel point,representing the number of non-face pixels on the image sample,/->The probability label corresponding to the non-face pixel point can be 0,/for the probability label corresponding to the non-face pixel point>The face probability of the second face surface point corresponding to the non-face pixel point.
For each target radiation ray, the computer equipment can acquire the distance between each sampling point and the virtual shooting point on the current target radiation ray, multiply the distance between the sampling point and the virtual shooting point by the face probability corresponding to the sampling point for each sampling point on the current target radiation ray to obtain the multiplication result corresponding to the sampling point, add the multiplication results corresponding to all the sampling points on the current target radiation ray to obtain the addition result, and divide the addition result by the sum of the distances between all the sampling points and the virtual shooting points on the target radiation ray to obtain the face probability of the face surface point corresponding to the current target radiation ray.
In the above embodiment, a specific mode of calculating model loss is provided, first, face segmentation processing is performed on an image sample to obtain face pixel points and non-face pixel points; then, a first face surface point corresponding to the face pixel point is obtained, and rendering loss is determined according to the pixel value of the face pixel point and the target color information of the first face surface point; acquiring a second face surface point corresponding to the non-face pixel point, and determining mask loss according to the probability label corresponding to the non-face pixel point and the face probability of the second face surface point; finally, model loss is calculated based on the rendering loss and the mask loss. The rendering loss can ensure that the target color information of the face surface points calculated in the actual application process is closer to the pixel values of the corresponding pixel points on the original image, so that the rendered three-dimensional face model is more vivid, and the mask loss can ensure that the face probability output by the neural radiation field model in the actual application process is more accurate.
In some embodiments, referring to fig. 6, the object may input a segment of face video to the computer device, and the computer device may use each video frame in the face video as an image sample, and perform the following processing on each video frame: the computer equipment obtains shooting parameters corresponding to the current video frame, determines virtual shooting points and virtual image planes of the current video frame in a three-dimensional space based on the shooting parameters, constructs a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determines target radiation rays based on the radiation rays, and respectively samples on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray, one of which is shown in fig. 5 Sample point +.>Carrying out standardization treatment to obtain a standardized sampling point +.>Normalized sample point +.>The three-dimensional coordinates of the target object in the current video frame, the face gesture and the expression parameter are input into a nerve radiation field model to be trained, and a standardized sampling point +.>Face probability belonging to face surface points +.>And high-level features, namely, facial gestures, expression parameters and standardized sampling points of a target object in the current video frame are added>Three-dimensional coordinates of (2) face probability->And inputting the high-level features into a neural rendering model to be trained to obtain a standardized sampling point ++>Color information of (3); for each target radiation ray, determining a face surface point corresponding to the current target radiation ray based on the face probability that each sampling point on the current target radiation ray corresponds to a standardized sampling point belonging to the face surface point, and determining target color information of the face surface point corresponding to the current target radiation ray based on the color information of each sampling point on the current target radiation ray; the mask loss can be calculated based on the face probability output by the neural radiation field model to be trained, the rendering loss is calculated based on the color information output by the neural rendering model to be trained, the model loss is calculated based on the rendering loss and the mask loss, the parameters of the neural radiation field model to be trained and the neural rendering model are adjusted based on the model loss, and the neural radiation field model to be trained and the neural rendering model are stopped until the stopping condition of model training is met, so that the trained neural radiation field model and neural rendering model are obtained.
In some embodiments, referring to fig. 7, after obtaining a neural radiation field model and a neural rendering model after training is completed, an object may input a video to be processed to a computer device, for each frame of image in the video to be processed, construct a target radiation ray based on a shooting parameter corresponding to the image, sample on each target radiation ray to obtain a plurality of sampling points on each target radiation ray, perform standardization processing on each sampling point to obtain a standardized sampling point, input three-dimensional coordinates, a face pose and an expression parameter of the standardized sampling point into the neural radiation field model to perform face prediction to obtain a face probability and a high-level feature of the standardized sampling point belonging to the face surface point, and input the three-dimensional coordinates, the face probability and the high-level feature of the face pose, the expression parameter, and the standardized sampling point into the neural rendering model to be trained to obtain color information of the standardized sampling point. For each target radiation ray, determining a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray, and determining target color information of the face surface point corresponding to the current radiation ray based on the color information corresponding to each sampling point on the current radiation ray; and constructing a three-dimensional face model based on the face surface points corresponding to the target radiation rays and the target color information of the face surface points.
It should be understood that, although the steps in the flowcharts related to the above embodiments are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.
Based on the same inventive concept, the embodiment of the application also provides an image reconstruction device for realizing the image reconstruction method. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitation in the embodiments of the image reconstruction apparatus provided in the following may be referred to the limitation of the image reconstruction method hereinabove, and will not be repeated here.
In one embodiment, as shown in fig. 8, there is provided an image reconstruction apparatus including:
the acquiring module 801 is configured to acquire shooting parameters corresponding to an image to be processed, determine a virtual shooting point and a virtual image plane of the image to be processed in a three-dimensional space based on the shooting parameters, construct a plurality of radiation rays based on the virtual shooting point and the virtual image plane, determine target radiation rays based on the radiation rays, and sample each target radiation ray based on the virtual image plane to obtain a plurality of sampling points on each target radiation ray;
the extraction module 802 is configured to extract a face pose and expression parameters of a target object in an image to be processed;
the nerve radiation processing module 803 is configured to perform nerve radiation processing on the three-dimensional coordinates, the face pose and the expression parameters of the current sampling point for each sampling point, so as to obtain a face probability that the current sampling point belongs to a face surface point;
the construction module 804 is configured to determine, for each target radiation ray, a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray, and construct a three-dimensional face model based on the face surface points corresponding to each target radiation ray.
In some embodiments, the obtaining module 801 is specifically configured to determine an optical axis ray based on the virtual shooting point and a center point of the virtual image plane; translating the virtual image plane along the optical axis ray to a preset distance in a direction close to the virtual shooting point to obtain a low-beam image plane, and translating the virtual image plane along the optical axis ray to a preset distance in a direction far away from the virtual shooting point to obtain a high-beam image plane; and aiming at each target radiation ray, taking the intersection point of the current target radiation ray and the low beam image plane as a sampling starting point, taking the intersection point of the current target radiation ray and the high beam image plane as a sampling ending point, and sampling between the sampling starting point and the sampling ending point on the current target radiation ray to obtain a plurality of sampling points on the current target radiation ray.
In some embodiments, the extracting module 802 is specifically configured to input the image to be processed into a face feature extracting model, take a face pose output by the face feature extracting model as a face pose of a target object in the image to be processed, and take an expression parameter output by the face feature extracting model as an expression parameter of the target object in the image to be processed; the face feature extraction model is obtained by training based on face sample data, wherein the face sample data comprises a face image sample, and a labeling gesture and a labeling expression parameter of a face in the face image sample.
In some embodiments, the expression parameters include expression deformation parameters and shape deformation parameters, and the image reconstruction device further includes a standardization module, configured to perform fusion processing on the three-dimensional coordinates of the face pose and the current sampling point for each sampling point, so as to obtain a pose standardization result; carrying out fusion processing on the expression deformation parameters and the expression deformation network parameters to obtain an expression deformation standardization result; performing fusion processing on the shape deformation parameters and the shape deformation network parameters to obtain a shape deformation standardization result; carrying out aggregation treatment on the gesture standardization result, the expression deformation standardization result and the shape deformation standardization result to obtain a standardization sampling point corresponding to the current sampling point; the nerve radiation processing module 803 is specifically configured to perform nerve radiation processing on three-dimensional coordinates, face pose and expression parameters of the current standardized sampling point for each standardized sampling point, so as to obtain a face probability that the current standardized sampling point belongs to a face surface point.
In some embodiments, the nerve radiation processing module 803 is specifically configured to input three-dimensional coordinates, a face pose, and expression parameters of a current sampling point into the nerve radiation field model, and use a probability output by the nerve radiation field model as a face probability that the current sampling point belongs to a face surface point; the nerve radiation field model is used for predicting whether the sampling points belong to face surface points.
In some embodiments, the constructing module 804 is further configured to calculate a normal gradient of the face probability at the current sampling point; inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinate of the current sampling point, the face probability and the high-level characteristic into the trained neural rendering model to obtain the color information of the current sampling point, wherein the high-level characteristic is output after the neural radiation field model processes the three-dimensional coordinate, the face gesture and the expression parameter of the current sampling point input into the neural rendering model; aiming at each target radiation ray, determining target color information of a face surface point corresponding to the current radiation ray based on the color information corresponding to each sampling point on the current radiation ray; and rendering the three-dimensional face model based on the target color information of each face surface point to obtain a rendered three-dimensional face model.
In some embodiments, the constructing module 804 is specifically configured to obtain distances between each sampling point and the virtual shooting point on the current target radiation ray; the distances between each sampling point and the virtual shooting point and the face probabilities corresponding to each sampling point are subjected to statistical processing to obtain the surface point distances; and determining the face surface point corresponding to the current target radiation ray based on the virtual shooting point and the surface point distance.
In some embodiments, the obtaining module 801 is further configured to respond to the special effect processing request, and display image input reminding information; receiving a target image input by an object based on image input reminding information, and taking the target image as an image to be processed; taking shooting parameters corresponding to the target image as shooting parameters corresponding to the image to be processed; and acquiring special effect processing parameters corresponding to the special effect processing request, and rendering the three-dimensional face model based on the special effect processing parameters to obtain a special effect image matched with the special effect processing request.
The respective modules in the above-described image reconstruction apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.
Based on the same inventive concept, the embodiment of the application also provides a processing device for reconstructing a model, which is used for realizing the processing method for reconstructing the model. The implementation of the solution provided by the apparatus is similar to the implementation described in the above method, so the specific limitation in the embodiments of the processing apparatus for reconstructing one or more models provided below may be referred to the limitation of the processing method for reconstructing a model hereinabove, and will not be described herein.
In one embodiment, as shown in fig. 9, there is provided a processing apparatus for reconstructing a model, including:
the acquiring module 901 is configured to acquire shooting parameters corresponding to an image sample, determine a virtual shooting point and a virtual image plane of the image sample in a three-dimensional space based on the shooting parameters, construct a plurality of radiation rays based on the virtual shooting point and the virtual image plane, determine target radiation rays based on the radiation rays, and sample each target radiation ray based on the virtual image plane to obtain a plurality of sampling points on each target radiation ray;
the extracting module 902 is configured to extract a face pose and expression parameters of a target object in the image sample;
the neural radiation field module 903 is configured to input, for each sampling point, three-dimensional coordinates, a face pose, and expression parameters of the current sampling point to a neural radiation field model to be trained, so as to obtain a face probability and a high-level feature that the current sampling point belongs to a face surface point; inputting the face gesture, the expression parameters, the three-dimensional coordinates of the current sampling point, the face probability and the high-level features into a neural rendering model to be trained to obtain the color information of the current sampling point;
the surface point determining module 904 is configured to determine, for each target radiation ray, a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray, and determine target color information of the face surface point based on the color information corresponding to each sampling point on the current target radiation ray;
A loss calculation module 905, configured to calculate model loss based on the target color information and the image samples of each face surface point;
and the parameter adjustment module 906 is used for performing parameter adjustment on the neural radiation field model to be trained and the neural rendering model based on model loss until stopping when the model training stopping condition is met, so as to obtain the neural radiation field model and the neural rendering model after training. The trained neural radiation field model and the neural rendering model are used for constructing a three-dimensional face model.
In some embodiments, the neural radiation field module 903 is further configured to calculate a normal gradient of the face probability at the current sampling point; inputting the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level features into a neural rendering model to be trained to obtain the color information of the current sampling point, wherein the method comprises the following steps: and inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinate of the current sampling point, the face probability and the high-level characteristic into a neural rendering model to be trained, and obtaining the color information of the current sampling point.
In some embodiments, the surface point determining module 904 is specifically configured to, for each sampling point on the current radiation ray, perform fusion processing on color information and face probability corresponding to the current sampling point, to obtain a fusion result corresponding to the current sampling point; performing aggregation treatment on fusion results corresponding to each sampling point on the current radiation ray to obtain a color aggregation result, and performing aggregation treatment on face probabilities corresponding to each sampling point on the current radiation ray to obtain a probability aggregation result; and determining target color information of the face surface points based on the color aggregation result and the probability aggregation result.
In some embodiments, the face surface points and the pixel points on the image sample are in one-to-one correspondence, and the loss calculation module 905 is specifically configured to perform face segmentation processing on the image sample to obtain face pixel points and non-face pixel points; acquiring a first face surface point corresponding to a face pixel point, and determining rendering loss according to the pixel value of the face pixel point and target color information of the first face surface point; acquiring a second face surface point corresponding to the non-face pixel point, and determining mask loss according to the probability label corresponding to the non-face pixel point and the face probability of the second face surface point; model loss is calculated based on the rendering loss and the mask loss.
In one embodiment, a computer device is provided, which may be a server or a terminal, and the internal structure thereof may be as shown in fig. 10. The computer device includes a processor, a memory, an Input/Output interface (I/O) and a communication interface. The processor, the memory and the input/output interface are connected through a system bus, and the communication interface is connected to the system bus through the input/output interface. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer equipment is used for storing data such as shooting parameters, facial gestures, expression parameters and the like. The input/output interface of the computer device is used to exchange information between the processor and the external device. The communication interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement an image reconstruction method or a processing method of reconstructing a model.
It will be appreciated by those skilled in the art that the structure shown in fig. 10 is merely a block diagram of some of the structures associated with the present application and is not limiting of the computer device to which the present application may be applied, and that a particular computer device may include more or fewer components than shown, or may combine certain components, or have a different arrangement of components.
In an embodiment, there is also provided a computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method embodiments described above when the computer program is executed.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored which, when executed by a processor, carries out the steps of the method embodiments described above.
In an embodiment, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the steps of the method embodiments described above.
It should be noted that, the object information (including, but not limited to, object device information, object personal information, etc.) and the data (including, but not limited to, data for analysis, stored data, presented data, etc.) related to the present application are both information and data authorized by the object or sufficiently authorized by each party, and the collection, use, and processing of the related data are required to comply with the related laws and regulations and standards of the related countries and regions.
Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in the various embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magnetic random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (Phase Change Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in the form of a variety of forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), and the like. The databases referred to in the various embodiments provided herein may include at least one of relational databases and non-relational databases. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processors referred to in the embodiments provided herein may be general purpose processors, central processing units, graphics processors, digital signal processors, programmable logic units, quantum computing-based data processing logic units, etc., without being limited thereto.
The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.
The above examples only represent a few embodiments of the present application, which are described in more detail and are not to be construed as limiting the scope of the present application. It should be noted that it would be apparent to those skilled in the art that various modifications and improvements could be made without departing from the spirit of the present application, which would be within the scope of the present application. Accordingly, the scope of protection of the present application shall be subject to the appended claims.
Claims (26)
1. A method of image reconstruction, the method comprising:
acquiring shooting parameters corresponding to an image to be processed, determining virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray;
Extracting face gestures and expression parameters of a target object in the image to be processed;
aiming at each sampling point, performing nerve radiation treatment on the three-dimensional coordinates of the current sampling point, the face gesture and the expression parameters to obtain the face probability that the current sampling point belongs to the face surface point;
for each target radiation ray, face surface points corresponding to the current target radiation ray are determined based on the face probability corresponding to each sampling point on the current target radiation ray, and a three-dimensional face model is constructed based on the face surface points corresponding to each target radiation ray.
2. The method according to claim 1, wherein the sampling on each target radiation ray based on the virtual image plane to obtain a plurality of sampling points on each target radiation ray includes:
determining an optical axis ray based on the virtual photographing point and a center point of the virtual image plane;
translating the virtual image plane along the optical axis ray to a preset distance in a direction close to the virtual shooting point to obtain a low-beam image plane, and translating the virtual image plane along the optical axis ray to a preset distance in a direction far away from the virtual shooting point to obtain a high-beam image plane;
And aiming at each target radiation ray, taking the intersection point of the current target radiation ray and the low beam image plane as a sampling starting point, taking the intersection point of the current target radiation ray and the high beam image plane as a sampling ending point, and sampling between the sampling starting point and the sampling ending point on the current target radiation ray to obtain a plurality of sampling points on the current target radiation ray.
3. The method according to claim 1, wherein the extracting facial pose and expression parameters of the target object in the image to be processed includes:
inputting the image to be processed into a face feature extraction model, taking the face gesture output by the face feature extraction model as the face gesture of a target object in the image to be processed, and taking the expression parameter output by the face feature extraction model as the expression parameter of the target object in the image to be processed; the face feature extraction model is obtained by training based on face sample data, wherein the face sample data comprises a face image sample, and a labeling gesture and a labeling expression parameter of a face in the face image sample.
4. The method of claim 1, wherein the expression parameters include an expression deformation parameter and a shape deformation parameter, the method further comprising:
For each sampling point, carrying out fusion processing on the human face gesture and the three-dimensional coordinates of the current sampling point to obtain a gesture standardization result; carrying out fusion processing on the expression deformation parameters and the expression deformation network parameters to obtain an expression deformation standardization result; performing fusion processing on the shape deformation parameters and the shape deformation network parameters to obtain a shape deformation standardization result;
performing aggregation treatment on the gesture standardization result, the expression deformation standardization result and the shape deformation standardization result to obtain a standardization sampling point corresponding to a current sampling point;
the step of performing nerve radiation processing on the three-dimensional coordinates of the current sampling point, the face gesture and the expression parameter aiming at each sampling point to obtain the face probability that the current sampling point belongs to the face surface point comprises the following steps:
and aiming at each standardized sampling point, performing nerve radiation processing on the three-dimensional coordinates of the current standardized sampling point, the face gesture and the expression parameter to obtain the face probability that the current standardized sampling point belongs to the face surface point.
5. The method according to claim 1, wherein the performing the neural radiation processing on the three-dimensional coordinates of the current sampling point, the face pose and the expression parameter to obtain a face probability that the current sampling point belongs to a face surface point includes:
Inputting the three-dimensional coordinates of the current sampling point, the face gesture and the expression parameters into a nerve radiation field model, and taking the probability output by the nerve radiation field model as the face probability that the current sampling point belongs to the face surface point; the nerve radiation field model is used for predicting whether the sampling points belong to face surface points.
6. The method according to claim 1, wherein the method further comprises:
solving a normal gradient of the face probability at the current sampling point;
inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level characteristic into a trained neural rendering model to obtain color information of the current sampling point, wherein the high-level characteristic is output after the neural radiation field model processes the three-dimensional coordinates of the current sampling point, the face gesture and the expression parameter which are input into the neural rendering model;
for each target radiation ray, determining target color information of a face surface point corresponding to the current target radiation ray based on the color information corresponding to each sampling point on the current target radiation ray;
and rendering the three-dimensional face model based on the target color information of each face surface point to obtain a rendered three-dimensional face model.
7. The method according to claim 1, wherein the determining the face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray includes:
obtaining the distance between each sampling point and the virtual shooting point on the current target radiation ray;
the distances between each sampling point and the virtual shooting point and the face probability corresponding to each sampling point are subjected to statistical processing to obtain a surface point distance;
and determining the face surface point corresponding to the current target radiation ray based on the distance between the virtual shooting point and the surface point.
8. The method according to claim 1, wherein the acquiring the shooting parameters corresponding to the image to be processed includes:
responding to the special effect processing request, and displaying the image input reminding information;
receiving a target image input by an object based on the image input reminding information, and taking the target image as the image to be processed; taking shooting parameters corresponding to the target image as shooting parameters corresponding to the image to be processed;
the method further comprises the steps of:
and acquiring special effect processing parameters corresponding to the special effect processing request, and rendering the three-dimensional face model based on the special effect processing parameters to obtain a special effect image matched with the special effect processing request.
9. A method of processing a reconstructed model, the method comprising:
acquiring shooting parameters corresponding to an image sample, determining virtual shooting points and virtual image planes of the image sample in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray;
extracting face gestures and expression parameters of a target object in the image sample;
inputting the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point to a neural radiation field model to be trained aiming at each sampling point to obtain the face probability and high-level characteristics of the current sampling point belonging to the face surface point; inputting the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level characteristic into a neural rendering model to be trained to obtain color information of the current sampling point;
for each target radiation ray, determining a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray, and determining target color information of the face surface point based on the color information corresponding to each sampling point on the current target radiation ray;
Calculating model loss based on the target color information of each face surface point and the image sample;
based on the model loss, carrying out parameter adjustment on the nerve radiation field model to be trained and the nerve rendering model until stopping when the model training stopping condition is met, and obtaining a trained nerve radiation field model and a nerve rendering model; the trained neural radiation field model and the neural rendering model are used for constructing a three-dimensional face model.
10. The method according to claim 9, wherein the method further comprises:
solving a normal gradient of the face probability at the current sampling point;
the step of inputting the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level feature to a neural rendering model to be trained to obtain color information of the current sampling point, comprises the following steps:
and inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level characteristic into a neural rendering model to be trained to obtain the color information of the current sampling point.
11. The method according to claim 9, wherein determining the target color information of the face surface point based on the color information corresponding to each sampling point on the current target radiation ray includes:
Aiming at each sampling point on the current target radiation ray, carrying out fusion processing on color information and face probability corresponding to the current sampling point to obtain a fusion result corresponding to the current sampling point;
performing aggregation treatment on fusion results corresponding to each sampling point on the current target radiation ray to obtain a color aggregation result, and performing aggregation treatment on face probabilities corresponding to each sampling point on the current target radiation ray to obtain a probability aggregation result;
and determining target color information of the face surface point based on the color aggregation result and the probability aggregation result.
12. The method of claim 9, wherein each face surface point corresponds to a pixel point on the image sample one-to-one, and wherein calculating model loss based on the target color information for each face surface point and the image sample comprises:
carrying out face segmentation processing on the image sample to obtain face pixel points and non-face pixel points;
acquiring a first face surface point corresponding to the face pixel point, and determining rendering loss according to the pixel value of the face pixel point and target color information of the first face surface point;
acquiring a second face surface point corresponding to the non-face pixel point, and determining mask loss according to a probability label corresponding to the non-face pixel point and the face probability of the second face surface point;
Based on the rendering loss and the mask loss, a model loss is calculated.
13. An image reconstruction apparatus, the apparatus comprising:
the device comprises an acquisition module, a sampling module and a sampling module, wherein the acquisition module is used for acquiring shooting parameters corresponding to an image to be processed, determining virtual shooting points and virtual image planes of the image to be processed in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting points and the virtual image planes, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image planes to obtain a plurality of sampling points on each target radiation ray;
the extraction module is used for extracting the facial pose and expression parameters of the target object in the image to be processed;
the nerve radiation processing module is used for carrying out nerve radiation processing on the three-dimensional coordinates, the face gesture and the expression parameters of the current sampling point aiming at each sampling point to obtain the face probability that the current sampling point belongs to the face surface point;
the construction module is used for determining face surface points corresponding to the current target radiation rays based on the face probabilities corresponding to the sampling points on the current target radiation rays for each target radiation ray, and constructing a three-dimensional face model based on the face surface points corresponding to the target radiation rays.
14. The apparatus of claim 13, wherein the obtaining module is specifically configured to:
determining an optical axis ray based on the virtual photographing point and a center point of the virtual image plane;
translating the virtual image plane along the optical axis ray to a preset distance in a direction close to the virtual shooting point to obtain a low-beam image plane, and translating the virtual image plane along the optical axis ray to a preset distance in a direction far away from the virtual shooting point to obtain a high-beam image plane;
and aiming at each target radiation ray, taking the intersection point of the current target radiation ray and the low beam image plane as a sampling starting point, taking the intersection point of the current target radiation ray and the high beam image plane as a sampling ending point, and sampling between the sampling starting point and the sampling ending point on the current target radiation ray to obtain a plurality of sampling points on the current target radiation ray.
15. The apparatus of claim 13, wherein the extraction module is specifically configured to:
inputting the image to be processed into a face feature extraction model, taking the face gesture output by the face feature extraction model as the face gesture of a target object in the image to be processed, and taking the expression parameter output by the face feature extraction model as the expression parameter of the target object in the image to be processed; the face feature extraction model is obtained by training based on face sample data, wherein the face sample data comprises a face image sample, and a labeling gesture and a labeling expression parameter of a face in the face image sample.
16. The apparatus of claim 13, wherein the expression parameters include an expression deformation parameter and a shape deformation parameter, the apparatus further comprising a normalization module for:
for each sampling point, carrying out fusion processing on the human face gesture and the three-dimensional coordinates of the current sampling point to obtain a gesture standardization result; carrying out fusion processing on the expression deformation parameters and the expression deformation network parameters to obtain an expression deformation standardization result; performing fusion processing on the shape deformation parameters and the shape deformation network parameters to obtain a shape deformation standardization result;
performing aggregation treatment on the gesture standardization result, the expression deformation standardization result and the shape deformation standardization result to obtain a standardization sampling point corresponding to a current sampling point;
the nerve radiation treatment module is specifically used for:
and aiming at each standardized sampling point, performing nerve radiation processing on the three-dimensional coordinates of the current standardized sampling point, the face gesture and the expression parameter to obtain the face probability that the current standardized sampling point belongs to the face surface point.
17. The apparatus according to claim 13, wherein the nerve radiation treatment module is specifically configured to:
Inputting the three-dimensional coordinates of the current sampling point, the face gesture and the expression parameters into a nerve radiation field model, and taking the probability output by the nerve radiation field model as the face probability that the current sampling point belongs to the face surface point; the nerve radiation field model is used for predicting whether the sampling points belong to face surface points.
18. The apparatus of claim 13, wherein the build module is further to:
solving a normal gradient of the face probability at the current sampling point;
inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level characteristic into a trained neural rendering model to obtain color information of the current sampling point, wherein the high-level characteristic is output after the neural radiation field model processes the three-dimensional coordinates of the current sampling point, the face gesture and the expression parameter which are input into the neural rendering model;
for each target radiation ray, determining target color information of a face surface point corresponding to the current target radiation ray based on the color information corresponding to each sampling point on the current target radiation ray;
and rendering the three-dimensional face model based on the target color information of each face surface point to obtain a rendered three-dimensional face model.
19. The apparatus of claim 13, wherein the construction module is specifically configured to:
obtaining the distance between each sampling point and the virtual shooting point on the current target radiation ray;
the distances between each sampling point and the virtual shooting point and the face probability corresponding to each sampling point are subjected to statistical processing to obtain a surface point distance;
and determining the face surface point corresponding to the current target radiation ray based on the distance between the virtual shooting point and the surface point.
20. The apparatus of claim 13, wherein the acquisition module is further configured to:
responding to the special effect processing request, and displaying the image input reminding information;
receiving a target image input by an object based on the image input reminding information, and taking the target image as the image to be processed; taking shooting parameters corresponding to the target image as shooting parameters corresponding to the image to be processed;
and acquiring special effect processing parameters corresponding to the special effect processing request, and rendering the three-dimensional face model based on the special effect processing parameters to obtain a special effect image matched with the special effect processing request.
21. A processing apparatus for reconstructing a model, the apparatus comprising:
The acquisition module is used for acquiring shooting parameters corresponding to an image sample, determining a virtual shooting point and a virtual image plane of the image sample in a three-dimensional space based on the shooting parameters, constructing a plurality of radiation rays based on the virtual shooting point and the virtual image plane, determining target radiation rays based on the radiation rays, and respectively sampling on each target radiation ray based on the virtual image plane to obtain a plurality of sampling points on each target radiation ray;
the extraction module is used for extracting the facial pose and expression parameters of the target object in the image sample;
the neural radiation field module is used for inputting the three-dimensional coordinates, the face gestures and the expression parameters of the current sampling point to a neural radiation field model to be trained aiming at each sampling point to obtain the face probability and the high-level characteristics of the current sampling point belonging to the face surface point; inputting the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level characteristic into a neural rendering model to be trained to obtain color information of the current sampling point;
the surface point determining module is used for determining a face surface point corresponding to the current target radiation ray based on the face probability corresponding to each sampling point on the current target radiation ray for each target radiation ray, and determining target color information of the face surface point based on the color information corresponding to each sampling point on the current target radiation ray;
The loss calculation module is used for calculating model loss based on the target color information of each face surface point and the image sample;
the parameter adjustment module is used for carrying out parameter adjustment on the nerve radiation field model to be trained and the nerve rendering model based on the model loss until the model training stopping condition is met, so as to obtain a trained nerve radiation field model and a nerve rendering model; the trained neural radiation field model and the neural rendering model are used for constructing a three-dimensional face model.
22. The apparatus of claim 21, wherein the neural radiation field module is further configured to:
solving a normal gradient of the face probability at the current sampling point;
the step of inputting the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level feature to a neural rendering model to be trained to obtain color information of the current sampling point, comprises the following steps:
and inputting the normal gradient, the face gesture, the expression parameter, the three-dimensional coordinates of the current sampling point, the face probability and the high-level characteristic into a neural rendering model to be trained to obtain the color information of the current sampling point.
23. The apparatus of claim 21, wherein the surface point determination module is specifically configured to:
aiming at each sampling point on the current target radiation ray, carrying out fusion processing on color information and face probability corresponding to the current sampling point to obtain a fusion result corresponding to the current sampling point;
performing aggregation treatment on fusion results corresponding to each sampling point on the current target radiation ray to obtain a color aggregation result, and performing aggregation treatment on face probabilities corresponding to each sampling point on the current target radiation ray to obtain a probability aggregation result;
and determining target color information of the face surface point based on the color aggregation result and the probability aggregation result.
24. The apparatus of claim 21, wherein each face surface point corresponds to a pixel point on the image sample one-to-one, and the loss calculation module is specifically configured to:
carrying out face segmentation processing on the image sample to obtain face pixel points and non-face pixel points;
acquiring a first face surface point corresponding to the face pixel point, and determining rendering loss according to the pixel value of the face pixel point and target color information of the first face surface point;
Acquiring a second face surface point corresponding to the non-face pixel point, and determining mask loss according to a probability label corresponding to the non-face pixel point and the face probability of the second face surface point;
based on the rendering loss and the mask loss, a model loss is calculated.
25. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 12 when the computer program is executed.
26. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 12.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311518981.4A CN117237547B (en) | 2023-11-15 | 2023-11-15 | Image reconstruction method, reconstruction model processing method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311518981.4A CN117237547B (en) | 2023-11-15 | 2023-11-15 | Image reconstruction method, reconstruction model processing method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117237547A CN117237547A (en) | 2023-12-15 |
| CN117237547B true CN117237547B (en) | 2024-03-01 |
Family
ID=89084756
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311518981.4A Active CN117237547B (en) | 2023-11-15 | 2023-11-15 | Image reconstruction method, reconstruction model processing method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117237547B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118334265B (en) * | 2024-03-11 | 2024-11-05 | 广州极点三维信息科技有限公司 | Three-dimensional model modeling method and device, electronic equipment and storage medium |
| CN117953165B (en) * | 2024-03-26 | 2024-06-18 | 合肥工业大学 | New human face view synthesis method and system based on nerve radiation field |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111325823A (en) * | 2020-02-05 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Method, device and equipment for acquiring face texture image and storage medium |
| CN112887698A (en) * | 2021-02-04 | 2021-06-01 | 中国科学技术大学 | High-quality face voice driving method based on nerve radiation field |
| CN113538659A (en) * | 2021-07-05 | 2021-10-22 | 广州虎牙科技有限公司 | Image generation method and device, storage medium and equipment |
| CN113793408A (en) * | 2021-09-15 | 2021-12-14 | 宿迁硅基智能科技有限公司 | Real-time audio-driven face generation method and device and server |
| CN113822969A (en) * | 2021-09-15 | 2021-12-21 | 宿迁硅基智能科技有限公司 | Method, device and server for training nerve radiation field model and face generation |
| CN115330947A (en) * | 2022-08-12 | 2022-11-11 | 百果园技术(新加坡)有限公司 | Three-dimensional face reconstruction method and device, equipment, medium and product thereof |
| CN115953513A (en) * | 2022-12-30 | 2023-04-11 | 北京元起点信息科技有限公司 | Method, device, equipment and medium for reconstructing drivable three-dimensional human head model |
| CN116894911A (en) * | 2023-03-28 | 2023-10-17 | 网易（杭州）网络有限公司 | Three-dimensional reconstruction method, three-dimensional reconstruction device, electronic equipment and readable storage medium |
| CN116912393A (en) * | 2023-07-28 | 2023-10-20 | 咪咕文化科技有限公司 | Face reconstruction method and device, electronic equipment and readable storage medium |
| CN116959058A (en) * | 2023-02-27 | 2023-10-27 | 腾讯科技（深圳）有限公司 | Three-dimensional face driving method and related device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111709288B (en) * | 2020-05-15 | 2022-03-01 | 北京百度网讯科技有限公司 | Face key point detection method and device and electronic equipment |
-
2023
- 2023-11-15 CN CN202311518981.4A patent/CN117237547B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111325823A (en) * | 2020-02-05 | 2020-06-23 | 腾讯科技（深圳）有限公司 | Method, device and equipment for acquiring face texture image and storage medium |
| CN112887698A (en) * | 2021-02-04 | 2021-06-01 | 中国科学技术大学 | High-quality face voice driving method based on nerve radiation field |
| CN113538659A (en) * | 2021-07-05 | 2021-10-22 | 广州虎牙科技有限公司 | Image generation method and device, storage medium and equipment |
| CN113793408A (en) * | 2021-09-15 | 2021-12-14 | 宿迁硅基智能科技有限公司 | Real-time audio-driven face generation method and device and server |
| CN113822969A (en) * | 2021-09-15 | 2021-12-21 | 宿迁硅基智能科技有限公司 | Method, device and server for training nerve radiation field model and face generation |
| CN115330947A (en) * | 2022-08-12 | 2022-11-11 | 百果园技术(新加坡)有限公司 | Three-dimensional face reconstruction method and device, equipment, medium and product thereof |
| CN115953513A (en) * | 2022-12-30 | 2023-04-11 | 北京元起点信息科技有限公司 | Method, device, equipment and medium for reconstructing drivable three-dimensional human head model |
| CN116959058A (en) * | 2023-02-27 | 2023-10-27 | 腾讯科技（深圳）有限公司 | Three-dimensional face driving method and related device |
| CN116894911A (en) * | 2023-03-28 | 2023-10-17 | 网易（杭州）网络有限公司 | Three-dimensional reconstruction method, three-dimensional reconstruction device, electronic equipment and readable storage medium |
| CN116912393A (en) * | 2023-07-28 | 2023-10-20 | 咪咕文化科技有限公司 | Face reconstruction method and device, electronic equipment and readable storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 三维人脸重建及其应用;郭玉东;《中国博士学位论文全文数据库信息科技辑》;第1-113页 * |
| 单张图像三维人脸重建方法综述;王静婷 等;《计算机工程与应用》;第59卷(第17期);第1-21页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117237547A (en) | 2023-12-15 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Shi et al. | A facial expression recognition method based on a multibranch cross-connection convolutional neural network | |
| CN109214343B (en) | Method and device for generating face key point detection model | |
| CN117237547B (en) | Image reconstruction method, reconstruction model processing method and device | |
| WO2020103700A1 (en) | Image recognition method based on micro facial expressions, apparatus and related device | |
| CN109948475B (en) | Human body action recognition method based on skeleton features and deep learning | |
| CN111680678B (en) | Target area identification method, device, equipment and readable storage medium | |
| CN111783620A (en) | Expression recognition method, device, equipment and storage medium | |
| CN112085835B (en) | Three-dimensional cartoon face generation method and device, electronic equipment and storage medium | |
| CN111667001B (en) | Target re-identification method, device, computer equipment and storage medium | |
| CN112395979A (en) | Image-based health state identification method, device, equipment and storage medium | |
| US20240119699A1 (en) | Detection of moment of perception | |
| WO2022052782A1 (en) | Image processing method and related device | |
| Wang et al. | Light attention embedding for facial expression recognition | |
| CN115222896B (en) | Three-dimensional reconstruction method, three-dimensional reconstruction device, electronic equipment and computer readable storage medium | |
| CN116012626A (en) | Material matching method, device, equipment and storage medium for building elevation image | |
| Qin et al. | Depth estimation by parameter transfer with a lightweight model for single still images | |
| CN113569809A (en) | Image processing method, device and computer readable storage medium | |
| Zhang et al. | Outdoor illumination estimation via all convolutional neural networks | |
| CN115909255B (en) | Image generation and image segmentation methods, devices, equipment, vehicle-mounted terminal and medium | |
| Wang et al. | Generative model with coordinate metric learning for object recognition based on 3D models | |
| CN117152352A (en) | Image processing method, deep learning model training method and device | |
| CN117037244A (en) | Face security detection method, device, computer equipment and storage medium | |
| CN116048763A (en) | Task processing method and device based on BEV multitasking model framework | |
| CN116343019A (en) | Target detection method for remote sensing image | |
| CN116645700B (en) | Feature extraction model processing method and device and feature extraction method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |