CN111047509B - Image special effect processing method, device and terminal - Google Patents
Image special effect processing method, device and terminal Download PDFInfo
- Publication number
- CN111047509B CN111047509B CN201911300750.XA CN201911300750A CN111047509B CN 111047509 B CN111047509 B CN 111047509B CN 201911300750 A CN201911300750 A CN 201911300750A CN 111047509 B CN111047509 B CN 111047509B
- Authority
- CN
- China
- Prior art keywords
- face
- image
- special effect
- training
- picture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 230000000694 effects Effects 0.000 title claims abstract description 138
- 238000003672 processing method Methods 0.000 title claims abstract description 13
- 230000011218 segmentation Effects 0.000 claims abstract description 52
- 238000000034 method Methods 0.000 claims abstract description 40
- 238000013507 mapping Methods 0.000 claims abstract description 35
- 238000012545 processing Methods 0.000 claims abstract description 28
- 238000012549 training Methods 0.000 claims description 64
- 230000001815 facial effect Effects 0.000 claims description 42
- 238000004590 computer program Methods 0.000 claims description 20
- 125000004122 cyclic group Chemical group 0.000 claims description 19
- 230000005012 migration Effects 0.000 claims description 11
- 238000013508 migration Methods 0.000 claims description 11
- 238000009877 rendering Methods 0.000 claims description 9
- 238000003860 storage Methods 0.000 claims description 8
- 239000000470 constituent Substances 0.000 claims 1
- 230000008569 process Effects 0.000 description 13
- 238000010586 diagram Methods 0.000 description 8
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 5
- 230000008859 change Effects 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 238000004519 manufacturing process Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000009466 transformation Effects 0.000 description 4
- 230000003044 adaptive effect Effects 0.000 description 3
- 238000004891 communication Methods 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000001514 detection method Methods 0.000 description 3
- 238000010606 normalization Methods 0.000 description 3
- 239000000427 antigen Substances 0.000 description 2
- 102000036639 antigens Human genes 0.000 description 2
- 108091007433 antigens Proteins 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 239000005660 Abamectin Substances 0.000 description 1
- RRZXIRBKKLTSOM-XPNPUAGNSA-N avermectin B1a Chemical compound C1=C[C@H](C)[C@@H]([C@@H](C)CC)O[C@]11O[C@H](C\C=C(C)\[C@@H](O[C@@H]2O[C@@H](C)[C@H](O[C@@H]3O[C@@H](C)[C@H](O)[C@@H](OC)C3)[C@@H](OC)C2)[C@@H](C)\C=C\C=C/2[C@]3([C@H](C(=O)O4)C=C(C)[C@@H](O)[C@H]3OC\2)O)C[C@H]4C1 RRZXIRBKKLTSOM-XPNPUAGNSA-N 0.000 description 1
- 239000011324 bead Substances 0.000 description 1
- 239000003086 colorant Substances 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000037308 hair color Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000002156 mixing Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 210000001747 pupil Anatomy 0.000 description 1
- 230000036548 skin texture Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/04—Context-preserving transformations, e.g. by using an importance map
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
- G06T2207/30201—Face
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Processing Or Creating Images (AREA)
- Image Analysis (AREA)
Abstract
The application is applicable to the technical field of image processing, and provides an image special effect processing method, an image special effect processing device and a terminal, wherein the method comprises the following steps: acquiring a face image to be processed and a first face semantic segmentation mask corresponding to the face special effect image; inputting the face image to be processed and the first face semantic segmentation mask into a dense mapping network to generate a preliminary face image; based on the preliminary face image and the face special effect image, a target face image is generated through a circulating generation countermeasure network, so that the efficiency is improved, the cost is reduced, and the large-scale quick realization of the science fiction movie special effect is possible.
Description
Technical Field
The application belongs to the technical field of image processing, and particularly relates to an image special effect processing method, device and terminal.
Background
The current production of science fiction movies requires special effects to achieve the science fiction image of the character, such as the nalmeyer in averda and the green giant in the revenge alliance. These characters are not present in reality and need to be created by special means, typically by computer-generated visual images (computer graphics, CG), creating a specific character model, and then rendering for the final effect by the designer. One character from the pandura star in averda consists of millions of CG patterns. In addition, based on the technology of motion capture or expression capture, an actor is required to wear a tracking suit full of position tracking mark points, and the post-production requires a lot of manpower and time for repeated work.
The method has the advantages that the film similar to the science fiction material is high in difficulty in making, the effect of the human face science fiction conversion is poor, the processing efficiency is extremely low, the cost is extremely high, and the making of the science fiction film special effect cannot be rapidly realized on a large scale.
Disclosure of Invention
The embodiment of the application provides an image special effect processing method, an image special effect processing device and an image special effect processing terminal, which are used for solving the problems that in the prior art, the manufacturing difficulty of a special effect film image is high, the effect of human face science fiction conversion is poor, the processing efficiency is extremely low, the cost is extremely high, and the manufacturing of a science fiction film special effect cannot be rapidly realized on a large scale.
A first aspect of an embodiment of the present application provides an image special effect processing method, including:
Acquiring a face image to be processed and a first face semantic segmentation mask corresponding to the face special effect image; the facial image to be processed is different from the facial features in the facial special effect image;
inputting the face image to be processed and the first face semantic segmentation mask into a dense mapping network to generate a preliminary face image; the preliminary face image is an image obtained by deforming a face in the face image to be processed according to the face effect in the face effect image;
Generating a target face image by circularly generating an countermeasure network based on the preliminary face image and the face special effect image; the target face image is an image obtained by performing special effect rendering on the preliminary face image according to the face special effect image.
A second aspect of an embodiment of the present application provides an image special effect processing apparatus, including:
The first acquisition module is used for acquiring the face image to be processed and a first face semantic segmentation mask corresponding to the face special effect image; the facial image to be processed is different from the facial features in the facial special effect image;
The first generation module is used for inputting the face image to be processed and the first face semantic segmentation mask into a dense mapping network to generate a preliminary face image; the preliminary face image is an image obtained by deforming a face in the face image to be processed according to the face effect in the face effect image;
The second generation module is used for generating a target face image through a cyclic generation countermeasure network based on the preliminary face image and the face special effect image; the target face image is an image obtained by performing special effect rendering on the preliminary face image according to the face special effect image.
A third aspect of an embodiment of the present application provides a terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, the processor implementing the steps of the method according to the first aspect when executing the computer program.
A fourth aspect of the embodiments of the present application provides a computer-readable storage medium storing a computer program which, when executed by a processor, implements the steps of the method according to the first aspect.
A fifth aspect of the application provides a computer program product for causing a terminal to carry out the steps of the method of the first aspect described above when the computer program product is run on the terminal.
From the above, in the embodiment of the application, the face with the facial special effect geometric feature and the original image individual feature is generated by using a pre-trained mask to the face generation network, and then the style migration is realized by using the cyclic countermeasure training, so that the science fiction special effect conforming to the semantic logic is finally realized.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a flowchart of a method for processing special effects of an image according to an embodiment of the present application;
FIG. 2 is a schematic diagram of generating a preliminary face image through a dense mapping network according to an embodiment of the present application;
FIG. 3 is a second flowchart of an image special effect processing method according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a process for performing cyclic countermeasure training in a cyclic generation countermeasure network according to an embodiment of the present application;
fig. 5 is a block diagram of an image special effect processing apparatus according to an embodiment of the present application;
fig. 6 is a block diagram of a terminal according to an embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth such as the particular system architecture, techniques, etc., in order to provide a thorough understanding of the embodiments of the present application. It will be apparent, however, to one skilled in the art that the present application may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It should be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the application herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this specification and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be further understood that the term "and/or" as used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes such combinations.
As used in this specification and the appended claims, the term "if" may be interpreted as "when..once" or "in response to a determination" or "in response to detection" depending on the context. Similarly, the phrase "if a determination" or "if a [ described condition or event ] is detected" may be interpreted in the context of meaning "upon determination" or "in response to determination" or "upon detection of a [ described condition or event ]" or "in response to detection of a [ described condition or event ]".
In particular implementations, the terminals described in embodiments of the application include, but are not limited to, other portable devices such as mobile phones, laptop computers, or tablet computers having a touch-sensitive surface (e.g., a touch screen display and/or a touch pad). It should also be appreciated that in some embodiments, the device is not a portable communication device, but a desktop computer having a touch-sensitive surface (e.g., a touch screen display and/or a touch pad).
In the following discussion, a terminal including a display and a touch sensitive surface is described. However, it should be understood that the terminal may include one or more other physical user interface devices such as a physical keyboard, mouse, and/or joystick.
The terminal supports various applications, such as one or more of the following: drawing applications, presentation applications, word processing applications, website creation applications, disk burning applications, spreadsheet applications, gaming applications, telephony applications, video conferencing applications, email applications, instant messaging applications, workout support applications, photo management applications, digital camera applications, digital video camera applications, web browsing applications, digital music player applications, and/or digital video player applications.
Various applications that may be executed on the terminal may use at least one common physical user interface device such as a touch sensitive surface. One or more functions of the touch-sensitive surface and corresponding information displayed on the terminal may be adjusted and/or changed between applications and/or within the corresponding applications. In this way, the common physical architecture (e.g., touch-sensitive surface) of the terminal may support various applications with user interfaces that are intuitive and transparent to the user.
It should be understood that, the sequence number of each step in this embodiment does not mean the execution sequence, and the execution sequence of each process should be determined by its function and internal logic, and should not limit the implementation process of the embodiment of the present application in any way.
In order to illustrate the technical scheme of the application, the following description is made by specific examples.
Referring to fig. 1, fig. 1 is a flowchart of an image special effect processing method according to an embodiment of the present application. As shown in fig. 1, a method for processing an image special effect, the method comprises the following steps:
step 101, acquiring a face image to be processed and a first face semantic segmentation mask corresponding to the face special effect image.
The face effect image is specifically, for example, an image having a science fiction effect or an image having a national wind effect. In the embodiment of the application, a specific image with an aver image will be exemplified.
Wherein the face image to be processed is different from the facial features in the face effect image.
The facial features may be, but are not limited to, specifically: facial features, facial shape features, facial color features, texture features, facial structure geometric features, and the like.
Step 102, inputting the face image to be processed and the first face semantic segmentation mask into a dense mapping network to generate a preliminary face image.
The preliminary face image is an image obtained by deforming a face in the face image to be processed according to the face effect in the face effect image.
Here, the dense mapping network is specifically a pre-trained dense mapping network. And in combination with the illustration of fig. 2, inputting a human face semantic segmentation mask corresponding to the averda image and a photograph of a normal person (namely a human face image to be processed) into a pre-trained generator in the dense mapping network, and carrying out forward propagation once to generate images conforming to geometric features of the averda human face and individual features of the normal human face, namely preliminary images conforming to geometric features of the face in the face special effect image and individual features of the face in the human face image to be processed. The individual characteristics are, for example, white pupil texture, eye bead color, skin tone, hair color, skin texture, etc.
Specifically, in the dense mapping network, the normalized result of the mean value and the variance of the feature parameters input by each channel of the first face semantic segmentation mask is matched to the mean value and the variance of the feature parameters input by each channel of the face image to be processed through each layer of feature layer in the dense mapping network.
In the process, the input channel of the first face semantic segmentation mask is different from the input channel of the face image to be processed, and the special effect of the first face semantic segmentation mask and the face features in the face image to be processed are mapped through each feature layer in the dense mapping network, so that the geometric features of the face in the face image to be processed are changed.
The part takes a dense mapping network as a framework, and constructs a deformation module for supervising facial feature transformation through a facial semantic segmentation mask, so that facial deformation is generated on a facial image.
And step 103, generating a target face image by circularly generating an countermeasure network based on the preliminary face image and the face special effect image.
The target face image is an image obtained by performing special effect rendering on the preliminary face image according to the face special effect image.
In the step, a cyclic countermeasure network is used to realize style migration among images, so that the face image to be processed after geometric deformation is changed in texture and color, and the face image is utilized in face specialization to achieve the purposes of generating accurate geometric deformation, retaining individual characteristics of the original character and attaching special effect colors.
The part takes a circularly generated countermeasure network as a framework, so that the style migration of the face image is generated.
In the implementation process, the first face semantic segmentation mask corresponding to the face special effect image is used as a proper face intermediate representation layer to perform face transformation with high fidelity, so that the facial features can be transformed semantically or geometrically, and the facial features can be transformed on both the real face and the synthesized face.
Further, the first face semantic segmentation mask is a mask image which is connected by vertexes of different composition parts.
The first facial semantic segmentation mask may be a mesh point diagram obtained by segmenting the facial special effect image into different component parts in advance, where the mesh point diagram includes vertex sets corresponding to the different component parts, the different component parts have vertices corresponding to themselves, and the different component parts are connected through connection points (vertices) at the same position. Based on the segmented face special effect image, a corresponding first face semantic segmentation mask is obtained, and the division of the component parts and the division of the vertexes in the first face semantic segmentation mask are consistent with those in the face special effect image.
Correspondingly, as an optional implementation manner, after generating the target face image by generating the countermeasure network in a circulating manner based on the preliminary face image and the face special effect image, the method further includes:
Receiving operation input for adjusting the position of a target vertex in the first face semantic segmentation mask; and responding to the operation input, and adjusting the position of the target vertex to a target position to obtain the updated first face semantic segmentation mask.
The target vertex is one or more selected from vertices of different composition parts. After the position of the target vertex is adjusted to the target position, the face composition part corresponding to the target vertex is changed, and the change is the change of the position or the change of the shape, so that the first face semantic segmentation mask is updated and changed. The design personnel can manually modify the facial semantic segmentation mask to personalize the required special effect image. The scheme has better autonomous control capability, and can automatically realize the adjustment of the science fiction image by manually modifying the face mask.
In the embodiment of the application, the face with the facial special effect geometric characteristics and the original image individual characteristics is generated by utilizing a pre-trained mask to a face generation network, and then the style migration is realized by using cyclic antigen training, so that the science fiction special effect conforming to semantic logic is finally realized.
The embodiment of the application also provides different implementation modes of the image special effect processing method.
Referring to fig. 3, fig. 3 is a flowchart two of an image special effect processing method according to an embodiment of the present application. As shown in fig. 3, a method for processing an image special effect, the method comprises the following steps:
Step 301, training the dense mapping network based on a set face image and a second face semantic segmentation mask corresponding to the set face image.
And (3) supervising and training a dense mapping network by using a second face semantic segmentation mask and a corresponding face picture, wherein a normalization method of adaptive instance normalization is used, and the average value and the variance of the feature layer of each channel after the face semantic segmentation mask and the corresponding face picture are input are normalized to be matched with the average value and the variance of each channel of the face picture. While mask variations are used from the encoder and alpha blending to enhance training.
The set face image is identical to the face features in the second face semantic segmentation mask.
In training a dense mapping network, model training is performed through paired images, and after training is completed, image structuring (geometric features) change is achieved through unpaired images.
Specifically, during training of the dense mapping network, affine parameters are obtained through training of formula x i,yi＝Encstyle(Ii,Mi). Wherein x i,yi is affine parameters including a face style, enc style is style coding, I i represents an input ith set face image, and M i represents an input ith face semantic segmentation mask.
By the formula:
Matching the face style of x i,yi to z i is implemented, and an adaptive instance normalization process is implemented, where AdaIN represents the adaptive process and z i represents a certain feature layer of the dense mapping network. After the feature layer parameters of the dense mapping network are subtracted by the average value and divided by the variance for standardization, the affine transformation of the face style on each feature layer of the face semantic segmentation mask is realized by combining the affine parameters of the face style, and the geometric feature deformation of the face image to be processed is realized.
Step 302, generating a set number of training face images through the dense mapping network, and obtaining a first picture domain based on the set number of training face images.
The picture field is a picture commonly provided with a certain image style characteristic.
As a specific embodiment, the generating, by the dense mapping network, a set number of training face images includes:
Acquiring preset face images and third face semantic segmentation masks respectively corresponding to the set number of face special effect training images; the facial features in the preset face image and the facial special effect training image are different; the third face semantic segmentation mask is respectively matched with the preset face images to be input into the dense mapping network, and the training face images with the set number are generated; the training face image is an image obtained by deforming a face in the preset face image according to the face effect in the face effect training image.
The dense mapping network is a trained network. The training face image is generated based on training face images with different facial features and face semantic segmentation masks.
Step 303, obtaining a second picture field based on a set number of training special effect images identical to the face special effect of the face special effect image.
The face effect is the same and indicates the effect style is the same, for example, all pictures of the avermectin style.
The picture field is a picture commonly provided with a certain image style characteristic.
Step 304, training the cyclic generation countermeasure network based on the first picture field and the second picture field.
As an optional implementation manner, the training the cyclic generation countermeasure network based on the first picture field and the second picture field includes:
based on the first picture field and the second picture field, the first generator, the first discriminator, the second generator and the second discriminator in the cyclic generation countermeasure network are combined and alternately trained.
The first generator is used for receiving picture input of the first picture domain and outputting a picture imitating the content of the second picture domain; the second generator is a generator for receiving picture input of the second picture domain and outputting pictures imitating the content of the first picture domain; the first discriminator is used for discriminating the picture from the first picture field and the picture which is generated by the second generator in a imitating way; the second discriminator is used for discriminating the picture from the second picture field and the picture generated by the first generator in a imitating way.
The generation result of the intensively mapped network after training can be used as a first picture field X, and the normal averda picture can be used as a second picture field Y to perform cyclic countermeasure training. Referring to fig. 4, G is a generator that accepts input of a picture field X and outputs content imitating a picture field Y; f is a generator which accepts the input of the picture field Y and outputs the content imitating the picture field X. D X is a discriminator that discriminates the pictures actually coming from X and the pictures emulated by generator F; d Y is a discriminator for discriminating the picture actually coming from Y and the picture imitated by generator G; the two groups of generator-discriminant combinations are trained alternately, so that the purpose of style migration is achieved.
The combination in the alternate training may be that the first generator and the second discriminator are combined, the second generator and the first discriminator are combined, and the two groups of devices after the combination are alternately trained based on the first picture field and the second picture field.
In the network, through the unpaired image training generated generator, color or texture rendering, style migration and affine transformation are realized.
Step 305, acquiring a face image to be processed and a first face semantic segmentation mask corresponding to the face special effect image.
The face image to be processed is different from facial features in the face effect image.
The implementation process of this step is the same as that of step 101 in the foregoing embodiment, and will not be repeated here.
Step 306, inputting the face image to be processed and the first face semantic segmentation mask into a dense mapping network to generate a preliminary face image.
The preliminary face image is an image obtained by deforming a face in the face image to be processed according to the face effect in the face effect image.
The implementation process of this step is the same as that of step 102 in the foregoing embodiment, and will not be repeated here.
Step 307, generating a target face image by generating a countermeasure network in a circulating way based on the preliminary face image and the face special effect image.
The target face image is an image obtained by performing special effect rendering on the preliminary face image according to the face special effect image.
The implementation process of this step is the same as that of step 103 in the foregoing embodiment, and will not be repeated here.
In the embodiment of the application, the face with the facial special effect geometric characteristics and the original image individual characteristics is generated by utilizing a pre-trained mask to a face generation network, and then the style migration is realized by using cyclic antigen training, so that the science fiction special effect conforming to semantic logic is finally realized.
Referring to fig. 5, fig. 5 is a block diagram of an image special effect processing apparatus provided in an embodiment of the present application, and for convenience of explanation, only a portion related to the embodiment of the present application is shown.
The image special effect processing apparatus 500 includes:
A first obtaining module 501, configured to obtain a face image to be processed and a first face semantic segmentation mask corresponding to the face special effect image; the facial image to be processed is different from the facial features in the facial special effect image;
A first generating module 502, configured to input the face image to be processed and the first face semantic segmentation mask to a dense mapping network, and generate a preliminary face image; the preliminary face image is an image obtained by deforming a face in the face image to be processed according to the face effect in the face effect image;
A second generating module 503, configured to generate a target face image by circularly generating an countermeasure network based on the preliminary face image and the face special effect image; the target face image is an image obtained by performing special effect rendering on the preliminary face image according to the face special effect image.
The apparatus further comprises:
And the first training module is used for training the dense mapping network based on the set face image and a second face semantic segmentation mask corresponding to the set face image.
The apparatus further comprises:
The second training module is used for generating a set number of training face images through the dense mapping network and obtaining a first picture domain based on the set number of training face images; obtaining a second picture domain based on a set number of training special effect images which are the same as the face special effect of the face special effect image;
training the cyclic generation countermeasure network based on the first picture field and the second picture field.
The second training module is specifically configured to:
Based on the first picture field and the second picture field, performing combined alternate training on a first generator, a first discriminator, a second generator and a second discriminator in the cyclic generation countermeasure network; the first generator is used for receiving picture input of the first picture domain and outputting a picture imitating the content of the second picture domain; the second generator is a generator for receiving picture input of the second picture domain and outputting pictures imitating the content of the first picture domain; the first discriminator is used for discriminating the picture from the first picture field and the picture which is generated by the second generator in a imitating way; the second discriminator is used for discriminating the picture from the second picture field and the picture generated by the first generator in a imitating way.
The second training module is also specifically configured to:
Acquiring preset face images and third face semantic segmentation masks respectively corresponding to the set number of face special effect training images; the facial features in the preset face image and the facial special effect training image are different;
The third face semantic segmentation mask is respectively matched with the preset face images to be input into the dense mapping network, and the training face images with the set number are generated; the training face image is an image obtained by deforming a face in the preset face image according to the face effect in the face effect training image.
The first facial semantic segmentation mask is a mask image formed by connecting vertexes of different component parts.
The apparatus further comprises:
The adjusting module is used for receiving operation input for adjusting the position of the target vertex in the first face semantic segmentation mask; and responding to the operation input, and adjusting the position of the target vertex to a target position to obtain the updated first face semantic segmentation mask.
The image special effect processing device provided by the embodiment of the application can realize each process of the embodiment of the image special effect processing method and achieve the same technical effect, and is not repeated here for avoiding repetition.
Fig. 6 is a block diagram of a terminal according to an embodiment of the present application. As shown in the figure, the terminal 6 of this embodiment includes: at least one processor 60 (only one is shown in fig. 6), a memory 61 and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, the processor 60 implementing the steps in any of the various method embodiments described above when executing the computer program 62.
The terminal 6 may be a computing device such as a desktop computer, a notebook computer, a palm computer, a cloud server, etc. The terminal 6 may include, but is not limited to, a processor 60, a memory 61. It will be appreciated by those skilled in the art that fig. 6 is merely an example of terminal 6 and is not intended to limit terminal 6, and may include more or fewer components than shown, or may combine certain components, or different components, e.g., the terminal may further include an input-output device, a network access device, a bus, etc.
The Processor 60 may be a central processing unit (Central Processing Unit, CPU), other general purpose Processor, digital signal Processor (DIGITAL SIGNAL Processor, DSP), application SPECIFIC INTEGRATED Circuit (ASIC), off-the-shelf Programmable gate array (Field-Programmable GATE ARRAY, FPGA) or other Programmable logic device, discrete gate or transistor logic device, discrete hardware components, or the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 61 may be an internal storage unit of the terminal 6, such as a hard disk or a memory of the terminal 6. The memory 61 may also be an external storage device of the terminal 6, such as a plug-in hard disk, a smart memory card (SMART MEDIA CARD, SMC), a Secure Digital (SD) card, a flash memory card (FLASH CARD) or the like, which are provided on the terminal 6. Further, the memory 61 may also include both an internal storage unit and an external storage device of the terminal 6. The memory 61 is used for storing the computer program and other programs and data required by the terminal. The memory 61 may also be used for temporarily storing data that has been output or is to be output.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-described division of the functional units and modules is illustrated, and in practical application, the above-described functional distribution may be performed by different functional units and modules according to needs, i.e. the internal structure of the apparatus is divided into different functional units or modules to perform all or part of the above-described functions. The functional units and modules in the embodiment may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit, where the integrated units may be implemented in a form of hardware or a form of a software functional unit. In addition, the specific names of the functional units and modules are only for distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working process of the units and modules in the above system may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and in part, not described or illustrated in any particular embodiment, reference is made to the related descriptions of other embodiments.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed terminal and method may be implemented in other manners. For example, the above-described terminal embodiments are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be other manners of division in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection via interfaces, devices or units, which may be in electrical, mechanical or other forms.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated modules/units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the present application may implement all or part of the flow of the method of the above embodiment, or may be implemented by a computer program to instruct related hardware, where the computer program may be stored in a computer readable storage medium, and when the computer program is executed by a processor, the computer program may implement the steps of each of the method embodiments described above. Wherein the computer program comprises computer program code which may be in source code form, object code form, executable file or some intermediate form etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a U disk, a removable hard disk, a magnetic disk, an optical disk, a computer Memory, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), an electrical carrier signal, a telecommunications signal, a software distribution medium, and so forth. It should be noted that the computer readable medium contains content that can be appropriately scaled according to the requirements of jurisdictions in which such content is subject to legislation and patent practice, such as in certain jurisdictions in which such content is subject to legislation and patent practice, the computer readable medium does not include electrical carrier signals and telecommunication signals.
The present application may also be implemented as a computer program product for implementing all or part of the procedures of the methods of the above embodiments, which when run on a terminal causes the terminal to perform the steps of the method embodiments described above.
The above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present application, and are intended to be included in the scope of the present application.
Claims (9)
1. An image special effect processing method, characterized by comprising the following steps:
Acquiring a face image to be processed and a first face semantic segmentation mask corresponding to the face special effect image; the facial image to be processed is different from the facial features in the facial special effect image;
inputting the face image to be processed and the first face semantic segmentation mask into a dense mapping network to generate a preliminary face image; the preliminary face image is an image obtained by deforming a face in the face image to be processed according to the face effect in the face effect image;
Generating a target face image by circularly generating an countermeasure network based on the preliminary face image and the face special effect image; the target face image is an image obtained by performing special effect rendering on the preliminary face image according to the face special effect image, and is a face image obtained after geometric deformation processing and style migration processing; the cyclic generation countermeasure network is used for realizing style migration between the preliminary face image and the face special effect image, so that the preliminary face image is changed in texture color;
The generating a target face image by generating an countermeasure network in a circulating way based on the preliminary face image and the face special effect image further comprises: generating a set number of training face images through the dense mapping network, and obtaining a first picture domain based on the set number of training face images, wherein the training face images are images obtained by deforming faces in preset face images according to face special effects in face special effect training images; obtaining a second picture domain based on a set number of training special effect images which are the same as the face special effect of the face special effect image; training the cyclic generation countermeasure network based on the first picture field and the second picture field.
2. The method for processing an image special effect according to claim 1, wherein before inputting the face image to be processed and the first face semantic segmentation mask to a dense mapping network to generate a preliminary face image, further comprises:
Training the dense mapping network based on a set face image and a second face semantic segmentation mask corresponding to the set face image.
3. The method of claim 1, wherein training the recurring generation countermeasure network based on the first picture field and the second picture field comprises:
Based on the first picture field and the second picture field, performing combined alternate training on a first generator, a first discriminator, a second generator and a second discriminator in the cyclic generation countermeasure network;
The first generator is used for receiving picture input of the first picture domain and outputting a picture imitating the content of the second picture domain; the second generator is a generator for receiving picture input of the second picture domain and outputting pictures imitating the content of the first picture domain; the first discriminator is used for discriminating the picture from the first picture field and the picture which is generated by the second generator in a imitating way; the second discriminator is used for discriminating the picture from the second picture field and the picture generated by the first generator in a imitating way.
4. The method of image special effects processing according to claim 1, wherein the generating a set number of training face images through the dense mapping network includes:
Acquiring the preset face image and third face semantic segmentation masks corresponding to the set number of face special effect training images respectively; the facial features in the preset face image and the facial special effect training image are different;
And respectively matching the third face semantic segmentation masks with the preset face images to input the face images into the dense mapping network, and generating the training face images with the set number.
5. The image special effect processing method according to claim 1, wherein the first face semantic segmentation mask is a mask image in which vertices of different constituent parts are joined.
6. The method according to claim 5, wherein after generating the target face image by generating the countermeasure network in a loop based on the preliminary face image and the face effect image, further comprising:
Receiving operation input for adjusting the position of a target vertex in the first face semantic segmentation mask;
And responding to the operation input, and adjusting the position of the target vertex to a target position to obtain the updated first face semantic segmentation mask.
7. An image special effect processing apparatus, characterized by comprising:
The first acquisition module is used for acquiring the face image to be processed and a first face semantic segmentation mask corresponding to the face special effect image; the facial image to be processed is different from the facial features in the facial special effect image;
The first generation module is used for inputting the face image to be processed and the first face semantic segmentation mask into a dense mapping network to generate a preliminary face image; the preliminary face image is an image obtained by deforming a face in the face image to be processed according to the face effect in the face effect image;
The second generation module is used for generating a target face image through a cyclic generation countermeasure network based on the preliminary face image and the face special effect image; the target face image is an image obtained by performing special effect rendering on the preliminary face image according to the face special effect image, and is a face image obtained after geometric deformation processing and style migration processing; the cyclic generation countermeasure network is used for realizing style migration between the preliminary face image and the face special effect image, so that the preliminary face image is changed in texture color;
The image special effect processing device further comprises a second training module for: generating a set number of training face images through the dense mapping network, and obtaining a first picture domain based on the set number of training face images, wherein the training face images are images obtained by deforming faces in preset face images according to face special effects in face special effect training images; obtaining a second picture domain based on a set number of training special effect images which are the same as the face special effect of the face special effect image; training the cyclic generation countermeasure network based on the first picture field and the second picture field.
8. A terminal comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 6 when the computer program is executed.
9. A computer readable storage medium storing a computer program, characterized in that the computer program when executed by a processor implements the steps of the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911300750.XA CN111047509B (en) | 2019-12-17 | 2019-12-17 | Image special effect processing method, device and terminal |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911300750.XA CN111047509B (en) | 2019-12-17 | 2019-12-17 | Image special effect processing method, device and terminal |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111047509A CN111047509A (en) | 2020-04-21 |
| CN111047509B true CN111047509B (en) | 2024-07-05 |
Family
ID=70236796
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911300750.XA Active CN111047509B (en) | 2019-12-17 | 2019-12-17 | Image special effect processing method, device and terminal |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111047509B (en) |
Families Citing this family (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111275784B (en) * | 2020-01-20 | 2023-06-13 | 北京百度网讯科技有限公司 | Method and device for generating image |
| CN113642359B (en) * | 2020-04-27 | 2023-11-14 | 北京达佳互联信息技术有限公司 | Face image generation method and device, electronic equipment and storage medium |
| CN111833238B (en) * | 2020-06-01 | 2023-07-25 | 北京百度网讯科技有限公司 | Image translation method and device and image translation model training method and device |
| CN111738122B (en) * | 2020-06-12 | 2023-08-22 | Oppo广东移动通信有限公司 | Image processing method and related device |
| CN111709875B (en) * | 2020-06-16 | 2023-11-14 | 北京百度网讯科技有限公司 | Image processing method, device, electronic equipment and storage medium |
| CN111709878B (en) * | 2020-06-17 | 2023-06-23 | 北京百度网讯科技有限公司 | Face super-resolution implementation method and device, electronic equipment and storage medium |
| CN111833242A (en) * | 2020-07-17 | 2020-10-27 | 北京字节跳动网络技术有限公司 | Face transformation method and device, electronic equipment and computer readable medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110189249A (en) * | 2019-05-24 | 2019-08-30 | 深圳市商汤科技有限公司 | A kind of image processing method and device, electronic equipment and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109712144A (en) * | 2018-10-29 | 2019-05-03 | 百度在线网络技术（北京）有限公司 | Processing method, training method, equipment and the storage medium of face-image |
| CN109815893B (en) * | 2019-01-23 | 2021-03-26 | 中山大学 | Color face image illumination domain normalization method based on cyclic generation countermeasure network |
| CN109961507B (en) * | 2019-03-22 | 2020-12-18 | 腾讯科技（深圳）有限公司 | Face image generation method, device, equipment and storage medium |
| CN110503703B (en) * | 2019-08-27 | 2023-10-13 | 北京百度网讯科技有限公司 | Method and apparatus for generating image |
-
2019
- 2019-12-17 CN CN201911300750.XA patent/CN111047509B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110189249A (en) * | 2019-05-24 | 2019-08-30 | 深圳市商汤科技有限公司 | A kind of image processing method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111047509A (en) | 2020-04-21 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111047509B (en) | Image special effect processing method, device and terminal | |
| WO2020192568A1 (en) | Facial image generation method and apparatus, device and storage medium | |
| CN113327278B (en) | Three-dimensional face reconstruction method, device, equipment and storage medium | |
| US11455765B2 (en) | Method and apparatus for generating virtual avatar | |
| WO2020150689A1 (en) | Systems and methods for realistic head turns and face animation synthesis on mobile device | |
| AU2013266192A1 (en) | Systems and methods for generating a 3-D model of a virtual try-on product | |
| CN110148191A (en) | The virtual expression generation method of video, device and computer readable storage medium | |
| CN112330527A (en) | Image processing method, image processing apparatus, electronic device, and medium | |
| CN115601484B (en) | Virtual character face driving method and device, terminal equipment and readable storage medium | |
| US20180276870A1 (en) | System and method for mass-animating characters in animated sequences | |
| CN115147265A (en) | Virtual image generation method and device, electronic equipment and storage medium | |
| JP7418370B2 (en) | Methods, apparatus, devices and storage media for transforming hairstyles | |
| CN106548117B (en) | A kind of face image processing process and device | |
| CN115861515A (en) | Three-dimensional face reconstruction method, computer program product and electronic device | |
| CN115512193A (en) | Facial expression generation method and device | |
| CN112714337A (en) | Video processing method and device, electronic equipment and storage medium | |
| CN112634444B (en) | Human body posture migration method and device based on three-dimensional information, storage medium and terminal | |
| CN113223128B (en) | Method and apparatus for generating image | |
| CN114820908B (en) | Virtual image generation method and device, electronic equipment and storage medium | |
| CN116363331B (en) | Image generation method, device, equipment and storage medium | |
| US11062504B1 (en) | Method for generating model of sculpture of face, computing device, and non-transitory storage medium | |
| CN118781235A (en) | Expression driving method, system and device for virtual face model and storage medium | |
| CN115713458A (en) | Face replacement method, face replacement device, electronic equipment and storage medium | |
| CN118864666A (en) | Digital person customization method and system | |
| CN118411746A (en) | Face fusion method, system, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |