CN112200031A - Network model training method and equipment for generating image corresponding word description - Google Patents
Network model training method and equipment for generating image corresponding word description Download PDFInfo
- Publication number
- CN112200031A CN112200031A CN202011033394.2A CN202011033394A CN112200031A CN 112200031 A CN112200031 A CN 112200031A CN 202011033394 A CN202011033394 A CN 202011033394A CN 112200031 A CN112200031 A CN 112200031A
- Authority
- CN
- China
- Prior art keywords
- image
- coding vector
- network model
- sample
- training
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000012549 training Methods 0.000 title claims abstract description 108
- 238000000034 method Methods 0.000 title claims abstract description 57
- 239000013598 vector Substances 0.000 claims abstract description 151
- 238000013528 artificial neural network Methods 0.000 claims description 42
- 230000006870 function Effects 0.000 claims description 28
- 238000002372 labelling Methods 0.000 abstract description 10
- 230000004913 activation Effects 0.000 description 9
- 238000003062 neural network model Methods 0.000 description 9
- 238000013508 migration Methods 0.000 description 8
- 230000005012 migration Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 6
- 238000013527 convolutional neural network Methods 0.000 description 4
- 238000013507 mapping Methods 0.000 description 4
- 238000013135 deep learning Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 230000008569 process Effects 0.000 description 3
- 230000000007 visual effect Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000001419 dependent effect Effects 0.000 description 2
- 230000036544 posture Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
- G06N3/084—Backpropagation, e.g. using gradient descent
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Data Mining & Analysis (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Biomedical Technology (AREA)
- Life Sciences & Earth Sciences (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
- Image Processing (AREA)
Abstract
Compared with the prior art, the network model training method and device for generating the image corresponding to the caption firstly obtain an image training set, then determine the image coding vector of the image in the sample and the caption coding vector corresponding to the image based on the sample in the image training set, then associate the image coding vector and the caption coding vector to obtain the associated feature vector of the sample, finally input the associated feature vector into the network model for training, and finish the training of the network model when the loss function of the network model meets the preset condition to obtain the trained network model. By the method, a network model which can be used for automatically labeling the text descriptions on the images is obtained, and the labor cost of manually labeling the text descriptions is greatly reduced. The network model is applied to the pedestrian re-identification task, so that the identification performance of pedestrian re-identification can be improved, and the practical application value is achieved.
Description
Technical Field
The application relates to the technical field of computer vision processing, in particular to a technology for generating a text description corresponding to an image.
Background
The pedestrian re-identification technology in the field of computer vision is used for identifying and matching the same target pedestrian in images acquired by different monitoring equipment, and has important significance in research and application in the fields of intelligent security and the like. In a real scene, due to the influence of factors such as human body posture, shooting angle change and lighting conditions, how to accurately identify and match a target pedestrian becomes a very challenging problem in the pedestrian re-identification technology.
In recent years, with the successful application of deep learning related algorithms in the field of computer vision and the continuous appearance of large-scale databases, many pedestrian re-identification methods utilize auxiliary information such as human body postures, attributes of various parts of the human body, and word descriptions on images in addition to extracting features provided by the images, so as to improve the effect of pedestrian re-identification.
The image description can provide specific and comprehensive information, so that the image description is semantically richer than the visual attribute, and the position descriptions of different images to the same target pedestrian are generally more consistent. Therefore, the pedestrian re-identification method utilizes the text description of the image, the problem of pedestrian appearance difference in the pedestrian re-identification method can be reduced, and the identification and matching accuracy is improved.
However, the method for labeling the characters of a plurality of images of the pedestrian re-identification method, especially the image data set of the pedestrian re-identification method by adopting deep learning, needs to invest high labor cost and has low efficiency, so that the application of the character description of the images in the pedestrian re-identification method is limited.
Disclosure of Invention
The application aims to provide a method and equipment for training a network model for generating a text description corresponding to an image, and the method and equipment are used for solving the technical problems that in the prior art, manual text marking of an image data set based on pedestrian re-recognition of deep learning is high in labor cost and low in efficiency.
According to one aspect of the application, a network model training method for generating image corresponding word descriptions is provided, wherein the method comprises the following steps:
acquiring an image training set, wherein each sample in the image training set comprises an image and a text description corresponding to the image;
determining an image coding vector of an image in the sample and a text coding vector of a text description corresponding to the image based on the sample in the image training set;
correlating the image coding vector and the character coding vector to obtain a correlation characteristic vector of the sample;
and inputting the associated feature vectors into a neural network for training, and finishing the training of the neural network when the loss function of the neural network meets a preset condition so as to obtain a trained network model.
Optionally, wherein the method further comprises:
acquiring a target image of a to-be-generated caption;
determining an image coding vector of the target image based on the target image;
and inputting the image coding vector into the trained network model to obtain the text description corresponding to the target image.
Optionally, wherein before the determining an image coding vector of the target image based on the target image, the method further comprises:
unifying the picture style of the target image with the picture style in the image training set.
Optionally, the inputting the image coding vector into the trained network model to obtain the caption corresponding to the target image includes:
inputting the image coding vector into the trained network model, and extracting attribute characteristic information of the target image;
generating scattered character description information based on the attribute characteristic information of the target image;
and integrating the scattered text description information to obtain the text description corresponding to the target image.
According to another aspect of the present application, there is also provided a network model training apparatus for generating a corresponding caption of an image, wherein the apparatus includes:
the device comprises a first device, a second device and a third device, wherein the first device is used for acquiring an image training set, and each sample in the image training set comprises an image and a word description corresponding to the image;
the second device is used for determining an image coding vector of an image in the sample and a text coding vector of a text description corresponding to the image based on the sample in the image training set;
the third device is used for correlating the image coding vector with the character coding vector to obtain a correlation characteristic vector of the sample;
and the fourth device is used for inputting the associated feature vectors into a neural network for training, and finishing the training of the neural network when the loss function of the neural network meets a preset condition so as to obtain a trained network model.
Compared with the prior art, the network model training method and device for generating the image corresponding to the caption firstly obtain an image training set, then determine the image coding vector of the image in the sample and the caption coding vector corresponding to the image based on the sample in the image training set, then associate the image coding vector and the caption coding vector to obtain the associated feature vector of the sample, finally input the associated feature vector into the network model for training, and finish the training of the network model when the loss function of the network model meets the preset condition to obtain the trained network model. By the method, a network model which can be used for automatically labeling the text descriptions on the images is obtained, and the labor cost of manually labeling the text descriptions is greatly reduced. The network model is applied to the pedestrian re-identification task, so that the identification performance of pedestrian re-identification can be improved, and the practical application value is achieved.
Drawings
Other features, objects and advantages of the invention will become more apparent upon reading of the detailed description of non-limiting embodiments made with reference to the following drawings:
FIG. 1 illustrates a flow diagram of a network model training method for generating image-corresponding text descriptions, according to one aspect of the subject application;
FIG. 2 illustrates a schematic diagram of a network model architecture for generating corresponding textual descriptions of images, in accordance with an aspect of the subject application;
FIG. 3 illustrates a CNN module convolutional layer structure diagram for generating a network model of an image corresponding to a caption, according to an aspect of the subject application;
FIG. 4 illustrates a schematic diagram of a network model training apparatus for generating image corresponding text descriptions, according to an aspect of the subject application;
the same or similar reference numbers in the drawings identify the same or similar elements.
Detailed Description
The present invention is described in further detail below with reference to the attached drawing figures.
In a typical configuration of the present application, each module and trusted party of the system includes one or more processors (CPUs), input/output interfaces, network interfaces, and memory.
The memory may include forms of volatile memory in a computer readable medium, Random Access Memory (RAM) and/or non-volatile memory, such as Read Only Memory (ROM) or flash memory (flash RAM). Memory is an example of a computer-readable medium.
Computer-readable media, including both non-transitory and non-transitory, removable and non-removable media, may implement information storage by any method or technology. The information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), other types of Random Access Memory (RAM), Read Only Memory (ROM), Electrically Erasable Programmable Read Only Memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), Digital Versatile Discs (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include non-transitory computer readable media (transient media), such as modulated data signals and carrier waves.
In order to further explain the technical means and effects adopted by the present application, the following description clearly and completely describes the technical solution of the present application with reference to the accompanying drawings and preferred embodiments.
FIG. 1 illustrates a flow diagram of a network model training method for generating image-corresponding text descriptions in one aspect of the application, wherein the method of an embodiment comprises:
s11, acquiring an image training set, wherein each sample in the image training set comprises an image and a text description corresponding to the image;
s12, determining image coding vectors of images in the samples and character coding vectors of the character descriptions corresponding to the images based on the samples in the image training set;
s13, correlating the image coding vector and the character coding vector to obtain a correlation characteristic vector of the sample;
s14, inputting the associated feature vectors into a neural network for training, and finishing the training of the neural network when the loss function of the neural network meets a preset condition to obtain a trained network model.
In the present application, the method is performed by a device 1, the device 1 is a computer device and/or a cloud, the computer device includes but is not limited to a personal computer, a notebook computer, an industrial computer, a network host, a single network server, a plurality of network server sets; the Cloud is made up of a large number of computers or web servers based on Cloud Computing (Cloud Computing), which is a type of distributed Computing, a virtual supercomputer consisting of a collection of loosely coupled computers.
The computer device and/or cloud are merely examples, and other existing or future devices and/or resource sharing platforms, as applicable to the present application, are also intended to be included within the scope of the present application and are hereby incorporated by reference.
In this embodiment, in step S11, the apparatus 1 acquires an image training set, where each sample in the image training set includes an image and a caption corresponding to the image, and the caption is usually saved in the form of an annotation file corresponding to the image. The image training set may be an existing pedestrian re-identification data set, each sample in the existing pedestrian re-identification data set includes an image and a caption corresponding to the image, for example, a CUHK-PEDES data set which is well known in the field of pedestrian re-identification, or an acquired image data set suitable for a pedestrian re-identification task, and then preprocessing is performed, that is, manually labeling the caption on each image in the image data set is traversed, the image and the caption corresponding to the image are used as one sample, and all samples form the image training set.
Continuing in this embodiment, in step S12, after the device 1 acquires the image training set, each sample in the image training set is processed, wherein the images in the samples are encoded to obtain image encoding vectors of the images.
For example, images in the samples are input into a trained VGG16 module to perform image coding in combination with a neural network model of the ReLU activation function, so as to obtain image coding vectors corresponding to the images, and the lengths of the image coding vectors can be set according to actual needs, for example, set to 512 dimensions. The trained neural network model of the VGG16 in combination with the ReLU activation function can be obtained after training based on actual conditions, for example, the neural network pre-training of the VGG16 in combination with the ReLU activation function is performed on the existing ImageNet image data set, so as to give a better initialization to network parameters, so that the training process can be converged faster, and a disclaimer (dropout) operation can be added to the network to reduce the overfitting phenomenon of the neural network which may occur in the training.
After the device 1 acquires the image training set, each sample in the image training set is processed, and besides the images in the samples are encoded to obtain the image encoding vectors of the images, the word descriptions corresponding to the images in the samples are encoded in a word/word direction according to the words/words to obtain the word encoding vectors of the word descriptions corresponding to the images.
For example, the text description corresponding to the image in the sample is input into the basic RNN neural network model for text coding, so as to obtain a text coding vector of the text description corresponding to the image, and the length of the text coding vector can be set according to actual needs, but should be the same as the length of the image coding vector of the image, for example, set to 512 dimensions.
In this embodiment, in step S13, the device 1 associates the image coding vector of the obtained image with the text coding vector of the text description corresponding to the image, and obtains an associated feature vector of the corresponding sample.
The associating the image coding vector of the obtained image and the text coding vector of the text description corresponding to the image may be splicing the image coding vector of the obtained image and the text coding vector of the text description corresponding to the image, for example, splicing a 512-dimensional image coding vector and a 512-dimensional text coding vector to obtain a 1024-dimensional associated feature vector of the corresponding sample; or corresponding channel addition may be performed on the obtained image coding vector of the image and the text coding vector of the text description corresponding to the image, for example, corresponding channel addition is performed on the 512-dimensional image coding vector and the 512-dimensional text coding vector to obtain a 512-dimensional associated feature vector corresponding to the sample; the obtained image coding vector of the image and the text coding vector of the text description corresponding to the image may be associated based on a weighted mapping (attribute) method, for example, a 512-dimensional image coding vector and a 512-dimensional text coding vector are associated based on a weighted mapping method to obtain a 512-dimensional associated feature vector corresponding to the sample.
Continuing in this embodiment, in step S14, the device 1 inputs the associated feature vector into a neural network for training, and when the loss function of the neural network satisfies a preset condition, completes the training of the neural network to obtain a trained network model. And constructing the association relation between the images in the sample of the image training set and the corresponding text descriptions of the images through a neural network.
The preset condition includes that the numerical value of the loss function of the neural network meets a preset threshold, or within a preset training cycle number (epoch), although the numerical value of the loss function of the neural network does not meet the preset threshold, the numerical value of the loss function of the neural network does not decrease.
The trained network model is used for automatically labeling the images in the image data set of the pedestrian re-recognition task with the corresponding text descriptions of the images.
One network model structure for generating corresponding text descriptions of images of one embodiment is shown in FIG. 2. For example, the device 1 obtains an image training set, where a caption labeled to an image of one sample is a woman in red shirt, and inputs the image into a trained VGG16 to perform image coding in combination with a neural network model of a ReLU activation function, so as to obtain a 512-dimensional image coding vector; acquiring a caption label corresponding to the image, and inputting the caption into a basic RNN neural network model for character coding to obtain a 512-dimensional character coding vector of the caption corresponding to the image; then, 512-dimensional image coding vectors and 512-dimensional character coding vectors are spliced into 1024-dimensional associated feature vectors, a neural network formed by a CNN module and a classification layer is input for training, wherein the CNN module is formed by 3 layers of convolution layers and GLU activation functions, the structure of each layer of convolution layer is shown in figure 3, weight normalization, residual connection and weight abandon operations are added into each layer of convolution layer, so that the neural network training effect is better, the CNN module outputs 512-dimensional vectors as the input of the classification layer, the classification layer performs downsampling on the input 512-dimensional vectors to obtain 256-dimensional vectors, and the feature vectors are classified through a normalized softmax function.
And when the numerical value of the loss function of the neural network meets a preset condition, finishing the training of the neural network to obtain a trained network model for automatically labeling the text description corresponding to the image in the image data set of the pedestrian re-recognition task.
For example, the neural network is trained using a cross-entropy loss function (cross-entropy loss function), and optimized using a back propagation algorithm based on gradient descent, wherein the optimizer may use RMSProp, and the value of the cross-entropy loss function may be calculated based on the following formula:
wherein,
LIDis a cross entropy loss function;
k is the total number of classified categories;
ykprobability of being the kth class;
theta is an optimized initial parameter;
f is the feature extracted by the neural network.
Optionally, the method for training a network model for generating a caption corresponding to an image further includes:
s15 (not shown) acquiring a target image of a caption to be generated;
s16 (not shown) determining an image coding vector for the target image based on the target image;
s17 (not shown) inputs the image coding vector into the trained network model to obtain the corresponding text description of the target image.
In one embodiment, in step S15, the device 1 acquires a target image to be recognized of the pedestrian re-recognition task, wherein the target image to be recognized has no corresponding text description.
Continuing in this embodiment, in step S16, device 1 performs image coding on the obtained target image to be recognized, and obtains an image coding vector corresponding to the target image to be recognized. For example, the target image to be recognized is input into a trained VGG16 to perform image coding in combination with a neural network model of a ReLU activation function, so as to obtain an image coding vector corresponding to the image, and the length of the image coding vector may be set according to actual needs, for example, set to 512 dimensions. The trained neural network model of the VGG16 in combination with the ReLU activation function can be obtained after training based on actual conditions, for example, the neural network pre-training of the VGG16 in combination with the ReLU activation function is performed on the existing ImageNet image data set, so as to give a better initialization to network parameters, so that the training process can be converged faster, and a disclaimer (dropout) operation can be added to the network to reduce the overfitting phenomenon of the neural network which may occur in the training.
If the picture style of the target image to be recognized is greatly different from the picture style of the images in the sample of the image training set, for example, the difference of the brightness, exposure, color saturation, hue and the like of the image shot is large, the generalization capability of the trained network model obtained based on the image training set (for example, CUHK-PEDES) to the caption prediction of the target image to be recognized is poor.
Optionally, before the step S16, the method further includes:
s18 (not shown) unifies the picture style of the target image with the picture style in the training set of images.
In order to enable the trained network model obtained based on the image training set to have good prediction generalization capability on the word description of the target image to be recognized, before image coding is performed on the target image to be recognized, a style migration technology is adopted to unify the picture style of the target image and the picture style in the image training set so as to eliminate the picture style difference of different images and improve the generalization capability of the trained network model prediction. For example, the picture style of the target image to be recognized may be migrated to the picture style of the sample image of the image training set by using a SPGAN (simple prediction cycle-dependent adaptive Network) method, for example, the picture style of the image in the Duke-MTMC data set for the pedestrian re-recognition task may be migrated to the picture style of the CUHK-PEDES data set by using the SPGAN method with the image in the Duke-MTMC data set as the target image. Other image style migration methods, such as CycleGAN, Pix2Pix and other image style migration methods, may also be adopted, and any other image style migration method, such as being applicable to the present application, is within the scope of the present application without limitation.
Continuing in this embodiment, in step S17, the device 1 inputs the obtained image coding vector corresponding to the target image to be recognized into the trained network model to obtain the caption corresponding to the target image to be recognized, and further saves the obtained caption corresponding to the target image to be recognized as the annotation file corresponding to the target image to be recognized.
Optionally, wherein the step S17 includes:
inputting the image coding vector into the trained network model, and extracting attribute characteristic information of the target image;
generating scattered character description information based on the attribute characteristic information of the target image;
and integrating the scattered text description information to obtain the text description corresponding to the target image.
For example, images in the Duke-MTMC data set for the pedestrian re-recognition task after the image style migration are encoded into image encoding vectors and then input into a trained network model, the network model extracts attribute features from the image encoding vectors, predicts the interpretative scattered word (word) character descriptions of the target images according to the attribute features, and integrates the scattered word (word) character descriptions into corresponding character descriptions by combining language styles (grammars and rules) of the character descriptions corresponding to sample images in the image training set learned by training.
Furthermore, by traversing each image of the image data set where the target image is located by adopting the method, the description can be automatically labeled for the image data set where the target image is located efficiently and accurately, and the image data set with the description label can be obtained. For example, the method is adopted to traverse each image in the Duke-MTMC data set, so that the caption annotation of the Duke-MTMC data set can be efficiently and accurately obtained, and the Duke-MTMC data set with the caption annotation is obtained.
Furthermore, the obtained image data set with the caption (such as the Duke-MTMC data set with the caption) can be used in a pedestrian re-identification task, so that the visual image features are provided for the existing neural network model for assisting the pedestrian re-identification task by using the caption, and meanwhile, the caption with rich semantic information is provided, so that the better performance is obtained based on the pedestrian re-identification task.
FIG. 4 is a schematic diagram of a network model training apparatus for generating image-corresponding text descriptions according to another aspect of the present application, wherein the apparatus comprises:
a first device 41, configured to obtain an image training set, where each sample in the image training set includes an image and a caption corresponding to the image;
a second device 42, configured to determine, based on the samples in the image training set, an image coding vector of an image in the samples and a text coding vector of a text description corresponding to the image;
a third device 43, configured to associate the image coding vector and the text coding vector to obtain an associated feature vector of the sample;
and a fourth device 44, configured to input the associated feature vector into a neural network for training, and complete training of the neural network when a loss function of the neural network meets a preset condition, so as to obtain a trained network model.
The first device 41 acquires an image training set, where each sample in the image training set includes an image and a caption corresponding to the image, and the caption is usually saved in the form of an annotation file corresponding to the image. The image training set may be an existing pedestrian re-identification data set, each sample in the existing pedestrian re-identification data set includes an image and a caption corresponding to the image, for example, a CUHK-PEDES data set which is well known in the field of pedestrian re-identification, or an acquired image data set suitable for a pedestrian re-identification task, and then preprocessing is performed, that is, manually labeling the caption on each image in the image data set is traversed, the image and the caption corresponding to the image are used as one sample, and all samples form the image training set.
Wherein the second means 42 processes each sample in the training set of images, wherein the images in the samples are encoded to obtain an image encoding vector for the image.
Optionally, wherein the second device 42 comprises:
a VGG16 network module to determine an image encoding vector for an image based on the image of the sample in the training set of images;
and the RNN module is used for determining a word coding vector of the word description corresponding to the image based on the word description corresponding to the image of the sample in the image training set.
The VGG16 network module comprises a VGG16 module and a ReLU activation function, and the VGG16 network module performs image coding on images in input image training set samples to obtain image coding vectors corresponding to the images. The length of the image coding vector can be set according to actual needs, for example, 512 dimensions.
The RNN module carries out character/word direction coding on the word descriptions corresponding to the images in the input image training set samples according to characters/words so as to obtain character coding vectors of the word descriptions corresponding to the images. The length of the text encoding vector can be set according to actual needs, but should be the same as the length of the image encoding vector of the image, for example, set to 512 dimensions.
The third device 43 correlates the image coding vector of the obtained image with the text coding vector of the text description corresponding to the image to obtain a correlation feature vector corresponding to the sample.
The associating the image coding vector of the obtained image and the text coding vector of the text description corresponding to the image may be splicing the image coding vector of the obtained image and the text coding vector of the text description corresponding to the image, for example, splicing a 512-dimensional image coding vector and a 512-dimensional text coding vector to obtain a 1024-dimensional associated feature vector of the corresponding sample; or corresponding channel addition may be performed on the obtained image coding vector of the image and the text coding vector of the text description corresponding to the image, for example, corresponding channel addition is performed on the 512-dimensional image coding vector and the 512-dimensional text coding vector to obtain a 512-dimensional associated feature vector corresponding to the sample; the obtained image coding vector of the image and the text coding vector of the text description corresponding to the image may be associated based on a weighted mapping (attribute) method, for example, a 512-dimensional image coding vector and a 512-dimensional text coding vector are associated based on a weighted mapping method to obtain a 512-dimensional associated feature vector corresponding to the sample.
The fourth device 44 inputs the associated feature vector into a neural network for training, and when the loss function of the neural network meets a preset condition, completes the training of the neural network to obtain a trained network model. The relevance relation between the images in the sample of the image training set and the text descriptions corresponding to the images is established through the neural network, and the trained network model can be used for automatically labeling the text descriptions corresponding to the images in the image data set of the pedestrian re-recognition task.
Optionally, the apparatus further comprises:
fifth means 45 (not shown) for acquiring a target image of the caption to be generated;
sixth means 46 (not shown) for determining an image coding vector for the target image based on the target image;
seventh means 47 (not shown) for inputting the image coding vector into the trained network model to obtain the corresponding text description of the target image.
The fifth device 45 obtains a target image to be recognized of the pedestrian re-recognition task, where the target image to be recognized has no corresponding text description.
The sixth device 46 performs image coding on the obtained target image to be identified, and obtains an image coding vector corresponding to the target image to be identified. The length of the image coding vector can be set according to actual needs, for example, set to 512 dimensions.
The seventh device 47 inputs the image coding vector corresponding to the target image to be recognized into the trained network model to obtain the caption corresponding to the target image to be recognized, and further saves the caption corresponding to the target image to be recognized as the annotation file corresponding to the target image to be recognized.
Furthermore, each image of the image data set where the target image is located is traversed, so that the image data set where the target image is located can be automatically labeled with the caption efficiently and accurately, and the image data set with the caption label is obtained. For example, the method is adopted to traverse each image in the Duke-MTMC data set, so that the caption annotation of the Duke-MTMC data set can be efficiently and accurately obtained, and the Duke-MTMC data set with the caption annotation is obtained.
Optionally, the apparatus may further comprise:
eighth means 48 (not shown) for unifying the picture style of the target image with the picture style in the training set of images before the target image is encoded.
The method can eliminate the picture style difference of different images and improve the generalization capability of the trained network model prediction. For example, the picture style of the target image to be recognized may be migrated to the picture style of the sample image of the image training set by using a SPGAN (simple prediction cycle-dependent adaptive Network) method, for example, the picture style of the image in the Duke-MTMC data set for the pedestrian re-recognition task may be migrated to the picture style of the CUHK-PEDES data set by using the SPGAN method with the image in the Duke-MTMC data set as the target image. Other image style migration methods, such as CycleGAN, Pix2Pix and other image style migration methods, may also be adopted, and any other image style migration method, such as being applicable to the present application, is within the scope of the present application without limitation.
Further, the apparatus may further include:
ninth means 49 (not shown) for using the obtained image dataset with the caption annotations for the pedestrian re-identification task.
For example, the ninth device 49 uses the Duke-MTMC data set with the caption for the pedestrian re-recognition task, and provides the visual image features for the existing neural network model that uses the caption to assist the pedestrian re-recognition task, and at the same time provides the caption with rich semantic information, so as to obtain better performance based on the pedestrian re-recognition task.
According to yet another aspect of the present application, there is also provided a computer readable medium having stored thereon computer readable instructions executable by a processor to implement the foregoing method.
According to yet another aspect of the present application, there is also provided an apparatus for optimizing a predicted radar echo image, wherein the apparatus comprises:
one or more processors; and
a memory storing computer readable instructions that, when executed, cause the processor to perform operations of the method as previously described.
For example, the computer readable instructions, when executed, cause the one or more processors to: firstly, obtaining an image training set, wherein each sample in the image training set comprises an image and a text description corresponding to the image, and then determining an image coding vector of the image in the sample and a text coding vector of the text description corresponding to the image based on the samples in the image training set; then, correlating the image coding vector and the character coding vector to obtain a correlation characteristic vector of the sample; and then inputting the associated feature vectors into a neural network for training, and finishing the training of the neural network when the loss function of the neural network meets a preset condition so as to obtain a trained network model. The target image of the text description to be generated can be continuously obtained; then determining an image coding vector of the target image based on the target image; and then inputting the image coding vector into the trained network model to obtain the corresponding word description of the target image. And unifying the picture style of the target image with the picture style in the image training set.
It will be evident to those skilled in the art that the invention is not limited to the details of the foregoing illustrative embodiments, and that the present invention may be embodied in other specific forms without departing from the spirit or essential attributes thereof. The present embodiments are therefore to be considered in all respects as illustrative and not restrictive, the scope of the invention being indicated by the appended claims rather than by the foregoing description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. Any reference sign in a claim should not be construed as limiting the claim concerned. Furthermore, it is obvious that the word "comprising" does not exclude other elements or steps, and the singular does not exclude the plural. A plurality of units or means recited in the apparatus claims may also be implemented by one unit or means in software or hardware. The terms first, second, etc. are used to denote names, but not any particular order.
Claims (9)
1. A method for training a network model for generating a caption corresponding to an image, the method comprising:
acquiring an image training set, wherein each sample in the image training set comprises an image and a text description corresponding to the image;
determining an image coding vector of an image in the sample and a text coding vector of a text description corresponding to the image based on the sample in the image training set;
correlating the image coding vector and the character coding vector to obtain a correlation characteristic vector of the sample;
and inputting the associated feature vectors into a neural network for training, and finishing the training of the neural network when the loss function of the neural network meets a preset condition so as to obtain a trained network model.
2. The method of claim 1, further comprising:
acquiring a target image of a to-be-generated caption;
determining an image coding vector of the target image based on the target image;
and inputting the image coding vector into the trained network model to obtain the text description corresponding to the target image.
3. The method of claim 2, wherein prior to said determining an image coding vector for the target image based on the target image, the method further comprises:
unifying the picture style of the target image with the picture style in the image training set.
4. The method of claim 2 or 3, wherein the inputting the image coding vector into the trained network model to obtain the corresponding caption of the target image comprises:
inputting the image coding vector into the trained network model, and extracting attribute characteristic information of the target image;
generating scattered character description information based on the attribute characteristic information of the target image;
and integrating the scattered text description information to obtain the text description corresponding to the target image.
5. A network model training apparatus for generating a corresponding caption of an image, the apparatus comprising:
the device comprises a first device, a second device and a third device, wherein the first device is used for acquiring an image training set, and each sample in the image training set comprises an image and a word description corresponding to the image;
the second device is used for determining an image coding vector of an image in the sample and a text coding vector of a text description corresponding to the image based on the sample in the image training set;
the third device is used for correlating the image coding vector with the character coding vector to obtain a correlation characteristic vector of the sample;
and the fourth device is used for inputting the associated feature vectors into a neural network for training, and finishing the training of the neural network when the loss function of the neural network meets a preset condition so as to obtain a trained network model.
6. The apparatus of claim 5, wherein the second means comprises:
a VGG16 network module to determine an image encoding vector for an image based on the image of the sample in the training set of images;
and the RNN module is used for determining a word coding vector of the word description corresponding to the image based on the word description corresponding to the image of the sample in the image training set.
7. The apparatus according to claim 5 or 6, characterized in that it further comprises;
the fifth device is used for acquiring a target image of the text description to be generated;
sixth means for determining an image coding vector for the target image based on the target image;
and the seventh device is used for inputting the image coding vector into the trained network model so as to obtain the word description corresponding to the target image.
8. A computer-readable medium comprising, in combination,
stored thereon computer readable instructions executable by a processor to implement the method of any one of claims 1 to 4.
9. An apparatus, characterized in that the apparatus comprises:
one or more processors; and
memory storing computer readable instructions that, when executed, cause the processor to perform the operations of the method of any of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011033394.2A CN112200031A (en) | 2020-09-27 | 2020-09-27 | Network model training method and equipment for generating image corresponding word description |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011033394.2A CN112200031A (en) | 2020-09-27 | 2020-09-27 | Network model training method and equipment for generating image corresponding word description |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112200031A true CN112200031A (en) | 2021-01-08 |
Family
ID=74008447
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011033394.2A Pending CN112200031A (en) | 2020-09-27 | 2020-09-27 | Network model training method and equipment for generating image corresponding word description |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112200031A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113240115A (en) * | 2021-06-08 | 2021-08-10 | 深圳数联天下智能科技有限公司 | Training method for generating face change image model and related device |
| CN113436062A (en) * | 2021-07-28 | 2021-09-24 | 北京达佳互联信息技术有限公司 | Image style migration method and device, electronic equipment and storage medium |
| CN113780679A (en) * | 2021-09-27 | 2021-12-10 | 深圳供电局有限公司 | Load prediction method and device based on ubiquitous power Internet of things |
| CN114549935A (en) * | 2022-02-25 | 2022-05-27 | 北京百度网讯科技有限公司 | Information generation method and device |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110516530A (en) * | 2019-07-09 | 2019-11-29 | 杭州电子科技大学 | A kind of Image Description Methods based on the enhancing of non-alignment multiple view feature |
| CN111339340A (en) * | 2018-12-18 | 2020-06-26 | 顺丰科技有限公司 | Training method of image description model, image searching method and device |
| CN111507406A (en) * | 2020-04-17 | 2020-08-07 | 上海眼控科技股份有限公司 | Method and equipment for optimizing neural network text recognition model |
-
2020
- 2020-09-27 CN CN202011033394.2A patent/CN112200031A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111339340A (en) * | 2018-12-18 | 2020-06-26 | 顺丰科技有限公司 | Training method of image description model, image searching method and device |
| CN110516530A (en) * | 2019-07-09 | 2019-11-29 | 杭州电子科技大学 | A kind of Image Description Methods based on the enhancing of non-alignment multiple view feature |
| CN111507406A (en) * | 2020-04-17 | 2020-08-07 | 上海眼控科技股份有限公司 | Method and equipment for optimizing neural network text recognition model |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113240115A (en) * | 2021-06-08 | 2021-08-10 | 深圳数联天下智能科技有限公司 | Training method for generating face change image model and related device |
| CN113436062A (en) * | 2021-07-28 | 2021-09-24 | 北京达佳互联信息技术有限公司 | Image style migration method and device, electronic equipment and storage medium |
| CN113780679A (en) * | 2021-09-27 | 2021-12-10 | 深圳供电局有限公司 | Load prediction method and device based on ubiquitous power Internet of things |
| CN113780679B (en) * | 2021-09-27 | 2024-04-16 | 深圳供电局有限公司 | Load prediction method and device based on ubiquitous power Internet of things |
| CN114549935A (en) * | 2022-02-25 | 2022-05-27 | 北京百度网讯科技有限公司 | Information generation method and device |
| CN114549935B (en) * | 2022-02-25 | 2024-05-07 | 北京百度网讯科技有限公司 | Information generation method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111797893B (en) | Neural network training method, image classification system and related equipment | |
| US11144587B2 (en) | User drawing based image search | |
| US10963504B2 (en) | Zero-shot event detection using semantic embedding | |
| WO2020228446A1 (en) | Model training method and apparatus, and terminal and storage medium | |
| CN112200031A (en) | Network model training method and equipment for generating image corresponding word description | |
| JP2017062781A (en) | Similarity-based detection of prominent objects using deep cnn pooling layers as features | |
| CN106919652B (en) | Short-sighted frequency automatic marking method and system based on multi-source various visual angles transductive learning | |
| CN113836992B (en) | Label identification method, label identification model training method, device and equipment | |
| CN113344206A (en) | Knowledge distillation method, device and equipment integrating channel and relation feature learning | |
| CN111542841A (en) | System and method for content identification | |
| CN117033609B (en) | Text visual question-answering method, device, computer equipment and storage medium | |
| He et al. | Aggregating local context for accurate scene text detection | |
| Islam et al. | Convolutional auto-encoder and independent component analysis based automatic place recognition for moving robot in invariant season condition | |
| Fang et al. | Adversarial multi-task deep learning for signer-independent feature representation | |
| CN116597267A (en) | Image recognition method, device, computer equipment and storage medium | |
| US20230154221A1 (en) | Unified pretraining framework for document understanding | |
| Vilaça et al. | Improving audiovisual content annotation through a semi-automated process based on deep learning | |
| CN114842251A (en) | Training method and device of image classification model, image processing method and device and computing equipment | |
| Liu et al. | Infrared ship video target tracking based on cross-connection and spatial transformer network | |
| Lou et al. | Mse-net: Pedestrian attribute recognition using mlsc and se-blocks | |
| Sreenivasulu et al. | Adaptive inception based on transfer learning for effective visual recognition | |
| Sharma et al. | Optical Character Recognition Using Hybrid CRNN Based Lexicon-Free Approach with Grey Wolf Hyperparameter Optimization | |
| Niu et al. | Robust place recognition based on salient landmarks screening and convolutional neural network features | |
| Xu et al. | Deep Neural Network‐Based Sports Marketing Video Detection Research | |
| CN118447340B (en) | Method and equipment for carrying out spatial modeling on image class relation based on prototype network |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |