KR20220122566A - Text recognition model training method, text recognition method, and apparatus - Google Patents

Abstract

The present invention relates to a method for training a text recognition model, a method for recognizing text, and a device for recognizing text. The present invention relates to the field of artificial intelligence technology and, specifically, to the field of deep learning and computer vision technology, and the present invention can be applied to scenarios such as optical character recognition. The method for training a text recognition model comprises the steps of: performing mask prediction on a part of an obtained first sample image to obtain a predicted complete image corresponding to the first sample image; performing mask prediction on a part of the texts in an obtained second sample image to obtain predicted text content for the part of the texts; training a model according to the predicted complete image and the predicted text content to obtain a pre-training model; and generating a text recognition model based on the pre-training model, wherein the text recognition model is for performing text recognition on the image to be recognized. The present invention enables the pre-training model to learn stronger image vision reasoning ability and text meaning reasoning ability. Therefore, when the text recognition model generated based on the pre-training model performs the text recognition, the accuracy and the reliability of the text recognition are improved.

Description

TEXT RECOGNITION MODEL TRAINING METHOD, TEXT RECOGNITION METHOD, AND APPARATUS

This application relates to the field of artificial intelligence (AI) technology, specifically to the field of deep learning and computer vision technology, and can be applied to scenarios such as optical character recognition (OCR), in particular text It relates to a training method of a recognition model, a text recognition method and an apparatus.

OCR technology has received widespread interest and is being applied in various industries such as education, finance, healthcare, transportation and insurance.

In the prior art, by combining OCR technology and deep learning technology to build a text recognition model, text recognition may be performed on an image based on the text recognition model.

However, the text recognition model generally relies on vision information, and since the text content in the image is discriminated based on the vision information, the recognition accuracy is lower.

The present application provides a method for training a text recognition model, a text recognition method, and an apparatus for improving the reliability of text recognition.

According to a first aspect of the present application, there is provided a method for training a text recognition model, the method comprising:

performing mask prediction on a partial image among the obtained first sample images to obtain a predicted full image corresponding to the first sample image;

performing the mask prediction on the partial text in the obtained second sample image to obtain predicted text content corresponding to the partial text;

A pre-training model is obtained by training based on the predicted full image and the predicted text content, and a text recognition model is generated based on the pre-training model, wherein the text recognition model performs text recognition for a recognition target image. Including; step for carrying out.

According to a second aspect of the present application, there is provided a text recognition method, the method comprising:

acquiring a recognition target image, wherein the recognition target image includes text;

Performing text recognition on the recognition target image based on a previously trained text recognition model to obtain text content in the recognition target image;

Here, the text recognition model is obtained based on the method according to the first aspect.

According to a third aspect of the present application, there is provided an apparatus for training a text recognition model, the apparatus comprising a prediction unit, a training unit, and a generating unit,

the prediction unit performs mask prediction on the partial images in the obtained first sample images to obtain a prediction full image corresponding to the first sample image;

the prediction unit is further configured to perform the mask prediction on the partial text in the obtained second sample image to obtain a predicted text content corresponding to the partial text;

the training unit trains on the basis of the predicted full image and the predicted text content to obtain a pre-training model;

The generating unit generates a text recognition model based on the pre-training model, wherein the text recognition model is for performing text recognition on a recognition target image.

According to a fourth aspect of the present application, there is provided a text recognition apparatus, the apparatus comprising:

an acquiring unit for acquiring a recognition object image, wherein the recognition object image includes text;

a recognition unit configured to perform text recognition on the recognition object image based on a text recognition model trained in advance to obtain text content in the recognition object image; and

Here, the text recognition model is obtained by the method according to the first aspect.

According to a fifth aspect of the present application, there is provided an electronic device, the electronic device comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; and

instructions executable by the at least one processor are stored in the memory, and the instructions are executed by the at least one processor to cause the at least one processor to perform the method according to the first aspect or the second aspect. make it possible

According to a sixth aspect of the present application, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions cause a computer to perform the method according to the first or second aspect.

According to a seventh aspect of the present application, there is provided a computer program, wherein the computer program is stored in a readable storage medium, and at least one processor of an electronic device can read the computer program from the readable storage medium, The at least one processor executes the computer program to cause the electronic device to perform the method according to the first aspect or the second aspect.

Based on the mask prediction of the present application, the prediction full image corresponding to the first sample image is obtained, the prediction text content of the partial text in the second sample image is obtained based on the mask prediction, and the prediction full image and the prediction text contents are obtained. Based on the technical method of creating a pre-training model by combining them, and creating a text recognition model based on the pre-training model, the pre-training model learns stronger image vision reasoning ability and text meaning reasoning ability, and accordingly When the text recognition model generated based on the training model performs text recognition, the accuracy and reliability of text recognition are improved.

It should be understood that the contents described in this section are not intended to specify key or important features of the embodiments of the present application, and are not intended to limit the scope of the present application. Other features of the present application can be easily understood from the following specification.

The accompanying drawings are provided to more fully understand the present method, and are not limited to the present application. here,
1 is a view according to a first embodiment of the present application.
2 is a view according to a second embodiment of the present application.
3 is a view according to a third embodiment of the present application.
4 is a view according to a fourth embodiment of the present application.
5 is a view according to a fifth embodiment of the present application.
6 is a view according to a sixth embodiment of the present application.
7 is a view according to a seventh embodiment of the present application.
8 is a view according to an eighth embodiment of the present application.
9 is a view according to a ninth embodiment of the present application.
10 is a block diagram of an electronic device for implementing a training method of a text recognition model and a text recognition method according to an embodiment of the present application.

The following describes exemplary embodiments of the present application in conjunction with the accompanying drawings, and includes various details of the embodiments of the present application for easy understanding, which should be regarded as exemplary only. Accordingly, it should be understood that various changes and modifications may be made to the embodiments described herein by those of ordinary skill in the art without departing from the scope and spirit of the present application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted from the description below.

When building a text recognition model by combining OCR technology and deep learning, it can be implemented using the "module separation" method, or the "end-to-end model" method.

Exemplarily, the "module separation" method refers to building a text detection module, an information extraction module, and a text recognition module, and combining these three modules to build a text recognition model.

When using the "module separation" method, each module must be built in advance and each module must be combined. The recognition accuracy of the built text recognition model results in a lower disadvantage.

Illustratively, the method of "end-to-end model" obtains one prediction result from the input end to the output end, for example, inputs an image to the input end, and obtains the predicted text content for the image at the output end.

However, using the "end-to-end model" approach, data labeling must be performed, e.g. labeling against the true textual content of an image, and the data provided for training must be relatively valid, so a trained text recognition model of lower reliability.

The text recognition model obtained by training based on any one of the methods described above generally performs only two types of judgments, and when there is a different class field demand for different vertical types, the text recognition model, especially the number of classified channels must be redesigned, the text recognition model must also be retrained, and cannot be multiplexed.

For example, image character detection model (EAST), segmented character detection model (DB), and text detector (LOMO) in OCR technology are generally used as text type and non-text type. It can only be used for both types of judgment. If it is necessary to solve the field recognition demand of the user's interest in a specific specific vertical type, the number of classification classes should be increased.

In some embodiments, a new text recognition model may be acquired by training through a class detection extension method, for example, a separate language model may be added on the basis of an existing text recognition model to perform field classification.

For example, if the text recognition model is an end-to-end text detection and recognition (FOTS) and text detection and recognition model (Mask Text Spotter) in OCR technology, such as Bidirectional Encoder Representation from Transformers (BERT) A separate language model must be added to obtain a new text recognition model, and since a separate language model is added, additional training must be added, which results in a higher training cost and lower efficiency.

In order to avoid at least one of the technical problems described above, the inventor of the present application has come to the inventive idea of the present application through creative labor. Specifically, training using the "end-to-end model" method to obtain a pre-training model, that is, performing end-to-end pre-training on the model base, and performing pre-training by combining the vision dimension and semantic dimension and create a text recognition model based on the pre-trained base.

Based on the above-described inventive idea, the present application provides a training method of a text recognition model, a text recognition method and apparatus, and relates to the field of artificial intelligence technology, specifically to the field of deep learning and computer vision technology, OCR, etc. can be applied to the scenario of , to improve the reliability of text recognition of the text recognition model.

1 is a view according to a first embodiment of the present application. As shown in FIG. 1 , the training method of the text recognition model provided in this embodiment includes the following steps ( S101 - S1013 ).

S101: Mask prediction is performed on a partial image of the obtained first sample image to obtain a prediction full image corresponding to the first sample image.

Illustratively, the subject of this embodiment may be a training apparatus for text recognition model (hereinafter, abbreviated as 'training apparatus'), and the training apparatus may be a server (eg, a cloud server, a local server, or a server cluster). It may be a terminal, a computer, a processor, a chip, etc., which is not limited in this embodiment.

Here, mask prediction refers to performing mask processing (or 'occlusion processing') on partial images or text, etc., and using complete images or texts such as images or text before mask processing, that is, before occlusion processing. refers to restoration.

Correspondingly, for the above steps, obtaining a first sample image including text, performing mask processing on a partial image of the first sample image, and performing a mask processing on a complete first sample image (that is, based on the image after mask processing) It can be understood as predicting the prediction pool image).

In other words, for the above step, as an image reconstruction task (mask image modeling), it can be understood as performing image reconstruction on the first sample image by combining the method of mask prediction.

S102: Perform mask prediction on the partial text in the obtained second sample image to obtain the predicted text content corresponding to the partial text.

Combining the above analysis, for the above steps, a second sample image including text is obtained, mask processing is performed on partial text in the second sample image, and a masked portion is based on the text after mask processing. It can be understood as predicting the textual content of the text (ie predictive text content).

In other words, as for the above step, as a text reconstruction task (mask OCR modeling), text reconstruction is performed on the second sample image by combining the method of mask prediction, specifically, reconstruction on partial text in the second sample image. I can understand.

It should be particularly noted that the first sample image and the second sample image may be the same image or different images, and the present embodiment is not limited thereto.

S103: Obtain a pre-training model by training based on the predicted full image and the predicted text content, and generate a text recognition model based on the pre-training model.

Here, the text recognition model is for performing text recognition on an image to be recognized.

The pre-training model may be understood as a base of the text recognition model or may be understood as a hidden layer of the text recognition model.

As can be seen by combining the above analysis, the pre-training model is obtained by training based on image reconstruction and text reconstruction, so that the pre-training model learns stronger image vision reasoning ability and text meaning reasoning ability, The text recognition model generated based on the pre-training model has stronger accuracy and reliability.

In this embodiment, end-to-end model training can be implemented, that is, prediction results corresponding to each can be output directly based on the first sample image and the second sample image, for example, in the first sample image. The corresponding prediction result is a prediction full image, and the prediction result corresponding to the second sample image is the predicted text content, such as performing text detection on the second sample image based on artificial or OCR technology to obtain text. There is no need to add other steps, which improves training efficiency and saves training resources and costs.

As can be seen based on the above analysis, the embodiment of the present application provides a training method of a text recognition model, and performs mask prediction on a partial image in the obtained first sample image, so that the first sample image and Obtaining the corresponding prediction full image, performing mask prediction on the partial text in the obtained second sample image, obtaining the prediction text content corresponding to the partial text, and training based on the prediction full image and the prediction text content acquiring a pre-training model, and generating a text recognition model based on the pre-training model, wherein the text recognition model is for performing text recognition on an image to be recognized, in this embodiment, mask prediction Acquire a prediction full image corresponding to the first sample image based on Through the technical feature of creating a text recognition model based on the pre-training model and generating a text recognition model based on the pre-training model, the pre-training model learns stronger image vision reasoning ability and text semantic reasoning ability, and accordingly generated based on the pre-training model When the proposed text recognition model performs text recognition, it improves the accuracy and reliability of text recognition.

2 is a view according to a second embodiment of the present application. As shown in Fig. 2, the training method of the text recognition model provided in this embodiment includes steps S201 to S203.

S201: Acquire a target object.

Here, the target object includes a first sample image and a second sample image.

It should be understood that, in order to avoid complicated description, repeated descriptions of the same technical features as those of the above-described embodiments are omitted in the present embodiment.

S202: Partial objects among the target objects are randomly covered, and prediction results are obtained by predicting the partial objects of the target objects based on the objects not covered among the target objects.

Here, if the target object is the first sample image, the partial object of the target object is the partial image, and the prediction result is the prediction full image.

If the target object is the second sample image, the partial object of the target object is the partial text, and the prediction result is the predicted text content.

In some embodiments, obtaining a prediction result by predicting a partial obscured object among the target objects based on the non-occluded object among the target objects includes the following steps.

Step 1: A first target characteristic is obtained by extracting a target feature corresponding to an unobscured target from among the target targets.

Step 2: Predicting a partial object hidden among the target objects based on the first target feature to obtain a prediction result.

Here, if the target object is a first sample image, the first object characteristic is a first vision characteristic. If the target object is a second sample image, the first object characteristic is the first semantic characteristic.

S203: Obtain a pre-training model by training based on the predicted full image and the predicted text content, and generate a text recognition model based on the pre-training model.

Here, the text recognition model is for text recognition with respect to an image to be recognized.

In order for the reader to more fully understand the implementation principle of the present application, the above-described embodiment (the embodiment shown in FIGS. 1 and 2 ) with reference to FIG. 3 will be described in detail below.

3 is a view according to a third embodiment of the present application. As shown in FIG. 3 , the training method of the text recognition model provided in this embodiment includes the following steps ( S301 - S307 ).

S301: Acquire a first sample image.

Similarly, in order to avoid complicated description, in the present embodiment, repeated description of the same technical features as in the above-described embodiment of the present embodiment will be omitted.

S302: Randomly cover partial images in the first sample image.

It should be understood that the training of the network model is generally a process of repeated training, and in this embodiment, each repetition training randomly covers partial images of the first sample image, so the quantity of the first sample image is may be one, and, of course, the number of the first sample images may be plural, which is not limited in this embodiment.

S303: Based on the non-occluded image among the first sample images, prediction is performed on the partial image of the first sample image to obtain a prediction full image.

Illustratively, after random occlusion on the first sample image, partial images in the first sample image are occluded and other partial images are not occluded, so that the complete first sample image (ie, prediction full image) can be determined.

In this embodiment, by combining the method of "random occlusion + prediction" to determine the prediction full image, the indeterminacy in the training process can be increased, so that the pre-training model obtained by training is a complete image to restore reliability and improve reliability.

Here, S302 - S303 may be implemented based on a mask automatic encoder (MAE). In other words, a prediction full image may be output by inputting the first sample image to the mask auto-encoder.

In some embodiments, S303 may include the following steps.

Step 1: A first vision feature is obtained by extracting a vision feature corresponding to an unobscured image from among the first sample images.

Here, vision features include such things as texture features, contour features, color features, and shape features, which are not listed here.

Correspondingly, the first vision feature refers to such as a texture feature, a contour feature, a color feature, and a shape feature corresponding to an unoccluded image of the first sample image.

Step 2: Acquire a predicted full image by predicting a partial image in the first sample image based on the first vision feature.

In this embodiment, combining vision features such as texture features, contour features, color features, and shape features corresponding to the non-occluded image to obtain a predictive full image is to obtain a predictive full image based on the vision context. Thus, it corresponds to acquiring a pre-training model capable of completing the learning of the context knowledge of the visual cue by training.

In some embodiments, the second step may include the following sub-steps.

First sub-step: estimating a vision feature corresponding to an obscured partial image of the first sample image based on the first vision feature to obtain a second vision feature.

Illustratively, referring to the above-described analysis, for the sub-step, the predicted and obscured portion based on vision features such as texture features, contour features, color features, and shape features corresponding to the non-occluded image It may be understood as acquiring vision features corresponding to the image, such as, for example, texture features, contour features, color features, and shape features.

Second sub-step: Determine an obscured partial image of the first sample image based on the second vision characteristic.

Illustratively, after acquiring vision features, such as texture features, contour features, color features, and shape features, corresponding to the occluded partial image, the occluded partial image may be supplemented and restored based on the vision feature. .

A third sub-step: a prediction full image is generated based on the non-occluded image among the first sample images and the determined partial image of the first sample image.

Referring to the above analysis, after supplementing and repairing the occluded partial image, the occluded partial image is restored immediately, and splicing for the non-occluded partial image and the reconstructed occluded partial image, the prediction full image , that is, by reconstructing the first sample image, so that the prediction full image and the first sample image are highly consistent, thereby improving the accuracy and reliability of the prediction full image.

S304: Acquire a second sample image.

As can be seen with reference to the above analysis, the first sample image and the second sample image may be the same image, and correspondingly, if the first sample image and the second sample image are the same image, the corresponding step is omitted. can do.

S305: Randomly hide partial text in the second sample image.

Similarly, the training of the network model is generally a process of repetitive training, and in this embodiment, each repetition training is to randomly cover the partial text of the second sample image, so the quantity of the second sample image can be one. And, of course, the number of second sample images may be plural, and the present embodiment is not limited thereto.

For example, partial words or partial phrases in the second sample image may be randomly hidden.

S306: Based on the non-occluded text in the second sample image, predict for the hidden text in the second sample image, and obtain predictive text content.

Illustratively, after randomly covering the second sample image, the partial text in the second sample image is hidden, and the other partial text is not hidden, wherein the text content of the partial text that is hidden based on the unhidden text ( That is, predictive text content) can be determined.

In this embodiment, by combining the method of "random occlusion + prediction", by determining the text content, indeterminacy in the training process can be increased, so that the pre-training model obtained by training restores the complete image. improve reliability.

Here, S305 - S306 may be implemented based on a Masked Language Model (MLM). In other words, the predicted text content may be output by inputting the second sample image to the mask language model.

In some embodiments, S306 may include the following steps.

Step 1: A first semantic feature is obtained by extracting a semantic feature corresponding to the text that is not covered by the second sample image.

Here, the semantic characteristic refers to the characteristic of a logical relationship between each character string. Correspondingly, for the first semantic characteristic, it can be understood as the characteristic of the logical relationship between each character string included in the unmasked text, and the characteristic of the relational relationship between each character (letter and/or word) in the unmasked text. can also be understood as

Second step: based on the first semantic feature, predict the hidden partial text in the second sample image to obtain the predicted text content.

In this embodiment, obtaining the predictive text content by combining semantic features such as a logical relationship between each character string corresponding to the non-occluded text is to acquire the predictive text content based on the semantic context, train it to obtain the semantic queue It corresponds to acquiring a pre-training model that can complete the context knowledge learning of

In some embodiments, the second step may include the following sub-steps.

First sub-step: Predicting a semantic feature corresponding to the partial text hidden in the second sample image based on the first semantic feature to obtain a second semantic feature.

Illustratively, referring to the above-described analysis, for the sub-step, predicting corresponding to the hidden partial text based on semantic characteristics, such as the characteristic of the logical relationship between each character string, corresponding to the unhidden text For example, it can be understood as acquiring semantic characteristics such as characteristics of a logical relationship between each character string.

Second sub-step: based on the second semantic feature, generate predictive text content.

Illustratively, after acquiring semantic features corresponding to the non-occluded text, such as, for example, a feature of a logical relationship between each character string, the semantic feature of the hidden partial text may be supplemented and restored based on the semantic feature.

Referring to the above analysis, after supplementing and restoring the semantic feature of the hidden partial text, the semantic feature of the hidden partial text is immediately restored, and the text content corresponding to the semantic feature (that is, the predictive text content) is By determining, the predictive text content and the text content of the hidden partial text are highly consistent, thereby improving the accuracy and reliability of the predictive text content.

S307: Obtain a pre-training model by training based on the predicted full image and the predicted text content, and generate a text recognition model based on the pre-training model.

Here, the text recognition model is for performing text recognition on an image to be recognized.

4 is a view according to a fourth embodiment of the present application. As shown in FIG. 4 , the training method of the text recognition model provided in this embodiment includes the following steps ( S401 - S405 ).

S401: Mask prediction is performed on a partial image of the obtained first sample image to obtain a prediction full image corresponding to the first sample image.

Similarly, in order to avoid complicated description, in the present embodiment, repeated description of the same technical features as in the above-described embodiment of the present embodiment will be omitted.

S402: Perform mask prediction on the partial text in the obtained second sample image to obtain predicted text content corresponding to the partial text.

S403: Acquire a pre-training model by training based on the predicted full image and the predicted text content.

Illustratively, based on the predicted full image and the predicted text content, the basic network model may be trained to obtain a pre-training model.

For example, based on the predicted full image and the predicted text content, the pre-training model may be obtained by adjusting the model parameters of the basic network model.

Here, the basic network model may be a Vision Transformer (ViT), a neural network model (Backbone), for example, a convolutional neural network model (CNN), or other network models, but not limited in this embodiment. does not

S404: Acquire a task to be recognized and a training image.

Here, the training image includes text.

Here, the recognition target task may be determined based on the recognition demand of the text recognition model. For example, the recognition target task may be a character detection task, a text recognition task, a field classification task, or other recognition tasks. It may be, and it is not listed here.

S405: Based on the task to be recognized and the training image, train the pre-training model to obtain a text recognition model.

Here, the text recognition model is for performing text recognition on an image to be recognized.

As can be seen with reference to the above analysis, the pre-training model not only has a model for completing the context knowledge learning of the vision cue, but also has a model for learning the semantic cue context knowledge, that is, the pre-training model is Since it is a multimodal feature extraction base, the text recognition model obtained by training by combining the pre-training model not only has a vision cue-based context knowledge recognition capability, but also has a semantic cue-based context knowledge recognition capability.

In addition, by training the pre-training model by combining the tasks to be recognized, training based on different recognition demands to obtain a corresponding text recognition model, thereby improving the flexibility and diversity of the text recognition model obtained by training, , can be applied to various recognition scenarios, and satisfy different recognition needs.

In some embodiments, the pre-training model (i.e., multimodal feature extraction base) is combined with a text detection network model (Efficient and Accuracy Scene Text, EAST), a segmentation-based character detection network (Differentiable Binarization, DB), and a text detection network (Look More Than). Once, LOMO), etc., to implement the character detection task of the text recognition model; Also, for example, the pre-training model can be loaded into a convolutional recurrent neural network (CRNN), where the convolutional recurrent neural network can use a Connectionist Temporal Classification (CTC) decoding method, and , may use an Attention decoding method, may use a Transformer decoding method, etc., and implement the text recognition task of the text recognition model therefrom; Also, for example, the pre-training model can be loaded into a Fully Connected (FC) model or a Convolutional Neural Networks (CNN) model to implement a field classification task of a text recognition model.

In some embodiments, S405 may include the following steps.

Step 1: By inputting the training image as a pre-training model, multi-modal feature maps corresponding to the training image are obtained.

Referring to the above-described analysis, the multimodal feature map is for representing a plurality of dimensional features of a training image, for example, a vision dimension feature and a semantic dimension feature. For example, the multimodal feature map may indicate image features and semantic features corresponding to the training image.

In some embodiments, the multimodal feature map may be represented by (d*h*w), where d represents the feature channel quantity, and h and w represent the height and width of the multimodal feature map.

Step 2: Create a text recognition model based on the task to be recognized and the multimodal feature map.

In this embodiment, the multimodal feature map may represent the features of the training image from a plurality of dimensions, and thus may not only represent the vision features of the training image, but may also represent the semantic features of the training image and represent vision features and semantic features have stronger reliability and fullness, so the text recognition model created by combining multimodal feature maps has stronger reliability and accuracy.

In some embodiments, the second step may include the following sub-steps.

First sub-step: Acquire a prediction recognition result according to a task to be recognized in a training image based on the multimodal feature map.

For example, a prediction recognition result (prediction text result) may be obtained by inputting a multimodal feature map into a convolutional recurrent neural network.

Second sub-step: based on the preset true recognition result of the training image and the predictive recognition result, build a text recognition model.

Here, the true recognition result may be obtained by labeling the training image in advance, and in this embodiment, the labeling method is not limited, for example, an artificial labeling method or an automatic labeling method.

Illustratively, it is possible to calculate a loss value between the true recognition result and the predicted recognition result, and if the loss value is greater than (or equal to) a preset loss threshold, training is repeatedly performed, and conversely, if the loss value is If it is less than the preset loss threshold, the text recognition model is completed, or, if the number of iterations reaches the preset number of iterations, the text recognition model is completed.

For example, if it is necessary to train a text recognition model to perform text recognition on a train ticket, the training image is a train ticket image, and the train ticket image is input as a pre-training model, and the multimodal feature map of the train ticket image is output, input the multimodal feature map to, for example, a convolutional recurrent neural network, and output prediction recognition results such as "date, train number, seat number" in the train ticket image, for example, By comparing "date, train number, seat number" (that is, the true recognition result), training to obtain a text recognition model, the text recognition model obtained by training is "date, train number, seat number" in the recognition target ticket image " can be used for the recognition of text content.

5 is a view according to a fifth embodiment of the present application. As shown in FIG. 5 , the text recognition method provided in this embodiment includes the following steps ( S501 - S502 ).

S501: Acquire a recognition target image.

Here, the recognition target image includes text.

Exemplarily, the subject of this embodiment may be a text recognition apparatus, and the text recognition apparatus may be the same apparatus as the training apparatus, or may be a different apparatus from the training apparatus, but this embodiment is not limited thereto.

S502: Perform text recognition on the recognition target image based on the pre-trained text recognition model to obtain text content in the recognition target image.

Here, the text recognition model is obtained based on the training method of the text recognition model according to any one of the above-described embodiments.

In some embodiments, S502 may include the following steps.

Step 1: Determine the multimodal feature map of the image to be recognized based on the text recognition model.

Step 2: Determine the text content in the recognition target image based on the multimodal feature map.

Here, the multimodal feature map of the recognition target image is for representing the vision characteristics and semantic features of the recognition target image.

Illustratively, referring to the above analysis, the text recognition model includes a pre-training model, and if the text recognition model is obtained by loading the pre-training model into a convolutional recurrent neural network and training, that is, the text recognition model is a convolutional model. It further includes a recurrent neural network, and this embodiment can be understood as follows.

An image to be recognized is input to the pre-training model, a multimodal feature map is output, and the multimodal feature map is input to a convolutional recurrent neural network to output text content in the image to be recognized.

6 is a view according to a sixth embodiment of the present application. 6, the training apparatus 600 of the text recognition model provided in this embodiment includes a prediction unit 601, a training unit 602, and a generation unit 603,

the prediction unit 601 performs mask prediction on the partial images in the obtained first sample images to obtain a prediction full image corresponding to the first sample image;

The prediction unit 601 also performs mask prediction on the partial text in the obtained second sample image to obtain the predicted text content corresponding to the partial text;

the training unit 602 trains on the basis of the predicted full image and the predicted text content to obtain a pre-training model;

The generating unit 603 generates a text recognition model based on the pre-training model, wherein the text recognition model is for performing text recognition on a recognition target image.

7 is a view according to a seventh embodiment of the present application. As shown in FIG. 7 , the training apparatus 700 of the text recognition model provided in the present embodiment includes a prediction unit 701 .

The prediction unit 701 is for performing mask prediction on a partial image among the obtained first sample images, to obtain a prediction full image corresponding to the first sample image.

The prediction unit 701 is also for performing mask prediction on the partial text in the obtained second sample image, to obtain the predicted text content corresponding to the partial text.

Referring to FIG. 7 , in some embodiments, the prediction unit 701 includes:

an obscuring sub-unit 7011 that randomly covers a partial target among target targets;

and a prediction sub-unit 7012 configured to obtain a prediction result by predicting a partial object hidden among the target objects based on the non-occluded object among the target objects.

Here, if the target object is a first sample image, a partial object of the target object is a partial image, and the prediction result is a prediction full image; If the target object is the second sample image, the partial object of the target object is the partial text, and the prediction result is the predicted text content.

In some embodiments, the prediction sub-unit 7012 includes:

an extraction module for obtaining a first target characteristic by extracting a target characteristic corresponding to an unobscured target among target targets;

and a prediction module configured to obtain a prediction result by predicting a partial object hidden among the target objects based on the first target feature.

wherein if the target object is a first sample image, the first object feature is a first vision feature; If the target object is a second sample image, the first object characteristic is the first semantic characteristic.

In some embodiments, the target object is a first sample image, the first object characteristic is a first vision characteristic; The prediction module is

a first prediction sub-module configured to obtain a second vision feature by predicting a vision feature corresponding to a partial image obscured in the first sample image based on the first vision feature;

a first determining sub-module configured to determine an obscured partial image of the first sample image based on the second vision characteristic;

and a first generating sub-module configured to generate a full prediction image based on an unobscured image among the first sample images and a partial image that is obscured from among the determined first sample images.

In some embodiments, the target object is a second sample image, and the first object characteristic is a first semantic characteristic; The prediction module is

a second prediction submodule configured to obtain a second semantic feature by predicting a semantic feature corresponding to the partial text hidden in the second sample image based on the first semantic feature;

and a second generation sub-module configured to generate the predicted text content based on the second semantic characteristic.

The training apparatus 700 of the text recognition model is

a training unit 702, configured to train on the basis of the predicted full image and the predicted text content to obtain a pre-training model;

Generate a text recognition model based on the pre-training model, wherein the text recognition model further includes a generating unit 703 for performing text recognition on a recognition target image.

Referring to FIG. 7 , in some embodiments, generating unit 703 includes:

Acquire a task to be recognized and a training image, wherein the training image includes an acquiring sub-unit 7031 including text;

and a training sub-unit 7032 configured to train a pre-training model based on the recognition target task and the training image to obtain a text recognition model.

In some embodiments, the training sub-unit 7032 comprises:

an input module for inputting a training image as a pre-training model to obtain a multimodal feature map corresponding to the training image;

and a generation module for generating a text recognition model based on the recognition target task and the multi-modal feature map.

In some embodiments, the generating module comprises:

a third prediction sub-module for predicting a prediction recognition result according to a task to be recognized in a training image based on the multimodal feature map;

and a building sub-module for building a text recognition model based on the preset true recognition result of the training image and the predictive recognition result.

8 is a view according to an eighth embodiment of the present application. As shown in FIG. 8 , the text recognition apparatus 800 provided in this embodiment includes:

an acquiring unit 801 for acquiring a recognition object image, wherein the recognition object image includes text;

and a recognition unit 802 that performs text recognition on the recognition object image based on the text recognition model trained in advance to obtain text content in the recognition object image.

Here, the text recognition model is obtained based on the training method of the text recognition model according to any one of the above-described embodiments.

As can be seen with reference to FIG. 8 , in some embodiments, the recognition unit 802 comprises:

a first determining unit 8021 for determining a multimodal feature map of the recognition target image based on the text recognition model;

and a second determining unit 8022 that determines text content in the recognition target image based on the multimodal feature map.

Here, the multimodal feature map of the recognition target image is for representing the vision characteristics and semantic features of the recognition target image.

9 is a view according to a ninth embodiment of the present application. As shown in FIG. 9 , the electronic device 900 according to the present application may include a processor 901 and a memory 902 .

memory 902 is for storing programs; Memory 902 may include volatile memory, such as static random-access memory (SRAM), double data rate synchronous dynamic random access memory (Double Data Rate Synchronous Dynamic Random Access) random-access memory (RAM), such as Memory, DDR SDRAM, and the like; The memory may include a non-volatile memory, for example, a flash memory. The memory 902 is for storing a computer program (application program, function module, etc. for implementing the above method), computer instructions, etc. ) may be stored in The above-described computer program, computer instructions, data, etc. may be called by the processor 901 .

The above-described computer programs, computer instructions, and the like may be stored in one or a plurality of memories 902 by dividing the sections. In addition, the above-described computer programs, computer instructions, etc. may be called by the processor 901 .

The processor 901 is for executing a computer program stored in the memory 902 to implement each step of the method according to the above-described embodiment.

Specifically, reference may be made to the relevant description in the above-described method examples.

The processor 901 and the memory 902 may be configured separately or may be integrally integrated. When the processor 901 and the memory 902 are separate components, the memory 902 and the processor 901 may be coupled to each other through the bus 903 .

The electronic device of this embodiment can perform the technical solution of the above-described methods, and since the specific implementation process and the technical principle are the same, repeated description is omitted here.

In the technical solution of this application, the collection, storage, use, processing, transmission, provision and disclosure of relevant user personal information (eg, face image, etc.) does not deviate

According to an embodiment of the present application, the present application further provides an electronic device, a readable storage medium, and a computer program product.

According to an embodiment of the present application, the present application further provides a computer program, the computer program is stored in a readable storage medium, at least one processor of the electronic device can read the computer program from the readable storage medium, At least one processor executes a computer program so that the electronic device performs the method according to any one of the above-described embodiments.

10 is a block diagram illustrating an exemplary electronic device 1000 capable of performing an embodiment of the present application. Electronic device means various types of digital computers such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, large computers, and other suitable computers. Electronic devices may refer to various types of mobile devices such as personal information terminals, cellular phones, smart phones, wearable devices, and other similar computing devices. The elements disclosed herein, their connections and relationships, and their functions, are illustrative only and are not intended to limit what is disclosed in the text and/or the required implementation of the present application.

10, the device 1000, based on a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003, It includes a computing unit 1001 capable of performing a variety of suitable operations and processing. The RAM 1003 may further store various programs and data necessary for the operation of the device 1000 . The computing unit 1001 , the ROM 1002 , and the RAM 1003 are connected to each other via a bus 1004 . An input/output (I/O) interface 1005 is also coupled to the bus 1004 .

A plurality of members of the device 1000 are connected to the I/O interface 1005 and include, for example, an input unit 1006 such as a keyboard, a mouse, and the like; output units 1007 such as, for example, various types of displays, speakers, etc.; a storage unit 1008 such as, for example, a magnetic disk, an optical disk, or the like; and a communication unit 1009 such as, for example, a network card, modem, wireless communication transceiver, and the like. The communication unit 1009 allows the device 1000 to exchange information/data with other devices via a computer network such as the Internet and/or various communication networks.

The computing unit 1001 may be various general-purpose and/or dedicated processing modules with processing and computing capabilities. Some examples of the computing unit 1001 include a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computational chips, a computing unit executing various machine learning model algorithms, and a digital signal processor (DSP). , and any suitable processor, controller, microcontroller, and the like. The computing unit 1001 performs each of the above-described methods and processes, for example, a training method of a text recognition model and a text recognition method. For example, in some embodiments, the training method of the text recognition model, the text recognition method may be implemented as a computer software program and explicitly stored in a machine-readable medium such as the storage unit 1008 . In some embodiments, portions or all of the computer program may be loaded and/or installed on the device 1000 via the ROM 1002 and/or the communication unit 1009 . When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001 , one or a plurality of steps of the training method of the text recognition model and the text recognition method described above may be performed. Alternatively, in other embodiments, computing unit 1001 may be configured to perform a text recognition method, a training method of a text recognition model, via any other suitable manner (eg, via firmware).

Various embodiments of the systems and techniques described herein include digital electronic circuit systems, integrated circuit systems, field programmable arrays (FPGAs), dedicated integrated circuits (ASICs), dedicated standard products (ASSPs), system-on-a-chip systems (SOCs). , load programmable logic device, computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs, wherein the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, and The processor may be a dedicated or general purpose programmable processor, and receives data and instructions from a storage system, at least one input device, and at least one output device, and transmits data and instructions to the storage system, the at least one input device, and may be transmitted to the corresponding at least one output device.

Program code for implementing the method of the present application may be written in any combination of one or a plurality of programming languages. Such program code may be provided to a processor or controller of a general-purpose computer, dedicated computer, or other programmable data processing device so that, when the program code is executed by the processor or controller, the functions/operations specified in the flowcharts and/or block diagrams are performed. have. The program code may run entirely on the machine or partially on the machine, as a standalone software package, partially on the machine and partly on a remote machine, or entirely on the remote machine or server.

In the context of this application, a machine-readable medium may be a tangible medium and may contain or store a program used by or in combination with an instruction execution system, apparatus, or apparatus. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or appliances, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory. (EPROM or Flash memory), optical fiber, portable compact read-only memory (CD-ROM), optical storage, magnetic storage, or any combination of the foregoing.

To provide for interaction with a user, the systems and techniques described herein may be implemented on a computer, the computer comprising a display device (eg, a CRT (cathode tube) or LCD for displaying information to the user. (liquid crystal display) monitor); and a keyboard and a pointing device (eg, a mouse or a trackball), wherein the user can provide input to the computer through the keyboard and the pointing device. Other types of devices may provide for interaction with the user. For example, the feedback provided to the user may be any form of sensing feedback (eg, visual feedback, auditory feedback, or tactile feedback); An input from the user may be received through any form (sound input, voice input, or tactile input).

The systems and techniques described herein provide a computing system that includes a background member (eg, as a data server), or a computing system that includes an intermediate member (eg, an application server), or a computing system that includes a front end member. system (eg, a user computer having a graphical user interface or Internet browser, through which a user may interact with embodiments of the systems and technologies described herein), or such a background It may be practiced in any combination of computing systems including members, intermediate members, or front end members. Any form or medium of digital data communication (eg, a communication network) may connect the members of the system to one another. Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

A computer system may include a client and a server. A client and server are typically remote from each other and typically interact through a communications network. Creates a client-server relationship through computer programs running on corresponding computers and having a client-server relationship to each other. The server may be a cloud server, also called a cloud computing server or a cloud host, and is a host product in a cloud computing service system, and is connected to an existing physical host and a virtual private server (“Virtual Private Server” or “VPS” for short). Address the shortcomings of existing management difficulties and weak service expansion. The server may be a server of a distributed system, or a server combined with a block chain.

It should be understood that steps may be rearranged, added, or deleted using the various types of processes described above. For example, each step described in the present application may be performed in parallel, sequentially, or in a different order, and as long as the technical solution disclosed in this application can achieve a desired result, this specification not limited about

The specific embodiments described above are not limited to the protection scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made based on design requirements and other factors. All modifications, equivalent substitutions and improvements made within the spirit and principle of the present application should be included within the protection scope of the present application.

Claims (23)

In the training method of a text recognition model,
performing mask prediction on a partial image among the obtained first sample images to obtain a predicted full image corresponding to the first sample image;
performing the mask prediction on the partial text in the obtained second sample image to obtain predicted text content corresponding to the partial text;
A pre-training model is obtained by training based on the predicted full image and the predicted text content, and a text recognition model is generated based on the pre-training model, wherein the text recognition model performs text recognition for a recognition target image. A method of training a text recognition model, comprising: a step to perform. The method of claim 1, wherein the mask prediction comprises:
randomly obscuring a partial object among the target objects;
Based on the non-occluded target among the target targets, predicting the partial target of the target target to obtain a prediction result; includes,
wherein, if the target object is a first sample image, a partial object of the target object is a partial image, and the prediction result is the prediction full image; and if the target object is a second sample image, a partial object of the target object is a partial text, and the prediction result is the content of the predicted text. 3. The method of claim 2,
The step of obtaining a prediction result by predicting a partial object covered among the target objects based on the non-occluded object among the target objects,
obtaining a first target characteristic by extracting a target characteristic corresponding to an unobscured target among the target targets;
Based on the first target characteristic, predicting a partial target of the target target, and obtaining the prediction result; includes,
wherein if the target object is a first sample image, the first object feature is a first vision feature; If the target object is a second sample image, the first object feature is a first semantic feature. 4. The method of claim 3,
the target object is a first sample image, and the first object feature is a first vision feature; The step of obtaining the prediction result by predicting the partial object hidden among the target objects based on the first target feature,
obtaining a second vision characteristic by predicting a vision characteristic corresponding to an obscured partial image of the first sample image based on the first vision characteristic;
determining an obscured partial image of the first sample image based on the second vision characteristic;
The method of training a text recognition model comprising: generating the predicted full image based on an image that is not obscured from among the first sample images and the determined partial image of the first sample image. 5. The method of claim 3 or 4,
the target object is a second sample image, and the first object feature is a first semantic feature; The step of obtaining the prediction result by predicting a partial object hidden among the target objects based on the first target characteristic,
obtaining a second semantic feature by predicting a semantic feature corresponding to the partially hidden text in the second sample image based on the first semantic feature;
and generating the predicted text content based on the second semantic feature. 5. The method according to any one of claims 1 to 4,
The step of generating a text recognition model based on the pre-training model comprises:
acquiring a task to be recognized and a training image, wherein the training image includes text;
Training method of a text recognition model comprising a; training on the pre-training model based on the recognition target task and the training image to obtain the text recognition model. 7. The method of claim 6,
The step of obtaining the text recognition model by training on the pre-training model on the basis of the recognition target task and the training image,
obtaining a multimodal feature map corresponding to the training image by inputting the training image as the pre-training model;
Generating the text recognition model based on the recognition target task and the multi-modal feature map; training method of a text recognition model comprising a. 8. The method of claim 7,
The step of generating the text recognition model based on the recognition target task and the multi-modal feature map comprises:
predicting a prediction recognition result according to the recognition target task of the training image based on the multimodal feature map;
The training method of a text recognition model comprising a; based on a preset true recognition result of the training image and the predictive recognition result, building the text recognition model. In the text recognition method,
acquiring a recognition target image, wherein the recognition target image includes text;
Performing text recognition on the recognition target image based on a previously trained text recognition model to obtain text content in the recognition target image;
Here, the text recognition model is a text recognition method, characterized in that it is obtained based on the training method of the text recognition model according to any one of claims 1 to 4. 10. The method of claim 9,
The step of performing text recognition on the recognition target image based on a pre-trained text recognition model to obtain text content in the recognition target image includes:
determining a multi-modal feature map of the recognition target image based on the text recognition model, and determining text content in the recognition target image based on the multi-modal feature map,
Here, the multi-modal feature map of the recognition target image is a text recognition method for representing vision features and semantic features of the recognition target image. In the training apparatus of the text recognition model,
a prediction unit, a training unit, and a generating unit;
the prediction unit performs mask prediction on the partial images in the obtained first sample images to obtain a prediction full image corresponding to the first sample image;
The prediction unit is further configured to perform the mask prediction on the partial text in the obtained second sample image to obtain a predicted text content corresponding to the partial text;
the training unit trains on the basis of the predicted full image and the predicted text content to obtain a pre-training model;
and the generating unit generates a text recognition model based on the pre-training model, wherein the text recognition model is configured to perform text recognition on a recognition target image. The method of claim 11 , wherein the prediction unit comprises:
an obscuring sub-unit randomly covering a partial target among the target targets;
A prediction sub-unit configured to obtain a prediction result by predicting a partial object hidden among the target objects based on the non-occluded object among the target objects; and
wherein, if the target object is a first sample image, a partial object of the target object is a partial image, and the prediction result is the prediction full image; When the target object is a second sample image, the partial object of the target object is a partial text, and the prediction result is the content of the predicted text. The method of claim 12, wherein the prediction sub-unit,
an extraction module for obtaining a first target characteristic by extracting a target characteristic corresponding to an unobscured target among the target targets;
A prediction module for obtaining the prediction result by predicting a partial object hidden among the target objects based on the first target feature;
wherein if the target object is a first sample image, the first object feature is a first vision feature; If the target object is a second sample image, the first object feature is a first semantic feature. 14. The method of claim 13,
the target object is a first sample image, and the first object feature is a first vision feature; The prediction module is
a first prediction sub-module configured to obtain a second vision feature by predicting a vision feature corresponding to an obscured partial image of the first sample image based on the first vision feature;
a first determining sub-module configured to determine an obscured partial image of the first sample image based on the second vision characteristic;
The training apparatus of a text recognition model comprising a; a first generation sub-module for generating the predicted full image based on an image that is not obscured from among the first sample images and the determined partial image of the first sample image. 15. The method of claim 13 or 14,
the target object is a second sample image, and the first object feature is a first semantic feature; The prediction module is
a second prediction submodule configured to obtain a second semantic feature by predicting a semantic feature corresponding to the partial text hidden in the second sample image based on the first semantic feature;
and a second generation sub-module configured to generate the predicted text content based on the second semantic feature. The method according to any one of claims 11 to 14, wherein the generating unit comprises:
an acquiring sub-unit for acquiring a task to be recognized and a training image, wherein the training image includes text;
and a training sub-unit configured to acquire the text recognition model by training the pre-training model based on the recognition target task and the training image. The method of claim 16, wherein the training sub-unit,
an input module for inputting the training image as the pre-training model to obtain a multimodal feature map corresponding to the training image;
A text recognition model training apparatus comprising a; a generating module for generating the text recognition model based on the recognition target task and the multi-modal feature map. The method of claim 17, wherein the generating module comprises:
a third prediction sub-module for predicting a prediction recognition result according to the recognition target task of the training image based on the multimodal feature map;
Training apparatus for a text recognition model comprising a; a building sub-module for constructing the text recognition model based on a preset true recognition result of the training image and the predictive recognition result. A text recognition device comprising:
an acquiring unit for acquiring a recognition object image, wherein the recognition object image includes text;
a recognition unit configured to perform text recognition on the recognition object image based on a previously trained text recognition model to obtain text content in the recognition object image; and
Here, the text recognition model is a text recognition apparatus, characterized in that it is obtained based on the training method of the text recognition model according to any one of claims 1 to 4. The method of claim 19, wherein the recognition unit,
a first determining unit for determining a multimodal feature map of the recognition target image based on the text recognition model;
a second determining unit that determines text content in the recognition target image based on the multimodal feature map;
Here, the multi-modal feature map of the recognition target image is a text recognition apparatus for indicating vision features and semantic features of the recognition target image. In electronic devices,
at least one processor; and
a memory communicatively coupled to the at least one processor; and
An instruction executable by the at least one processor is stored in the memory, and the instruction is executed by the at least one processor, so that the at least one processor according to any one of claims 1 to 4 to perform a training method of a text recognition model; Alternatively, the electronic device characterized in that the at least one processor performs the text recognition method according to claim 9 . A non-transitory computer readable storage medium storing computer instructions, the computer instructions causing a computer to perform the method of training a text recognition model according to any one of claims 1 to 4; Alternatively, the computer instruction causes the computer to perform the text recognition method according to claim 9 . In a computer program stored in a computer-readable storage medium,
implementing the training method of a text recognition model according to any one of claims 1 to 4 when the computer program is executed by a processor; or a computer program for implementing the text recognition method according to claim 9 when said computer instructions are executed by a processor.

Images