CN114037671B

CN114037671B - Microscopic hyperspectral leukocyte detection method based on improvement FASTER RCNN

Info

Publication number: CN114037671B
Application number: CN202111282117.XA
Authority: CN
Inventors: 李良军; 曾凡一; 黄杰; 苏颖; 石爽; 蔡虹
Original assignee: Second Hospital of Dalian Medical University
Current assignee: Second Hospital of Dalian Medical University
Priority date: 2021-11-01
Filing date: 2021-11-01
Publication date: 2024-10-29
Anticipated expiration: 2041-11-01
Also published as: CN114037671A

Abstract

The invention belongs to the technical field of medical treatment, and provides a white blood cell detection method based on an improved FASTER RCNN microscopic hyperspectral image, which uses a hyperspectral microscope to obtain a white blood cell detection image of a blood smear and carries out identification and classification on white blood cells based on an improved FASTER RCNN. And (3) obtaining pseudo-color images and spectrum data of the blood smear by using a hyperspectral microscope, marking different types of white blood cells, and manufacturing a data set. The traditional FASTER RCNN network is improved, namely the VGG16 in the FASTER RCNN original network is replaced by Resnet and is used as a new pseudo-color image feature extraction network; and establishing a spectrum data extraction module aiming at blood smear hyperspectral data, extracting spectrum features by utilizing a one-dimensional convolutional neural network, and fusing white blood cell image features and spectrum features by utilizing a FASTER RCNN network on the basis of the improvement, so as to finally realize white blood cell identification and classification. Compared with the traditional FASTER RCNN network, the invention has obvious improvement on the identification precision and classification accuracy of the white blood cells.

Description

Microscopic hyperspectral leukocyte detection method based on improvement FASTER RCNN

Technical Field

The invention belongs to the technical field of medical treatment, and particularly relates to a microscopic hyperspectral leukocyte detection method based on improvement FASTER RCNN.

Background

White blood cells are an important component of blood, produced by bone marrow and lymphoid tissue, and function to combat viral and bacterial infections. Can be classified into neutrophils, lymphocytes, monocytes, eosinophils, and basophils. Traditional manual microscopic detection methods are overly complex, and manual differential counting of leukocytes is more prone to error, particularly when detecting large numbers of samples. In recent years, corresponding computer vision algorithms and systems have been widely used in the field of automatic classification and detection of blood cells.

Computer vision inspection can be divided into two major categories at present, the first category is to simply stack traditional algorithm modules according to the order of preprocessing, feature extraction and classification. Rawat et al propose a new technique for distinguishing acute lymphoblastic leukemia (Acute Lymphoblastic Leukemia) from healthy lymphocytes, using an SVM classifier to classify fusion features including texture and shape with an accuracy of 89.8%. Mohapatra et al adopts a two-stage color segmentation strategy based on fuzzy clustering to segment white blood cells in an image, and proposes two new shape features of Hausdorff dimension and outline feature, and uses SVM to classify lymphatic nuclei, so that the accuracy reaches 95%. Alferez et al utilized color component clustering and watershed transformation to segment peripheral blood cell images, and extracted 113 features altogether for different types of lymphocyte recognition. The accuracy of the training set is 98.07%, and the accuracy of the verification set is 85.33%.

However, most of the conventional pattern recognition algorithms are used for sequentially completing the representation and classification modules, the super parameters of the conventional pattern recognition algorithms are usually given by experience before training, and the generalization capability and stability of the model cannot be ensured when the sample space is small or the inter-class gap is not obvious.

The second category is to analyze pathological pictures based on a deep learning algorithm, and the computer can extract multidimensional features more comprehensively and automatically without manually designating the extraction of a certain feature, so that after a large amount of training, the targets in the images can be identified and classified more accurately, efficiently and stably, and strong strength is shown in a plurality of fields, particularly in the medical field with a large amount of image analysis and processing, and the method has the advantage incomparable with the traditional machine learning algorithm. Boldu et AL in the study of acute leukemia detection, a dual-module deep learning architecture (AL Net) was used to identify and classify abnormal promyelocytic and blast cells, respectively, and the accuracy of identification of myeloid leukemia and lymphoid leukemia reached 93.7% and 100%, respectively. SHAKARAMI et al used a fast and efficient YOLOv (FED) detector with EFFICIENTNET convolutional neural network as the backbone, enabling blood cell identification based on BCCD datasets. Finally, the classification accuracy of platelets, erythrocytes and leukocytes was 90.25%, 80.41% and 98.92%, respectively.

In the published patent, it is referred to the detection of cells using FASTER RCNN networks as follows:

Patent CN109598224a proposes a method for detecting white blood cells in bone marrow section based on region recommended convolution, which uses Resnet and FPN as a characteristic extraction network of FASTER RCNN, and bone marrow section image as input, and the accuracy can reach more than 90%. Patent CN110059672a proposes a method for performing class-increasing learning on a microscope cell image detection model by using incremental learning, calculates the distance between an intermediate feature layer obtained by predicting new class cells by using a FASTER RCNN model trained by an old class cell image and an intermediate feature layer of an incremental model, and obtains a new class-increasing prediction model. Patent CN110580699a proposes a pathological image cell nucleus detection method based on an improved FASTER RCNN algorithm, uses ZF as a FASTER RCNN feature extraction network, processes sample pictures by data enhancement transformation, difficult sample mining and small target detection optimization, achieves a monitoring speed of 1 s/sheet, and has good robustness. Patent CN111598849a proposes a pathological image cell counting method, device and medium based on target detection, which refers to two different FASTER RCNN models trained by bockbone to perform cell counting detection on pathological images, and fuses the detection results to accurately, rapidly and efficiently realize pathological cell counting.

However, these methods are only based on gray scale or RGB images with cellular space characteristics, and are susceptible to various environmental factors such as optical conditions of a microscope, thickness of a slide, etc. The hyperspectral imaging technology is a combination of the traditional imaging technology and the spectrum technology, so that not only the spatial characteristics of a monitored target but also the spectrum characteristics of the monitored target can be extracted. In the medical field, it has been used for the identification and diagnosis of acute lymphoblastic leukemia, food borne pathogens and cancer. Although hyperspectral imaging provides rich spectral information on the basis of spatial features, hundreds of narrow continuous bands make the data dimension redundant, requiring manual extraction of spectral values of a region of interest (ROI), which is extremely cumbersome in practical monitoring. Therefore, there is a strong need for an automatic, fast and efficient method of white blood cell detection based on hyperspectral imaging.

According to the invention, the original VGG16 characteristic extraction network FASTER RCNN is replaced by Resnet, except for stacking on a simple convolution layer, the cross arrangement of two basic blocks of Conv Block and Identity Block in the network can avoid losing important characteristics due to mismatch between the number of convolution layers and data, and the generalization of the network is far superior to that of VGG16. Besides the replacement of the feature extraction network, the invention is different from the original FASTER RCNN network frame in that the biggest innovation point of the patents and researches is that a complete set of spectrum feature extraction network and feature fusion network are constructed and embedded into FASTER RCNN, thereby realizing the novel automatic white blood cell identification and classification technology based on the combined detection of space and spectrum features.

Disclosure of Invention

In order to solve the technical problems, the invention provides a microscopic hyperspectral leukocyte detection method based on an improved FASTER RCNN algorithm, which adds a detection network branch of spectrum data in a traditional FASTER RCNN network architecture, realizes the joint detection of the spatial characteristics and the spectral characteristics of blood images, and effectively improves the classification precision.

The technical scheme of the invention is as follows:

the microscopic hyperspectral leukocyte detection method based on the improvement FASTER RCNN comprises the following steps:

1. Establishing a leukocyte dataset: collecting a pseudo-color image (an image synthesized by 667.4,557.2 and 440.5 wave bands) and a spectrum image of a blood smear by using a microscopic hyperspectral system, correcting the color of the pseudo-color image, marking five types of white blood cells by frame selection and category, correcting the black and white of the spectrum image, and manufacturing a training and verification data set;

2. And (3) constructing a pseudo-color image feature extraction network: resnet18 which can extract more effective features is adopted as a feature extraction network of FASTER RCNN;

3. Constructing an RPN proposal network, which is responsible for automatically generating an initial target priori frame, carrying out preliminary classification on the priori frame, and then finally screening according to the priori frame to obtain a proposal frame, and intercepting corresponding features at a public feature layer;

4. And constructing a suggested frame coordinate return branch, constructing a suggested frame coordinate return network branch by adopting a full connection layer (Dense), and decoding the suggested frame coordinate return network branch to obtain coordinate values of the target feature on the original image.

5. And (3) constructing a spectrum data extraction module: using FASTER RCNN to detect four coordinate values obtained by a network terminal positioning frame coordinate branch as input of a spectrum extraction module, intercepting full-band data of a corresponding region in a spectrum image according to the coordinate, and then calculating an average spectrum value of the full-band data to obtain an average spectrum curve as input of a spectrum feature extraction network;

6. Constructing a spectrum feature extraction network, additionally constructing a one-dimensional spectrum data feature extraction network, and outputting one-dimensional spectrum features;

7. Construction of a feature fusion network: the characteristics extracted from the tail end of the spectrum characteristic extraction network are fused with a FASTER RCNN tail end of a characteristic layer based on cell pseudo-color image classification;

8. Construction of cell sorting layer: calculating the obtained one-dimensional fusion characteristics by using the full-connection layer and the softmax classification layer to obtain a final cell classification result;

Further, in step (1), the hyperspectral camera cannot obtain a conventional RGB image due to the limitation of the band range, so that the RGB image, i.e. the pseudo-color image, is synthesized by extracting 667.4,557.2 and 440.5 three bands representing R, G, and B channels, respectively. The Gamma correction is used for correcting the color of the pseudo-color image, so that the influence of various environmental factors, such as the optical condition of a microscope, the thickness of a slide and the like, on the brightness of images in different batches is reduced. Black and white correction of spectral images specifically refers to: collecting a pair of blank slide spectral images as white reference images for correction, namely, defining the spectral reflectivity of a blank slide as the highest value; and covering a lens cover of the spectrum camera, closing a light source, and collecting a spectrum image in a pure black environment as a corrected black reference image, namely, defining the spectrum reflectivity in the dark environment as the minimum value. And then carrying out normalization processing on the data of all the spectral images based on the black-white correction data to obtain corrected spectral images. Each white blood cell within the image field of view is framed and based on the corresponding class label, the final dataset includes for each image: a pseudo color image; a spectral image; a property file (containing the coordinates of the positioning frame and the classification label of the white blood cells contained in the image).

Further, in the step (2), the feature extraction network Resnet of the pseudo-color image is specifically built by using a small convolution kernel of 3*3, two basic blocks of Conv Block and Identity Block in the network are arranged in a crossing manner, the Conv Block is used for compressing the image, the Identity Block is not used for changing the size of the image, and the key point is that the two feature extraction modules combine the extracted features with the original features, thereby not only ensuring the extraction of important features, but also ensuring that the original features are not missed due to deepening of the layer number. The spectrum feature extraction network is built in the form of Conv1D+ MaxPooling D layer based on 1*3 convolution kernels.

Further, in step (3), the common feature obtained by the pseudo-color image after passing through the feature extraction network has two application paths, one of which is to sweep it through a 3*3-size convolution to obtain a feature of a deeper layer, and then to divide it into two network branches, one of which is used to generate 9 prior frames for each point. And secondly, generating a deviation value of each priori frame and the true annotation frame. These two sets of features are then passed sequentially into propasal and ROI Pooling layers for preliminary classification of the prior frames and screening of the final suggested frames, respectively. The second purpose of the public feature layer is to input the public feature layer into the ROI Pooling layer, intercept corresponding feature blocks according to the screened suggestion boxes, unify the feature fast sizes and obtain final target features.

Further, in step (4) and step (5), after the pseudo color image is input to FASTER RCNN network and features are extracted by Resnet, the pseudo color image passes through RPN network, the network determines those areas in the image are cell bodies, and feature blocks with different sizes are obtained, and after ROI Pooling layers, the feature blocks proposed by the network are converted into uniform sizes. Then FASTER RCNN is divided into two branches, one branch is a return network for predicting the coordinates of the positioning frame and is used for learning and returning the four coordinates of the predicted cell positioning frame and the deviation value of the coordinates of the labeling positioning frame in the attribute file, and the other branch is a classification network for five classifications of cells. The spectrum data is stored in a picture file of a raw type, the file type is that reflectivity values of all pixels in the spectrum picture corresponding to all wave bands are arranged end to form a one-dimensional list, the one-dimensional list is rearranged into a three-dimensional data block according to a format of length multiplied by height multiplied by wave band number of an image, then the corresponding pixel points and the reflectivity values of the corresponding pixel points in all wave bands are cut out by utilizing four coordinate values of a cell prediction coordinate frame obtained based on a pseudo-color image, and the reflectivity average value of all the pixel points is obtained under each wave band to obtain an average spectrum curve which is used as input of a spectrum characteristic network.

Further, in the step (6), the average spectrum curve of each cell obtained by the spectrum data extraction module is one-dimensional data, so that a spectrum-oriented feature extraction network is built by using Conv 1D. The network is built by a Conv1D+ Max Pooling structure, 350-dimensional spectrum data are compressed, after a feature layer is obtained, the Flatten operation is carried out, a one-dimensional full feature layer is obtained, a full-connection Dense layer is connected, and then the layer is connected with a pseudo-color image feature layer in a FASTER RCNN tail cell classification branch.

Further, in the step (7), before the classification of FASTER RCNN is performed by using the softmax layer, a plurality of feature blocks obtained by the RPN layer are subjected to a flat operation to be changed into a one-dimensional feature layer, the last layer of the spectrum feature extraction network is also a one-dimensional feature layer, and the two feature layers are spliced according to the weight of 1:1, so that a fusion feature layer comprising space and spectrum features is obtained.

Further, in the step (8), after a one-dimensional fusion feature layer is obtained, the features are transferred into a full-connection layer and a classification layer of softmax, so that classification based on the fusion features is realized.

Compared with the prior art, the invention provides a microscopic hyperspectral leukocyte detection method based on improvement FASTER RCNN, and particularly creates innovation in the aspect of feature Fusion detection of multi-sensor data, and provides a multi-class data feature Fusion detection method, namely a multi-feature Fusion network Fusion-Net.

Drawings

FIG. 1 is a schematic block diagram of the present invention;

FIG. 2A is a microscopic hyperspectral image of class 1 (neutrophils) and class 5 (monocytes) cells;

FIG. 2B is a microscopic hyperspectral image of class 1 (neutrophils) and class 3 (basophils) cells;

FIG. 2C is a microscopic hyperspectral image of class 1 (neutrophils), class 3 (basophils) and class 4 (lymphocytes) cells;

FIG. 2D is a microscopic hyperspectral image of class 1 (neutrophils) and class 2 (eosinophils) cells;

FIG. 3Resnet is a schematic diagram of a feature extraction network; (a) is Conv Block; (b) is Identity Block; (c) is Resnet < 18 >;

FIG. 4 is a spectral feature extraction network;

FIG. 5 is a schematic diagram of a network structure of FASTER RCNN after modification;

FIG. 6A is a graph showing the results of conventional FASTER RCNN-class 1 (neutrophils) and 5-class 5 (monocytes) cell detection classification;

FIG. 6B is a graph showing the results of conventional FASTER RCNN-based classification of class 1 (neutrophils) and class 3 (basophils) cells;

FIG. 6C is a graph showing the results of conventional FASTER RCNN-based classification of class 1 (neutrophils), class 3 (basophils) and class 4 (lymphocytes) cells;

FIG. 6D is a graph showing the results of conventional FASTER RCNN-based classification of class 1 (neutrophils) and class 2 (eosinophils) cells;

FIG. 7A is a graph showing the results of the detection classification of FASTER RCNN cells of class 1 (neutrophils) and class 5 (monocytes) after modification of the present invention;

FIG. 7B is a graph showing the results of the detection classification of FASTER RCNN versus class 1 (neutrophils) and class 3 (basophils) cells after modification of the present invention;

FIG. 7C is a graph showing the results of the detection classification of FASTER RCNN cells of class 1 (neutrophils), class 3 (basophils) and class 4 (lymphocytes) after modification of the invention;

FIG. 7D is a graph showing the results of the detection classification of FASTER RCNN versus class 1 (neutrophils) and class 2 (eosinophils) cells after modification of the present invention.

Detailed Description

The following describes the embodiments of the present invention further with reference to the drawings and technical schemes.

As shown in fig. 1, a microscopic hyperspectral leukocyte detection method based on the improvement FASTER RCNN comprises the following specific implementation steps:

s1, establishing a leukocyte data set:

the hyperspectral camera is arranged on a three-eye student microscope, and the wave band used by the hyperspectral camera is a near infrared short wave light region between 382nm and 1020 nm. The method comprises the steps of finding a visual field containing white blood cells through an eyepiece, enabling a detection target in the visual field to be clearly visible through focusing and other operations, and then collecting images and spectrum information in the visual field by using a hyperspectral camera.

And carrying out color correction on the acquired pseudo-color image. And Gamma correction is adopted, so that the influence of various environmental factors, such as the optical condition of a microscope, the thickness of a slide and the like, on the brightness of images of different batches is reduced. The gamma correction is a method of editing a gamma curve of an image to perform nonlinear tone editing of the image, and increases the ratio of dark color portions to light color portions in an image signal to thereby improve the image contrast effect.

And performing box selection and category labeling on the five types of white blood cells. And using marking software to carry out frame selection on each cell on each image, giving corresponding classification labels, and generating an attribute file containing four coordinates of the positioning frame and the corresponding labels. The principle of the frame selection is that background elements of non-leucocytes appear in the frame as little as possible, and when two leucocytes are overlapped, the content of non-own cell elements in the frame is reduced as much as possible.

Black and white correction is performed on the spectral image. To reduce the sample position differences and the effect of the light source on the image, the original hyperspectral image is corrected to the reflectance mode. Collecting a pair of blank slide spectral images as white reference images for correction, namely, defining the spectral reflectivity of a blank plectrum to be the highest value (the reflectivity is close to 100%); the lens cover of the spectrum camera is covered, the light source is turned off, and a spectrum image under a pure black environment is collected and used as a corrected black reference image, namely, the spectrum reflectivity under the dark environment is regulated to be the minimum value (the reflectivity is close to 0%). And then carrying out normalization processing on the data of all the spectral images based on the black-white correction data to obtain corrected spectral images. The correction process is as follows:

R＝(I-B)/(W-B)×100％

Wherein I represents an original image; w represents a full white calibration image; b represents a full black calibration image; r represents the corrected relative image.

For each image, the final dataset includes: a pseudo color image; a spectral image; a property file (containing the coordinates of the positioning frame and the classification label of the white blood cells contained in the image). The image of five types of white blood cells collected is shown in fig. 2.

S2, pseudo color image feature extraction network:

Resnet18 was employed as the feature extraction network for FASTER RCNN. As shown in fig. 3, resnet has two basic blocks, namely Conv Block and Identity Block, wherein the dimensions of Conv Block input and output are different, so that the Conv Block cannot be serially connected, and the effect of the Conv Block is to change the dimension of a network; the Identity Block input dimension and the output dimension are the same and can be connected in series to deepen the network. The pseudo-color image feature extraction network of the improvement FASTERRCNN only comprises four image size compressions, and the content after the fifth compression is used after ROI Pooling layers. The specific structure is shown in the Resnet block diagram in fig. 3, and after the 600×600×3 picture at the time of input is sent to Resnet, a feature block with a size of 38×38×256 is obtained.

S3.rpn proposed network:

As shown in the network structure schematic diagram of FASTER RCNN after improvement in fig. 5, the public feature obtained by the pseudo-color image after passing through the feature extraction network has two application ways, one of which is to sweep it through one 3*3-sized convolution to obtain a feature of a deeper layer, and then to divide it into two network branches, one of which is to sweep it through 9-channel convolution with a size of 1*1, that is, the number of prior frames generated by each pixel point in FASTER RCNN is 9. The second is to sweep using 36-channel convolution of size 1*1, i.e., each a priori frame contains 4 coordinate variables, so 9*4 =36 channels are produced per pixel. Both sets of features are input to proposal of FASTER RCNN, where the objects in each prior box are initially classified, i.e., whether the object is included or not is determined, and then the object to be detected is found in the prior box determined to be an object. At this time, the feature sizes of the extracted targets are different, and then the sizes of the prior frames are unified at ROI Pooling layers to obtain the final suggestion frame. Simultaneously, the common features extracted from the previous feature network are also input into the ROI Pooling layers, the features of 14 x 14 are cut out from the common features according to the suggested frame coordinates after the unified size, and then the features of each suggested frame are subjected to the fifth compression of the original Resnet to obtain the features of 7*7. So far, the whole RPN network is ended, and a plurality of characteristic blocks which are judged to be detection targets by the RPN network are obtained.

S4-S5, suggesting a frame coordinate return branch and a spectrum data extraction module:

After obtaining a number 7*7 of feature blocks through the RPN network, the flag operation is performed to become one-dimensional data, and then there are two branches, namely a return branch of the coordinates of the suggestion frame, which is constructed by a full connection layer, as shown in fig. 5, and is used for adjusting the corresponding suggestion frame, and 20 (the number of classification×4) neurons are included, that is, each cell includes 4 coordinate parameters no matter which class is divided into. Finally, after decoding the layer of data on the original image, coordinate values of four vertexes can be obtained, one of the purposes is to draw a recognition frame on the original image, and the other is to use four coordinate values obtained by suggesting frame coordinate branches as input of a spectrum extraction module. The spectrum data extraction module is written by using OpenCV and mainly comprises three parts: reading and arranging spectrum data in a raw file; intercepting a corresponding data block according to the coordinate value of the suggestion frame; and carrying out average calculation on the intercepted spectrum data on each wave band to obtain average spectrum data of the target in each suggestion frame, wherein the average spectrum data is used as input of a spectrum characteristic extraction network.

S6, spectral feature extraction network:

the average spectrum curve of each cell obtained through the spectrum data extraction module is one-dimensional data, so that a spectrum-oriented characteristic extraction network is built by using Conv 1D. A specific network structure is shown in fig. 4. The network is built by a Conv1D+ Max Pooling structure, 350-dimensional spectrum data are compressed, after a characteristic layer of 1 x 173 x 64 is obtained, a flat operation is carried out, a full characteristic layer of 1 x 5568 is obtained, a full connection Dense layer is connected, and then the layer is connected with a pseudo-color image characteristic layer in a FASTER RCNN tail cell classification branch.

S7, feature fusion network:

As shown in fig. 5, before classifying by softmax in the classification branch of FASTER RCNN, the features obtained by the RPN layer are subjected to the flatten operation to become a one-dimensional feature layer, the last layer of the spectral feature extraction network is also a one-dimensional feature layer, and the two feature layers are spliced according to a weight of 1:1, so that a fused feature layer comprising spatial and spectral features is obtained.

S8, cell classification network:

after a one-dimensional fusion feature layer is obtained, the features are transferred into a full-connection layer and a classification layer of softmax, so that classification based on the fusion features is realized.

The above is all the steps of the flow of the invention, and during training, the data set is randomly divided into a training set and a testing set according to the ratio of 7:3. Meanwhile, a conventional FASTER RCNN network which is characterized by VGG16 extraction and is trained only based on pseudo-color images is used as a control group, the detection result is shown in fig. 6, detection confusion can occur on two white blood cells which are closer to each other, two white blood cells are used as one cell for framing, and due to interference of a large number of red blood cells on a blood picture, part of white blood cells cannot be identified, besides the identification accuracy, the conventional FASTER RCNN network classifies all the identified white blood cells into one type, and the fact that the conventional FASTER RCNN network is poor in identification and classification accuracy is reflected in the condition that a data set is smaller. Fig. 7 shows the improved FASTER RCNN network detection result based on the combined detection of spatial and spectral features, and the same test image, each white blood cell is accurately identified and positioned, which shows the advantage of Resnet in feature extraction, and each white blood cell is accurately classified into 5 classes, which is because the white blood cells have obvious advantages in the inter-class distinction of white blood cells under the conditions that the difference between the white blood cells on the image is small and the data set is not huge enough.

Claims

1. The microscopic hyperspectral leukocyte detection method based on the improvement FASTER RCNN is characterized by comprising the following steps:

(1) Establishing a leukocyte dataset: collecting a pseudo-color image and a spectrum image of a blood smear by using a microscopic hyperspectral system, carrying out color correction on the pseudo-color image, carrying out frame selection and category marking on five types of white blood cells, carrying out black-and-white correction on the spectrum image, and manufacturing a training data set and a verification data set;

The pseudo-color image is synthesized by images corresponding to three wavebands 667.4, 557.2 and 440.5, the three wavebands correspond to R, G and B channels of the image respectively, and Gamma is used for correcting the color of the pseudo-color image;

The black-and-white correction of the spectrum image specifically refers to: collecting a pair of blank slide spectral images as white reference images for correction, namely, defining the spectral reflectivity of a blank slide as the highest value; covering a lens cover of a spectrum camera, closing a light source, and collecting a spectrum image in a pure black environment as a corrected black reference image, namely, defining the spectrum reflectivity in the dark environment as the minimum value; then, carrying out normalization processing on the data of all the spectrum images based on the white reference image and the black reference image to obtain corrected spectrum images; each white blood cell in the image view is framed and selected based on the corresponding classification label; for each image, the final dataset comprises a pseudo-color image, a spectral image, a property file containing the location frame coordinates and classification tags of the white blood cells contained in the image;

(2) Constructing a pseudo color image feature extraction network: resnet18 is adopted as a feature extraction network of FASTER RCNN;

Resnet 18 the construction is carried out by using a 3*3 small convolution kernel, a characteristic extraction module in the network is a Conv Block and an Identity Block which are arranged in a crossing way, the Conv Block is used for compressing an image, the Identity Block cannot change the size of the image, and the two characteristic extraction modules combine the extracted characteristics with original characteristics to obtain a common characteristic layer;

The characteristic extraction network of the spectrum image is built in a form of Conv1D+ MaxPooling D layer based on 1*3 convolution kernels;

(3) Constructing an RPN proposal network, which is responsible for automatically generating an initial target priori frame, carrying out preliminary classification on the target priori frame, and then finally screening according to the target priori frame to obtain a proposal frame, and intercepting corresponding features at a public feature layer;

The public feature layer obtained after the pseudo-color image passes through the image feature extraction network has two applications, and the first purpose of the public feature layer is to divide the public feature layer into two network branches after carrying out 3*3-dimension convolution once, wherein one network branch is used for generating 9 priori frames for each point; secondly, generating a deviation value of each priori frame and a true annotation frame; then the two groups of features are sequentially transmitted into propasal layers and ROI Pooling layers to respectively carry out preliminary classification of the prior frame and screening of the final suggestion frame; the second purpose of the public feature layer is to input the public feature layer into the ROI Pooling layer, intercept corresponding feature blocks according to the selected suggestion frame, and unify the feature fast size to obtain the final target feature;

(4) Building suggested box coordinates return network branches: constructing a suggestion frame coordinate return network branch by adopting a full-connection layer, and decoding the suggestion frame coordinate return network branch to obtain a coordinate value of a target feature block on an original image;

(5) Constructing a spectrum data extraction module: intercepting corresponding spectrum data according to coordinate values obtained by returning the suggested frame coordinates to the network branches; average calculation is carried out on the intercepted spectrum data on each wave band to obtain average spectrum data of the target in each suggestion frame, and the average spectrum data is used as input of a spectrum characteristic extraction network;

after the pseudo-color image is input into FASTER RCNN network and features are extracted through Resnet, the RPN network is used for judging which areas in the image are cell bodies and obtaining feature blocks with different sizes, and after the image passes through ROI Pooling layers, the feature blocks proposed by the network are converted into uniform sizes; then FASTER RCNN is divided into two branches, one branch is a return network for predicting the coordinates of the positioning frame and is used for learning and returning the four coordinates of the predicted cell positioning frame and the deviation value of the coordinates of the marking positioning frame in the attribute file; the other branch is a classification network of five classifications of cells; the spectrum data is stored in a picture file of a raw type, the file type is that reflectivity values of all pixels in the spectrum picture corresponding to all wave bands are arranged end to form a one-dimensional list, the data is rearranged into a three-dimensional data block according to a format of length multiplied by height multiplied by wave band number of an image, then the corresponding pixel points and the reflectivity values of the corresponding pixel points in all wave bands are cut out by utilizing four coordinate values of a cell prediction coordinate frame obtained based on a pseudo-color image, and the reflectivity average value of all the pixel points is obtained under each wave band to obtain an average spectrum curve which is used as input of a spectrum characteristic network;

(6) Constructing a spectrum feature extraction network: additionally constructing a one-dimensional spectrum data feature extraction network, and outputting one-dimensional spectrum features;

The average spectrum curve of each cell obtained by the spectrum data extraction module is one-dimensional data, so that a spectrum-oriented characteristic extraction network is built by using Conv 1D; the network is built by a Conv1D+ Max Pooling structure, 350-dimensional spectrum data are compressed, after a feature layer is obtained, the method is subjected to the Flatten operation, a one-dimensional full feature layer is obtained, a full-connection Dense layer is connected, and then the full-connection Dense layer is connected with a pseudo-color image feature layer in a FASTER RCNN tail cell classification branch;

(7) Construction of a feature fusion network: the characteristics extracted from the tail end of the spectrum characteristic extraction network are fused with a characteristic layer of FASTER RCNN tail end based on cell pseudo-color image classification, and then the whole classification network is connected, so that the purpose of classification based on fusion characteristics of cell image space and spectrum is realized;

before classifying by using a softmax layer in a FASTER RCNN classification branch, performing a flatten operation on a feature block obtained by an RPN layer to obtain a one-dimensional feature layer, wherein the last layer of a spectrum feature extraction network is a one-dimensional feature layer, splicing the two feature layers according to a weight of 1:1 to obtain a fusion feature layer containing space and spectrum features, and then connecting the softmax classification layer to realize classification based on fusion features.