CN115984231A - Method for distinguishing cancer patient specimen properties based on optical image and application - Google Patents
Method for distinguishing cancer patient specimen properties based on optical image and application Download PDFInfo
- Publication number
- CN115984231A CN115984231A CN202310037801.4A CN202310037801A CN115984231A CN 115984231 A CN115984231 A CN 115984231A CN 202310037801 A CN202310037801 A CN 202310037801A CN 115984231 A CN115984231 A CN 115984231A
- Authority
- CN
- China
- Prior art keywords
- training
- model
- optical image
- layer
- patient
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Image Analysis (AREA)
Abstract
The invention discloses a method for distinguishing the property of a cancer patient sample based on an optical image, which comprises the following steps: s1, acquiring an optical image of a sample of a patient in a training group, preprocessing the optical image, and establishing a positive and negative sample data set for lymph node metastasis state identification of the patient in the training group; s2, selecting a basic network architecture, and constructing a prediction model through pooling layer down-sampling and cross-layer splicing fusion; s3, constructing a good and malignant classification network structure loading training data set through a basic network architecture, and performing 3-fold cross training to obtain a lymph-Net deep learning network model; s4, acquiring positive and negative sample data sets of the patients in the test group, and identifying the properties of the samples on the positive and negative sample data sets of the patients in the test group through the lymph-Net deep learning network model; and S5, analyzing the prediction result of the S4 through GCAM thermodynamic diagram, observing the attention area of the different-scale convolutional layer extraction features of the lymph-Net deep learning network model and the prediction result basis of the fusion model, and analyzing the reasons of correctness and mistakes of the prediction result.
Description
Technical Field
The invention relates to the technical field of video image processing, in particular to a method for distinguishing the property of a cancer patient specimen based on an optical image and application thereof.
Background
Cancer, as a global high-incidence disease, seriously affects the physical and mental health of patients. In many cancers, the boundary of the tumor is difficult to distinguish by the surgeon during the operation, so that the surgical incision margin still remains after the operation, and early relapse is caused. Therefore, it is important to make a quick and accurate diagnosis of the specimen during the operation. Genetic information of the cancer foci can be used to determine prognosis and guide treatment. Furthermore, lymphatic metastasis is an important pathway for the malignant progression of cancer. The identification of lymph node metastasis is crucial for the staging of solid tumors and is an important prognostic factor. Complete lymph node dissection, proven important for accurate staging of patients in many cancers.
However, the lymph node metastasis rate of a patient is low in cancers such as early gastric cancer and early breast cancer. Many patients with lymph node metastases therefore receive unnecessary lymph node dissection. Sentinel Lymph Nodes (SLNs) are the first node or group of nodes through which cancer patients develop lymph node metastases, i.e., SLNs do not develop metastases, and other regional lymph nodes are theoretically free of tumor metastases. SLN tracer is injected around areola or tumor, and the distribution of the tracer is detected or visualized by special equipment at the armpit part, so that the SLN can be identified. Near-infrared fluorescence imaging has been introduced into surgery as an aid to surgeons in recent years. ICG fluorescence imaging is currently used for various cancers such as breast cancer, ovarian cancer, digestive tract tumor, glioma and the like. Although the ICG fluorescence imaging technology is more intuitive and sensitive and has a higher recognition rate, whether metastasis occurs or not cannot be identified at the naked eye level. Therefore, it is necessary to explore the relationship between optical images (including but not limited to ICG fluorescence images) and specimen properties.
The current preoperative imaging methods (including ultrasound, X-ray, CT, PET-CT, MRI, etc.) are not ideal in identifying the benign and malignant tumor of the patient, the genomic information of the cancer focus, the regional lymph node metastasis status, the nature of the tumor boundary or the prediction of the surgical margin, etc. before the operation. Intraoperative identification of patient surgical specimens (tumors/nodules), surgical margins, regional lymph node properties depends on intraoperative frozen pathology results, frozen sections are complex and time consuming (40 minutes to 1 hour), there is a high false negative rate, freezing 10 or hundreds of samples during surgery is impractical, which limits the development of real-time diagnosis in oncology, particularly for multiple tissue samples. Furthermore, grading of tumor tissue and acquisition of genomic level information during surgery is not possible. Post-operative acquisition of pathological results is a gold standard for patient surgical specimens (tumors/nodules), tumor boundaries or surgical margins, regional lymph node properties, and genomic information identification, but this technique is time consuming, subject to high inter-observer variability, and examining large numbers of specimens (tumors/nodules/lymph nodes) can result in overloading the use of labor, space, and equipment and supplies, increasing the cost of healthcare.
Disclosure of Invention
The invention provides a method for identifying the property of a cancer patient sample based on an optical image and an application thereof, which are used for solving the problems.
In order to achieve the purpose, the invention adopts the following technical scheme:
a method for discriminating a property of a cancer patient specimen based on an optical image, comprising the steps of:
s1, acquiring an optical image of a sample of a patient in a training group, preprocessing the optical image, and establishing a positive and negative sample data set for lymph node metastasis state identification of the patient in the training group;
s2, selecting a basic network architecture, and constructing a prediction model through pooling layer down-sampling and cross-layer splicing fusion;
s3, constructing a good and malignant classification network structure through a basic network architecture, downloading weights of a pre-training model, setting initial parameters, loading a training data set, and performing 3-fold cross training to obtain a lymph-Net deep learning network model;
s4, acquiring positive and negative sample data sets of the patients in the test group, identifying the properties of the samples on the positive and negative sample data sets of the patients in the test group through the lymph-Net deep learning network model, and outputting a prediction result of a single optical image in real time, wherein the prediction result comprises a prediction category and a prediction probability value, and if a single sample has a plurality of optical video images which are continuously shot, an average value is obtained through weighting and weighing to serve as the prediction result;
and S5, analyzing the prediction result of the S4 through GCAM thermodynamic diagram, observing the attention area of the different scale convolutional layer extraction features of the lymph-Net deep learning network model and the prediction result basis of the fusion model, and analyzing the reasons of the correctness and the mistake of the prediction result.
Preferably, step S1 is specifically:
s11, collecting optical images of the patient specimens of the training group and pathological results of the patients;
and S12, cutting the optical image, removing redundant information in shooting to obtain multi-frame optical images of the patient specimen of the training group, and constructing a positive and negative sample training data set for lymph node metastasis state identification by combining case results.
Preferably, the ResNet-152 neural network is adopted as the basic network architecture of the sentinel lymph node metastasis prediction model in the step S2.
Preferably, step S2 is specifically:
s21, performing feature extraction on the optical images of the patient specimens of the training set through a ResNet-152 neural network to obtain an effective feature map for predicting the properties of the patient specimens of the training set;
s22, extracting convolution layer feature vectors at different depths in the ResNet-152 neural network for splicing and fusion to obtain multi-scale fusion features;
and S23, converting the fusion characteristics into two classification results through an average pooling layer and a full connection layer, and outputting the results, and operating the output value of the softmax function by using the max function to obtain the prediction classification of the patients in the training group.
Preferably, step S21 is specifically:
s211, inputting optical images of the patient specimens of the training set into a ResNet-152 neural network, and performing feature extraction from shallow to deep through superposition of a convolution layer and a batch normalization layer of a multi-layer bottleneck layer structure, wherein the shallow bottleneck layer extracts morphological features of the specimens, the deep bottleneck layer extracts high-level semantic features of the specimens, the batch normalization layer can effectively ensure regularization of a model and relieve the problem of gradient disappearance, and the final convolution layer outputs a complete feature map;
s212, after the last convolution layer, connecting a self-adaptive average pooling layer to select the features in all the feature maps, and solving the average value of each feature map as an effective feature;
and S213, carrying out appropriate selection splicing and fusion on the effective features in each feature map in the shallow feature map and the deep feature map respectively to form an effective feature map for predicting the properties of the patient samples in the training set.
Preferably, step S22 is specifically:
s221, extracting feature maps to be fused after convolution of effective feature maps predicted by the sample properties of the patient in the training set on a convolutional layer with the ResNet-152 neural network depth of [33, 115, 477 and 509 ];
s222, the size of the feature graph to be fused of the shallow feature is adjusted through the self-adaptive average pooling layer, then the feature graphs to be fused on 4 scales are spliced and fused, 3840 fusion feature graphs are obtained, and the shallow convolution layer and the deep convolution layer occupy the same weight coefficient for result prediction.
Preferably, step S3 is specifically:
s31, constructing a benign and malignant classification network structure through a resnet-152 neural network, and loading pre-training weights of the resnet-152 neural network on positive and negative sample training data sets of the patients in a training group to initialize a prediction model;
s32, randomly dividing a positive and negative sample data set of a training group patient into a training data set and a verification data set according to the proportion of 7 by adopting a 3-fold cross verification mode, pre-training a model by using a small-scale training data set, finely adjusting model parameters, then training the model by using a large-batch training data set, accelerating the convergence of the model parameters, and obtaining a lymph-Net deep learning network model;
s33, training and testing the verification data set by adopting the lymph-Net deep learning network model, constructing a confusion matrix and an ROC curve, collecting the loss value, accuracy, precision, recall rate and F1 score of each round of training and testing, making a corresponding visual curve, and verifying the model stability of the lymph-Net deep learning network model.
Preferably, step S4 is specifically:
s41, acquiring an optical image of a sample of a patient in a test group, performing shape normalization and center cutting pretreatment, establishing a positive and negative sample data set for lymph node metastasis state identification of the patient in the test group, and defining a negative label as 1 and a positive label as 0;
s42, loading the lymph-Net deep learning network model weight obtained after 3-fold cross validation, performing model validation on positive and negative sample data sets identified by the lymph node metastasis state of the patients in the test group, and respectively outputting the prediction category and the prediction probability value of a single image;
s43, according to continuous frames of single samples of the patients in the test group appearing in the optical image video, the positive and negative probabilities of each sample are weighted and averaged by all the values of the positive and negative probabilities of all the frames appearing in the samples, a weight coefficient is set according to the proportion of single images in a single sample optical image set, the predicted probability value is multiplied by the weight coefficient to serve as the predicted proportion of the single images, and the sum of all the proportions of the single sample image set serves as the final prediction result.
A method for distinguishing the nature of the specimen of cancer patient based on optical image is used for real-time intraoperatively distinguishing the nature of specimen of cancer patient, the nature of surgical margin, tumor boundary, distinguishing the regional lymph node metastasis state of cancer patient and distinguishing the genome information of cancer focus of patient.
Compared with the background art, the invention has the following advantages by adopting the technical scheme:
1. the invention provides a method for distinguishing the nature of a cancer patient sample based on an optical image and an application thereof, which can automatically capture the optical image of an operation sample (tumor/nodule), a tumor boundary/operation margin and a regional lymph node to analyze the characteristics of the image by establishing a lymph-Net deep learning network model, provide accurate real-time visual information, help a surgeon to distinguish the nature of the patient sample, the nature of the operation margin, the tumor boundary, distinguish the regional lymph node metastasis state of the cancer patient and distinguish the genome information of a cancer focus of the patient, quickly provide a judgment result matched with pathological information in the operation, quickly and sensitively report the nature of the cancer patient sample in real time and with high specificity, guide the surgeon to carry out quick and complete operation decision in the operation, reduce postoperative complications, reduce the operation risk of the patient, save the medical burden of the patient, reduce the complicated workload of the doctor, and provide guarantee for the effective treatment decision and accurate stage division of the patient.
2. The invention provides a method for distinguishing the sample properties of a cancer patient based on an optical image and application thereof.A lymph-Net deep learning network model is suitable for optical images of different fluorescent dyes and different optical devices, and is effectively applied to breast cancer patients, gynecological cancer patients, ovarian cancer patients, brain tumor patients, digestive system tumor patients, gastric cancer patients, liver cancer patients and the like, and various solid tumor patients such as head and neck squamous cell carcinoma.
Drawings
FIG. 1 is a flow chart of the method steps of the present invention;
FIG. 2 is a schematic diagram of a network architecture according to the present invention;
FIG. 3 is a diagram of intraoperative fluorescence video acquisition of the present invention;
FIG. 4 is a network architecture of the present invention;
FIG. 5 is a schematic diagram of a convolutional layer multi-scale feature fusion process of the present invention;
FIG. 6 is a heat map of two different modes of the model of the present invention;
FIG. 7 is a heat map of the effect of the model of the present invention;
FIG. 8 shows AUC results of the model of the invention (Lymph-Net).
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not limit the invention.
Examples
Referring to fig. 1 to 7, the present invention discloses a method for discriminating the property of a cancer patient specimen based on an optical image, comprising the following steps:
s1, acquiring an optical image of a sample of a patient in a training group (in the embodiment, the optical image adopts a fluorescence dynamic video image, but is not limited to the fluorescence dynamic video image in practical application), preprocessing the optical image, and establishing a positive and negative sample data set for lymph node metastasis state identification of the patient in the training group;
the step S1 specifically comprises the following steps:
s11, injecting a tracer to a patient in a training group by taking a case as a unit, stripping surface fat of a specimen in an operation, and collecting an in-vitro NI area optical image of the specimen of the patient in the training group and a pathological result of the patient;
the method specifically comprises the following steps: the same amount of tracer (methylene blue 10mg/mL stock solution, 1mL stock solution, diluted to 2mL with physiological saline, and diluted to 5mg/mL final concentration, 25mg ICG and 10mL physiological saline were prepared to 2.5mg/mL ICG stock solution, 1mL ICG stock solution, diluted to 2mL with physiological saline, and diluted to 1.25mg/mL final concentration) was injected into the patients in the training group using the DPM H2000 fluorescence endoscope imaging system. After injecting tracer agent, each patient carries out in-vivo imaging, then a surgeon distinguishes and separates the in-vitro specimen, the surface fibroadipose tissue of the specimen is stripped, and before the frozen section pathological examination in the pathology department, a DPM H2000 fluorescence endoscope camera system is used for carrying out video recording on all the samples to be examined in a dark box, so as to obtain the in-vitro NI area optical image of the samples of the patient in the training set.
In this embodiment, the optical image adopts the following criteria:
the camera integrated system using the DPM H2000 fluorescence endoscope has full high-definition video output: 1920 × 1080P, 16 aspect ratio, 4 × 1/3inch wafer, fluorescence sensitivity: 0.305ug/mL, device power: 150VA, exposure time 20ms. The imaging system was calibrated for white balance prior to each imaging.
Image standard: the imaging distance is 1cm, and the reference is that a lens is clean, an image is clear, and focusing is proper.
The format is as follows: uncompressed TS-MPEG2 Transport (. TS) format dynamic video images derived from DPM.
And S12, cutting the optical image, removing redundant information in shooting to obtain a multi-frame optical image of the patient specimen of the training group, and constructing a positive and negative sample training data set for lymph node metastasis state identification by combining case results.
S2, selecting a basic network architecture, and constructing a prediction model through pooling layer down-sampling and cross-layer splicing fusion;
in the step S2, a ResNet-152 neural network is adopted as a basic network framework of the sentinel lymph node metastasis prediction model.
Firstly, selecting 4 different basic neural network models of vgg, densenet, efficientnet and resnet, and respectively performing primary screening on a sample data set; comprehensively comparing the accuracy, the processing rate and the loss value performance of 4 types of basic neural networks, finally selecting a ResNet-152 neural network as a basic network architecture of a sentinel lymph node metastasis prediction model, respectively intercepting feature maps at a conv2_ x layer, a conv3_ x layer, a conv4_ x layer and a conv5_ x layer of the Resnet-152 neural network, and respectively downsampling the feature maps through a self-adaptive average pooling layer, wherein the sizes of the feature maps are respectively 1/8,1/4 and 1/2 of the original sizes; splicing and fusing the feature graphs of 4 scales to serve as final output of the network;
modifying the output dimension of the full-connection layer of the last layer of the ResNet-152 neural network to be 2, corresponding to the negativity and the positivity of lymph node metastasis, using cross entropy loss as a loss function, using a random gradient descent method to adjust the network parameters of each round, setting an initial learning rate and a weight attenuation coefficient, and preventing overfitting.
The step S2 specifically comprises the following steps:
s21, performing feature extraction on the optical images of the patient specimens of the training set through a ResNet-152 neural network to obtain an effective feature map for predicting the properties of the patient specimens of the training set;
step S21 specifically includes:
s211, inputting optical images of the patient specimens of the training set into a ResNet-152 neural network, and performing feature extraction from shallow to deep through superposition of a convolution layer and a batch normalization layer of a multi-layer bottleneck layer structure, wherein the shallow bottleneck layer extracts morphological features of the specimens, the deep bottleneck layer extracts high-level semantic features of the specimens, the batch normalization layer can effectively ensure regularization of a model and relieve the problem of gradient disappearance, and the final convolution layer outputs a complete feature map;
s212, after the last convolution layer, connecting a self-adaptive average pooling layer to select the features in all the feature maps, and solving the average value of each feature map as an effective feature;
and S213, carrying out appropriate selection splicing and fusion on the effective features in each feature map in the shallow feature map and the deep feature map respectively to form an effective feature map for predicting the properties of the patient samples in the training set.
S22, extracting convolution layer feature vectors at different depths in the ResNet-152 neural network for splicing and fusion to obtain multi-scale fusion features;
step S22 specifically includes:
s221, convolving the effective characteristic diagrams predicted by the properties of the patient samples in the training set with convolutional layers with the depth of ResNet-152 neural networks [33, 115, 477 and 509] respectively, and then extracting characteristic diagrams to be fused;
s222, the size of the feature graph to be fused of the shallow feature is adjusted through the self-adaptive average pooling layer, then the feature graphs to be fused on 4 scales are spliced and fused, 3840 fusion feature graphs are obtained, and the shallow convolution layer and the deep convolution layer occupy the same weight coefficient for result prediction. Meanwhile, the space dimensionality can be compressed by self-adaptive average pooling, and meanwhile, the mean value of the corresponding dimensionality is taken out, so that some useless features can be restrained to a certain extent.
And S23, converting the fusion characteristics into two classification results through an average pooling layer and a full connection layer, and outputting the results, and operating the output value of the softmax function by using the max function to obtain the prediction classification of the patients in the training group.
S3, constructing a good and malignant classification network structure through a basic network architecture, downloading weights of a pre-training model, setting initial parameters, loading a training data set, and performing 3-fold cross training to obtain a lymph-Net deep learning network model;
the step S3 specifically comprises the following steps:
s31, constructing a benign and malignant classification network structure through a resnet-152 neural network, and loading pre-training weights of the resnet-152 neural network on positive and negative sample training data sets of the patients in a training group to initialize a prediction model;
s32, randomly dividing a positive and negative sample data set of a training group patient into a training data set and a verification data set according to the proportion of 7 by adopting a 3-fold cross verification mode, pre-training a model by using a small-scale training data set, finely adjusting model parameters, then training the model by using a large-batch training data set, accelerating the convergence of the model parameters, and obtaining a lymph-Net deep learning network model;
the method comprises the steps that 224 x 224 image blocks are randomly selected in 256 x 256 images during each training of a training data set, rotation angles are randomly set for the selected image blocks, saturation, contrast and brightness are randomly enhanced according to a certain probability, and a learning rate and a learning attenuation strategy are adaptively adjusted after each iteration.
S33, training and testing the verification data set by adopting the lymph-Net deep learning network model, adjusting the training frequency of the model according to the type and the number of samples and the size of the data set, reducing the iteration times, predicting and constructing a confusion matrix and an ROC curve on the test set after all the iteration times are finished, collecting the loss value, the accuracy, the precision, the recall rate and the F1 score of each round of training and testing, making a corresponding visual curve, taking the relevant indexes of the data as the performance standard of a classifier, and verifying the model stability of the lymph-Net deep learning network model.
S4, acquiring positive and negative sample data sets of the patients in the test group, identifying the sample properties on the positive and negative sample data sets of the patients in the test group through a lymph-Net deep learning network model, and outputting the prediction result of a single optical image in real time, wherein the prediction result comprises the prediction category and the prediction probability value, and if a single sample has a plurality of continuously shot optical video images, the average value is obtained through weighted weight to serve as the prediction result;
the step S4 specifically comprises the following steps:
s41, obtaining an optical image of a test group patient sample, carrying out shape normalization and center cutting pretreatment, establishing a positive and negative sample data set for lymph node metastasis state identification of the test group patient, and defining a negative label as 1 and a positive label as 0;
s42, loading the lymph-Net deep learning network model weight obtained after 3-fold cross validation, performing model validation on positive and negative sample data sets identified by the lymph node metastasis state of the patients in the test group, and respectively outputting the prediction category and the prediction probability value of a single image;
s43, according to continuous frames of single samples of the patients in the test group appearing in the optical image video, the positive and negative probabilities of each sample are weighted and averaged by all the values of the positive and negative probabilities of all the frames appearing in the samples, a weight coefficient is set according to the proportion of single images in a single sample optical image set, the predicted probability value is multiplied by the weight coefficient to serve as the predicted proportion of the single images, and the sum of all the proportions of the single sample image set serves as the final prediction result.
And S5, analyzing the prediction result of the S4 through GCAM thermodynamic diagram, observing the attention area of different scale convolutional layer extraction features of the lymph-Net deep learning network model and the prediction result basis of the fusion model, and analyzing the reasons of the correctness and the mistake of the prediction result.
The GCAM thermodynamic diagram can show the activation value distribution area of the neural network to the image in a weight mode, shows the interpretability of the model, needs to pay attention to the effectiveness of feature extraction for predicting the performance of the lymph-Net deep learning network model, and shows that the part effective for model prediction is red, and the part with low prediction percentage is blue.
Explicability of the lymph-Net deep learning network model is shown by using two different thermodynamic diagram visualization modes, GCAM thermodynamic diagram shows effective areas for model prediction, and GCAM can locate relevant image areas but lacks the capability of highlighting fine details. And the combination of GCAM and Guided Backproperation visualization can realize the fusion of the visualization effect of predicting the attention area and the image texture details.
The invention also discloses a method for distinguishing the property of the cancer patient specimen based on the optical image, which is applied to distinguishing the property of the cancer patient specimen, the property of surgical margin, the tumor boundary, distinguishing the regional lymph node metastasis state of the cancer patient and distinguishing the genome information of the cancer focus of the patient in real-time operation.
In this embodiment, the optical image adopts the following criteria:
the all-in-one camera system using the DPM H2000 fluorescence endoscope has full high-definition video output: 1920 × 1080P, 16 aspect ratio, 4 × 1/3inch wafer, fluorescence sensitivity: 0.305ug/mL, device power: 150VA, exposure time 20ms. The imaging system was calibrated for white balance prior to each imaging.
Image standard: the imaging distance is 1cm, and the reference is that a lens is clean, an image is clear, and focusing is proper.
The format is as follows: uncompressed TS-MPEG2 Transport (.ts) format motion video pictures derived from DPM.
In this embodiment, after the method is adopted, the specimen is predicted, and the prediction result is as described in table 1 below:
TABLE 1 prediction results of the samples according to this example
As can be seen from Table 1, the method of this example has an accuracy of 0.99 or more in the determination of the specimen properties and a predictive value (accuracy) of 0.9961. The recall rate is 0.9961, and the higher recall rate indicates that the detection rate of the model for positive samples is higher, which indicates that the model can effectively distinguish among samples with different properties, and the difference of the characteristics can be obviously distinguished through the model.
The model effect of the embodiment can be intuitively explained through different GCAM thermodynamic diagrams. GCAM is a two-dimensional feature score network associated with a particular output category, with each position of the grid representing the importance of that category. For a picture of a specimen fluorescence image input to the lymphet, the technique can present the correlation degree of each position in the picture and the characteristics of the specimen in a thermodynamic diagram form, and is helpful for knowing which local position of an original image leads the lymphet model to make a final classification decision.
As shown in fig. 7, in the GCAM thermodynamic diagram in the specimen fluorescence image analysis of this embodiment, the color of the fluorescence diagram is the GCAM portion, and different colors indicate that the model focuses on different parts of the diagram, and the judgment made based on these parts is equivalent to an explanation of the model, and the shade of the color represents the strength of the focus.
While the invention has been described with reference to specific preferred embodiments, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined in the following claims. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (9)
1. A method for discriminating a property of a cancer patient specimen based on an optical image, comprising the steps of:
s1, acquiring optical images of samples of patients in a training group, preprocessing the optical images, and establishing positive and negative sample data sets for lymph node metastasis state identification of the patients in the training group;
s2, selecting a basic network architecture, and constructing a prediction model through pooling layer down-sampling and cross-layer splicing fusion;
s3, constructing a good and malignant classification network structure through a basic network architecture, downloading weights of a pre-training model, setting initial parameters, loading a training data set, and performing 3-fold cross training to obtain a lymph-Net deep learning network model;
s4, acquiring positive and negative sample data sets of the patients in the test group, identifying the properties of the samples on the positive and negative sample data sets of the patients in the test group through the lymph-Net deep learning network model, and outputting a prediction result of a single optical image in real time, wherein the prediction result comprises a prediction category and a prediction probability value, and if a single sample has a plurality of optical video images which are continuously shot, an average value is obtained through weighting and weighing to serve as the prediction result;
and S5, analyzing the prediction result of the S4 through GCAM thermodynamic diagram, observing the attention area of the different scale convolutional layer extraction features of the lymph-Net deep learning network model and the prediction result basis of the fusion model, and analyzing the reasons of the correctness and the mistake of the prediction result.
2. The method for discriminating the property of the cancer patient specimen based on the optical image as claimed in claim 1, wherein the step S1 is specifically as follows:
s11, collecting optical images of the patient specimens of the training group and pathological results of the patients;
and S12, cutting the optical image, removing redundant information in shooting to obtain a multi-frame optical image of the patient specimen of the training group, and constructing a positive and negative sample training data set for lymph node metastasis state identification by combining case results.
3. The method as claimed in claim 1, wherein the ResNet-152 neural network is used as the basic network architecture of the sentinel lymph node metastasis prediction model in step S2.
4. The method for discriminating the property of the cancer patient specimen based on the optical image as claimed in claim 3, wherein the step S2 is specifically as follows:
s21, performing feature extraction on the optical images of the patient specimens of the training set through a ResNet-152 neural network to obtain an effective feature map for predicting the properties of the patient specimens of the training set;
s22, extracting convolution layer feature vectors at different depths in the ResNet-152 neural network for splicing and fusion to obtain multi-scale fusion features;
and S23, converting the fusion characteristics into two classification results through an average pooling layer and a full connection layer, and outputting the results, and operating the output value of the softmax function by using the max function to obtain the prediction classification of the patients in the training group.
5. The method for discriminating the property of the cancer patient specimen based on the optical image as claimed in claim 4, wherein the step S21 is specifically as follows:
s211, inputting optical images of the patient specimens of the training set into a ResNet-152 neural network, and performing feature extraction from shallow to deep through superposition of a convolution layer and a batch normalization layer of a multi-layer bottleneck layer structure, wherein the shallow bottleneck layer extracts morphological features of the specimens, the deep bottleneck layer extracts high-level semantic features of the specimens, the batch normalization layer can effectively ensure regularization of a model and relieve the problem of gradient disappearance, and the final convolution layer outputs a complete feature map;
s212, after the last convolution layer, connecting a self-adaptive average pooling layer to select the features in all the feature maps, and solving the average value of each feature map as an effective feature;
s213, carrying out appropriate selection splicing and fusion on the effective features in each feature map in the superficial layer feature map and the deep layer feature map respectively to form an effective feature map for predicting the property of the patient specimen in the training set.
6. The method for discriminating the property of the cancer patient specimen based on the optical image as claimed in claim 4, wherein the step S22 is specifically as follows:
s221, extracting feature maps to be fused after convolution of effective feature maps predicted by the sample properties of the patient in the training set on a convolutional layer with the ResNet-152 neural network depth of [33, 115, 477 and 509 ];
s222, the size of the feature graph to be fused of the shallow feature is adjusted through the self-adaptive average pooling layer, then the feature graphs to be fused on 4 scales are spliced and fused, 3840 fusion feature graphs are obtained, and the shallow convolution layer and the deep convolution layer occupy the same weight coefficient for result prediction.
7. The method according to claim 1, wherein the step S3 is to identify the property of the cancer patient specimen by using an optical image, and comprises:
s31, constructing a benign and malignant classification network structure through a resnet-152 neural network, and loading pre-training weights of the resnet-152 neural network on positive and negative sample training data sets of the patients in a training group to initialize a prediction model;
s32, randomly dividing a positive and negative sample data set of a training group patient into a training data set and a verification data set according to the proportion of 7 by adopting a 3-fold cross verification mode, pre-training a model by using a small-scale training data set, finely adjusting model parameters, then training the model by using a large-batch training data set, accelerating the convergence of the model parameters, and obtaining a lymph-Net deep learning network model;
s33, training and testing the verification data set by adopting the lymph-Net deep learning network model, constructing a confusion matrix and an ROC curve, collecting the loss value, the accuracy rate, the precision rate, the recall rate and the F1 score of each round of training and testing, making a corresponding visualization curve, and verifying the model stability of the lymph-Net deep learning network model.
8. The method according to claim 1, wherein the step S4 is to identify the property of the cancer patient specimen by using an optical image, and comprises:
s41, acquiring an optical image of a sample of a patient in a test group, performing shape normalization and center cutting pretreatment, establishing a positive and negative sample data set for lymph node metastasis state identification of the patient in the test group, and defining a negative label as 1 and a positive label as 0;
s42, loading the lymph-Net deep learning network model weight obtained after 3-fold cross validation, performing model validation on positive and negative sample data sets identified by the lymph node metastasis state of the patients in the test group, and respectively outputting the prediction category and the prediction probability value of a single image;
s43, according to continuous frames of single samples of the patients in the test group appearing in the optical image video, the positive and negative probabilities of each sample are weighted and averaged by all the values of the positive and negative probabilities of all the frames appearing in the samples, a weight coefficient is set according to the proportion of single images in a single sample optical image set, the predicted probability value is multiplied by the weight coefficient to serve as the predicted proportion of the single images, and the sum of all the proportions of the single sample image set serves as the final prediction result.
9. The method for real-time identification of cancer patient specimen properties based on optical images as claimed in claim 1 is applied to intraoperative real-time identification of cancer patient specimen properties, surgical margin properties, tumor boundary, differentiation of cancer patient regional lymph node metastasis status, and identification of genomic information of cancer focus of a patient.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310037801.4A CN115984231A (en) | 2023-01-10 | 2023-01-10 | Method for distinguishing cancer patient specimen properties based on optical image and application |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310037801.4A CN115984231A (en) | 2023-01-10 | 2023-01-10 | Method for distinguishing cancer patient specimen properties based on optical image and application |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115984231A true CN115984231A (en) | 2023-04-18 |
Family
ID=85962446
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310037801.4A Pending CN115984231A (en) | 2023-01-10 | 2023-01-10 | Method for distinguishing cancer patient specimen properties based on optical image and application |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115984231A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118052814A (en) * | 2024-04-15 | 2024-05-17 | 吉林大学 | AI technology-based full-automatic specimen pretreatment system and method |
-
2023
- 2023-01-10 CN CN202310037801.4A patent/CN115984231A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN118052814A (en) * | 2024-04-15 | 2024-05-17 | 吉林大学 | AI technology-based full-automatic specimen pretreatment system and method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Li et al. | Colonoscopy polyp detection and classification: Dataset creation and comparative evaluations | |
| Wang et al. | Using deep convolutional neural networks for multi-classification of thyroid tumor by histopathology: a large-scale pilot study | |
| US5627907A (en) | Computerized detection of masses and microcalcifications in digital mammograms | |
| CN111655116A (en) | Image diagnosis support device, data collection method, image diagnosis support method, and image diagnosis support program | |
| CN111278348A (en) | Diagnosis support method, diagnosis support system, diagnosis support program, and computer-readable recording medium storing diagnosis support program for disease based on endoscopic image of digestive organ | |
| Dov et al. | Thyroid cancer malignancy prediction from whole slide cytopathology images | |
| CN114782307A (en) | Enhanced CT image colorectal cancer staging auxiliary diagnosis system based on deep learning | |
| Domingues et al. | Computer vision in esophageal cancer: a literature review | |
| CN110916666A (en) | Imaging omics feature processing method for predicting recurrence of hepatocellular carcinoma after surgical resection based on multi-modal MRI (magnetic resonance imaging) images | |
| CN116630680B (en) | Dual-mode image classification method and system combining X-ray photography and ultrasound | |
| CN112071418B (en) | Gastric cancer peritoneal metastasis prediction system and method based on enhanced CT image histology | |
| yahia Ibrahim et al. | An enhancement technique to diagnose colon and lung cancer by using double CLAHE and deep learning | |
| CN115984231A (en) | Method for distinguishing cancer patient specimen properties based on optical image and application | |
| Raju et al. | An advanced diagnostic ColoRectalCADx utilises CNN and unsupervised visual explanations to discover malignancies | |
| CN117173460A (en) | Multitasking joint optimization method and device for detecting prostate sporadic cancer | |
| KR102559616B1 (en) | Method and system for breast ultrasonic image diagnosis using weakly-supervised deep learning artificial intelligence | |
| Perek et al. | Self supervised contrastive learning on multiple breast modalities boosts classification performance | |
| Wang et al. | Three feature streams based on a convolutional neural network for early esophageal cancer identification | |
| Liu et al. | The predictive accuracy of CT radiomics combined with machine learning in predicting the invasiveness of small nodular lung adenocarcinoma | |
| Guedri et al. | Detection of the breast cancer cells by a smart camera connected to a cellular Microscopy | |
| Schreuder et al. | Algorithm combining virtual chromoendoscopy features for colorectal polyp classification | |
| Janapati et al. | A Brief Survey on Feature Extraction Models for Brain Tumor Detection | |
| Priya et al. | Artificial Intelligence Techniques and Methodology Available for Lung Cancer Detection | |
| CN115116625B (en) | Pancreatic cancer lymph node metastasis stage prediction method, device and storage medium | |
| WO2024075274A1 (en) | Cell classification device, cell classification method, and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |