JP7197584B2 - Methods for storing and retrieving digital pathology analysis results - Google Patents

Description

[0001] This application claims the benefit of the filing date of U.S. Provisional Patent Application No. 62/595,143, filed December 6, 2017, the entire disclosure of which is incorporated herein by reference. incorporated into.

One embodiment of the present invention, for example, relates to a method for storing and retrieving digital pathology analysis results.

[0002] Digital pathology involves scanning an entire glass slide of pathological tissue or cells into a digital image interpretable on a computer screen. These images will later be processed by image processing algorithms or interpreted by a pathologist. To examine tissue sections (substantially transparent), tissue sections are prepared using colored histochemical stains that selectively bind to cellular components. The use of staining or stained cell structures by clinicians or computer-aided diagnosis (CAD) algorithms identifies morphological markers of disease and guides treatment accordingly. Monitoring assays enables processes as diverse as diagnosing disease, evaluating response to treatment, and developing new drugs to combat disease.

[0003] Immunohistochemical (IHC) slide staining can be used to identify proteins in cells of tissue sections, thus reducing the presence of various cell types, such as cancer cells and immune cells in biological tissues. Widely used in research. Thus, in the case of immune response studies, IHC staining can be used in studies to understand the distribution and localization of differentially expressed biomarkers of immune cells (such as T cells or B cells) in cancer cells. For example, tumors often contain an infiltrate of immune cells, which may prevent or favor tumor growth.

[0004] In-situ hybridization (ISH) is used to confirm the presence or absence of conditions such as genetic abnormalities or amplification of oncogenes in cells that appear morphologically malignant, particularly when observed under a microscope. Available. ISH detects or localizes a target nucleic acid target gene in a cell or tissue sample by employing labeled DNA or RNA probe molecules that are antisense to the target gene sequence or transcript. ISH is performed by exposing a cell or tissue sample immobilized on a glass slide to a labeled nucleic acid probe specifically hybridizable to a given target gene in the sample. Multiple target genes can be analyzed simultaneously by exposing a cell or tissue sample to multiple nucleic acid probes that have been labeled with multiple different nucleic acid tags. By utilizing labels with different emission wavelengths, simultaneous multicolor analysis can be performed in one step on a single target cell or tissue sample.

One embodiment of the present invention, for example, relates to a method for storing and retrieving digital pathology analysis results.

[0005] The present disclosure relates, among other things, to automated systems and methods for analyzing and storing data associated with irregularly shaped organisms (eg, fibroblasts or macrophages). The present disclosure also provides a mid-resolution or medium-resolution analysis technique, ie, a technique that groups pixels with similar characteristics (e.g., staining intensity, presence or absence of staining, and/or texture) into "sub-regions." Automated systems and methods for using to analyze and store data associated with living organisms.

[0006] In digital pathology, images are acquired from biological specimens (eg, tissue specimens) mounted on glass slides and stained for identification of biomarkers. Biological samples can be evaluated under a high power microscope or automatically analyzed by digital pathology algorithms that detect and classify organisms of interest. For example, objects of interest can be cells, blood vessels, glands, tissue regions, and the like. Any derived information may be stored in a database for later retrieval, including statistics of the presence or absence of biological structures of interest, spatial relationships, and/or staining properties. You can As will be appreciated by those skilled in the art, storing and retrieving analysis results for distinct cells (eg, tumor cells or immune cells) is relatively easy. Such cells can be represented by a point at the center position of each cell and stored in a database (see, eg, FIG. 4). Similarly, a living body of well-defined size and shape (e.g., a blood vessel) can be represented by a simple outline, the coordinates of which can be stored in a database for later retrieval and/or further analysis. (Also referred to herein as a "polygon" or "polygon contour").

[0007] On the other hand, some biological structures of interest (eg, fibroblasts or macrophages) have irregular shapes. Groups of such cells may extend around each other or around other cells (see Figure 5). As a result, it is often difficult to accurately identify these irregularly shaped cells individually by an observer or by an automated algorithm. Instead, these cells are very often distinguished only by their respective staining cytoplasmic or membrane localizations, without identification of individual cells.

[0008] Although such irregularly shaped structures could be analyzed and stored using high resolution analysis, such techniques require substantial computational resources (computation time and/or storage resources). often become Indeed, high-resolution analytical techniques that store all pixel information of the biological structure of interest (e.g., analysis results for all pixels) consume software and hardware resources (e.g., memory and processors to process or display the information). ) are too numerous to ultimately provide meaningful results for a particular organism.

[0009] Such irregular structures could also be analyzed using low-resolution analysis, where such low-resolution data representation "lumps" multiple individual cells into a single object. ” and stored in the database. As an example, FIGS. 6A and 6B show a tumor (yellow, 620) represented by a large polygonal outline (red, 630) surrounding a group of related cells to the exclusion of undesirable area “pores” (turquoise, 640). ) and an example of an IHC image stained for fibroblasts (purple, 610). In this example, the analysis results are averaged over a large area (red outline, 630) that may contain many individual cells with varying characteristics (eg, shape, size, staining intensity, etc.). For example, with respect to FIG. 6B, the contoured FAP-positive area is 928.16 μm ² with a calculated fibroblast activation protein (FAP)-positive average intensity of 0.26. Given an average intensity in such a large pixel area, an average intensity of 0.26 is too coarse to be indicative and representative of FAP positivity overall in this image. Without wishing to be bound by any particular theory, it is believed that this low-resolution analysis approach can lead to reduced accuracy if stored results are later utilized in downstream processing. Thus, due to the heterogeneity of such stained cells, the method may not locally present real details of such regions of biological structures of interest.

[00010] In contrast to the high-resolution and low-resolution analysis methods described above, the present disclosure uses a medium-resolution analysis technique to analyze images with similar image characteristics (eg, at least one of texture, intensity, or color). Systems and methods are provided for deriving data corresponding to irregularly shaped cells by segmenting an image into a plurality of sub-regions that correspond to each other.

[00011] In view of the above, one aspect of the present disclosure is a method of storing image analysis data derived from an image of a biological specimen having at least one stain, comprising: (a) from the image, one or (b) segmenting the image into a plurality of sub-regions, each sub-region being substantially different in at least one of staining presence/absence, staining intensity, or local texture; (c) generating a plurality of representation objects based on the plurality of segmented sub-regions; and (d) associating each of the plurality of representation objects with a derived feature metric. and (e) storing the coordinates of each representation object in a database along with the association-derived feature metrics. It will be appreciated by those skilled in the art that at least steps (a) and (b) may be performed in any order. In some embodiments, segmenting the image into subregions includes deriving superpixels. In some embodiments, superpixels are derived by grouping pixels with local k-means clustering and (ii) merging small isolated regions into large nearest neighbor superpixels using a connected component algorithm. Without wishing to be bound by any particular theory, superpixels (as subregions) are each perceptually consistent units, i.e., the color and texture of all pixels in the superpixel. can be uniform, it is considered perceptually meaningful. In some embodiments, connected component labeling scans an image and groups its pixels into components based on pixel connectivity. That is, all pixels of a connected component share similar pixel intensity values and are connected to each other in some way.

[00012] In some embodiments, the step of segmenting the image into a plurality of subregions is superimposing a sampling grid over the image, the sampling grid defining non-overlapping areas having a predetermined size and shape. to do, including In some embodiments, the subregions have a size of M×N, where M ranges from 50 pixels to 100 pixels and N ranges from 50 pixels to approximately 100 pixels.

[00013] In some embodiments, the representational object includes outlines of subregions that meet a predetermined staining intensity threshold. In some embodiments, representation objects include seed points. In some embodiments, the seed points are derived by computing centroids of each of the multiple sub-regions. In some embodiments, the derived feature metric is staining intensity, and the average staining intensity of all pixels within each generated representational object outline is computed. In some embodiments, the derived feature metric is an expression score, and an average expression score corresponding to an area within each generated subregion is associated with a plurality of generated representational objects. In some embodiments, the method further includes retrieving the stored coordinates and associated feature metric data from the database and projecting the retrieved data onto the image. In some embodiments, analysis results (eg, intensity, area) within a corresponding sub-region can be stored in the form of average pixel measurements representing pixel data for that sub-region.

[00014] In some embodiments, the biological sample is stained with a membrane stain. In some embodiments, biological samples are stained with membrane stains and nuclear stains. In some embodiments, the biological sample is stained with at least FAP and the one or more derived feature metrics comprise at least one of FAP staining intensity or FAP percent positivity. In some embodiments, the average FAP percent positivity is calculated for all pixels in the sub-region. In some embodiments, the average FAP staining intensity is calculated for all pixels within the sub-region. In some embodiments, samples are stained with FAP and H&E. In some embodiments, the sample is stained with FAP and another nuclear or membrane stain.

[00015] In some embodiments, an image received as input is first separated into image channel images (eg, image channel images of a particular stain). In some embodiments, a region of interest is selected prior to image analysis.

[00016] Another aspect of the present disclosure is a system for deriving data corresponding to irregularly shaped cells from an image of a biological sample comprising at least one stain, comprising: (i) one or more (ii) a memory coupled to the one or more processors for (a) deriving one or more feature metrics from an image when executed by the one or more processors; (b) generating a plurality of sub-regions within the image, each sub-region having pixels with similar attributes selected from color, brightness, and/or texture; c) computing a set of representation objects based on the plurality of generated sub-regions; and (d) associating one or more derived feature metrics from the image with the calculated coordinates of each of the set of computed representation objects. and a memory storing computer-executable instructions that cause one or more processors to perform operations including: In some embodiments, the sub-regions are (i) neighboring pixels, (ii) pixels with perceptually meaningful similar properties (e.g., color, brightness, and/or texture), and (iii) biological formed by grouping pixels that are substantially homogenous with respect to their biological properties (eg, biological structures, staining properties of biological structures, cellular properties, groups of cells). In some embodiments, the subregion pixels have similar properties and descriptive statistics with respect to the organism of interest (e.g., irregularly shaped cells, including but not limited to fibroblasts and macrophages). have

[00017] In some embodiments, segmenting the image into a plurality of subregions includes deriving superpixels. In some embodiments, superpixels are derived using either a graph-based approach or a gradient ascent-based approach. In some embodiments, superpixels are derived by grouping pixels with local k-means clustering and (ii) merging small isolated regions into large nearest neighbor superpixels using a connected component algorithm.

[00018] In some embodiments, the representational object includes outlines of sub-regions that meet a predetermined staining intensity threshold. In some embodiments, representation objects include seed points. In some embodiments, the system further includes instructions for storing the coordinates of the one or more derived feature metrics and association computational representation objects in a database. In some embodiments, the one or more derived feature metrics comprise at least one expression score selected from percent positivity, H-score, or staining intensity. In some embodiments, data corresponding to irregularly shaped cells is derived for a region of interest within the image. In some embodiments, the region of interest is an area of the image annotated by a medical professional.

[00019] Another aspect of the disclosure is a non-transitory computer-readable medium storing instructions for analyzing data associated with an irregularly shaped living body, the instructions comprising: (a) a biological sample; instructions for deriving one or more feature metrics from an image, the biological sample comprising at least one stain; (c) instructions for computing a plurality of representational objects based on a sequence of partitioned sub-regions; d) associating one or more derived feature metrics from the image with calculated coordinates of each of the plurality of computational representation objects.

[00020] In some embodiments, dividing the image into a series of subregions includes computing superpixels. In some embodiments, superpixels are computed using one of a normalized cut algorithm, an agglomerative clustering algorithm, a quick shift algorithm, a turbo pixel algorithm, or a simple linear iterative clustering algorithm. In some embodiments, the superpixels are generated using simple iterative clustering, with the superpixel size parameter set from about 40 pixels to about 400 pixels, and the density parameter set from about 10 to about 100. be. In some embodiments, superpixels are computed by grouping pixels with local k-means clustering and (ii) merging small isolated regions into large nearest neighbor superpixels using a connected component algorithm.

[00021] In some embodiments, the biological sample is stained with at least FAP and the one or more derived feature metrics comprise at least one of FAP staining intensity or FAP percent positiveness. In some embodiments, the average FAP percent positivity is calculated for all pixels in the sub-region. In some embodiments, the average FAP staining intensity is calculated for all pixels within the sub-region. In some embodiments, the representation object includes at least one of a polygon outline and seed points. In some embodiments, the memory includes instructions for storing the coordinates of the one or more derived feature metrics and association computation representation objects in a database. In some embodiments, the memory further includes instructions for projecting the stored information onto an image of the biological sample.

[00022] Applicants believe that the systems and methods described herein provide an improved solution for storing biological analysis results that cannot be defined in a single location or geometry for each object of interest. showed that. Further, applicants believe that the systems and methods described herein can reduce storage space for storing analysis results compared to pixel-level high-resolution analysis techniques. This is because the analysis results for a particular pixel and its surrounding pixels are collectively stored in sub-regions, and pixels in sub-regions have similar characteristics or attributes (eg, color, brightness, texture). Furthermore, Applicants have demonstrated that the generated sub-regions can scale from image complexity from thousands of pixels to a smaller and more manageable number of sub-regions that allow for significantly faster readout and reporting of analytical results. We believe that the above system and method are computationally efficient because we can reduce . Applicants also believe that the sub-regions are neither too small nor too large for the storage and representation of the analysis results, and thus are expressively efficient. Finally, Applicants believe that the systems and methods disclosed herein can improve accuracy, especially when compared to low-resolution analytical techniques. The generated sub-region is to describe the biologically relevant properties or statistical information of the object of interest as compared to storing information from the large-region representation (i.e. the sub-regions are stained including pixels that are as uniform in intensity and texture as possible). These and other advantages are discussed elsewhere herein.

[00023] For a general understanding of the features of the present disclosure, reference is made to the drawings. The same reference numbers are used throughout the drawings to identify the same elements.

[00024] FIG. 7 illustrates a representative digital pathology system comprising an image acquisition device and a computer system, according to some embodiments. [00025] FIG. 7 illustrates various modules that may be utilized in a digital pathology system or digital pathology workflow, according to some embodiments. [00026] FIG. 7 is a flowchart illustrating various steps for deriving image analysis data and related image analysis data by generation subregions, according to some embodiments. [00027] FIG. 10 illustrates an example of a digital pathology image of liver cancer cells at a high level of resolution, wherein after image analysis processing and classification, the analysis results (e.g., cell can be stored and retrieved from the database and displayed, with each annotation point representing the readout information (eg, the presence or absence of biological structures of interest, spatial relationships, and descriptive statistics of staining properties). [00028] FIG. 5A shows the appearance of fibroblasts with different appearance (e.g., irregular cell size, shape, and borders) and morphologically heterogeneous, showing normal fibroblasts. be. FIG. 5B illustrates the different appearance (eg, irregular size, shape, and boundaries of cells) and morphologically heterogeneous appearance of fibroblasts. FIG. 5C shows the different appearance (e.g., irregular cell size, shape, and boundaries) and morphologically heterogeneous appearance of fibroblasts with hematoxylin and eosin staining of normal activated fibroblasts ( H&E) shows an image. FIG. 5D. Different appearance (e.g., irregular cell size, shape, and boundaries) and morphologically heterogeneous appearance of fibroblasts with hematoxylin and eosin staining (H&E) of activated fibroblasts. It is the figure which showed the image. [00029] Figure 6A shows an example of immunohistochemistry (IHC) of fibroblasts associated with tumor cells, where fibroblasts (610) stain purple and tumors (620) FIG. 10 : Stained yellow, fibroblasts contact other cells and can have a very irregular shape extending opposite or around other cells. [00030] Figure 6B shows an example of a low-resolution polygonal outline (red, 630) of an area with positive fibroblast expression in turquoise and an exclusion area (hole, 640). [00031] FIG. 7 illustrates a simple shaped (eg, circular) subregion (710) that may be associated with image data using the medium resolution techniques described herein. [00032] Fig. 10 shows an example of superpixels generated with SLIC of a fibroblast area on an IHC image. [00033] Higher magnification original IHC image with tumor cells (830) stained yellow and fibroblasts (840) stained purple. [00034] FIG. 7 illustrates an initial shape of a superpixel that looks similar to a square before adjustment of regularization parameters, according to some embodiments. [00035] FIG. 7 illustrates a final representation of superpixels with adjusted regularization parameters of the SLIC algorithm, according to some embodiments. [00036] FIG. 16 illustrates polygonal outlines (black, 910) of subregions (here superpixels) belonging to a region of interest (fibroblast region), according to some embodiments. [00037] FIG. 16 depicts the polygonal outline (black, 920) and central seed (green dot, 930) of a subregion (superpixel) belonging to an organism of interest (fibroblast), according to some embodiments. [00038] Head and neck cancer tissue stained purple with fibroblast activation protein (FAP) for fibroblasts (1010) and pancytokeratin (PanCK) yellow for epithelial tumors (1020) is a diagram showing an example of a full-slide IHC image of . [00039] Fig. 10 shows an example polygonal outline annotated with analysis results of superpixels (blue, 1030) belonging to fibroblast regions that can be stored in a database. [00040] Fig. 11 shows an example of a central seed with an annotated analysis of superpixels (red, 1140) belonging to a fibroblast area that can be stored in a database. [00041] Fig. 11 shows an example of a histogram plot of FAP intensity read out from whole slide superpixels. [00042] FIG. 7 is a flowchart illustrating steps for region selection, according to some embodiments. [00043] Fig. 6 illustrates six different annotation shapes and regions within an image of a biological sample; [00044] Concordance was shown for the percentage of FAP positive areas between FAP+ areas determined using (i) the high resolution analysis approach and (ii) the exemplary medium resolution (subregion) approach described herein. It is a diagram.

[00045] In any method claimed herein involving more than one step or action, unless expressly stated otherwise, the order of the steps or actions is the order in which the steps or actions are listed. It is to be understood that this is not necessarily so.

[00046] As used herein, the singular terms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Similarly, the word "or" is intended to include "and" unless the context clearly dictates otherwise. The term "includes" is defined inclusively such that "includes A or B" means including A, B, or A and B.

[00047] As used herein and in the claims, "or" shall be understood to have the same meaning as "and/or" as defined above. For example, when separating items in a list, "or" or "and/or" is interpreted as being inclusive, and multiple or list elements and optionally additional unlisted items, and may include more than one. otherwise expressly specified terms such as "only one of" or "exactly one of" or when used in a claim The term "consisting of" alone indicates the inclusion of exactly one element of a plurality or list of elements. Generally, the term "or" herein is exclusive when preceded by an exclusive term such as "either," "one of," "only one of," or "exactly one of." are to be construed only as indicating alternatives (ie, "one or the other but not both"). As used in the claims, "consisting essentially of" shall have its ordinary meaning as used in the field of patent law.

[00048] The terms "comprising," "including," "having," etc. are used interchangeably and have the same meaning. Similarly, the terms "comprises," "includes," "has," etc. are used interchangeably and have the same meaning. Specifically, each of these terms is defined consistently with the general definition in U.S. patent law of "comprising," and thus an open definition meaning "at least the following." It is to be interpreted as a term and not to exclude additional features, limitations, aspects, and the like. Thus, for example, "a device having components a, b, and c" means that the device comprises at least components a, b, and c. . Similarly, the phrase "a method involving steps a, b, and c" means that the method includes at least steps a, b, and c. Further, although steps and processes may be described in a particular order herein, those skilled in the art will recognize that the order of steps and processes may vary.

[00049] As used herein and in the claims, the phrase "at least one" refers to one or more elements of a list and any one of the elements in the list of elements. or at least one element selected from a plurality, but not necessarily including at least one of every and every element specifically listed in the list of elements, nor excluding any combination of elements in the list of elements shall be understood. Also, according to this definition, elements other than those specifically identified in the list of elements to which the phrase "at least one" refers may or may not be related. May be present as an option. Thus, as one non-limiting example, "at least one of A and B" (or equivalently, "at least one of A or B"). A or B" or "at least one of A and/or B"), in one embodiment, at least one (optionally two or more) of A (B is absent (optionally including elements other than B)), and in another embodiment, at least one (optionally two or more) B (A is absent (optionally optionally including elements other than A)), and in yet another embodiment, at least one (optionally two or more) A and at least one (optionally two or more) B (optionally optionally including other elements) and the like.

[00050] As used herein, the term "biological sample" (used interchangeably herein with the terms "biological specimen" or "specimen") or A "tissue sample" (used interchangeably herein with the term "tissue specimen") is a biomolecule (protein, peptide , nucleic acids, lipids, carbohydrates, or combinations thereof). Other examples of organisms include mammals (e.g., domestic animals such as humans, cats, dogs, horses, cows, and pigs, and laboratory animals such as mice, rats, and primates), insects, annelids, spiders. Including morphs, marsupials, reptiles, amphibians, bacteria and fungi. Biological samples include tissue samples (e.g., tissue sections and needle biopsies of tissues), cell samples (e.g., cytological smears such as Pap smears or blood smears, or cells obtained by microdissection). sample), or cell fractions, cell debris, or organelles (eg, cells lysed and cellular components separated by centrifugation, etc.). Other examples of biological samples include blood, serum, urine, semen, fecal material, cerebrospinal fluid, interstitial fluid, mucus, tears, sweat, pus (obtained, for example, by surgical or needle biopsy). d) biopsy tissue, nipple aspirate, earwax, breast milk, vaginal fluid, saliva, swabs (such as buccal swabs), or any material containing biomolecules derived from a first biological sample. In certain embodiments, the term "biological sample" herein refers to a sample (homogenized or liquefied sample) made from a tumor or portion thereof obtained from a subject.

[00051] As used herein, the term "biomarker" or "marker" refers to a measurable indicator of some biological state or condition. In particular, biomarkers may be proteins or peptides (eg, surface proteins) that are particularly stainable and indicative of a cell's biological characteristics (eg, cell type or cell physiological state). Immune cell markers are biomarkers that selectively characterize mammalian immune responses. Biomarkers may be adapted for use in determining the body's response to a disease or disease treatment or determining whether a subject is predisposed to a disease or illness. With respect to cancer, biomarkers represent biological substances that indicate the presence or absence of cancer in the body. A biomarker may be a molecule secreted by a tumor or a specific response of the body to the presence of cancer. Genetic, epigenetic, proteomic, glycomic, and imaging biomarkers can be used for cancer diagnosis, prognosis, and epidemiology. Such biomarkers can be tested in non-invasively collected biological fluids such as blood or serum. Multiple gene- and protein-based biomarkers are already in use in patient care, including AFP (liver cancer), BCR-ABL (chronic myeloid leukemia), BRCA1/BRCA2 (breast/ovarian cancer), BRAF V600E ( melanoma/colorectal cancer), CA-125 (ovarian cancer), CA19.9 (pancreatic cancer), CEA (colorectal cancer), EGFR (non-small cell lung cancer), HER-2 (breast cancer), KIT ( gastrointestinal stromal tumors), PSA (prostate specific antigen), S100 (melanoma), and many others. Biomarkers are diagnostic (to identify early-stage cancer) and/or prognostic (to predict cancer progression and/or predict a subject's response to a particular treatment and/or the likelihood of cancer recurrence) It is considered useful as a diagnostic method.

[00052] As used herein, the term "image data", as understood herein, refers to raw image data acquired from a biological tissue sample, such as by an optical sensor or sensor array, or pre-processed contains the processed image data. In particular, the image data may include pixel matrices. As used herein, the term "immunohistochemistry" refers to methods that detect the interaction of antigens with specific binding agents, such as antibodies, to determine the presence or distribution of antigens in a sample. The sample is contacted with the antibody under conditions that allow antibody-antigen binding. Antibody-antigen binding can be detected by a detectable label conjugated to the antibody (direct detection) or by a detectable label conjugated to a secondary antibody that specifically binds to the primary antibody (indirect detection). . As used herein, "mask" is a derivative of digital image in which each pixel of the mask has a binary value (e.g., "1" or "0" (or "true" or "false"). )). By superimposing the mask on the digital image, in a separate processing step applied to the digital image, all pixels of the digital image mapped to a particular one of the binary value mask pixels are obscured, deleted, or Ignored or filtered out. For example, a mask is generated from the original digital image by assigning an intensity value above the true or false threshold to all pixels in the original image, and removing all pixels that overlap the "false" masked pixels. obtain. A "multi-channel image", as understood herein, is defined as the specific fluorescent dyes, quantum dots, chromogens, etc. that make up one of the channels of the multi-channel image due to their fluorescence or detectability in different spectral bands. includes digital images obtained from biological tissue samples in which different biological structures, such as nuclei and tissue structures, are simultaneously stained by .

[00053] Overview
[00054] Applicants have developed a system and method for storing analytical results of irregularly shaped organisms, such as fibroblasts or macrophages, in non-transitory memory, such as a database. Analysis results may later be retrieved from a database or memory for further analysis or use in other downstream processes. Analysis results may also be projected onto the input image or other derived image, or visualized by other means. The present disclosure also provides for storing analysis results at an adjustable level of detail and It can facilitate reporting. It is believed that this can improve efficiency and accuracy compared to the low-resolution analysis approach described herein, where average analysis results from global regions of interest are saved.

[00055] As described elsewhere herein, the disclosed systems and methods are based on a medium-resolution analysis approach that stores analysis results using small locally similar regions (sub-regions). Subregions can be simple (e.g., circle, square) or complex (e.g., superpixel) and are used to store local analysis results for each subregion across the entire slide. used for Sub-regions defined by the present medium-resolution approach have similar (or homogenous) properties (e.g., presence or absence of staining (ie, presence or absence of a particular stain), staining intensity (ie, relative intensity (or amount) of staining) , pixels with local texture (ie, information about the spatial distribution of color or intensity in an image or selected regions of an image) are grouped to allow identification of irregularly shaped objects. In some embodiments, the subregions in the medium resolution approach have sizes ranging from approximately 50 to approximately 100 pixels or pixel areas from approximately ^{2,500 pixels2} to approximately 10,000 pixels2. Of course, the subregions may have any size, and this size may be based on the type of analysis to be performed and/or the type of cells investigated.

[00056] It will be appreciated by those skilled in the art that the mid-level approach lies between the high-resolution and low-resolution analysis approaches described herein, so that data are collected at the subregion level and The sub-regions are proportionally smaller than the regions of interest in the low-resolution approach and clearly larger than the pixels in the high-resolution analysis approach. "High resolution analysis" means that image data is captured at the pixel level or substantially at the pixel level. On the other hand, "low-resolution analysis" refers to region-level analysis, such as regions having a size of at least 500 pixels by 500 pixels or areas having a size greater than 250,000 ^pixels2 . As will be appreciated by those skilled in the art, low resolution analytical techniques will encompass many organisms (eg, multiple irregularly shaped cells).

[00057] The present disclosure may represent analysis and storage of living organisms having irregular shapes and/or sizes, such as fibroblasts or macrophages. It is to be understood that the present disclosure is not limited to fibroblasts or macrophages, but is extendable to any living organism whose size or shape is not well defined.

[00058] With respect to fibroblasts, these are cells that make up the structural framework or stroma composed of extracellular matrix and collagen in animal tissues. These cells are the most common type of connective tissue in animals and are important for wound healing. Fibroblasts come in a variety of shapes and sizes, as well as activated and non-activated forms (see, eg, FIGS. 5A-5D). Fibroblasts are the activated form (the suffix "blast" denotes metabolically active cells), whereas fibrosis-associated cells are considered less active. However, both fibroblasts and fibrosis-associated cells are not designated as distinct and are sometimes simply referred to as fibroblasts. Fibroblasts are morphologically distinguishable from fibrosis-associated cells by the abundance and relatively large size of the rough endoplasmic reticulum. In addition, fibroblasts are thought to make contact with their respective neighboring cells, and these contacts are thought to be attachments that can distort the morphology of the detached cells. The medium-resolution analysis approach presented here can account for these morphological differences and is considered optimal for storing information about fibroblasts, macrophages, and other irregular organisms.

[00059] FIG. 1 illustrates a digital pathology system 200 for imaging and analyzing specimens, according to some embodiments. The digital pathology system 200 includes an imaging device 12 (eg, a device having means for scanning a microscope slide carrying a specimen) and a computer 14 such that the imaging device 12 and the computer are integrated (eg, network 20). directly or indirectly). Computer system 14 includes desktop computers, laptop computers, tablets, etc., digital electronic circuits, firmware, hardware, memory, computer storage media, computer programs or instruction sets (e.g., programs stored in memory or storage media). case), one or more processors (including programmed processors), and any other hardware, software, or firmware modules, or combinations thereof. For example, computer system 14 shown in FIG. 1 may comprise a computer having display 16 and housing 18 . Computers can store digital images in binary form (locally, such as in memory, on a server, or another network-connected device). A digital image can also be divided into rows and columns of pixels. A pixel may contain a digital value of one or more bits defined by a bit depth. One of ordinary skill in the art will appreciate that other computing devices or systems may be adapted for use, and the computer system described herein may include additional components (e.g., sample analysis equipment). , microscopes, other imaging systems, automated slide preparation equipment, etc.). Some of these additional components and the various computers, networks, etc. available are described elsewhere herein.

[00060] In general, imager 12 (or other image source with pre-scan images stored in memory) includes, but is not limited to, one or more image capture devices. Image capture devices include cameras (e.g., analog cameras, digital cameras, etc.), optical elements (e.g., one or more lenses, sensor focusing lens groups, microscope objectives, etc.), imaging sensors (e.g., charge-coupled devices (CCD ), complementary metal oxide semiconductor (CMOS) image sensors, etc.), photographic film, and the like. In digital embodiments, the image capture device may include multiple lenses that cooperate to demonstrate on-the-fly focusing. An image sensor (eg, a CCD sensor) can capture a digital image of the specimen. In some embodiments, imager 12 is a brightfield imaging system, a multispectral imaging (MSI) system, or a fluorescence microscope system. Digitized tissue data is available from VENTANA MEDICAL SYSTEMS, Inc., for example. (Tucson, Arizona) or other suitable imaging equipment, such as the VENTANA iScan HT scanner. Additional imaging devices and systems are described elsewhere herein. Those skilled in the art will appreciate that digital color images acquired by imaging device 12 may conventionally be composed of primary color pixels. Each colored pixel can be encoded on three digital components each containing the same number of bits, each component being a primary color (commonly red, green, or blue, also denoted by the term "RGB" components). ).

[00061] Figure 2 provides an overview of the various modules utilized in the digital pathology system disclosed above. In some embodiments, the digital pathology system employs a computing device 200 or computer-implemented method having one or more processors 203 and at least one memory 201, wherein the at least one memory 201 comprises one or store non-transitory computer readable instructions that, upon execution by multiple processors, cause one or more processors to execute instructions (or stored data) in one or more modules (eg, modules 202 and 205-209) doing.

[00062] Referring to FIGS. 2 and 3, the present disclosure provides analysis of irregularly shaped organisms, such as fibroblasts or macrophages, and/or storage of analysis results in non-transitory memory, such as a database. A computer-implemented method is provided. The method includes, for example, (a) operating the image acquisition module 202 to generate or receive multi-channel image data (e.g., an acquired image of a biological sample stained with one or more stains) (step 300); (b) operating the image analysis module 205 to derive one or more metrics from features in the acquired image (step 310); and (c) operating the segmentation module 206. , segmenting the acquired image into a plurality of sub-regions (step 320); and (d) operating representational object generation module 207 to generate polygons, center seeds, or other objects that identify the sub-regions. (step 330); (e) operating the labeling module 208 to associate one or more derived metrics with the generated representation object (step 340); and storing in database 209 (step 350). As will be appreciated by those skilled in the art, the workflow may incorporate additional modules or databases. For example, operation of the image processing module may apply a particular filter to the acquired image, or may identify particular histological and/or morphological structures within the tissue sample. A region of interest selection module may also be used to select a particular portion of the image for analysis. Similarly, operation of the separation module may provide image channel images corresponding to particular stains or biomarkers.

[00063] Image Acquisition Module
[00064] In some embodiments, as a first step, referring to FIG. 2, the digital pathology system 200 operates the image acquisition module 202 to extract a biological sample having one or more stains. image or image data. In some embodiments, the received or acquired images are RGB images or multispectral images (eg, multiplexed brightfield and/or darkfield images). In some embodiments, the captured images are stored in memory 201 .

[00065] An image or image data (used interchangeably herein) may be captured by imaging device 12, such as in real-time. In some embodiments, the image is obtained from an instrument such as a microscope capable of capturing image data of a specimen-bearing microscope slide as described herein. In some embodiments, the image is acquired using a 2D scanner that can scan image tiles or a line scanner that can scan the image line by line, such as the VENTANA DP 200 scanner. Alternatively, the image may be a previously acquired (eg, scanned) image stored in memory 201 (or retrieved from a server over network 20 for that matter).

[00066] A biological sample may be stained by application of one or more stains, and as a result the image or image data includes signals corresponding to each of the one or more stains. Thus, the systems and methods described herein are capable of estimating or normalizing a single stain (eg, hematoxylin), but not limiting the number of stains within a biological sample. In fact, a biological sample may be stained in multiplex assays for two or more stains in addition or inclusion of any counterstain.

[00067] It will be appreciated by those skilled in the art that the biological sample may be stained for different types of nuclear and/or cell membrane biomarkers. Methods for staining tissue structures and guiding in the selection of suitable stains for various purposes are described, for example, in Sambrook et al. See "Molecular Cloning: A Laboratory Manual", Cold Spring Harbor Laboratory Press (1989) and Ausubel et al. "Current Protocols in Molecular Biology", Greene Publishing Associates and Wiley-Intersciences (1987), the disclosures of each of which are incorporated herein by reference.

[00068] As a non-limiting example, in some embodiments, tissue samples are stained in an IHC assay for the presence of one or more biomarkers, including fibroblast activation protein (FAP). be. Overexpression of FAP in fibroblast cell lines is thought to promote malignant behavior. Stromal fibroblasts, an essential component of the tumor microenvironment and often designated as cancer-associated fibroblasts (CAFs), are controlled by multiple mechanisms such as proliferation, angiogenesis, invasion, persistence, and immunosuppression. , can promote tumorigenesis and progression. Without wishing to be bound by any particular theory, it is believed that cancer cells activate stromal fibroblasts and induce expression of FAP, which affects cancer cell proliferation, invasion and migration. Conceivable. FAP is thought to be heavily expressed in reactive stromal fibroblasts in 90% of human epithelial cell carcinomas, including breast, lung, colorectal, ovarian, pancreatic, and head and neck. For this reason, the amount of FAP most likely presents an important prognostic value for the clinical behavior of tumors (it is some kind of metric that can be associated with generated subregions or representational objects after derivation). is an example).

[00069] Chromogenic stains include hematoxylin, eosin, fast red, or 3,3'-diaminobenzidine (DAB). It will also be appreciated by those skilled in the art that any biological sample may be stained with one or more fluorophores. In some embodiments, tissue samples are stained with a primary stain (eg, hematoxylin). In some embodiments, tissue samples are stained in IHC assays for particular biomarkers. The sample may also be stained with one or more fluorescent dyes.

[00070] A typical biological sample is processed in an automated staining/assay platform that applies a stain to the sample. There are a variety of commercial products on the market suitable for use as a staining/assay platform, one example being Ventana Medical Systems, Inc.; (Tucson, AZ) Discovery™ product. Camera platforms also include brightfield microscopes (such as the Ventana iScan HT or VENTANA DP 200 scanners from Ventana Medical Systems, Inc.) or any microscope with one or more objectives and a digital imaging device. Other techniques for capturing images at different wavelengths may also be used. Other camera platforms suitable for imaging stained biological specimens are known in the art and are commercially available from companies such as Zeiss, Canon and Applied Spectral Imaging. Also, such platforms are readily adaptable for use in the systems, methods, and apparatus of the present disclosure.

[00071] In some embodiments, the input image is masked so that only tissue regions are present in the image. In some embodiments, a tissue region mask is generated to mask non-tissue regions from tissue regions. Some embodiments automatically or A tissue region mask may be formed by semi-automatic (ie, with minimal user input) exclusion. Those skilled in the art will appreciate that, in addition to masking non-tissue regions from tissue regions, the tissue masking module may also mask any necessary tissue, such as a portion of tissue identified as belonging to a particular tissue type or suspected tumor region. Other areas of interest may be masked depending on . In some embodiments, a segmentation technique is used to generate a tissue region masked image by masking tissue regions from non-tissue regions in the input image. Suitable segmentation techniques include those known from the prior art ("Digital Image Processing", Third Edition, Rafael C. Gonzalez, Richard E. Woods, chapter 10, page 689 and "Handbook of Medical Imaging", Processing and Analysis, Isaac N. Bankman Academic Press, 2000, chapter 2). In some embodiments, image segmentation techniques are utilized to distinguish between the digitized tissue data in the image and the slide, where the tissue corresponds to the foreground and the slide to the background. In some embodiments, the component computes an area of interest (AoI) in the entire slide image, thereby limiting the amount of background non-tissue area analyzed, while all tissue regions in the AoI are To detect. For example, a wide range of image segmentation techniques (e.g., HSV color-based image segmentation, Lab image segmentation, mean-shift color image segmentation, region growing, level setting method, fast marching method, etc.) may be used. Also, based at least in part on the segmentation, the component can generate a tissue foreground mask that can be used to identify portions of the digitized slide data corresponding to the tissue data. Alternatively, the component can generate a background mask that is used to identify portions of the digitized slide data that do not correspond to tissue data.

[00072] This identification may be enabled by image analysis operations such as edge detection. A tissue region mask may be used to remove non-tissue background noise in an image (eg, non-tissue regions). In some embodiments, the tissue region mask is generated such that pixels whose luminance is above a given threshold are set to 1 and pixels below the threshold are set to 0 to generate the tissue region mask. , computing the luminance of the low-resolution input image, generating the luminance image, applying a standard deviation filter to the luminance image, generating a filtered luminance image, and applying a threshold to the filtered luminance image. Including (but not limited to) these actions. For additional information and examples regarding the generation of tissue region masks, see "An Image Processing Method and System for Analyzing a Multi-Channel Image Obtained from a Biological Tissue Sample Being Stained by Multi-Stains." PCT/EP/2015/062015 titled PCT/EP/2015/062015 entitled Image Processing Methods and Systems for Analyzing Multichannel Images Obtained from Targeted Tissue Samples, the entire disclosure of which is incorporated herein by reference. .

[00073] In some embodiments, a region of interest identification module is used to select a portion of a biological sample for which an image or image data is to be acquired (eg, a region of interest with a high concentration of fibroblasts). It may be possible to FIG. 13 is a flowchart illustrating steps for region selection, according to some embodiments. At step 420, the region selection module receives the identified region of interest or field of view. In some embodiments, the region of interest is identified by a user of the disclosed system or another system communicatively coupled to the disclosed system. Alternatively, in another embodiment, the region selection module retrieves the location or identification of the region of interest from storage/memory. In some embodiments, as shown in step 430, the region selection module automatically generates a field of view (FOV) or region of interest (ROI), for example by methods described in PCT/EP2015/062015, The entire disclosure content of which is incorporated herein by reference. In some embodiments, the region of interest is automatically determined by the system in the image or based on some predetermined criterion or characteristic of the image (e.g., a biological sample stained with three or more stains). , identify areas of the image that contain only two stains). At step 440, the region selection module outputs the ROI.

[00074] Image Analysis Module
[00075] In some embodiments, certain metrics (eg, FAP-positive area, FAP-positive intensity) are derived from features in the image received as input (step 300) (see Figure 3). . The derived metric may be adapted to be correlated with the subregions generated herein (steps 320, 330, and 340), and the metric (or its mean, standard deviation, etc.) and The subregion locations may be stored together in a database (step 350) for later retrieval and/or downstream processing. The procedures and algorithms described herein are useful for deriving metrics from various types of cells or cell nuclei, including deriving metrics from fibroblasts and/or macrophages, and/or Alternatively, it may be configured to classify cell nuclei.

[00076] In some embodiments, the metric is the detection of nuclei in the input image and/or the detected nuclei (such as image patches around the detected nuclei) and/or cell membranes (of course the (depending on the biomarkers utilized). In other embodiments, the metric is derived by analyzing plasma membrane staining, cytoplasmic staining, and/or enhanced staining (eg, to distinguish between membranous and non-membrane stained areas). As used herein, the term "cytoplasmic staining" refers to groups of pixels arranged in a pattern having morphological characteristics of the cytoplasmic region of a cell. As used herein, the term "membrane staining" refers to groups of pixels arranged in a pattern having the morphological characteristics of a cell membrane. As used herein, the term "punctate staining" refers to groups of pixels that store localized staining intensities that appear as scattered spots/dots on the membrane area of the cell. Those skilled in the art will appreciate that the nucleus, cytoplasm, and membrane of cells have different characteristics, so that differently stained tissue samples may reveal different biological characteristics. Indeed, it will be appreciated by those skilled in the art that certain receptors on the cell surface may have localized staining patterns to the membrane or cytoplasm. For this reason, the 'membrane' staining pattern analytically differs from the 'cytoplasmic' staining pattern. Similarly, 'cytoplasmic' and 'nuclear' staining patterns are analytically different. For example, stromal cells can be stained strongly with FAP, while tumor epithelial cells can be stained strongly with EpCAM, while cytokeratins can be stained with PanCK. Thus, during image analysis, the use of different stains may allow different cell tumors to be differentiated and differentiated, and different metrics may be derived.

[00077] U.S. Patent No. 7,760,927 ("' '927 patent), the entire disclosure of which is incorporated herein by reference. For example, the '927 patent describes an automated method for simultaneously identifying multiple pixels in an input image of biomarker-stained biological tissue, comprising multiple pixels in the foreground of the input image for simultaneous identification of cytoplasmic and membrane pixels. considering a first color plane of pixels, wherein the input image has been processed to remove a background portion of the input image and to remove a counter-staining component of the input image; and a digital image. and using a selected pixel from the foreground and its eight neighboring pixels, at the same time the selected pixel is a cytoplasmic pixel in the digital image. and determining whether it is a cell membrane pixel or a transition pixel using a decision threshold level. Further, the '927 patent states that the step of determining simultaneously using the selected pixel and its eight neighboring pixels includes determining the square root of the product of the selected pixel and its eight neighboring pixels; comparing the product to a decision threshold level; incrementing a first counter for the cell membrane, a second counter for the cytoplasm, or a third counter for the transition pixel based on the comparison; the first counter, the second counter; or determining whether the third counter exceeds a predetermined maximum value, and if so, classifying the selected pixel based on the counter exceeding the predetermined maximum value. . In addition to nuclear scoring, the '927 patent provides examples for cytoplasmic and membrane scoring based on computational cytoplasmic pixel volume index, cytoplasmic pixel median intensity, membrane pixel volume, and membrane pixel median intensity, etc., respectively.

[00078] Another method for identifying and/or scoring membrane, nuclear, and other cell features of interest is described in PCT Publication WO2017/037180 ("'180 Publication"), the entire disclosure of which is hereby incorporated by reference. incorporated herein by reference. Additionally, the '180 publication describes a method for quantifying membrane staining of a specimen of interest in a biological sample when areas of membrane staining are mixed with cytoplasmic staining and/or enhanced staining. To accomplish this, the '180 publication describes (A) segmenting a digital image of a tissue or cytological sample into multiple distinct regions based on specimen staining patterns, wherein the multiple regions comprise at least one two composite staining regions, i.e. regions of the image having analyte positive staining in the first biological compartment mixed with analyte positive staining in at least one second biological compartment; (B) separately from step (A), a candidate biological compartment, i.e. at least the first biological compartment; and (C) separately from steps (A) and (B), categorizing pixel clusters corresponding to analyte staining into high-intensity bins, low-intensity bins, and background-intensity bins. generating an analyte intensity map by segmenting into bins; and (E) quantifying analyte staining in analytically relevant portions of the composite staining region to mix analyte staining and staining of analytically distinct biological compartments. quantify the analyte staining of the biological compartment in the marked areas (e.g., areas where (i) diffuse membrane staining was mixed with cytoplasmic staining or (ii) areas where diffuse membrane staining was mixed with enhanced staining) Describe the method. Pixels in any identified compartment can then be quantified so that the compartment or area of staining intensity quantification can be determined. The '180 publication also describes scoring membrane-specific expression levels.

[00079] In some embodiments, scoring is performed on the classified nuclei to obtain a percentage positivity metric or H-score metric for a particular biomarker. By identifying the nuclei, the corresponding cells can be identified. In other embodiments, cells are scored by associating each relevant nucleus with the surrounding stained membrane. Based on the presence or absence of a staining membrane around the nucleus, for example, no staining (no staining membrane found around the nucleus), partial staining (a portion of the cell's nucleus is surrounded by a staining membrane), or complete staining (nuclear Entirely surrounded by a staining membrane) cells may be classified.

[00080] In some embodiments, tumor nuclei are automatically identified by first identifying candidate nuclei and then automatically distinguishing between tumor nuclei and non-tumor nuclei. Many methods are known in the art for identifying candidate nuclei in images of tissue. For example, methods based on radial symmetry, as described in Ruifrok et al. As also described herein, for hematoxylin image channels or biomarker image channels acquired using color deconvolution as described by Parvin et al. Automatic candidate nucleus detection can be performed by applying a method based on the radial symmetry of In one exemplary embodiment, nuclear detection operation based on radial symmetry is used, as described in co-pending patent application WO2014/140085A1, assigned to the assignee of the present invention, the entire contents of which are incorporated herein by reference. incorporated into the book. Other methods are described in US Patent Application Publication No. 2017/0140246, the disclosure of which is incorporated herein by reference.

[00081] After candidate nuclei are identified, they are analyzed separately to distinguish tumor nuclei from other candidate nuclei. Other candidate nuclei may be further classified (eg, by distinguishing between lymphocyte and interstitial nuclei). In some embodiments, a trained supervised classifier is applied to identify tumor nuclei. For example, a trained supervised classifier is applied to classify candidate nuclei as tumor nuclei or non-tumor nuclei in test images after identifying tumor nuclei by training on nuclear features. Optionally, the trained supervised classifier may be trained separately to distinguish different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei. In some embodiments, the trained supervised classifier used to identify tumor nuclei is a random forest classifier. For example, the training of a random forest classifier consists of (i) generating a training set of tumor and non-tumor nuclei, (ii) extracting features for each nucleus, and (iii) based on the extracted features, It can be done by training a random forest classifier to distinguish between tumor nuclei. Application of the trained random forest classifier may then classify the nuclei into tumor and non-tumor nuclei in the test image. Optionally, the random forest classifier may be separately trained to distinguish different classes of non-tumor nuclei, such as lymphocyte nuclei and stromal nuclei.

[00082] In some embodiments, images received as input are processed to perform nuclear center (seed) detection and/or nucleus segmentation. For example, one may be instructed to detect the nuclear center based on radially symmetric voting using techniques commonly known to those skilled in the art (Parvin, Bahram, et al. "Iterative voting for influence of Structural saliency and characterization of subcellular events,” Image Processing, IEEE Transactions on 16.3 (2007): 615-623 (all disclosures are incorporated herein by reference) see). In some embodiments, after the center of the nucleus is detected by detecting nuclei using radial symmetry, the nuclei are classified based on the intensity of the stain around the cell center. For example, image dimensions may be computed within the image, and one or more votes at each pixel are accumulated by adding the sum of the dimensions within the selected region. Mean shift clustering may be used to find the local centers in the regions representing the actual locations of the nuclei. Nucleus detection based on radially symmetric voting is performed on color image intensity data and explicitly uses a priori domain knowledge that nuclei are oval-shaped patches of varying size and eccentricity. To achieve this, along with the color intensity in the input image, image gradient information is also used for radial symmetry voting, and in combination with the adaptive segmentation process, provides accurate detection and localization of cell nuclei. As used herein, a "gradient" is a pixel intensity gradient calculated for a particular pixel, for example by considering the intensity value gradients of a set of pixels surrounding the particular pixel. Each gradient may have a particular "orientation" with respect to a coordinate system whose x-axis and y-axis are defined by two orthogonal edges of the digital image. For example, nuclear seed detection involves defining the seed as a point that is assumed to reside inside the cell nucleus and serves as the starting point for localizing the cell nucleus. In the first step, the seed points associated with each cell nucleus are detected by using a robust method based on radial symmetry to detect oval-shaped spots, structures resembling cell nuclei. Radial symmetric methods operate on gradient images by using a kernel-based voting procedure. A vote response matrix is generated by processing each pixel that has accumulated votes through a voting kernel. The kernel is based on the gradient direction computed at that particular pixel, the expected range of maximum and minimum kernel sizes, and the voting kernel angle (typically in the range [π/4, π/8]). In the resulting vote space, local maxima with vote values higher than a predetermined threshold are saved as seed points. Irrelevant seeds may be discarded in subsequent segmentation or classification processes.

[00083] Nuclei may be identified using other techniques known to those skilled in the art. For example, the image dimensions may be computed from a particular image channel of one of the H&E or IHC images, and each pixel around the specified dimensions may include the A number of votes may be assigned based on the total size within the region. Alternatively, a mean shift clustering operation may be performed to find local centers within the voting image that represent the actual locations of the nuclei. In other embodiments, the use of nucleus segmentation may segment the entire nucleus based on the currently known center of that nucleus via morphological operations and local thresholding. In still other embodiments, the kernels may be detected through the use of model-based segmentation (i.e., by learning a shape model of the kernels from the training data set and using this as prior knowledge, segment the nuclei in the test image).

[00084] In some embodiments, the nuclei are then segmented using separately computed thresholds for each. For example, Otsu's method may be used for segmentation in the region around the identified nucleus, since pixel intensities in the nucleus region are expected to vary. Those skilled in the art will appreciate that Otsu's method, which is used to determine the optimal threshold by minimizing the within-class variance, is known to those skilled in the art. More specifically, Otsu's method is used to automatically perform clustering-based image thresholding, ie, reduction of gray-level images to binary images. The algorithm assumes that the image contains two classes of pixels (foreground and background pixels) following a bimodal histogram. Then, by calculating the optimal threshold that separates the two classes such that the joint scatter (within-class variance) of the two classes is minimized or equal (because the sum of the pairwise squared distances is constant), each within-class variance is maximum.

[00085] In some embodiments, these systems and methods include automatically analyzing spectral and/or shape features of identified nuclei in images that identify nuclei of non-tumor cells. For example, in a first step, a speckle may be identified in the first digital image. As used herein, a "blob" can be, for example, a region of a digital image in which some characteristic (e.g., intensity or gray value) is constant or varies within a given range of values. be. In a sense, all pixels in a speckle are considered similar to each other. For example, speckles may be identified using a derivative-based method and an extremum-based method of a function of position on the digital image. A nuclear speck is a speck whose pixel and/or outline indicates that it was probably produced by a nucleus stained by the first stain. For example, evaluation of the radial symmetry of a speckle can determine whether the speckle should be identified as a nuclear speckle or any other structure (eg staining artifact). For example, if the speckle has a very long shape and is not radially symmetrical, the speckle may be identified as a staining artifact rather than as a nuclear speckle. According to this embodiment, a speckle identified as a "nuclear speckle" may represent a set of pixels that may be identified as a candidate nucleus and further analyzed to determine if the nuclear speckle represents a nucleus. In some embodiments, any kind of nuclear speck is used directly as a "identifying nucleus." In some embodiments, a filtering operation is applied to the identified nuclei or nuclear speckles to identify nuclei that do not belong to biomarker-positive tumor cells and remove said identified non-tumor nuclei from the list of identified nuclei , or do not add the nucleus to the list of identified nuclei from the beginning. For example, analysis of additional spectral and/or shape features of the discriminating nuclear speck may determine whether the nucleus or nuclear speck is the nucleus of a tumor cell. For example, the nuclei of lymphocytes are larger than those of other tissue cells (eg lung cells). If the tumor cells are derived from lung tissue, lymphocyte nuclei are identified by identifying all nuclear patches of minimal size or diameter that are much larger than the average size or diameter of normal lung cell nuclei. be. Identified nuclear patches for lymphocyte nuclei may be removed (ie, "filtered out") from the set of identified nuclei. Filtering out the nuclei of non-tumor cells may improve the accuracy of this method. Biomarkers allow the generation of intensity signals in the first digital image that are not derived from tumor cells, since non-tumor cells may also represent biomarkers to some extent. By identifying nuclei that do not belong to tumor cells and filtering out all identified nuclei, the accuracy of identifying biomarker-positive tumor cells may be improved. These and other methods are described in US Patent Application Publication No. 2017/0103521, the entire disclosure of which is incorporated herein by reference. In some embodiments, once a seed is detected, a locally adaptive thresholding method may be used to generate a speckle around the detection center. In some embodiments, other methods may be implemented, for example, the use of marker-based watershed algorithms may identify nuclear speckles around detected nuclear centers. . These and other methods are described in co-pending application PCT/EP2016/051906 (published as WO2016/120442), the entire disclosure of which is incorporated herein by reference.

[00086] The system can use at least one image attribute metric and at least one morphology metric to determine whether a feature in an image corresponds to a structure of interest (referred to as a "feature metric"). collectively). Image attribute metrics (derived from features in the image) include, for example, color, color balance, intensity, and the like. Morphological metrics (derived from features in the image) include, for example, feature size, feature color, feature orientation, feature shape, relationship or distance between features (e.g., adjacent features), feature relative to different anatomy Relation or distance etc. are mentioned. As described herein, image attribute metrics, morphological metrics, and other metrics can be used for classifier training. Specific examples of metrics derived from image features are given below.

[00087] (A) Metrics derived from morphological features
[00088] As used herein, a "morphology feature" is a feature indicative of, for example, the shape or size of the nucleus. Without wishing to be bound by any particular theory, morphological features are believed to provide some essential information regarding the size and shape of a cell or its nucleus. For example, morphological features can be computed by applying various image analysis algorithms to pixels contained in or surrounding a nuclear spot or seed. In some embodiments, morphological features include area, minor and major axis lengths, perimeter, radius, solidity, and the like.

[00089] (B) Metrics Derived from Appearance Features
[00090] As used herein, an "appearance feature" is defined, for example, by comparing pixel intensity values of pixels contained in or surrounding a nuclear speckle or seed used to identify a nucleus. Features computed for a particular nucleus, where pixel intensities to be compared are derived from different image channels (eg, background channel, channel for biomarker staining, etc.). In some embodiments, metrics derived from appearance features are pixel intensities and gradient magnitude percentile values (e.g., 10th, 50th, and 95th percentiles) computed from different image channels. quantile value). For example, first, the X-percentile values (X=10, 50, 95) is identified. Computing an appearance feature metric is considered advantageous because the derived metric represents properties of the nucleus area, as well as the membrane area around the nucleus.

[00091] (C) Metrics Derived from Background Features
[00092] A "background feature" is, for example, a feature that indicates appearance and/or presence or absence of stain in the cytoplasm and a cell membrane feature of the cell containing the nucleus from which the background feature was extracted from the image. For example, by identifying seeds representing nuclear spots or nuclei and analyzing pixel areas directly adjacent to identified cell clusters (e.g., 20 pixel (approximately 9 micron) thick ribbons around nuclear spot borders), Background features and corresponding metrics of the nucleus and corresponding cells shown in the digital image by capturing the appearance and presence of stains in the cytoplasm and membrane of the cell, together with the area directly adjacent to this nucleated cell. can be computed. These metrics are similar to nuclear appearance features, but are computed on ribbons approximately 20 pixels (approximately 9 microns) thick around each nuclear boundary, and thus integral with areas immediately adjacent to cells with discriminating nuclei. , incorporates the appearance and presence of stains in the cytoplasm and membrane of the cell. Without wishing to be bound by any particular theory, the ribbon size is chosen so that it captures a sufficient amount of background tissue area around the nuclei that can be used to provide useful information for identifying nuclei. This is because it is considered that These features are described in J. Am. Kong, et al. "A comprehensive framework for classification of nuclei in digital microscopy imaging: Application to diffuse gliomas", ISBI, p. 2, p. 2128-2131, the entire disclosure of which is incorporated herein by reference. These features could be used to determine whether the surrounding tissue is stromal or epithelial (such as in H&E stained tissue samples). Also, without wishing to be bound by any particular theory, it is believed that these background features capture useful membrane staining patterns when the tissue sample is stained with a suitable membrane stain.

[00093] (D) Metrics Derived from Color
[00094] In some embodiments, color-derived metrics include color ratios R/(R+G+B) or color principal components. In other embodiments, color-derived metrics include local statistics (mean/median/variance/standard deviation) for each color and/or color intensity correlations within the local image window.

[00095] (E) Metrics Derived from Intensity Features
[00096] Neighboring cell groups with certain characteristic values are set between the black and white tones of gray colored cells represented in the histopathological slide image. Since the correlation of color features defines an instance of a size class, the intensity of these colored cells determines which cells are affected by clusters of dark cells around them. Examples of texture features are described in PCT Publication WO2016/075095, the entire disclosure of which is incorporated herein by reference.

[00097] (F) Spatial features
[00098] In some embodiments, the spatial features include the local density of cells, the average distance between two neighboring detected cells, and/or the distance from the cell to the segmented region.

[00099] (G) Metrics Derived from Nuclear Features
[000100] Those skilled in the art will appreciate that metrics can also be derived from nuclear features. Computation of such kernel features is described in Xing et al. "Robust Nucleus/Cell Detection and Segmentation in Digital Pathology and Microscopy Images: A Comprehensive Review", IEEE Rev 2 Eg-Biomed 9. 263, January 2016, the entire disclosure of which is incorporated herein by reference. Of course, other features known to those skilled in the art may also be considered and used as a basis for feature computation.

[000101] After derivation of feature metrics, the features may be used alone or in combination with training data (e.g., during training, in conjunction with ground truth discriminators provided by observing experts according to procedures known to those skilled in the art). example cells are presented), nuclei or cells may be classified. In some embodiments, the system may comprise a classifier trained based at least in part on a training set or reference slide set for each biomarker. Those skilled in the art will appreciate that different slide sets can be used for training classifiers for each biomarker. Thus, for a single biomarker, we get a single classifier after training. It should also be appreciated by those skilled in the art that there is variability between image data obtained from different biomarkers, so training a different classifier for each different biomarker may result in better performance on unknown test data. can be guaranteed, and the biomarker species of the test data will be grasped. Classifiers to be trained can be selected based, at least in part, on how best to handle training data variability in, for example, tissue type, staining protocol, and other features of interest for slide interpretation.

[000102] In some embodiments, the classification module is a support vector machine ("SVM"). In general, SVM is a classification technique, based on statistical learning theory, in which a nonlinear input dataset is transformed into a high-dimensional linear feature space by kernels in the nonlinear case. Without wishing to be bound by any particular theory, it is believed that a support vector machine projects a training data set E representing two different classes by a kernel function K into a high dimensional space. In this transformed data space, the nonlinear data is transformed so that flat lines (discrimination hyperplanes) can be generated that separate classes to maximize class separation. Then K projects the test data into a higher dimensional space and sorts them based on their position relative to the hyperplane. A kernel function K defines how the data is projected into a high-dimensional space.

[000103] In another embodiment, the classification is performed using the AdaBoost algorithm. AdaBoost is an adaptive algorithm that combines many weak classifiers to generate a strong classifier. To generate a probability density function for each individual texture feature Φj (j∈{1, . pixels with stains of 1000 psi or belonging to a specific tissue type) are used. Then, for each Φj, we generate a likelihood scene Lj=(Cj, 1jε{1, . These are combined by the AdaBoost algorithm to give a strong classifier Πj=ΣTi=1αjilji, where for every pixel cjεCj Πj(cj) is the joint likelihood that pixel cj belongs to class ωT and αji is , are the weights determined during training of the features Φi, and T is the number of iterations.

[000104] In some embodiments, the use of derived stain intensity values, the number of specific nuclei, or other classification results allows various marker expression scores, such as percentage positivity or H-scores (herein is used interchangeably with the term "expression score") (ie, from the classification features, the expression score may be calculated). The scoring method is described in co-pending patent application WO2014/102130A1 "Image analysis for breast cancer prognosis," filed Dec. 19, 2013, assigned to the assignee of the present invention and 3/2014. Further described in WO 2014/140085 A1 "Tissue object-based machine learning system for automated scoring of digital whole slides" filed on May 12. and the entire contents of each are incorporated herein by reference. For example, a score (eg, whole slide score) can be determined based at least in part on the number of biomarker-positive tumor cells/biomarker-positive non-tumor cells. In some embodiments, for each detected nuclear speckle, the average speckle intensity, color, and geometric features (such as the area and shape of the detected nuclear speckle) may be computed. Plaques are classified into tumor nuclei and non-tumor cell nuclei. The number of discriminating nuclei output corresponds to the total number of biomarker-positive tumor cells detected in the FOV, as indicated by the tumor nuclei count.

[000105] In some embodiments, again for staining with FAP, a characteristic metric is derived and the percentage of FAP positive or negative cells (e.g., positively or negatively stained stromal cells) (e.g., A classifier is trained such that a percentage positive expression score) can be revealed. In some embodiments, staining areas of 10% or less of the tumor cells are assigned a score of 0, areas of greater than 11% to 25% or less of the tumor cells are assigned a score of 1, and areas of greater than 26% to 50% of the tumor cells are assigned a score of 0. A 2 may be assigned to areas below 51% of tumor cells and a 3 to areas above 51% of tumor cells. Regarding staining intensity, zero/weak staining (negative control) was assigned a score of 0, weak staining clearly stronger than the negative control level was assigned a score of 1, moderately intense staining was assigned a score of 2, and strong staining was assigned. may be assigned 3. In some embodiments, a final score of 3 or higher may be recognized as indicative of positive expression of FAP.

[000106] Segmentation module
[000107] The medium resolution analysis approach employs a segmentation algorithm that generates sub-regions within the input image defined to capture biologically meaningful regions of interest. After the image analysis module 205 derives metrics from the input image (step 310), the segmentation generation module 206 is utilized to segment the input image into a plurality of sub-regions (step 320).

[000108] In some embodiments, segmentation is performed on a single channel image (eg, the "purple" channel in the separated FAP image). Separation methods are known to those skilled in the art (see, for example, Zimmermann "Spectral Imaging and Linear Unmixing in Light Microscopy", Adv Biochem Engin/Biotechnol (2005) 95:245-265 and C. L. Lawson and R. J. Hanson, "Solving least squares Problems", Prentice Hall, 1974, Chapter 23, p. the contents of which are incorporated herein by reference). Other separation methods are disclosed herein (Ruifok et. al. "Quantification of histochemical staining by color deconvolution", Anal Quant Cytol Histol. 2001 Aug. ;23(4):291-9, the entire disclosure of which is incorporated herein by reference).

[000109] In some embodiments, the generated subregions are of a predetermined size or within a specified range within an image processing algorithm (e.g., parameters of the SLIC superpixel generation algorithm as described herein). Capture information of an area of the input image that has a size.

[000110] In some embodiments, an input image is segmented into subregions having a predetermined shape, size, area, and/or spacing. For example, subregions (710) may be oval, circular, square, rectangular, etc., as shown in FIG. In some embodiments, the oval, circular, square, or rectangular subregions may have sizes ranging from 50 pixels to approximately 100 pixels and have similar properties or characteristics (e.g., color, It may also have some other size so that groups of pixels with (brightness, and/or texture) are selected. In some embodiments, the subregions may be non-overlapping and generated via a sampling grid. As used herein, the term "sampling grid" refers to a network of horizontal and vertical lines that are uniformly spaced and superimposed on an image and are ultimately used to locate points that do not overlap points in the image. Regarding. In some embodiments, any number of adjacent positions established by horizontal and vertical lines may be used to define image segments. In some embodiments, subregions are distributed across the image to capture a representative sample of relevant regions for analysis (eg, areas where irregularly shaped cells are the dominant feature).

[000111] In other embodiments, the input image is segmented by applying a series of algorithms to the image, such as global thresholding filters, locally adaptive thresholding filters, morphological operations, and watershed transforms. These filters may be operated sequentially or in any order deemed necessary by those skilled in the art. Of course, any filter may be applied iteratively until the desired result is achieved. In some embodiments, application of the first filter to the input image results in the possibility of having nuclei, such as removing white image areas (corresponding to areas in unstained or nearly unstained tissue samples). Remove areas with low . In some embodiments, this is achieved by applying a global thresholding filter. In some embodiments, global thresholding is based on the median and/or standard deviation computed on the first principal component channel (eg, analogous to the grayscale channel). By determining a global threshold, any white image areas representing unstained or near-unstained areas where nuclei are likely to be absent could be discarded. A filter is then applied to the image to selectively remove artifacts (eg, small speckles, small discontinuities, other small objects) and/or fill holes. In some embodiments, the application of morphological operators removes artifacts and/or fills holes. In some embodiments, a distance-based watershed is applied based on a binary image introduced as input (eg, a binary image resulting from a previous filtering step).

[000112] In some embodiments, the input image is segmented into superpixels. A superpixel algorithm is thought to divide an image into many segments (groups of pixels) that represent perceptually meaningful entities. Each superpixel is obtained by a low-level grouping process and has perceptually consistent units. That is, all pixels in the organism included in the superpixel are stained or not (e.g., the pixels present in the superpixel are of a particular type of stain), staining intensity (e.g., the pixels have a constant relative intensity value or value). ), and texture (eg, pixels have a particular spatial arrangement of color or intensity). Local analysis results for each superpixel can be stored and reported to represent the analysis results on the digital pathology image.

[000113] A superpixel is a collection of pixels with similar attributes such as color, brightness, and texture. An image can be composed of a fixed number of superpixels that contain multiple combinatorial features of pixels and that can preserve the edge information of the original image. Compared to single pixels, superpixels contain rich characteristic information and can greatly reduce the complexity of image post-processing and greatly speed up image segmentation. Superpixels are also useful for estimating and determining probabilities with small neighborhood models.

[000114] A superpixel algorithm is a method of grouping pixels into meaningful atomic regions of similar size. Without wishing to be bound by any particular theory, superpixels are often located at important boundaries in images and tend to take on non-standard or unique shapes when they contain prominent object features. Therefore, it is considered effective. Consistent with the desire for information acquisition and storage in medium-resolution analysis, superpixels are positioned between the pixel level and the object level and are perceptually meaningful groups of pixels that do not comprehensively represent an image object. gives us more information than pixels. A superpixel may be understood as a form of image segmentation that over-segments an image with a short computation time. Superpixel contours have been found to follow natural image boundaries well, as most structures in the image are preserved. Subsequent processing tasks are of reduced complexity and computation time because the image features are computed for each superpixel instead of each pixel. For this reason, superpixels may be useful as a preprocessing step for object-level analysis such as image segmentation.

[000115] Without wishing to be bound by any particular theory, superpixels are believed to over-segment an image by forming dense and uniform groups of pixels that have similar characteristics, for example in color or shape. . Several superpixel approaches have been developed in the past. These can be classified into (i) graph-based approaches and (ii) gradient ascent-based approaches. In graph-based approaches, each pixel is considered a node in the graph. Edge weights are defined between all node pairs that are proportional to their similarities. The superpixel segments are then extracted by formulating and minimizing a cost function defined on the graph. In the gradient ascent-based approach, iterative mapping of pixels into feature space delineates dense regions representing clusters. Each iteration obtains a better segmentation until convergence by refining each cluster.

[000116] Many superpixel algorithms have been developed, such as normalized cut, agglomerative clustering, quickshift, and turbopixel algorithms. The normalized cut algorithm recursively partitions the graph of all pixels by using contour and texture cues and minimizing a cost function defined on the edge of the partition boundary. This results in very regular and visually pleasing superpixels (Jianbo Shi and Jitendra Malik, "Normalized cuts and image segmentation", IEEE Transactions on Pattern Analysis and Machine Intelligence, (PAMI), 22(8):888-905, Aug 2000 (the entire disclosure of which is incorporated herein by reference)). Alastair Moore, Simon Prince, Jonathan Warrell, Umar Mohammed, and Graham Jones "Superpixel Lattices", IEEE Computer Vision and Pattern Recognition (CVPR), 2008. Splitting an image vertically into smaller regions or horizontally We describe a method to generate grid-following superpixels by finding the optimal path or seam. The optimal path is found by using the graph cut method. (Shai Avidan and Ariel Shamir, "Seam carving for content-aware image resizing," ACM Transactions on Graphics (SIGGRAPH), 26(3), 2007 (disclosure see incorporated herein by reference). Quickshift, A. Vedaldi and S. Soatto, "Quick shift and kernel methods for mode seeking," European Conference on Computer Vision (ECCV), 2008, the disclosure of which is incorporated herein by reference. ) uses a mode search segmentation scheme. This initializes the segmentation using the medoid shift procedure. We then move each point in the feature space to the nearest neighbor that increases the Parzen density estimate. Turbo-pixel methods use a level-set-based geometric flow to gradually extend a set of seed positions (A. Levinshtein, A. Stereo, K. Kutulakos, D. Fleet, S. Dickinson, and K. Siddiqi, "Turbopixels: Fast superpixels using geometric flows," IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 2009, the disclosure of which is incorporated herein by reference. incorporated) see). Geometric flow relies on local image gradients for the purpose of regular distribution of superpixels on the image plane. Turbopixel superpixels are constrained to have uniform size, density, and boundary following unlike other methods. For yet another method of generating superpixels, see Radhakrishna Achanta, "SLIC Superpixels Compared to State-of-the-art," Journal of Latex Class Files, Vol. 6, No. 1, December 2011, the entire disclosure of which is incorporated herein by reference.

[000117] A superpixel algorithm called simple linear iterative clustering (SLIC) has been introduced, which outperforms state-of-the-art superpixel methods in both boundary following and efficiency. SLIC has two steps. First, superpixels are generated by grouping pixels by a local k-means clustering (KMC) method, whose distances are measured as Euclidean distances integrated with data and spatial distances. Second, through the use of the Connected Component Algorithm (CCA), the generated small isolated regions are eliminated by merging them as the nearest large superpixel.

[000118] K-means clustering aims to divide n observations into k clusters, with each observation belonging to the cluster closest to the mean and serving as a cluster prototype. Connected component labeling works by scanning an image pixel by pixel (top to bottom, left to right) and identifying connected pixel regions, i.e. regions of adjacent pixels that share the same set of intensity values V ( For binary images V={1}, but in gray-level images V takes a range of values (eg V={51, 52, 53, . . . , 77, 78, 79 , 80})). Connected component labeling works on binary or gray-level images and allows for different measures of connectivity. However, in the following, we assume a binary input image and 8-connectivity. The connected component labeling operator is by moving along the row to a point p where V={1}, where p denotes the pixel to be labeled at any stage of the scanning process. Scan an image. If this is true, then the 4 neighbors of p passed through during the scan (i.e., (i) the left, (ii) above, and (iii and iv) the two upper diagonal neighbors of p) to inspect. Based on this information, the labeling of p is done as follows. If 4 neighbors are all 0 then assign a new label to p, if only 1 neighbor is V={1} then assign that label to p and if there are 2 or more neighbors V={ 1}, assign one of the labels to p and note the equivalent.

[000119] After the scan is complete, the equivalence label pairs are stored in equivalence classes and a unique label is assigned to each class. As a final step, the image undergoes a second scan, during which each label is replaced by the label assigned to its equivalence class. For display purposes, each label may be a different gray level or color.

[000120] SLIC is an adaptation of k-means for superpixel generation, but with two important differences. (i) By limiting the search space to an area proportional to the superpixel size, the number of distance calculations in the optimization is dramatically reduced (this allows the number of pixels complexity is expected to decrease linearly). (ii) A weighted distance measure combines color and spatial proximity while controlling superpixel size and density (Achanta, et al., SLIC Superpixels Compared to State-of-the-Art Superpixel Methods). SLIC Super-Pixel by Comparison with Advanced Super-Pixel Method,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. ).

[000121] SLIC considers image pixels in 5D space defined by their L*a*b values in the CIELAB color space and their respective x and y coordinates. Pixels in 5D space are clustered based on adaptive k-means clustering that integrates color similarity and proximity in the image plane. This clustering is based on a distance measure D that measures color similarity (dc) in L*a*b space and pixel proximity (ds) in x,y space. The latter is normalized by the grid spacing (S) which defines the square root of the total number of image pixels divided by the number of superpixels (k). Superpixel density and regularity are controlled by a constant m. This parameter serves as a weighting criterion between spatial distance (dc) and spectral distance (ds). Larger m gives more spatial proximity weighting, resulting in denser superpixels with less boundary tracking to spectral contours in the image.

[000123] The SLIC algorithm may be applied as follows. Let N _p be the number of pixels in a given image (or portion or region of interest thereof) and let k be the number of superpixels to generate. The main steps of the SLIC algorithm are then as follows.

[000124] (1) Initialize the cluster centers. Let k cluster centers on a regular grid be

After being spaced by a pixel, move these cluster centers to the location with the lowest gradient in the 3x3 neighborhood. Without wishing to be bound by any particular theory, it is believed that this is to avoid setting superpixel centers on edges and to reduce the chance of setting superpixel seeds on noisy pixels.

[000125] (2) Allocate pixels. A local KMC designates each pixel to the nearest cluster center in the local search space.
[000126] (3) Update cluster centers. Set each cluster center as the average of all pixels in the corresponding cluster.

[000127] (4) Repeat steps (2) and (3) until no clusters change, or until another given criterion is met.
[000128] (5) Post-processing. If the size of the isolation region is less than the minimum size S _min , use CCA to reassign the isolation region to a nearby superpixel.

[000129] In step (2) of the SLIC method, local KMC is applied, where each pixel is associated with the closest cluster center whose location the search area covers. In conventional KMC, the search area for each cluster center is the entire image, so the distance from each cluster center to every pixel in the image is calculated. However, in local KMC, the cluster center search space is limited to a local 2S×2S square region. Therefore, SLIC only computes the distance from each cluster center to pixels within its search area.

[000130] In local KMC, Euclidean distance is used in clustering. Let z _i be the data of the i-th cluster center at spatial position (x _i , y _i ). Let _zj be the intensity of the pixel in the central search area. Then the integrated distance between this pixel and the center is:

[000132] where d _f = |z _i −z _j | and

are the intensity and the distance between the pixel and the center, respectively, and m is a regularization parameter that weights the relative contribution of d _f and d _s to the integrated distance _DI . A larger m indicates that _ds is more significant than _df . An equivalent integrated metric _DI that directly describes the contribution of these two metrics can be given by:

[000134] where N _f is the average intensity of the whole screen and wε[0,1] is the regularization parameter. In this context, w and ( ₁ -w) are the ratios of the normalized intensity and each spatial distance in DI, respectively.

[000135] In some embodiments, the parameter k of the SLIC algorithm specifies the number of approximately equally sized superpixels. In some embodiments, the density parameter m can be set to control the trade-off between superpixel homogeneity and boundary following. Without wishing to be bound by any particular theory, by varying the density parameter, regularly shaped superpixels are generated in non-textured regions and highly irregular superpixels are generated in textured regions. considered to be obtained. Again, without wishing to be bound by any particular theory, it is believed that the parameter m allows weighting of the relative importance between color similarity and spatial proximity. When m is large, spatial proximity is more important and the resulting superpixels are denser (ie, smaller area-to-perimeter ratio). If m is small, the resulting superpixels will follow the image boundaries more closely, but will be irregular in size and shape.

[000136] In some embodiments, both the size and density parameters of the superpixel are adjusted. In some embodiments, superpixel sizes ranging from approximately 40 pixels to approximately 400 pixels are used. In other embodiments, superpixel sizes ranging from approximately 60 pixels to approximately 300 pixels are used. In still other embodiments, superpixel sizes ranging from approximately 70 pixels to approximately 250 pixels are used. In yet another embodiment, superpixel sizes ranging from approximately 80 pixels to approximately 200 pixels are used.

[000137] In some embodiments, the compactness parameter ranges from approximately 10 to approximately 100. In other embodiments, the compactness parameter ranges from approximately 20 to approximately 90. In other embodiments, the compactness parameter ranges from approximately 40 to approximately 80. In other embodiments, the compactness parameter ranges from approximately fifty to approximately eighty.

[000138] FIG. 8A shows an example of superpixels generated using SLIC, as described herein, where the superpixels are aligned to suit the localized features of the region of interest without overlap and gaps. is segmented. Furthermore, each superpixel subregion has a specific final shape depending on the local intensity (810) and direction (820) of the presence of biomarker expression. Thus, superpixels are perceptually meaningful to such biological structures of interest. Figures 8B, 8C, and 8D show, respectively, the original IHC image at high magnification, the initialization of the superpixel generation process, and the locally homogeneous final superpixels, whose shape regularity is As mentioned above, it has been adjusted by the technical parameters of the SLIC algorithm.

[000139] representation object generation module
[000140] After the sub-regions have been generated by the sub-region generation module (step 320), the representational objects or points of interest are determined for each sub-region (step 330) using module 207 . In some embodiments, the representational object is a subregion or superpixel outline for a cell or group of cells of interest (eg, fibroblasts or macrophages). In other embodiments, the representation objects are seed points. As described herein, the purpose of the present disclosure is to characterize cells of interest (irregularly shaped cells) based on subregions with similar staining, staining intensity, and/or local texture, and to is to automatically store in the database the sub-regions of the homogenous characteristics of A representation object or its coordinates is one way of storing a generated subregion. Figures 9A and 9B provide examples of polygonal outlines and center seeds of superpixels containing the organism of interest.

[000141] In some embodiments, algorithms are utilized that separate sub-regions of different color or texture and form boundaries that coincide with dominant edges in the image so that the organism of interest (e.g., fiber Boundaries representing irregularly sized or shaped cells, such as blasts or macrophages, are generated. In some embodiments, a thresholding algorithm (e.g., Otsu's method, mean clustering, etc.) is applied such that subregions with no stain are excluded and only subregions with a threshold amount of stain are provided as representation objects. ) may be applied to the stain channel image. In some embodiments, a threshold parameter (eg, a threshold staining parameter provided by a pathologist) may be used to generate a binary mask of sub-regions. In some embodiments, a sequence designed to enhance the image such that (i) sub-regions unlikely to represent objects of interest are separated from (ii) sub-regions representing cells with objects of interest Segmentation is achieved by applying a filter of Additional filters may be selectively applied for artifact removal, small speckle removal, small discontinuity removal, hole filling, and large speckle segmentation.

[000142] In some embodiments, irregularly shaped cells are removed, such as by removing white image areas (corresponding to areas in unstained or nearly unstained tissue samples) in the stain channel binary image. Regions that are unlikely to have subregions that identify are removed. In some embodiments, this is achieved by applying a global thresholding filter. Thresholding converts the intensity image (I) into a binary image (I′) by assigning the value 1 or 0 to all pixels if the intensity is above or below some threshold (here a global threshold). The method used to convert. In other words, global thresholding is applied to divide the pixels according to their intensity values. In some embodiments, global thresholding is based on the median and/or standard deviation computed on the first principal component channel (eg, analogous to the grayscale channel). By determining a global threshold, any white image areas representing unstained or nearly unstained areas where irregularly shaped cells are likely not present could be discarded.

[000143] In some embodiments, for the FAP stain, 1) isolate the violet channel, 2) identify FAP-positive regions by thresholding the violet channel, and 3) perform superpixel segmentation on the violet channel. 4) applying a feature metric to the superpixel object, a boundary can be formed. In some embodiments, the presence or absence of FAP-positive regions may be identified using supervised generation rules trained on ground truth obtained from pathologists. In some embodiments, the FAP positive threshold parameter may be supplied by the pathologist, such as by identifying a threshold on a training set of images. A binary mask may be generated by using this threshold parameter. These methods are described by Auranuch Lorsakul et al. "Automated whole-slide analysis of multiplex-brightfield IHC images for cancer cells and carcinoma-associated fibroblasts", Proc. SPIE 10140, Medical Imaging 2017: Digital Pathology, 1014007 (03/01/2017), the entire disclosure of which is incorporated herein by reference.

[000144] In some embodiments, boundaries of subregions are traced. For example, an algorithm may be provided that traces the outer boundaries of sub-regions as well as "holes" inside or between sub-regions. In some embodiments, subregion boundaries are generated by forming boundary traces using a matlab function called bwboundaries (https://www.mathworks.com/help/images/ref/bwboundaries. html).

[000145] After bounding, the bounding trace is converted to a polygonal outline consisting of x,y coordinates. The x, y coordinates of the trace boundaries may be stored in memory or in a database, e.g. the row and column coordinates of all pixels of the trace boundaries of the subregion object are determined and stored. may

[000146] In some embodiments, the seed points are derived by calculating or calculating the center of gravity or center of mass of each sub-region. A person skilled in the art knows how to determine the centroid of an irregular object. After calculation, the centroids of the subregions are labeled and/or the x,y coordinates of the seeds are stored in a memory or database. In some embodiments, the location of the centroid or center of mass may be superimposed on the input image.

[000147] Labeling Module
[000148] After the segmentation module 206 has been used to generate the sub-regions and the module 207 has been used to compute the representational objects, the labeling module 208 is used to apply the metrics derived from the image analysis module 202, such as Step 310), representation objects are annotated, labeled, or associated with data (step 330). The tagging module 208 may generate a database 209, which is a non-transitory memory for storing data, as described herein. In some embodiments, database 209 stores the image received as input, the coordinates of any polygons and/or seed points, and any associated data or labels from image analysis (see FIG. 11).

[000149] In this regard, a data vector may be stored for each segmented sub-region of the image. For example, a data vector may be stored for each subregion, including any representational objects and associated image analysis data. As an example, the data points 'a', 'b', and 'c' are the coordinates of the represented object, and 'x', 'y', and 'z' are the metrics derived by image analysis (or specific , then the database contains [a, b, c, x, y, z] ¹ , [a, b, c, x, y, z] ² , [a , b, c, x, y, z] ^N . where N is the number of subregions generated by segmentation module 206 .

[000150] In some embodiments, the data from the image analysis module represents individual pixels within the image. Those skilled in the art will appreciate that the data for all pixels within a particular sub-region can be averaged to give the average value of the pixel data within the sub-region. For example, each individual pixel may have a constant intensity. The intensities of all pixels in a particular sub-region may be averaged to give the average pixel intensity for that sub-region. The average pixel of the sub-region may be associated with the representational object of the sub-region, and the data may be stored together in memory.

[000151] With respect to staining with FAP, the FAP-positive area may be given another feature/measurement to the superpixel object. FAP positive area refers to the sum of pixels with FAP intensity above a set threshold. For threshold selection, see Auranuch Lorsakul et al. "Automated whole-slide analysis of multiplex-brightfield IHC images for cancer cells and carcinoma-associated fibroblasts", Proc. SPIE 10140, Medical Imaging 2017: Digital Pathology, 1014007 (03/01/2017), the entire disclosure of which is incorporated herein by reference.

[000152] As an example of the data stored by the labeling module, and for staining a biological sample with a FAP biomarker, the average intensity of FAP stain within a sub-region is derived by image analysis of a particular sub-region. and the FAP stain intensity may be stored in a database along with the coordinates of any representational object in the sub-region. Similarly, the image analysis may be adapted to derive a particular expression score for a subregion, such as a FAP expression score, and the FAP expression score for that subregion is a representation of that particular subregion. It may be stored together with the object. Besides the average intensity score and average expression score of image portions within any sub-region, other parameters may be stored, such as the distance between seed points, distinguishing between tumor cells and irregularly shaped cells. (eg, distance between tumor cells and fibroblasts), and FAP-positive area.

[000153] In some embodiments, by way of example, analysis results (eg, average local intensity, positive staining area) computed within the corresponding superpixel are applied to each corresponding polygon outline and seed. For full slide images, these representational objects (eg, polygon outlines and seeds) with their respective analysis results are stored in the database in xy coordinates. FIG. 10A Head and neck cancer stained purple with fibroblast activation protein (FAP) for fibroblasts (1010) and pancytokeratin (PanCK) for epithelial tumors (1020) yellow. An example of a whole-slide IHC image of tissue is shown. Figures 10B and 11 show examples of polygonal outlines and seeds, respectively, annotated with analysis results of superpixels belonging to fibroblast regions that can be stored in the database.

[000154] Data Readout or Projection Module
[000155] Those skilled in the art will appreciate that the stored analytical results and associated biological features can be later retrieved and the data reported in a variety of formats (e.g., histogram plots of analytical results). Or it may be visualized. More specifically, representational object coordinate data and associated image analysis data may be retrieved from database 209 and used for separate analysis. In some embodiments, and by way of example, representational objects can be retrieved from a database for visualization or reporting of analysis results within whole slide images or within user annotation regions. Correlated or assigned image analysis results can be reported by plotting in a histogram of FAP intensities read out from all slide superpixels, as shown in FIG. Alternatively, data can be visualized in full slide images, field images, or portions of images that are otherwise annotated by a medical professional for review.

[000156] Other components that implement embodiments of the present disclosure
[000157] The system 200 of the present disclosure may be connected to a specimen processing device capable of performing one or more preparation processes on tissue specimens. Preparation processes include deparaffinizing the specimen, conditioning the specimen (e.g., cell conditioning), staining the specimen, performing an antigen readout, immunohistochemical staining (including labeling) or performing other reactions, and/or Performing in-situ hybridization (eg, SISH, FISH, etc.) staining (including labeling) or other reactions, as well as microscopy, microanalysis, mass spectroscopy, or other analytical methods. Other processes to create include, but are not limited to.

[000158] The processing device is capable of applying a fixative to the specimen. Fixatives include cross-linking agents (such as aldehydes (e.g., formaldehyde, paraformaldehyde, and glutaraldehyde), as well as non-aldehyde cross-linkers), oxidizing agents (e.g., metal ions and complexes such as osmium tetroxide and chromic acid), protein denaturants (e.g., acetic acid, methanol, and ethanol), fixatives of unknown mechanism (e.g., mercuric chloride, acetone, and picric acid), combination reagents (e.g., Carnoys fixative, methacan, Bouin's solution, B5 fixatives, Rothmann's solution, and Jandl's solution), microwave, and mixed fixatives (eg, excluded volume fixatives and vapor fixatives).

[000159] If the specimen is a paraffin-embedded sample, the sample may be deparaffinized by using a suitable deparaffinizing solution. Any number of substances may be applied to the specimen in succession after the paraffin is removed. These substances include pretreatment (e.g., reversal of protein cross-links, exposure of nucleic acids, etc.), denaturation, hybridization, washing (e.g., stringency washing), detection (e.g., visual or marker molecules). probes), for amplification (eg, amplification of proteins, genes, etc.), for counter staining, for encapsulation, and the like.

[000160] The specimen processor is capable of applying a wide range of substances to the specimen. These substances include, but are not limited to, stains, probes, reagents, washes, and/or modifiers. These substances can be fluids (eg, gases, liquids, or gas/liquid mixtures), and the like. Fluids can be solvents (eg, polar solvents, non-polar solvents, etc.), solutions (eg, aqueous or other types of solutions), and the like. Reagents include, but are not limited to, stains, wetting agents, antibodies (eg, monoclonal antibodies, polyclonal antibodies, etc.), antigen retrieval solutions (eg, aqueous or non-aqueous antigen buffers, antigen retrieval buffers, etc.). not. A probe can be an isolated nucleic acid or an isolated synthetic oligonucleotide attached to a detectable label or reporter molecule. Labels include radioisotopes, enzyme substrates, cofactors, ligands, chemiluminescent or fluorescent agents, haptens, and enzymes.

[000161] Specimen processors include Ventana Medical Systems, Inc.; Automated devices such as the BENCHMARK XT instrument and the SYMPHONY instrument sold by Co., Inc. are possible. Ventana Medical Systems, Inc. is the assignee of a number of U.S. patents that disclose systems and methods for performing automated analysis, including U.S. Patent Nos. 5,650,327; 6,352,861, 6,827,901, and 6,943,029, and U.S. Patent Application Publication Nos. 2003/0211630 and 2004/0052685, the entire contents of each of which are incorporated by reference. incorporated herein by. Alternatively, specimens can be processed manually.

[000162] After the specimen has been processed, the user can transfer the specimen support slide to the imaging device. In some embodiments, the imaging device is a bright field imaging slide scanner. As one of the bright field imaging devices, Ventana Medical Systems, Inc. There are iScan HT and DP200 (Griffin) bright field scanners sold by . In an automated embodiment, the imaging device is disclosed in International Patent Application No. PCT/US2010/002772 entitled "IMAGING SYSTEM AND TECHNIQUES" (Patent Publication No. WO2011/049608) or dated September 9, 2011. Digital Pathology as disclosed in U.S. Patent Application Serial No. 61/533,114 entitled "IMAGING SYSTEMS, CASSETTES, AND METHODS OF USING THE SAME" filed. Device. International Patent Application PCT/US2010/002772 and US Patent Application No. 61/533,114 are all incorporated herein by reference.

[000163] The imaging system or device may be a multispectral imaging (MSI) system or a fluorescence microscopy system. The imaging system used here is MSI. MSI generally equips a computerized microscope-based imaging system for the analysis of pathological specimens by making the spectral distribution of the image accessible at the pixel level. Although there are a variety of multispectral imaging systems, a common operational aspect of all these systems is the ability to construct multispectral images. A multispectral image is one that captures image data at specific wavelengths or spectral bandwidths across the electromagnetic spectrum. These wavelengths may be filtered out through the use of optical filters or other instruments capable of selecting predetermined spectral components, including electromagnetic radiation at wavelengths beyond the visible light range, such as infrared (IR). .

[000164] MSI systems include optical imaging systems, some of which include spectral selection systems that are adjustable to define a predetermined number N of discrete optical bands. The optical system may be configured to image a tissue sample transilluminated with a broadband light source onto a photodetector. The optical imaging system, in one embodiment, may include a magnifying system, such as a microscope, having a single optical axis generally spatially aligned with a single optical output of the optical system. The system constructs a series of images of tissue such that the spectral selection system is tuned or adjusted (eg, by a computer processor) such that images are acquired in different discrete spectral bands. The device may also include a display on which at least one visually perceptible image of tissue appears from the sequence of acquired images. The spectral selection system selects the spectrum of light transmitted from the source through the sample to the detector in response to either a diffraction grating, a group of optical filters such as thin film interference filters, or user input or preprogrammed processor commands. It may also include an optical dispersive element, such as any other system configured to select a particular passband from .

[000165] In an alternative embodiment, the spectral selection system defines a plurality of light outputs corresponding to N discrete spectral bands. A system of this type captures the transmitted light output from the optical system and, in an identified spectral band, images the sample onto the detection system along an optical path corresponding to the identified spectral band. At least a portion of this light output is spatially redirected along a spatially distinct optical path.

[000166] Embodiments of the subject matter and operations described herein may be digital electronic circuits or computer software, firmware, or hardware (including the structures disclosed herein and their structural equivalents), or any of these. can be implemented in one or more combinations of Embodiments of the subject matter described herein comprise one or more computer programs, i.e., computer program instructions encoded on a computer storage medium for execution by or controlling operation of a data processing apparatus. May be implemented as one or more modules. Any of the modules described herein may include logic that is executed by a processor. As used herein, "logic" refers to any information in the form of instruction signals and/or data that can be applied to affect the operation of a processor. Software is an example of logic.

[000167] Computer storage media can also be and include computer readable storage devices, computer readable storage substrates, random or sequential access memory arrays or devices, or combinations of one or more of these. is also possible. Moreover, a computer storage medium can be a source or destination of computer program instructions encoded in an artificially generated propagated signal, rather than a propagated signal. A computer storage medium can also be or be included in one or more separate physical components or media (eg, multiple CDs, disks, or other storage devices). be. The operations described herein may be implemented as operations performed by a data processing apparatus on data stored in one or more computer readable storage devices or received from other sources.

[000168] The term "programmed processor" includes all kinds of apparatus, devices, and machines for processing data, including, by way of example, programmable microprocessors, computers, systems-on-a-chip, or A plurality of them or a combination thereof can be mentioned. Devices include dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). In addition to hardware, the device includes code that generates an execution environment for the target computer program (for example, processor firmware, protocol stack, database management system, operating system, cross-platform runtime environment, virtual machine, or A code that constitutes one or more combinations of these). The device and execution environment can implement different computing model infrastructures, such as web services, distributed computing, and grid computing infrastructures.

[000169] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, such as a compiled or interpreted language, declarative or procedural language, It may also be deployed in any form, such as a single program or module, component, subroutine, object, or other unit suitable for use in a computing environment. Computer programs may, but need not, correspond to files in a file system. A program may be part of a file holding other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program of interest, or multiple coordinated It may be stored in a file (eg, a file containing one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer or on multiple computers located at one site or distributed across multiple sites and interconnected by a communications network.

[000170] The processes and logic flows described herein are performed by one or more programmable processors executing one or more computer programs and performing operations on input data and generating output. can be These processes and logic flows may be performed by dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits). Also, the apparatus may be implemented as dedicated logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits).

[000171] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from read-only memory, random-access memory, or both. The essential elements of a computer are a processor, which performs operations according to instructions, and one or more memory devices, which store instructions and data. Also, computers typically include, or have access to, one or more mass storage devices (e.g., magnetic, magneto-optical or optical discs) for storing data. It will be operatively coupled to receive data, transmit data, or both. However, a computer need not necessarily have such devices. Additionally, the computer may be connected to another device (e.g., a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device (e.g., a universal serial It can be embedded in a bus (USB flash drive), or many other devices). Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices). , magnetic disks (eg, internal or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by or embedded in dedicated logic circuitry.

[000172] To enable interaction with a user, embodiments of the subject matter described herein include display devices (e.g., LCD (Liquid Crystal Display), LED (Light Emitting Diode) displays, or on a computer having an OLED (organic light emitting diode) display) and a keyboard and pointing device (eg, mouse or trackball) through which the user may provide input to the computer. In some implementations, a touch screen may be used to display information and accept input from a user. Also, other types of devices may be used as well to allow interaction with the user. For example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback), and input from the user can be acoustic input, voice input, Or it can be received in any form, such as haptic input. Computers can also interact with users by sending and receiving documents to and from devices used by users (e.g., by sending web pages to a web browser on a user's client device in response to requests received from the web browser). is.

[000173] Embodiments of the subject matter described herein may include back-end components (eg, data servers), middleware components (eg, application servers), or front-end components (eg, one of the subject matter described herein). a client computer having a graphical user interface or web browser through which a user can interact with an embodiment), or any combination of one or more such back-end, middleware, or front-end components. can be The components of the system can be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include local area networks (“LAN”) and wide area networks (“WAN”), interconnected networks (eg, the Internet), and peer-to-peer networks (eg, ad-hoc peer-to-peer networks). peer network). For example, network 20 in FIG. 1 may include one or more local area networks.

[000174] A computer system may include any number of clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, the server sends data (e.g., HTML pages) to the client device (e.g., for the purpose of displaying data to and accepting user input from users interacting with the client device). . Data generated at the client device (eg, results of user interactions) may be received from the client device at the server.

[000175] Additional Separation Methods/Optional Separation Modules
[000176] Separation is a procedure in which the measured spectrum of a blend pixel is decomposed into a set of constituent spectra, or endmembers, and a set of corresponding fragments, or individuals, to indicate the proportion of each endmember present in that pixel. Specifically, in the separation process, local concentrations of individual stains can be determined by extracting stain-specific channels using known reference spectra for standard tissue and stain combinations. Separation may use a reference spectrum read from the control image or estimated from the image under observation. By separating the component signals of each input pixel, stain-specific channels can be read out and analyzed, such as the hematoxylin and eosin channels in H&E images or the diaminobenzidine (DAB) and counterstain (e.g., hematoxylin) channels in IHC images becomes. The terms "unmixing" and "color deconvolution" (or "deconvolution") and the like (e.g., "deconvolving", "unmixed") are used interchangeably in the art. be. In some embodiments, multiple images are separated by a separation module using linear separation. For linear separation see, for example, Zimmermann "Spectral Imaging and Linear Unmixing in Light Microscopy", Adv Biochem Engineering/Biotechnol (2005) 95:245-265 and C.I. L. Lawson andR. J. Hanson, "Solving least squares Problems," Prentice Hall, 1974, Chapter 23, p. 161, the entire disclosure of which is incorporated herein by reference. In linear stain separation, the measured spectrum (S(λ)) at any pixel is considered a linear mixture of stain spectral components, and the proportion of the individual stain color references (R(λ)) represented at that pixel or Equal to the sum of the weights (A).

[000177] S(λ) = A ₁ ·R ₁ (λ) + A ₂ ·R ₂ (λ) + A ₃ ·R ₃ (λ) ... A _i · R _i (λ)
[000178] More generally, it can be represented in the form of a matrix as follows.

[000179] S(λ) = ΣA _i R _i (λ) or S = R A
[000180] If there are M acquired channel images and N individual stains, then the columns of the M×N matrix R are the optimal color system, and the N×1 vector A is the unknown of the percentage of individual stains and the M×1 vector S is the multichannel spectral vector measured at the pixel. In these equations, the signal (S) of each pixel is measured during the acquisition of multiple images and a reference spectrum or optimum color system and derived as described herein. The various stain contributions (A _i ) can be determined by calculating the contribution to each point in the measured spectrum. In some embodiments, the solution is obtained using an inverse least-squares fitting technique that minimizes the squared difference between the measured and calculated spectra by solving the following set of equations.

[000181] [∂Σ _j {S(λ _j )−Σ _i A _i ·R _i (λ _j )} 2]/∂ A _i =0
[000182] In this equation, j represents the number of detection channels and i equals the number of stains. In solutions of linear equations, the weights (A) often sum to 1 due to conditional separability.

[000183] In another embodiment, the method described in WO 2014/195193 entitled "Image Adaptive Physiologically Plausible Color Separation" filed May 28, 2014. Separation is achieved using the method described in the entire disclosure of which is incorporated herein by reference. Generally, WO2014/195193 describes a separation method by separating component signals of an input image using iteratively optimized reference vectors. In some embodiments, the quality metric is determined by correlating image data from an assay against expected or ideal results inherent to the nature of the assay. For poor quality images or poor correlation for ideal results, one or more reference column vectors of matrix R are adjusted until the correlation indicates a good quality image that matches the physiological and anatomical requirements. Separation is iteratively repeated using the adjusted reference vector. Anatomical, physiological, and assay information may be used to define rules that are applied to the measured image data to determine quality metrics. This information includes how the tissue was stained, structures within the tissue intended to be stained or not intended to be stained, and relationships between structures, stains, and markers specific to the assay being processed. An iterative process yields stain-specific vectors that accurately identify the structures of interest and biologically relevant information, and that are free of noise and unwanted spectra and thus can produce images suitable for analysis. A reference vector is aligned within the search space. The search space defines the range of values that the reference vector can take to represent the stain. The search space may be determined by scanning a variety of representative training assays containing known or commonly occurring problems and determining a high quality reference vector set for the training assay. .

[000184] In another embodiment, the method described in WO2015/124772 entitled "Group Sparsity Model for Image Unmixing" filed Feb. 23, 2015. is used to achieve the separation, the entire disclosure of which is incorporated herein by reference. In general, WO2015/124772 describes separation using a group sparsity framework, where the proportion of stain contributions from multiple collocation markers is modeled within the "same group" and multiple non-collocation markers are modeled in different groups to provide the colocalization information of multiple colocation markers to the modeling group sparsity framework, and by solving the modeling framework using the group lasso , yields the least-squares solution within each group. This least-squares solution produces a sparse solution corresponding to the separation of the collocation markers and the separation of the non-collocation markers within the group. Further, WO2015/124772 inputs image data obtained from a biological tissue sample and retrieves from an electronic memory reference data describing the stain color for each of a plurality of stains, which can be localized in the biological tissue sample. by reading from an electronic memory collocation data describing a group of stains each comprising a stain group, each comprising a group for group Lasso criteria, at least one of which has a size of 2 or more, and using the reference data as a reference matrix, A separation method is described by computing the solution of the group Lasso criterion to obtain separated images. In some embodiments, the method of separating images is such that the percentage of stain contribution from localized markers is assigned within a single group and the percentage of stain contribution from delocalized markers is assigned within a separate group. and solving the group sparsity model using a separation algorithm to yield a least-squares solution within each group.

[000185] Example - Comparison of FAP positive area between high resolution and medium resolution methods
[000186] The precision of the FAP-positive area results was compared by experiments using:
[000187] 1) FAP positive high resolution analysis. For this measurement, all FAP-positive pixels after thresholding were accumulated at high magnification (20X) at a spatial resolution of 0.465 micrometer pixel size. The reported area selected from the pre-annotated regions was then determined as the FAP-positive area per pixel of the region of interest.

[000188] 2) Summing the FAP-positive areas of the FAP superpixel objects, seeds, or polygon outlines in the pre-annotated region yields the FAP-positive areas measured using the medium-resolution analysis approach described herein. calculated.

[000189] According to both methods, six different annotation areas (see Fig. 14), each with a different shape (large, small, circular, or odd shape, etc.) were analyzed. As shown in Figure 15 and the table below, there was no significant difference in the comparative results of the FAP-positive area measured using these two methods ( ^R2 = 0.99, p < 0.001).

[000190] In conclusion, summing the area features computed within a superpixel in a particular annotation equals the area computed directly by the high-resolution analytical technique within that annotation. The FAP-positive area results show that there is no significant difference in the computation by the two methods (with and without superpixels) with different geometries of the annotation regions.

[000191] All U.S. patents, U.S. patent application publications, U.S. patent applications, foreign patents, foreign patent applications, and non-patent publications referenced herein and/or published in application data sheets are is incorporated herein by reference. Aspects of the embodiments may be modified, as necessary to employ concepts of the various patents, applications, and publications, to provide yet further embodiments.

[000192] Although the present disclosure has been described with reference to a number of exemplary embodiments, it is understood that many other modifications and embodiments within the spirit and scope of the principles of the present disclosure can be devised by those skilled in the art. shall be understood. More particularly, reasonable variations and modifications in the component parts and/or arrangements of the subject combination constructions may be made within the scope of the foregoing disclosure, drawings, and appended claims without departing from the spirit of the present disclosure. is possible. Variations and improvements in component parts and/or arrangements, as well as alternative uses, will become apparent to those skilled in the art.

Claims (32)

1. A method of storing image analysis data derived from an image of a biological specimen having at least one stain, comprising:
(a) deriving one or more feature metrics from the image;
(b) segmenting the image into a plurality of sub-regions, each sub-region being substantially in at least one of the presence or absence of staining, staining intensity, or local texture; comprising uniform pixels in , each sub-region of the set of sub-regions identifying an amount of stain above a threshold ;
(c) generating a plurality of representation objects corresponding to a set of sub-regions of said plurality of sub-regions;
each representation object of the plurality of representation objects (i) identifies a particular cell type; and (ii) by a polygon outline generated from boundaries traced from corresponding subregions of the set of subregions . defined ,
a step;
(d) removing other regions of said image that exclude said plurality of sub-regions;
(e) associating each of the plurality of representational objects with one or more of the derived feature metrics;
(f) storing coordinates of each representation object of said plurality of representation objects in a database along with said association derived feature metric;
A method, including 2. The method of claim 1, wherein segmenting the image into the plurality of subregions comprises deriving superpixels. The superpixels (i) group pixels with local k-means clustering, and (ii) combine small isolated regions into large nearest-neighbor superpixels using a connected components algorithm. 3. The method of claim 2, wherein the method is derived by: segmenting the image into the plurality of sub-regions comprises superimposing a sampling grid over the image, the sampling grid defining non-overlapping areas having a predetermined size and shape; The method according to any one of claims 1-3. 5. Any one of claims 1 to 4, wherein the subregions have a size of MxN, where M ranges from 50 pixels to 100 pixels and N ranges from 50 pixels to approximately 100 pixels. The method described in . The method of any one of claims 1-5, wherein said specific cell types comprise fibroblasts or macrophages. A method according to any preceding claim, wherein each representation object of said plurality of representation objects is further identified by corresponding seed points. 8. The method of claim 7, wherein the corresponding seed points of the representation object are derived by computing centroids as corresponding sub-regions of the set of sub-regions. 7. The method of claim 6, wherein the derived one or more feature metrics comprise staining intensity, and an average staining intensity of all pixels within each generated representational object contour is computed. wherein the derived one or more feature metrics comprise expression scores, wherein an average expression score corresponding to an area within each generated subregion is the plurality of generated representational objects and The method of any one of claims 1-7, associated therewith. 8. The method of any one of claims 1 to 7, further comprising retrieving the stored coordinates and associated feature metric data from the database and projecting the associated feature metric data onto the image. described method. 1. A system for deriving data corresponding to irregularly shaped cells from an image of a biological sample comprising at least one stain, comprising: (i) one or more processors; a memory coupled to a processor of, when executed by said one or more processors,
(a) deriving one or more feature metrics from the image;
(b) generating a plurality of sub-regions in said image, each sub-region having pixels with similar properties selected from color, brightness and/or texture, of a set of sub-regions; each subregion identifying an amount of stain above a threshold ;
(c) computing a sequence of representation objects corresponding to a set of sub-regions of said plurality of sub-regions;
each representation object of said series of representation objects (i) identifies a particular cell type; and (ii) by a polygon outline generated from boundaries traced from corresponding subregions of said set of subregions . be defined and
(d) removing other regions of the image that exclude the plurality of sub-regions;
(e) for each of said series of representational objects, associating said one or more feature metrics derived from said images with calculated coordinates that identify the location of said objects;
a memory containing computer-executable instructions that cause the system to perform operations including: 13. The system of claim 12, wherein generating the plurality of subregions comprises deriving superpixels. 14. The system of claim 13, wherein the superpixels are derived using one of a graph-based approach or a gradient elevation based approach. 13-, wherein the superpixels are derived by (i) grouping pixels with local k-means clustering and (ii) merging small isolated regions into large nearest neighbor superpixels using a connected component algorithm. 15. The system according to any one of 14. 16. The system of any one of claims 12-15, wherein said specific cell type comprises fibroblasts or macrophages. A system according to any one of claims 12 to 16, wherein each expression object of said series of expression objects is further identified by a corresponding seed point. 18. The system of any one of claims 12-17, wherein the act further comprises storing the coordinates of the one or more derived feature metrics and association computational representation objects in a database. 19. Any one of claims 12-18, wherein the derived one or more feature metrics comprise at least one expression score selected from percent positivity, H-score, and staining intensity. A system as described in . The system of any one of claims 12-19, wherein data corresponding to irregularly shaped cells are derived for a region of interest within the image. 21. The system of claim 20, wherein the region of interest is an area of the image annotated by a medical professional. A non-transitory computer-readable medium storing instructions for analyzing data associated with an irregularly shaped living body, the instructions comprising:
(a) instructions for deriving one or more feature metrics from an image of a biological sample, said biological sample comprising at least one stain;
(b) dividing said image into a series of sub- regions by grouping pixels having similar properties, said similar properties being selected from color, brightness and/or texture; instructions , wherein the subregion identifies an amount of stain above a threshold ;
(c) an instruction to compute a plurality of representation objects corresponding to a set of sub-regions of said sequence of sub-regions;
each representation object of said series of representation objects (i) identifies a particular cell type; and (ii) by a polygon outline generated from boundaries traced from corresponding subregions of said set of subregions . defined ,
(d) instructions for removing other regions of the image that exclude the plurality of sub-regions;
(e) for each of said plurality of representational objects, instructions for associating said one or more feature metrics derived from said image with calculated coordinates identifying a position of said representational object;
A non-transitory computer-readable medium, including 23. The non-transitory computer-readable medium of claim 22, wherein dividing the image into a series of subregions comprises computing superpixels. 23. The superpixels are computed using one of a normalized cuts algorithm, an agglomerative clustering algorithm, a quick shift algorithm, a turbo pixel algorithm, or a simple linear iterative clustering algorithm. The non-transitory computer-readable medium as described in . 23-, wherein the superpixels were generated using simple iterative clustering, with a superpixel size parameter set from about 40 pixels to about 400 pixels, and a density parameter set from about 10 to about 100 25. The non-transitory computer readable medium of any one of Clauses 24. 23-, wherein the superpixels are computed by (i) grouping pixels with local k-means clustering and (ii) merging small isolated regions into larger nearest-neighbour superpixels using a connected component algorithm. 26. The non-transitory computer readable medium of any one of Clauses 25. 12. The biological sample is stained with at least fibroblast activation protein (FAP), and the one or more derived feature metrics comprise at least one of FAP staining intensity or FAP percent positivity. 27. The non-transitory computer readable medium of any one of Clauses 22-26. 28. The non-transitory computer readable medium of claim 27, wherein an average FAP percentage positivity is calculated for all pixels within the sub-region. 28. The non-transitory computer readable medium of Claim 27, wherein an average FAP staining intensity is calculated for all pixels within the subregion. 27. The non-transitory computer-readable medium of any one of claims 22-26, wherein each expression object of the plurality of expression objects is further identified by a corresponding seed point. 27. The non-transitory computer-readable medium of any one of claims 22-26, further comprising instructions for storing the coordinates of the one or more derived feature metrics and associated computational representation objects in a database. 32. The non-transitory computer-readable medium of claim 31, further comprising instructions for projecting stored information onto the image of the biological sample.

Images