CN112638234B - Image processing apparatus, image processing method, and computer readable medium - Google Patents
Image processing apparatus, image processing method, and computer readable medium Download PDFInfo
- Publication number
- CN112638234B CN112638234B CN201980057669.5A CN201980057669A CN112638234B CN 112638234 B CN112638234 B CN 112638234B CN 201980057669 A CN201980057669 A CN 201980057669A CN 112638234 B CN112638234 B CN 112638234B
- Authority
- CN
- China
- Prior art keywords
- image
- display
- images
- unit
- quality improvement
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 227
- 238000003672 processing method Methods 0.000 title claims description 9
- 230000006872 improvement Effects 0.000 claims abstract description 303
- 238000003384 imaging method Methods 0.000 claims description 144
- 230000008859 change Effects 0.000 claims description 38
- 230000010354 integration Effects 0.000 claims 2
- 230000009467 reduction Effects 0.000 abstract description 5
- 238000000034 method Methods 0.000 description 218
- FCKYPQBAHLOOJQ-UHFFFAOYSA-N Cyclohexane-1,2-diaminetetraacetic acid Chemical compound OC(=O)CN(CC(O)=O)C1CCCCC1N(CC(O)=O)CC(O)=O FCKYPQBAHLOOJQ-UHFFFAOYSA-N 0.000 description 207
- 230000008569 process Effects 0.000 description 162
- 238000012549 training Methods 0.000 description 110
- 238000004458 analytical method Methods 0.000 description 105
- 238000012014 optical coherence tomography Methods 0.000 description 91
- 230000033001 locomotion Effects 0.000 description 58
- 210000001508 eye Anatomy 0.000 description 53
- 230000003287 optical effect Effects 0.000 description 53
- 239000010410 layer Substances 0.000 description 46
- 238000003745 diagnosis Methods 0.000 description 40
- 238000012986 modification Methods 0.000 description 40
- 230000004048 modification Effects 0.000 description 40
- 238000010801 machine learning Methods 0.000 description 34
- 230000004044 response Effects 0.000 description 31
- 238000010586 diagram Methods 0.000 description 27
- 210000004204 blood vessel Anatomy 0.000 description 22
- 238000005259 measurement Methods 0.000 description 22
- 230000007704 transition Effects 0.000 description 18
- 238000013528 artificial neural network Methods 0.000 description 13
- 238000007689 inspection Methods 0.000 description 12
- 238000003860 storage Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 11
- 239000013307 optical fiber Substances 0.000 description 11
- 238000009826 distribution Methods 0.000 description 9
- 230000011218 segmentation Effects 0.000 description 9
- 210000001525 retina Anatomy 0.000 description 8
- 238000012935 Averaging Methods 0.000 description 7
- 238000013527 convolutional neural network Methods 0.000 description 7
- 239000006185 dispersion Substances 0.000 description 7
- 230000002159 abnormal effect Effects 0.000 description 6
- 230000002792 vascular Effects 0.000 description 6
- 210000004027 cell Anatomy 0.000 description 5
- 238000012790 confirmation Methods 0.000 description 5
- 238000013135 deep learning Methods 0.000 description 5
- 210000003128 head Anatomy 0.000 description 5
- 210000005036 nerve Anatomy 0.000 description 5
- 238000001514 detection method Methods 0.000 description 4
- 230000003902 lesion Effects 0.000 description 4
- 230000004913 activation Effects 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 3
- 239000003086 colorant Substances 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 238000007499 fusion processing Methods 0.000 description 3
- 238000005286 illumination Methods 0.000 description 3
- 210000003733 optic disk Anatomy 0.000 description 3
- 230000010287 polarization Effects 0.000 description 3
- 238000002583 angiography Methods 0.000 description 2
- 230000004397 blinking Effects 0.000 description 2
- 230000017531 blood circulation Effects 0.000 description 2
- 210000004556 brain Anatomy 0.000 description 2
- 210000005252 bulbus oculi Anatomy 0.000 description 2
- 238000007796 conventional method Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 239000000284 extract Substances 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 210000004072 lung Anatomy 0.000 description 2
- 238000010606 normalization Methods 0.000 description 2
- 210000000056 organ Anatomy 0.000 description 2
- 238000003825 pressing Methods 0.000 description 2
- 210000001747 pupil Anatomy 0.000 description 2
- 230000005855 radiation Effects 0.000 description 2
- 230000002207 retinal effect Effects 0.000 description 2
- 230000003595 spectral effect Effects 0.000 description 2
- 238000012706 support-vector machine Methods 0.000 description 2
- 239000002344 surface layer Substances 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 210000004127 vitreous body Anatomy 0.000 description 2
- 208000002874 Acne Vulgaris Diseases 0.000 description 1
- 208000003098 Ganglion Cysts Diseases 0.000 description 1
- 208000010412 Glaucoma Diseases 0.000 description 1
- 206010025421 Macule Diseases 0.000 description 1
- 208000005400 Synovial Cyst Diseases 0.000 description 1
- 206010000496 acne Diseases 0.000 description 1
- 206010064930 age-related macular degeneration Diseases 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000004323 axial length Effects 0.000 description 1
- 230000008901 benefit Effects 0.000 description 1
- 210000000601 blood cell Anatomy 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000015556 catabolic process Effects 0.000 description 1
- 239000002872 contrast media Substances 0.000 description 1
- 210000004087 cornea Anatomy 0.000 description 1
- 230000008878 coupling Effects 0.000 description 1
- 238000010168 coupling process Methods 0.000 description 1
- 238000005859 coupling reaction Methods 0.000 description 1
- 230000006378 damage Effects 0.000 description 1
- 238000006731 degradation reaction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 210000000416 exudates and transudate Anatomy 0.000 description 1
- 238000000799 fluorescence microscopy Methods 0.000 description 1
- 238000002599 functional magnetic resonance imaging Methods 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 210000002216 heart Anatomy 0.000 description 1
- 230000000004 hemodynamic effect Effects 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 238000009434 installation Methods 0.000 description 1
- 210000000936 intestine Anatomy 0.000 description 1
- 230000004410 intraocular pressure Effects 0.000 description 1
- 230000001678 irradiating effect Effects 0.000 description 1
- 238000005304 joining Methods 0.000 description 1
- 210000003734 kidney Anatomy 0.000 description 1
- 238000007562 laser obscuration time method Methods 0.000 description 1
- 210000000265 leukocyte Anatomy 0.000 description 1
- 230000031700 light absorption Effects 0.000 description 1
- 210000004185 liver Anatomy 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 208000002780 macular degeneration Diseases 0.000 description 1
- 238000002595 magnetic resonance imaging Methods 0.000 description 1
- 230000007257 malfunction Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 230000028161 membrane depolarization Effects 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 210000000496 pancreas Anatomy 0.000 description 1
- 230000035515 penetration Effects 0.000 description 1
- 230000000737 periodic effect Effects 0.000 description 1
- 210000000608 photoreceptor cell Anatomy 0.000 description 1
- 238000011176 pooling Methods 0.000 description 1
- 230000000306 recurrent effect Effects 0.000 description 1
- 238000011946 reduction process Methods 0.000 description 1
- 238000007670 refining Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 210000003786 sclera Anatomy 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000002603 single-photon emission computed tomography Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000010408 sweeping Methods 0.000 description 1
- 238000002604 ultrasonography Methods 0.000 description 1
Abstract
An image processing apparatus comprising: an image quality improvement unit configured to generate a second image subjected to at least one of noise reduction and contrast enhancement compared to the first image from the first image of the eye to be inspected by using a learning model; and a display control unit configured to display the first image and the second image on the display unit by switching, arranging, or superposing the first image and the second image side by side.
Description
Technical Field
The invention relates to an image processing apparatus, an image processing method, and a computer readable medium.
Background
An apparatus (OCT apparatus) using Optical Coherence Tomography (OCT) is put into practical use as a method for obtaining a tomographic image of an object to be examined such as a living body without damage and without wound. OCT apparatuses are widely used particularly as ophthalmic apparatuses for acquiring images for ophthalmic diagnosis.
In OCT, by causing light reflected from a measurement object and light reflected from a reference mirror to interfere with each other and analyzing the intensity (intensity) of the interference light, a tomographic image of the object to be examined can be obtained. Time domain OCT (TD-OCT) is known as one of such OCT. In TD-OCT, depth information of an object to be inspected is obtained by continuously changing the position of a reference mirror.
Spectral domain OCT (SD-OCT) and swept source OCT (SS-OCT) are also known. In SD-OCT, interference light obtained by causing light interference using low coherence light is divided, and depth information is replaced with frequency information to obtain frequency information. In SS-OCT, interference light is obtained by using light whose wavelength has been divided in advance by a wavelength scanning light source. Note that SD-OCT and SS-OCT are also collectively referred to as "fourier domain OCT (FD-OCT)".
By using OCT, a tomographic image based on depth information of the object to be examined can be acquired. Further, by integrating the acquired three-dimensional tomographic image in the depth direction and projecting the integrated image onto a two-dimensional plane, a front image of the measurement object can be generated. Conventionally, in order to improve the image quality of these images, the images are acquired and subjected to an averaging process a plurality of times. However, in this case, it takes time to perform imaging a plurality of times.
Patent document 1 discloses a technique of converting a previously acquired image into an image with a higher resolution by means of an Artificial Intelligence (AI) engine in order to cope with rapid development in medical technology and also conform to simple image capturing in an emergency. According to this technique, for example, an image acquired by performing imaging a fewer number of times can be converted into an image with higher resolution.
[ Reference List ]
[ Patent literature ]
Patent document 1: japanese patent laid-open No. 2018-5841
Disclosure of Invention
[ Technical problem ]
However, even if an image has high resolution, it cannot be said that the image is an image suitable for image diagnosis in some cases. For example, even when the resolution of an image is high, if there is a large amount of noise or low contrast or the like in the image, an object that should be observed cannot be appropriately recognized in some cases.
In this respect, an object of the present invention is to provide an image processing apparatus, an image processing method, and a computer-readable medium having a program stored thereon, which are capable of producing an image more suitable for image diagnosis than the conventional art.
[ Solution to the problem ]
An image processing apparatus according to an embodiment of the present invention includes: an image quality improvement unit configured to generate, from a first image of an eye to be inspected, a second image subjected to at least one of noise reduction and contrast enhancement compared to the first image, using a learning model; and a display control unit configured to cause the first image and the second image to be switched, juxtaposed or superimposed to be displayed on the display unit.
An image processing apparatus according to another embodiment of the present invention includes: an image quality improvement unit configured to generate, using the learning model, a second image from a first image, the first image being a front image generated based on information of a range of the eye to be inspected in a depth direction, the second image being subjected to at least one of noise reduction and contrast enhancement as compared with the first image; and a selection unit configured to select a learning model to be used by the image quality improvement unit from a plurality of learning models based on a range in a depth direction for generating the first image.
Further features of the invention will become apparent from the following description of exemplary embodiments with reference to the attached drawings.
Drawings
Fig. 1 is a diagram showing a schematic configuration of an OCT apparatus according to embodiment 1.
Fig. 2 is a diagram showing a schematic configuration of a control unit according to embodiment 1.
Fig. 3A is a diagram showing an example of training data according to embodiment 1.
Fig. 3B is a diagram showing an example of training data according to embodiment 1.
Fig. 4 is a diagram showing an example of the construction of the learning model according to embodiment 1.
Fig. 5 is a flowchart showing a series of image processing operations according to embodiment 1.
Fig. 6A is a diagram showing an example of a report screen displayed switching between images obtained before and after the image quality improvement processing.
Fig. 6B is a diagram showing an example of a report screen displayed switching between images obtained before and after the image quality improvement processing.
Fig. 7 is a diagram showing an example of a report screen in which images obtained before and after the image quality improvement process are displayed in juxtaposition.
Fig. 8A is a diagram showing an example of a report screen in which a plurality of images to which the image quality improvement process is applied are simultaneously displayed.
Fig. 8B is a diagram showing an example of a report screen in which a plurality of images to which the image quality improvement process is applied are simultaneously displayed.
Fig. 9 is a diagram showing a schematic configuration of a control unit according to embodiment 2.
Fig. 10 is a flowchart showing a series of image processing operations according to embodiment 2.
Fig. 11A is a diagram showing an example of changing the image quality improvement processing.
Fig. 11B is a diagram showing an example of changing the image quality improvement processing.
Fig. 12A is a diagram showing an example of a report screen in which a plurality of images to which the image quality improvement process is applied are simultaneously displayed.
Fig. 12B is a diagram showing an example of a report screen in which a plurality of images to which the image quality improvement process is applied are simultaneously displayed.
Fig. 13 is a flowchart showing a series of image processing operations according to embodiment 3.
Fig. 14 is a diagram showing a schematic configuration of a control unit according to embodiment 4.
Fig. 15 is a flowchart showing a series of image processing operations according to embodiment 4.
Fig. 16A is a diagram showing a configuration example of a neural network serving as a machine learning model according to modification 9.
Fig. 16B is a diagram showing a configuration example of a neural network serving as a machine learning model according to modification 9.
Fig. 17A is a diagram showing a configuration example of a neural network serving as a machine learning model according to modification 9.
Fig. 17B is a diagram showing a configuration example of a neural network serving as a machine learning model according to modification 9.
Fig. 18 is a diagram showing an example of a user interface according to embodiment 5.
FIG. 19A is a diagram showing an example of a plurality of OCTA en-face images.
FIG. 19B is a diagram showing an example of a plurality of OCTA en-face images.
Fig. 20A is a diagram showing an example of a user interface according to embodiment 5.
Fig. 20B is a diagram showing an example of a user interface according to embodiment 5.
Detailed Description
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.
However, the size, material, shape, and relative position of the components described in the following embodiments are indefinite, and may be changed according to the configuration of the apparatus to which the present invention is applied or according to various conditions. Furthermore, in the different figures, identical or functionally similar elements are denoted by the same reference numerals.
In the following embodiments, although an example in which an eye to be inspected is given as an object to be inspected, other organs of a person or the like may be taken as the object to be inspected. Further, an OCTA (OCT angiography) image of the eye to be inspected is described as an example of an image subjected to image quality improvement processing using a learning model related to a machine learning model (machine learning engine). Note that the term "OCTA" refers to angiography using OCT without contrast agent. In the OCTA, an OCTA image (frontal blood vessel image) is generated by integrating three-dimensional motion contrast data obtained based on depth information of a subject in a depth direction and projecting the integrated data onto a two-dimensional plane.
Here, the term "motion contrast data" refers to data obtained by repeatedly imaging substantially the same position of an object to be examined and detecting a change with time within the object during imaging. Note that the phrase "substantially the same location" refers to a location that is the same to the extent that motion contrast data may be generated, and includes a location that is slightly offset from the exact same location. The motion contrast data is obtained by calculating the phase, vector or intensity variation over time of the complex OCT signal, e.g. based on differences, ratios or correlations etc.
The gist concerning an image quality improvement process using a learning model concerning a machine learning model will now be mentioned. By subjecting an image to image quality improvement processing using a learning model related to a machine learning model, although a high-quality image can be obtained from a small number of images on the one hand, on the other hand, in some cases, a tissue that does not exist actually is visualized in the image, or a tissue that does exist originally is not visualized in the image. Therefore, there are the following problems: in an image subjected to image quality improvement by image quality improvement processing using a learning model, it is difficult to determine the authenticity of a visualized tissue.
Accordingly, in the following embodiments, an image processing apparatus is provided that can generate an image more suitable for image diagnosis than conventional techniques by using a machine learning model, and for such an image, the authenticity of a tissue visualized in the image can also be easily determined.
Note that although the OCTA image is described in the following embodiment, the image subjected to the image quality improvement process is not limited thereto, and may be a tomographic image, an intensity en-face image, or the like. Here, the term "en-face image" refers to a front image generated by projecting or integrating data within a predetermined depth range determined based on two reference planes onto a two-dimensional plane with respect to three-dimensional data of an object to be inspected. examples of en-face images include intensity en-face images based on intensity tomographic images and OCTA images based on motion contrast data.
Example 1
Hereinafter, an optical coherence tomography apparatus (OCT apparatus) and an image processing method according to embodiment 1 of the present invention are described with reference to fig. 1 to 7. Fig. 1 is a schematic configuration of an OCT apparatus according to the present embodiment.
The OCT apparatus 1 according to the present embodiment includes an OCT imaging unit 100, a control unit (image processing apparatus) 200, an input unit 260, and a display unit 270.
The OCT imaging unit 100 includes an imaging optical system of an SD-OCT apparatus, and acquires a signal including tomographic information of the eye E based on interference light generated by causing return light from the eye E irradiated with measurement light by the scanning unit and reference light corresponding to the measurement light to interfere with each other. An optical interference unit 110 and a scanning optical system 150 are provided in the OCT imaging unit 100.
The control unit 200 may control the OCT imaging unit 100, generate an image from a signal obtained from the OCT imaging unit 100 or other devices (not shown), and process the generated/acquired image. The display unit 270 is any display such as an LCD display, and can display a GUI for operating the OCT imaging unit 100 and the control unit 200, a generated image, an image subjected to any type of processing, and various information such as patient information.
The input unit 260 is used to operate the control unit 200 by operating the GUI and by inputting information. The input unit 260 includes, for example, a pointing device such as a mouse, a touch pad, a trackball, a touch panel display, or a stylus, and a keyboard. Note that in the case of using a touch panel display, the display unit 270 and the input unit 260 may be integrally formed with each other. Note that although in the present embodiment, it is assumed that the OCT imaging unit 100, the control unit 200, the input unit 260, and the display unit 270 are separate units from each other, some or all of these units may be integrally constituted with each other.
The optical interference unit 110 in the OCT imaging unit 100 is provided with a light source 111, a coupler 113, a collimator optical system 121, a dispersion compensating optical system 122, a mirror 123, a lens 131, a diffraction grating 132, an imaging lens 133, and a line sensor 134. The light source 111 is a low coherence light source that emits near infrared light. Light emitted from the light source 111 propagates through the optical fiber 112a and enters the coupler 113 as a spectroscopic unit. The light entering the coupler 113 is divided into measurement light and reference light, the measurement light travels toward the scanning optical system 150 side, and the reference light travels toward the reference light optical system side including the collimator optical system 121, the dispersion compensating optical system 122, and the reflecting mirror 123. The measurement light enters the optical fiber 112b and is guided to the scanning optical system 150. On the other hand, the reference light enters the optical fiber 112c and is introduced into the reference light optical system.
The reference light entering the optical fiber 112c exits from the fiber end, is incident on the dispersion compensating optical system 122 through the collimating optical system 121, and is guided to the reflecting mirror 123. The reference light reflected by the mirror 123 travels along the optical path in the opposite direction and enters the optical fiber 112c again. The dispersion compensating optical system 122 is a component for compensating the dispersion of the optical system with respect to the scanning optical system 150 and the eye E to be inspected, and matching the dispersion of the measurement light with the dispersion of the reference light. The mirror 123 is configured to be drivable in the optical axis direction of the reference light by a driving unit (not shown) controlled by the control unit 200, and can relatively change the optical path length of the reference light with respect to the optical path length of the measurement light and match the optical path lengths of the reference light and the measurement light.
On the other hand, the measurement light entering the optical fiber 112b exits from the optical fiber end and is incident on the scanning optical system 150. The scanning optical system 150 is an optical system configured to be relatively movable with respect to the eye E. The scanning optical system 150 is configured to be drivable in front, rear, upward, downward, left and right directions with respect to the eyeball axis of the eye E by a driving unit (not shown) controlled by the control unit 200, and to be alignable with respect to the eye E. Note that the scanning optical system 150 may be configured to include the light source 111, the coupler 113, a reference light optical system, and the like.
A collimator optical system 151, a scanning unit 152, and a lens 153 are disposed in the scanning optical system 150. Light emitted from the fiber end of the optical fiber 112b is substantially collimated by the collimating optical system 151 and incident on the scanning unit 152.
The scanning unit 152 has two galvanometer mirrors capable of rotating a mirror surface, one of which deflects light in the horizontal direction and the other of which deflects light in the vertical direction. The scanning unit 152 deflects the incident light according to the control of the control unit 200. In this way, the scanning unit 152 can scan the measurement light in two directions (i.e., a main scanning direction which is a direction (X direction) perpendicular to the paper surface and a sub scanning direction which is a direction (Y direction) parallel to the paper surface) on the fundus Er of the eye E. Note that the main scanning direction and the sub scanning direction are not limited to the X direction and the Y direction, and it is sufficient if the main scanning direction and the sub scanning direction are directions perpendicular to the depth direction (Z direction) of the eye E to be inspected and intersect each other. Thus, for example, the main scanning direction may be the Y direction, and the sub scanning direction may be the X direction.
The measurement light scanned by the scanning unit 152 forms an illumination spot on the fundus Er of the eye E via the lens 153. When the scanning unit 152 receives the in-plane deflection, each illumination spot moves (scans) on the fundus Er of the eye E. The return light of the measurement light reflected and scattered from the fundus Er at the position of the illumination spot travels along the optical path in the opposite direction, enters the optical fiber 112b, and returns to the coupler 113.
As described above, the reference light reflected by the mirror 123 and the return light of the measurement light from the fundus Er of the eye E return to the coupler 113, and interfere with each other to become interference light. The interference light passes through the optical fiber 112d and is emitted to the lens 131. The interference light is substantially collimated by the lens 131 and is incident on the diffraction grating 132. The diffraction grating 132 has a periodic structure and separates incident interference light. The divided interference light is imaged on the line sensor 134 through the imaging lens 133 that can change the focus state. The line sensor 134 outputs a signal corresponding to the intensity of light irradiated onto each sensor unit to the control unit 200. The control unit 200 may generate a tomographic image of the eye E to be inspected based on the interference signal output from the line sensor 134.
By the above-described series of operations, tomographic information about the depth direction at one point of the eye E to be inspected can be obtained. Such a series of operations is called "a-scan".
Further, by driving the galvanometer mirror of the scanning unit 152, interference light is generated at one point adjacent to the eye E, and tomographic information in the depth direction at one point adjacent to the eye E is acquired. By repeating this series of control, a-scan is performed a plurality of times in an arbitrary lateral direction (main scanning direction), two-dimensional tomographic information of the eye E to be inspected can be acquired in the foregoing lateral direction and depth direction. This operation is called "B-scan". The control unit 200 may construct one B-scan image by collecting a plurality of a-scan images based on interference signals acquired through a-scan. Hereinafter, the B-scan image is referred to as a "two-dimensional tomographic image".
In addition, by slightly driving the galvanometer mirror of the scanning unit 152 in the sub-scanning direction orthogonal to the main scanning direction, tomographic information at another position (adjacent scanning line) of the eye E to be inspected can be acquired. By repeating this operation to collect a plurality of B-scan images, the control unit 200 can acquire a three-dimensional tomographic image within a predetermined range of the eye E to be inspected.
Next, the control unit 200 is described with reference to fig. 2. Fig. 2 shows a schematic configuration of the control unit 200. The control unit 200 is provided with an obtaining unit 210, an image processing unit 220, a drive control unit 230, a memory 240, and a display control unit 250.
The obtaining unit 210 can obtain data of the output signal of the line sensor 134 corresponding to the interference signal of the eye E to be inspected from the OCT imaging unit 100. Note that the data of the output signal obtained by the obtaining unit 210 may be an analog signal or a digital signal. In the case where the obtaining unit 210 obtains an analog signal, the control unit 200 may convert the analog signal into a digital signal.
Further, the obtaining unit 210 may obtain tomographic data generated by the image processing unit 220, and various images such as a two-dimensional tomographic image, a three-dimensional tomographic image, a motion contrast image, and an en-face image. Here, the term "tomographic data" refers to data including information about a cross section of an object to be examined, and includes a signal obtained by subjecting an interference signal obtained by OCT to fourier transform, a signal obtained by subjecting a related signal to any processing, a tomographic image based on these signals, and the like.
In addition, the obtaining unit 210 obtains an image capturing condition group of an image to be subjected to image processing (for example, information on an image capturing date and time, an image capturing part name, an image capturing area, an image capturing angle, an image capturing system, image resolution and gradation, an image size, an image filter, and an image data format). Note that the image capturing condition group is not limited to the foregoing example of the image capturing condition group. Further, the imaging condition group does not necessarily include all the conditions mentioned in the foregoing example, and may include some of these conditions.
Specifically, the obtaining unit 210 obtains imaging conditions of the OCT imaging unit 100 when imaging a related image. Furthermore, the obtaining unit 210 may also obtain the set of imaging conditions stored in the data structure constituting the image, according to the data format of the image. Note that in the case where the image capturing conditions are not stored in the data structure of the image, the obtaining unit 210 may also obtain the image capturing information groups including the image capturing condition groups, respectively, from a storage device or the like that stores the image capturing conditions.
Further, the obtaining unit 210 may also obtain information for identifying the eye to be inspected, such as an object identification number, from the input unit 260 or the like. Note that the obtaining unit 210 may obtain various data, various images, or various information from the memory 240 or other devices (not shown) connected to the control unit 200. The obtaining unit 210 may store the obtained various data or images in the memory 240.
The image processing unit 220 may generate a tomographic image, an en-face image, or the like from the data obtained by the obtaining unit 210 or the data stored in the memory 240, and may perform image processing on the generated or obtained image. The image processing unit 220 is provided with a tomographic image generation unit 221, a motion contrast generation unit 222, an en-face image generation unit 223, and an image quality improvement unit 224.
The tomographic image generation unit 221 may subject the interference signal data obtained by the obtaining unit 210 to wave number conversion, fourier transform, absolute value conversion (acquisition of amplitude), and the like to generate tomographic data, and may generate a tomographic image of the eye E to be inspected based on the tomographic data. The interference signal data obtained by the obtaining unit 210 may be data of a signal output from the line sensor 134, or may be data of an interference signal obtained from the memory 240 or a device (not shown) connected to the control unit 200. Note that any known method may be employed as a method for generating a tomographic image, and a detailed description thereof is omitted here.
The tomographic image generation unit 221 may also generate a three-dimensional tomographic image based on the generated tomographic images of the plurality of sites. The tomographic image generation unit 221 can generate a three-dimensional tomographic image by arranging tomographic images of a plurality of sites side by side in one coordinate system, for example. Here, the tomographic image generation unit 221 may generate a three-dimensional tomographic image based on tomographic images of a plurality of sites obtained from the memory 240 or a device (not shown) connected to the control unit 200.
The motion contrast generation unit 222 may generate a two-dimensional motion contrast image using a plurality of tomographic images obtained by imaging substantially the same position. Further, the motion contrast generating unit 222 may generate a three-dimensional motion contrast image by arranging the generated two-dimensional motion contrast images of the respective parts side by side in one coordinate system.
In the present embodiment, the motion contrast generation unit 222 generates a motion contrast image based on a decorrelation value (decorrelation value) between a plurality of tomographic images obtained by imaging substantially the same position of the eye E.
Specifically, the motion contrast generation unit 222 acquires a plurality of tomographic images aligned with respect to a plurality of tomographic images obtained by imaging the substantially same position continuously with each other for imaging time. Note that various known methods may be used as the alignment method. For example, one reference image is selected among a plurality of tomographic images, the similarity to other tomographic images is calculated while changing the position and angle of the reference image, and the displacement amount of each tomographic image with respect to the reference image is calculated. Then, alignment of the plurality of tomographic images is performed by correcting the respective tomographic images based on the calculation result. Note that the processing for alignment may be performed by a component separate from the motion contrast generation unit 222. Furthermore, the alignment method is not limited to this method, and alignment may be performed by any known method.
The motion contrast generation unit 222 calculates a decorrelation value of each two tomographic images whose imaging times are continuous with each other among the plurality of tomographic images subjected to alignment using the following equation 1.
[ 1]
Here, a (x, z) represents the amplitude at the position (x, z) of the tomographic image a, and B (x, z) represents the amplitude at the same position (x, z) of the tomographic image B. The decorrelation value M (x, z) obtained as a result takes a value from 0 to 1, and becomes closer to 1 as the difference between the two amplitude values increases. Note that although the case of using a two-dimensional tomographic image on the X-Z plane has been described in the present embodiment, a two-dimensional tomographic image on the Y-Z plane or the like, for example, may be used. In this case, the position (x, z) may be replaced by the position (y, z) or the like. Note that the decorrelation value may be determined based on an intensity value of the tomographic image, or may be determined based on a value of an interference signal corresponding to the tomographic image.
The motion contrast generation unit 222 determines pixel values of the motion contrast image based on the decorrelation values M (x, z) at the respective positions (pixel positions), and generates the motion contrast image. Note that although in the present embodiment, the motion contrast generation unit 222 calculates a decorrelation value for tomographic images whose imaging times are continuous with each other, the method for calculating the motion contrast data is not limited thereto. The imaging times of the two tomographic images for which the decorrelation value M is obtained do not have to be continuous with each other, and it is sufficient if the imaging times related to the respective tomographic images corresponding to each other are within a predetermined time interval. Therefore, for example, in order to extract an object whose change with time is small, two tomographic images whose imaging interval is longer than a normal specified time may be extracted from the acquired plurality of tomographic images and a decorrelation value may be calculated. Further, instead of the decorrelation value, a variance value or a value obtained by dividing the maximum value by the minimum value (maximum value/minimum value) or the like may be determined.
Note that the method for generating a motion contrast image is not limited to the foregoing method, and any other known method may also be used.
The en-face image generating unit 223 may generate an en-face image (OCTA image) as a front image from the three-dimensional motion contrast image generated by the motion contrast generating unit 222. Specifically, the en-face image generation unit 223 may generate an OCTA image as a front image by projecting a three-dimensional motion contrast image on a two-dimensional plane based on, for example, two arbitrary reference planes of the eye E in the depth direction (Z direction). Further, the en-face image generating unit 223 may generate an intensity en-face image from the three-dimensional tomographic image generated by the tomographic image generating unit 221 in a similar manner.
More specifically, for example, the en-face image generating unit 223 determines representative values of pixel values in the depth direction at respective positions in the X-Y direction of the region surrounded by the two reference planes, determines the pixel values at the respective positions based on the representative values, and generates an en-face image. In this case, examples of the representative value include an average value, a median value, or a maximum value of pixel values in a range in the depth direction of the region surrounded by the two reference planes.
Note that the reference plane may be a plane along a layer boundary at the cross section of the eye E to be inspected, or may be a plane. The range in the depth direction between the reference planes used to generate the en-face image is hereinafter referred to as an "en-face image generation range". Further, the method for generating an en-face image according to the present embodiment is one example, and the en-face image generating unit 223 may generate an en-face image using any known method.
The image quality improving unit 224 uses a learning model described later to generate a high-quality OCTA image based on the OCTA image generated by the en-face image generating unit 223. Further, the image quality improving unit 224 may generate a high-quality tomographic image or a high-quality intensity en-face image based on the tomographic image generated by the tomographic image generating unit 221 or the intensity en-face image generated by the en-face image generating unit 223. Note that the image quality improving unit 224 may also generate a high-quality image based on various images obtained by the obtaining unit 210 from the memory 240 or other devices (not shown) connected to the control unit 200, not just on an OCTA image captured using the OCT imaging unit 100 or the like. In addition, the image quality improvement unit 224 may perform the image quality improvement processing on the three-dimensional motion contrast image or the three-dimensional tomographic image, not just the OCTA image or the tomographic image.
The drive control unit 230 can control the driving of components of the OCT imaging unit 100, such as the light source 111, the scanning optical system 150, the scanning unit 152, and the imaging lens 133, which are connected to the control unit 200. The memory 240 may store various data obtained by the obtaining unit 210, and various data and images such as tomographic images or OCTA images generated and processed by the image processing unit 220. Further, the memory 240 may store attributes (name, age, etc.) of the subject, information (axial length of eyeball, intraocular pressure, etc.) related to the eye to be inspected such as measurement results acquired using other inspection apparatuses, imaging parameters, image analysis parameters, and parameters set by the operator. Note that a configuration in which these images and information are stored in an external storage device (not shown) may also be employed. The memory 240 may also store programs for execution by the processor for performing the functions of the various components of the control unit 200.
The display control unit 250 may cause various information obtained by the obtaining unit 210 and various images such as tomographic images, OCTA images, and three-dimensional motion contrast images generated and processed by the image processing unit 220 to be displayed on the display unit 270. The display control unit 250 may also cause information or the like input by a user or the like to be displayed on the display unit 270.
The control unit 200 may be constructed, for example, by using a general-purpose computer. Note that the control unit 200 may be constituted by a dedicated computer using the OCT apparatus 1. The control unit 200 is equipped with a CPU (central processing unit) (not shown) or MPU (micro processing unit) and a storage medium including a memory such as an optical disk or ROM (read only memory). The components other than the memory 240 of the control unit 200 may be constituted by software modules executed by a processor such as a CPU or MPU. Furthermore, the components discussed may be formed of circuitry, such as an ASIC, that provides specific functionality, or a separate device. For example, the memory 240 may be constituted by any storage medium such as an optical disk or a memory.
Note that the control unit 200 may include one or more processors such as a CPU and a storage medium such as a ROM. Accordingly, the respective components of the control unit 200 may be configured to function in a case where at least one or more processors are connected to at least one storage medium and the at least one or more processors execute programs stored in the at least one storage medium. Note that the processor is not limited to a CPU or MPU, and may be a GPU (graphics processing unit) or the like.
Next, a learning model related to a machine learning model according to a machine learning algorithm such as deep learning according to the present embodiment is described with reference to fig. 3A to 4. The learning model according to the present embodiment generates and outputs an image subjected to the image quality improvement processing based on the input image according to the learning tendency.
In the present specification, the term "image quality improvement processing" refers to converting an input image into an image having an image quality more suitable for image diagnosis, and the term "high-quality image" refers to an image having an image quality more suitable for image diagnosis that has been converted. Here, the content of the image quality suitable for the image diagnosis depends on what is desired to be diagnosed using various image diagnoses. Therefore, although not being able to be generalized, the image quality suitable for image diagnosis includes, for example, the following image quality: the noise amount is low, the contrast is high, so that the imaging target is displayed in colors and gray scale which are easy to observe, the image size is large, and the resolution is high. In addition, the image quality suitable for image diagnosis may include image quality such that an object or gray scale rendered in the image generation process that does not exist in reality is removed from the image.
The term "learning model" refers to a model that is trained (learned) in advance using appropriate training data on a machine learning model according to any machine learning algorithm such as deep learning. However, it is assumed that the learning model is not a model that does not perform further learning, and is a model that may also perform incremental learning. Training data is composed of one or more pairs of input data and true values (correct answer data). In the present embodiment, a pair of input data and a true value (ground truth) is constituted by an OCTA image and an OCTA image obtained by subjecting a plurality of OCTA images including the aforementioned OCTA image to an averaging process such as addition averaging.
The average image subjected to the averaging process is a high-quality image suitable for image diagnosis because the pixels normally visualized in the source image group are enhanced. In this case, since the pixels of the normal visualization are enhanced, the generated high-quality image is a high-contrast image in which the difference between the low-intensity region and the high-intensity region is clear. In addition, for example, in the average image, random noise generated in each round of imaging can be reduced, and an area that is not rendered in the source image at a specific point in time can be subjected to interpolation processing using other source image groups.
Note that among the pairs constituting the training data, pairs that do not contribute to improvement of image quality may be removed from the training data. For example, if the image quality of a high-quality image as a true value included in a pair of training data is not suitable for image diagnosis, there is a possibility that: the images output by the learning model learned using the relevant training data will have image quality unsuitable for image diagnosis. Therefore, by removing the pair of image quality unsuitable for image diagnosis of the true value from the training data, the possibility that the learning model generates an image having image quality unsuitable for image diagnosis can be reduced.
Further, in the case where the average intensities or intensity distributions are greatly different in the image group as a pair, there is a possibility that: a learning model learned using the relevant training data will output an image unsuitable for image diagnosis and having an intensity distribution that is greatly different from that of a low-quality image. Thus, pairs of input data and true values with widely differing average intensities or intensity distributions may be removed from the training data.
In addition, in the case where the structures or positions of the imaging targets to be rendered in the image group as a pair are greatly different, there is a possibility that: a learning model learned using the relevant training data will output an image that is not suitable for image diagnosis and where the structure or position where the imaging target is rendered is greatly different from a low-quality image. Thus, such a pair of input data and true values may also be removed from the training data: the structure or position of the camera object to be rendered varies greatly between the input data and the true values.
By using the learning model learned in this way, in the case where an OCTA image obtained by one round of imaging (inspection) is input, the image quality improving unit 224 can generate a high-quality OCTA image with improved contrast, reduced noise, or the like by the averaging process. Accordingly, the image quality improving unit 224 can generate a high-quality image suitable for image diagnosis based on the low-quality image as the input image.
Next, an image used when learning will be described. Rectangular area images of a certain image size corresponding to the positional relationship are used to create an image group constituting a pair of the OCTA image 301 and the high-quality OCTA image 302 constituting the training data. The manner in which the image in question is created will now be described with reference to fig. 3A and 3B.
First, a case is described in which one pair group constituting training data is regarded as being constituted of an OCTA image 301 and a high-quality OCTA image 302. In this case, as shown in fig. 3A, a pair is formed in which the entire OCTA image 301 is regarded as input data and the entire high-quality OCTA image 302 is regarded as true value. Note that, although in the example shown in fig. 3A, pairs of input data and true values are formed by using the respective images as a whole, the pairs are not limited thereto.
For example, as shown in fig. 3B, a pair may be formed in which a rectangular area image 311 in the OCTA image 301 is employed as input data, and a rectangular area image 321 in the OCTA image 302 as a corresponding image capturing area is employed as a true value.
Note that, at the time of learning, the scanning range (imaging view angle) and the scanning density (a-scan number and B-scan number) may be normalized so that the image size is uniform, so that the rectangular area size at the time of learning may be uniform. Further, the rectangular region images shown in fig. 3A and 3B are examples of rectangular region sizes when each rectangular region size is used to learn independently of each other.
Further, the number of rectangular areas may be set to one in the example shown in fig. 3A, and may be set to a plurality of rectangular areas in the example shown in fig. 3B. For example, in the example shown in fig. 3B, a pair may also be constituted in which a rectangular area image 312 in the OCTA image 301 is employed as input data, and a rectangular area image 322 that is a corresponding image capturing area in the high-quality OCTA image 302 is employed as true value. Thus, rectangular region image pairs different from each other can be created from pairs composed of one OCTA image and one high quality OCTA image. Note that the content of the pair group constituting the training data can be enhanced by creating a large number of rectangular region image pairs while changing the position of the region to different coordinates in the OCTA image as the source image and the high-quality OCTA image.
Although the rectangular area is discretely shown in the example shown in fig. 3B, the OCTA image and the high-quality OCTA image as the source images may each be continuously and without a gap divided into groups of rectangular area images having uniform image sizes. Or an OCTA image and a high quality OCTA image as source images may each be divided into rectangular region image groups at random positions corresponding to each other. In this way, by selecting an image of a smaller area as a pair composed of input data and true values as rectangular areas, a large amount of pair data can be generated from the OCTA image 301 and the high-quality OCTA image 302 constituting the original pair. Thus, the time required to train the machine learning model can be shortened.
Next, a Convolutional Neural Network (CNN) that performs image quality improvement processing with respect to an input tomographic image is described with reference to fig. 4 as one example of a learning model according to the present embodiment. Fig. 4 shows an example of a construction 401 of a learning model used by the image quality improvement unit 224.
The learning model shown in fig. 4 is composed of a plurality of layer groups responsible for processing the input value group for output. Note that the types of layers included in the construction 401 of the learning model are a convolution layer, a downsampling layer, an upsampling layer, and a merging layer.
The convolution layer is a layer that performs convolution processing on an input value group according to set parameters such as the kernel size of a filter, the number of filters, a stride value (value of a stride), and an expansion value (dilation value). Note that the dimension of the kernel size of the filter may vary depending on the dimension of the input image.
The downsampling layer is a layer that performs the following processing: the number of sets of output values is made smaller than the number of sets of input values by refining or combining the sets of input values. Specifically, for example, a max-pooling process may be used as such a process.
The upsampling layer is a layer that performs the following processing: the number of sets of output values is made greater than the number of sets of input values by copying the sets of input values or adding values interpolated from the sets of input values. Specifically, for example, a linear interpolation process may be used as such a process.
The merging layer is a layer which performs the following processing: sets of output values such as a layer or sets of pixel values constituting an image are input from a plurality of sources, and the sets of values are combined by cascading or adding the sets of values.
Note that as parameters set for the convolution group included in the configuration 401 shown in fig. 4, for example, by setting the kernel size of the filter to a width of 3 pixels and a height of 3 pixels and setting the number of filters to 64, an image quality improvement process of a certain accuracy can be performed. But care should be taken in this respect because if the settings of the parameters for the group of layers and the group of nodes constituting the neural network are different, the degree to which the trend trained based on the training data will be reproducible in the output data will be different in some cases. In other words, in many cases, the appropriate parameters will vary depending on the form at the time of implementation, and thus the parameters may be changed to preferred values as needed.
In addition, there are also the following cases: CNNs can obtain better characteristics by changing the configuration of the CNN, not just by using the method of changing parameters as described above. The term "better characteristics" means, for example, an increase in the accuracy of the image quality improvement process, a decrease in the time taken for the image quality improvement process, and a decrease in the time required for training the training of the machine learning model.
Although not shown in the drawings, as a modification of the configuration of the CNN, for example, after a convolution layer or the like, a batch normalization layer (batch normalization layer) or an activation layer using a rectifying linear unit may be incorporated.
When data is input into a learning model of such a machine learning model, data according to the design of the machine learning model is output. For example, according to a trend of training a machine learning model using training data, output data having a high probability corresponding to input data is output. In the case of the learning model according to the present embodiment, when the OCTA image 301 is input, a high-quality OCTA image 302 is output according to the tendency of training the machine learning model using training data.
Note that, in the case of learning in such a manner that an image is divided into regions, the learning model outputs rectangular region images as high-quality OCTA images corresponding to the respective rectangular regions. In this case, first, the image quality improvement unit 224 divides the OCTA image 301 as an input image into rectangular region image groups based on the image size at the time of learning, and inputs the divided rectangular region image groups into the learning model. Thereafter, the image quality improvement unit 224 arranges the respective images of a set of rectangular area images as high-quality OCTA images output from the learning model according to the same positional relationship as that of the respective images of the rectangular area image set input into the learning model, and merges the rectangular area images. In this way, the image quality improvement unit 224 can generate the high-quality OCTA image 302 corresponding to the input OCTA image 301.
Next, a series of image processing operations according to the present embodiment will be described with reference to fig. 5 to 7. Fig. 5 is a flowchart showing a series of image processing operations according to the present embodiment.
First, in step S501, the obtaining unit 210 obtains a plurality of pieces of three-dimensional tomographic information obtained by imaging the eye E a plurality of times. The obtaining unit 210 may obtain tomographic information of the eye E to be inspected using the OCT imaging unit 100, or may obtain tomographic information from the memory 240 or other devices connected to the control unit 200.
Here, a case of obtaining tomographic information of the eye E to be inspected by using the OCT imaging unit 100 will be described. First, the operator sits a patient as an object in front of the scanning optical system 150, performs alignment, inputs patient information into the control unit 200, and the like, and then starts OCT imaging. The drive control unit 230 of the control unit 200 drives the galvanometer mirror of the scanning unit 152 to scan substantially the same position of the eye to be inspected a plurality of times, thereby obtaining a plurality of items of tomographic information (interference signals) at substantially the same position of the eye to be inspected. Thereafter, the drive control unit 230 slightly drives the galvanometer mirror of the scanning unit 152 in the sub-scanning direction orthogonal to the main scanning direction, and obtains a plurality of items of tomographic information at other positions (adjacent scanning lines) of the eye E to be inspected. By repeating this control, the obtaining unit 210 obtains a plurality of pieces of three-dimensional tomographic information within a predetermined range of the eye E to be inspected.
Next, in step S502, the tomographic image generation unit 221 generates a plurality of three-dimensional tomographic images based on the obtained plurality of three-dimensional tomographic information items. Note that in step S501, in the case where the obtaining unit 210 obtains a plurality of three-dimensional tomographic images from the memory 240 or other devices connected to the control unit 200, step S502 may be omitted.
The motion contrast generation unit 222 generates three-dimensional motion contrast data (three-dimensional motion contrast image) based on the plurality of three-dimensional tomographic images in step S503. Note that the motion contrast generation unit 222 may obtain pieces of motion contrast data based on three or more tomographic images acquired with respect to substantially the same position, and generate an average value of the pieces of motion contrast data as final motion contrast data. Note that in the case where the obtaining unit 210 obtains three-dimensional motion contrast data from the memory 240 or other devices connected to the control unit 200 in step S501, step S502 and step S503 may be omitted.
In step S504, for the three-dimensional motion contrast data, the en-face image generation unit 223 generates an OCTA image according to an instruction from the operator or based on a predetermined en-face image generation range. Note that in the case where the obtaining unit 210 obtains an OCTA image from the memory 240 or other devices connected to the control unit 200 in step S501, steps S502 to S504 may be omitted.
In step S505, the image quality improvement unit 224 performs image quality improvement processing on the OCTA image using the learning model. The image quality improvement unit 224 inputs the OCTA image into the learning model, and generates a high-quality OCTA image based on the output from the learning model. Note that, in the case where the learning model learns in such a manner that the image is divided into regions, the image quality improvement unit 224 first divides the OCTA image as the input image into rectangular region image groups based on the image size at the time of learning, and inputs the divided rectangular region image groups into the learning model. Thereafter, the image quality improvement unit 224 arranges each image of a set of rectangular area images as high-quality OCTA images output from the learning model according to the same positional relationship as that of each image of the set of rectangular area images input into the learning model, and combines these rectangular area images, thereby generating a final high-quality OCTA image.
In step S506, the display control unit 250 causes the display unit 270 to switch from displaying the original OCTA image (first image) to displaying the high quality OCTA image (second image) generated by the image quality improving unit 224. As described above, in the image quality improvement process using the machine learning model, in some cases, a blood vessel that does not actually exist is visualized in the OCTA image, or a blood vessel that does not exist originally is not visualized in the OCTA image. In this regard, by the display control unit 250 causing the display unit 270 to switch from displaying the original OCTA image to displaying the generated high-quality OCTA image, it is possible to facilitate determination of whether a blood vessel is a blood vessel newly generated by the image quality improvement process or a blood vessel also present in the original image. When the display processing by the display control unit 250 ends, a series of image processing operations end.
Next, a method for operating the control unit 200 is described with reference to fig. 6A to 7. Fig. 6A and 6B show examples of report screens displayed switching between images before and after the image quality improvement processing. A broken layer image 611 and an OCTA image 601 before the image quality improvement processing are shown on a report screen 600 shown in fig. 6A. The broken layer image 611 and the OCTA image 602 (high quality OCTA image) after the image quality improvement processing are shown on the report screen 600 shown in fig. 6B.
On the report screen 600 shown in fig. 6A, when the operator uses a mouse as one example of the input unit 260 and presses a right mouse button on the OCTA image 601, a pop-up menu 620 for selecting whether to perform image quality improvement processing is displayed. When the operator selects to perform the image quality improvement process on the pop-up menu 620, the image quality improvement unit 224 performs the image quality improvement process on the OCTA image 601.
Then, as shown in fig. 6B, the display control unit 250 causes the display on the report screen 600 to switch from displaying the OCTA image 601 before the image quality improvement processing to displaying the image OCTA 602 after the image quality improvement processing. Note that the pop-up menu 620 may also be opened by pressing the right mouse button again on the OCTA image 602, and causing the display to switch to displaying the OCTA image 601 before image quality is performed.
Note that although an example has been described in which the display is switched between images before and after the image quality improvement processing by using the pop-up menu 620 displayed in accordance with an operation in which the operator presses the right button of the mouse, any other method may be performed as a method of switching images, in addition to the pop-up menu. For example, image switching may also be performed using a button (for example, a button 3420 shown in fig. 18, 20A, and 20B), a pull-down menu, a radio button, a check box, or a keyboard operation provided on the report screen. In addition, switching of the display image may be performed by an operation of a mouse wheel or a touch operation on a touch panel display.
The operator can arbitrarily switch between displaying the OCTA image 601 before the image quality improvement process and the OCTA image 602 after the image quality improvement process by the above-described method. Therefore, the operator can easily view and compare the OCTA images before and after the image quality improvement process, and can easily confirm the change between the OCTA images caused by the image quality improvement process. Therefore, the operator can easily recognize a blood vessel that does not exist in reality and is visualized in the OCTA image by the image quality improvement process, or an originally existing blood vessel that disappears from the OCTA image due to the image quality improvement process, and can easily determine the authenticity of the tissue visualized in the image.
Note that although in the above-described display method, images before and after the image quality improvement processing are switched to be displayed, similar effects can be obtained by displaying these images in a juxtaposed or superimposed manner. Fig. 7 shows an example of a report screen in the case where images before and after the image quality improvement processing are displayed in a juxtaposed manner. On the report screen 700 shown in fig. 7, an OCTA image 701 before the image quality improvement processing and an OCTA image 702 after the image quality improvement processing are displayed in a juxtaposed manner.
Also in this case, the operator can easily view and compare the images before and after the image quality improvement process, and can easily confirm the change between the images caused by the image quality improvement process. Therefore, the operator can easily recognize a blood vessel that does not exist in reality and is visualized in the OCTA image by the image quality improvement process, or a blood vessel that exists originally and is disappeared from the OCTA image due to the image quality improvement process, and can easily determine the authenticity of the tissue visualized in the image. Note that in the case where images before and after the image quality improvement processing are displayed in a superimposed manner, the display control unit 250 may set transparency for at least one of the images before and after the image quality improvement processing, and cause the images before and after the image quality improvement processing to be displayed in a superimposed manner on the display unit 270.
Further, as described above, the image quality improvement unit 224 may also perform the image quality improvement process using the learning model on the tomographic image or the intensity en-face image instead of only the OCTA image. In this case, as a pair of training data of the learning model, such a pair may be used: the tomographic image or the intensity en-face image before the averaging is adopted as the input data, and the tomographic image or the intensity en-face image after the averaging is adopted as the true value. Note that in this case, the learning model may be a single learning model that is learned using training data such as an OCTA image or a tomographic image, or a plurality of learning models that are learned for respective kinds of images may be used as the learning model. In the case of using a plurality of learning models, the image quality improvement unit 224 may use a learning model corresponding to the kind of image that is the object of the image quality improvement process. Note that the image quality improvement unit 224 may perform image quality improvement processing using a learning model for a three-dimensional motion contrast image or a three-dimensional tomographic image, and training data in this case may also be prepared in the same manner as described above.
In fig. 7, a tomographic image 711 before the image quality improvement processing and a tomographic image 712 after the image quality improvement processing are displayed in a juxtaposed manner. Note that, similarly to the OCTA images before and after the image quality improvement processing as shown in fig. 6A and 6B, the display control unit 250 may cause tomographic images or intensity en-face images before and after the image quality improvement processing to be displayed on the display unit 270 in a switching manner. Further, the display control unit 250 may cause tomographic images or intensity en-face images before and after the image quality improvement process to be displayed in a superimposed manner on the display unit 270. Also in these cases, the operator can easily view and compare the images before and after the image quality improvement process, and can easily confirm the change between the images caused by the image quality improvement process. Accordingly, the operator can easily recognize the tissue that does not exist in reality and is visualized in the image by the image quality improvement process, or the tissue that exists originally and is lost from the image due to the image quality improvement process, and can easily determine the authenticity of the tissue visualized in the image.
As described above, the control unit 200 according to the present embodiment includes the image quality improvement unit 224 and the display control unit 250. The image quality improvement unit 224 generates a second image subjected to at least one of noise reduction and contrast enhancement compared to the first image from the first image of the eye to be inspected using the learning model. The display control unit 250 causes the first image and the second image to be switched, juxtaposed or superimposed to be displayed on the display unit 270. Note that the display control unit 250 may switch between the first image and the second image according to an instruction from the operator and display the relevant switch image on the display unit 270.
In this way, the control unit 200 may generate a high quality image from the source image with reduced noise and/or enhanced contrast. Accordingly, the control unit 200 may generate an image more suitable for image diagnosis, such as a clearer image or an image in which a site or lesion desired to be observed is enhanced, as compared to the conventional art.
Further, the operator can easily view and compare the images before and after the image quality improvement processing, and can easily confirm the variation between the images caused by the image quality improvement processing. Accordingly, the operator can easily recognize the tissue that does not exist in reality and is visualized in the image by the image quality improvement process, or the tissue that exists originally and is lost from the image due to the image quality improvement process, and can easily determine the authenticity of the tissue visualized in the image.
Although the average image is used as the true value of the training data for the learning model according to the present embodiment, the training data is not limited thereto. For example, a high-quality image obtained by performing maximum a posteriori processing (MAP estimation processing) for a source image group may be used as a true value of training data. In the MAP estimation process, likelihood functions are obtained based on probability densities of respective pixel values in a plurality of images, and true signal values (pixel values) are estimated using the obtained likelihood functions.
The high-quality image obtained by the MAP estimation process is a high-contrast image based on pixel values close to the true signal values. Further, since the estimated signal value is determined based on the probability density, noise generated randomly is reduced in the high-quality image obtained by the MAP estimation process. Therefore, by using the high-quality image obtained through the MAP estimation process as training data, the learning model can generate a high-quality image with reduced noise and high contrast suitable for image diagnosis from the input image. Note that, regarding a method for generating a pair of input data and true values of training data, a method similar to the case where an average image is used as training data may be performed.
Further, a high-quality image obtained by applying smoothing filter processing to the source image may be used as the true value of the training data. In this case, the learning model can generate a high-quality image with reduced random noise from the input image. In addition, an image obtained by applying the gradation conversion processing to the source image can also be used as a true value of the training data. In this case, the learning model can generate a high-quality image with enhanced contrast from the input image. Note that, regarding a method for generating a pair of input data and true values of training data, a method similar to the case where an average image is used as training data may be performed.
Note that the input data of the training data may be an image obtained from an image pickup device having the same image quality trend as the OCT image pickup unit 100. Further, the true value of the training data may be a high-quality image obtained by high-cost processing such as processing using a successive approximation method, or may be a high-quality image obtained by imaging an object to be inspected corresponding to input data using an imaging device having higher performance than the OCT imaging unit 100. Further, the true value may be a high-quality image obtained by performing a rule-based noise reduction process based on the structure of the object to be inspected or the like. Here, the noise reduction processing may include, for example, processing of replacing a high-intensity pixel which is only one pixel, which is obviously noise occurring in a low-intensity region, with an average value of adjacent low-intensity pixel values. Therefore, the learning model may employ, as the training data, an image imaged by an imaging apparatus having higher performance than an imaging apparatus for imaging an input image, or an image obtained by imaging processing involving a larger number of processes (number of processes) than imaging processing for obtaining an input image.
Note that although it has been described that the image quality improvement unit 224 generates a high-quality image with reduced noise or enhanced contrast by using a learning model, the image quality improvement process by the image quality improvement unit 224 is not limited thereto. As described above, it is sufficient that the image quality improvement unit 224 can generate an image having an image quality more suitable for image diagnosis by the image quality improvement processing.
Further, in the case where images before and after the image quality improvement processing are caused to be displayed on the display unit 270 in a juxtaposed manner, the display control unit 250 may enlarge and display any image among the images before and after the image quality improvement processing being displayed on the display unit 270 in a juxtaposed manner in accordance with an instruction from the operator. More specifically, for example, on the report screen 700 shown in fig. 7, if the operator selects the OCTA image 701, the display control unit 250 may enlarge-display the OCTA image 701 on the report screen 700. Further, if the operator selects the OCTA image 702 after the image quality improvement process, the display control unit 250 may enlarge-display the OCTA image 702 on the report screen 700. In this case, the operator can observe in more detail the image that the operator wishes to observe among the images before and after the image quality improvement processing.
In addition, in the case where the generation range of the en-face image such as the OCTA image is changed according to an instruction of the operator, the control unit 200 may change the display from the image displayed in the superimposed manner to the image based on the changed generation range and the image subjected to the image quality improvement. More specifically, when the operator changes the en-face image generation range through the input unit 260, the en-face image generation unit 223 generates an en-face image before the image quality improvement process based on the changed generation range. The image quality improving unit 224 generates a high-quality en-face image from the en-face image newly generated by the en-face image generating unit 223 using a learning model. Thereafter, the display control unit 250 causes the display unit 270 to change from displaying the en-face image before and after the image quality improvement processing being displayed in a juxtaposed manner to displaying the newly generated en-face image before and after the image quality improvement processing. In this case, while the operator arbitrarily changes the range in the depth direction that the operator wishes to observe, the operator can observe the en-face image before and after the image quality improvement process based on the changed range in the depth direction.
(Modification 1)
As described above, in an image subjected to the image quality improvement processing using the learning model, a tissue that does not exist in reality may be visualized, or a tissue that does exist originally may not be visualized. Therefore, since the operator performs image diagnosis based on such an image, misdiagnosis may occur. Accordingly, when the OCTA image or the tomographic image or the like after the image quality improvement processing is displayed on the display unit 270, the display control unit 250 may also display information that is largely: the image in question is an image subjected to image quality improvement processing using a learning model. In this case, the occurrence of misdiagnosis by the operator can be suppressed. Note that the display form may be any form as long as the form is such that it can be understood that the image is a high-quality image obtained using a learning model.
(Modification 2)
In embodiment 1, an example in which the image quality improvement process is applied to an OCTA image or a tomographic image or the like obtained by one round of imaging (inspection) is described. In this regard, the image quality improvement processing using the learning model can also be applied to a plurality of OCTA images or tomographic images obtained by performing imaging (examination) a plurality of times, or the like. In modification 2, a configuration is described with reference to fig. 8A and 8B in which images obtained by applying image quality improvement processing using a learning model to a plurality of OCTA images or tomographic images or the like are simultaneously displayed.
Fig. 8A and 8B each show an example of a time-series report screen for displaying a plurality of OCTA images obtained by imaging the same eye to be inspected a plurality of times with the lapse of time. On the report screen 800 shown in fig. 8A, a plurality of OCTA images 801 before the image quality improvement processing is performed are displayed side by side in chronological order. The report screen 800 further includes a pop-up menu 820, and the operator can select whether to apply the image quality improvement process by operating the pop-up menu 820 via the input unit 260.
If the operator selects to apply the image quality improvement process, the image quality improvement unit 224 applies the image quality improvement process using the learning model to all the OCTA images being displayed. Subsequently, as shown in fig. 8B, the display control unit 250 switches from displaying the plurality of OCTA images 801 being displayed to displaying the plurality of OCTA images 802 after the image quality improvement processing.
Further, if the operator selects not to apply the image quality improvement process on the pop-up menu 820, the display control unit 250 switches from displaying the plurality of OCTA images 802 after the image quality improvement process being displayed to displaying the plurality of OCTA images 801 before the image quality improvement process.
Note that in the present modification, an example has been described in which a plurality of OCTA images obtained before and after the image quality improvement processing using the learning model are simultaneously switched and displayed. However, a plurality of tomographic images or intensity en-face images or the like obtained before and after the image quality improvement processing using the learning model may be simultaneously switched and displayed. Note that the operation method is not limited to the method using the pop-up menu 820, and any operation method may be employed, for example, a method using a button, a drop-down menu, a radio button, or a check box provided on the report screen, or an operation for a keyboard, a mouse wheel, or a touch panel.
Example 2
The learning model outputs output data having a high probability corresponding to the input data according to the learning tendency. In this regard, when the learning model learns using, as training data, an image group having similar image quality trends to each other, an image subjected to more effective image quality improvement can be output for an image having the similar trend in question. Therefore, in embodiment 2, the image quality improvement process is performed more efficiently by performing the image quality improvement process with a plurality of learning models that undergo learning using training data composed of pairs of groups that are grouped for respective imaging conditions such as imaging sites or for respective en-face image generation ranges.
Hereinafter, the OCT apparatus according to the present embodiment is described with reference to fig. 9 and 10. Note that since the constitution of the OCT apparatus according to the present embodiment is the same as that of the OCT apparatus 1 according to embodiment 1 except for the control unit, the same components as those shown in fig. 1 are denoted by the same reference numerals as those in embodiment 1, and description of these components is omitted hereinafter. Hereinafter, the OCT apparatus according to the present embodiment will be described centering on the difference from the OCT apparatus 1 according to embodiment 1.
Fig. 9 shows a schematic configuration of a control unit 900 according to the present embodiment. Note that the components of the control unit 900 according to the present embodiment are the same as those of the control unit 200 according to embodiment 1 except for the image processing unit 920 and the selection unit 925. Accordingly, the same components as those shown in fig. 2 are denoted by the same reference numerals as those in embodiment 1, and description thereof is omitted hereinafter.
In addition to the tomographic image generation unit 221, the motion contrast generation unit 222, the en-face image generation unit 223, and the image quality improvement unit 224, a selection unit 925 is provided in the image processing unit 920 of the control unit 900.
The selection unit 925 selects a learning model to be used by the image quality improvement unit 224 from among a plurality of learning models based on the imaging condition of the image to be subjected to the image quality improvement process by the image quality improvement unit 224 or the en-face image generation range. The image quality improvement unit 224 performs image quality improvement processing on the target OCTA image, tomographic image, or the like using the learning model selected by the selection unit 925, and generates a high-quality OCTA image or a high-quality tomographic image.
Next, a plurality of learning models according to the present embodiment will be described. As described above, the learning model outputs output data having a high probability corresponding to input data according to the learning tendency. In this regard, when the learning model learns using, as training data, an image group having similar image quality trends to each other, an image that has undergone more effective image quality improvement can be output for an image having the similar trend in question. Thus, in the present embodiment, a plurality of learning models are prepared, which undergo learning using training data composed of pairs grouped according to imaging conditions including conditions such as imaging sites, imaging systems, imaging regions, imaging angles of view, scanning densities, and image resolutions, or generation ranges for the respective en-face images.
More specifically, for example, a plurality of learning models, such as a learning model employing an OCTA image in which a macular region is set as an imaging site as training data and a learning model employing an OCTA image in which an optic nerve head is set as an imaging site as training data, are prepared. Note that the macular region and the optic nerve head are each one example of an imaging region, and may include other imaging regions. Further, a learning model for which OCTA images for respective specific imaging regions of imaging regions such as a macular region or an optic nerve head are employed as training data may be prepared.
Further, for example, the visualization of structures such as blood vessels visualized in an OCTA image is significantly different between the case of photographing the retina at a wide angle of view and a low density and the case of photographing the retina at a narrow angle of view and a high density. Thus, a learning model that learns respective sets of training data according to the imaging view angle and the scanning density can be prepared. In addition, examples of the imaging system include an SD-OCT imaging system and an SS-OCT imaging system, and image quality, an imaging range, a penetration depth in a depth direction, and the like are different according to differences between these imaging systems. Therefore, a learning model that learns using training data according to the respective kinds of imaging systems can be prepared.
Furthermore, an OCTA image of blood vessels of all layers of the retina is often generated rarely once, and an OCTA image of blood vessels present only in a predetermined depth range is often generated. For example, for depth ranges such as the superficial, deep and outer layers of the retina and the superficial choroidal layer, an OCTA image is generated that extracts blood vessels within each depth range. On the other hand, the form of the blood vessel visualized in the OCTA image greatly differs depending on the depth range. For example, blood vessels visualized in the superficial layer of the retina form a low-density, thin and clear vascular network, whereas blood vessels visualized in the superficial choroidal layer are visualized at a high density, and it is difficult to clearly distinguish individual blood vessels. Thus, a learning model that learns using the respective training data sets according to the generation range of the en-face image such as the OCTA image can be prepared.
Although an example in which an OCTA image is used as training data is described here, similarly to embodiment 1, in the case of performing image quality improvement processing for a tomographic image or an intensity en-face image or the like, these images may be used as training data. In this case, a plurality of learning models are prepared which learn using respective training data sets according to imaging conditions of these images or the en-face image generation range.
Next, a series of image processing operations according to the present embodiment are described with reference to fig. 10. Fig. 10 is a flowchart showing a series of image processing operations according to the present embodiment. Note that description about the same processing as that in the series of image processing operations according to embodiment 1 is appropriately omitted.
First, in step S1001, similar to step S501 according to embodiment 1, the obtaining unit 210 obtains a plurality of pieces of three-dimensional tomographic information obtained by imaging the eye E a plurality of times. The obtaining unit 210 may obtain tomographic information of the eye E to be inspected using the OCT imaging unit 100, or may obtain tomographic information from the memory 240 or other devices connected to the control unit 200.
The obtaining unit 210 also obtains an imaging condition group related to tomographic information. Specifically, when imaging related to tomographic information is performed, the obtaining unit 210 may obtain imaging conditions such as an imaging location and an imaging system. Note that, according to the data format of the tomographic information, the obtaining unit 210 may obtain the set of imaging conditions stored in the data structure of the data constituting the tomographic information. Further, in the case where the imaging conditions are not stored in the data structure of the tomographic information, the obtaining unit 210 may obtain the imaging information group from a server or a database or the like in which a file describing the imaging conditions is stored. Further, the obtaining unit 210 may estimate the imaging information group from the tomographic information-based image by any known method.
Further, in the case where the obtaining unit 210 obtains a plurality of three-dimensional tomographic images, a plurality of three-dimensional motion contrast data, a plurality of OCTA images, or the like, the obtaining unit 210 obtains a set of imaging conditions related to the obtained images or data. Note that, in the case where the image quality improvement processing is performed using only a plurality of learning models that are learned using respective kinds of training data according to the generation range of the OCTA image or the intensity en-face image, the obtaining unit 210 does not need to obtain the imaging condition set of the tomographic image.
The processing of steps S1002 to S1004 is the same as that of steps S502 to S504 according to embodiment 1, and thus a description of the processing is omitted here. When the en-face image generating unit 223 generates an OCTA image in step S1004, the process advances to step S1005.
In step S1005, the selection unit 925 selects the learning model used by the image quality improvement unit 224 based on the set of imaging conditions or the generation range related to the generated OCTA image and information on training data related to a plurality of learning models. More specifically, for example, in the case where the imaging site in the OCTA image is an optical nerve head, the selection unit 925 selects a learning model that has been learned using the OCTA image of the optical nerve head as training data. Further, for example, in the case where the generation range of the OCTA image is a shallow layer of the retina, the selection unit 925 selects a learning model that has been learned using the OCTA image set as the generation range of the retina as training data.
Note that even if the set of imaging conditions or the generation range related to the generated OCTA image does not completely match with the information related to the training data of the learning model, the selection unit 925 may select the learning model that has been learned using an image having a similar tendency with respect to the image quality as the training data. In this case, for example, the selection unit 925 may include a table describing a correlation between the set of imaging conditions or the generation range related to the OCTA image and the learning model to be used.
In step S1006, the image quality improvement unit 224 performs image quality improvement processing on the OCTA image generated in step S1004 using the learning model selected by the selection unit 925, thereby generating a high-quality OCTA image. The method for generating the high-quality OCTA image is the same as in step S505 according to embodiment 1, and thus a description thereof is omitted here.
Step S1007 is the same as step S506 according to embodiment 1, and thus a description thereof is omitted here. When the high-quality OCTA image is displayed on the display unit 270 in step S1007, the series of image processing operations according to the present embodiment ends.
As described above, the control unit 900 according to the present embodiment includes the selection unit 925, and the selection unit 925 selects a learning model to be used by the image quality improvement unit 224 from a plurality of learning models. The selection unit 925 selects a learning model to be used by the image quality improvement unit 224 based on a range in the depth direction for generating the OCTA image to be subjected to the image quality improvement process.
For example, the selection unit 925 may select the learning model based on the display portion in the OCTA image to be subjected to the image quality improvement processing and the range in the depth direction for generating the OCTA image. Further, for example, the selection unit 925 may select a learning model to be used by the image quality improvement unit 224 based on an imaging portion including a display portion in the OCTA image to be subjected to the image quality improvement processing and a range in the depth direction for generating the OCTA image. In addition, for example, the selection unit 925 may select a learning model to be used by the image quality improvement unit 224 based on the imaging condition of the OCTA image to be subjected to the image quality improvement process.
Therefore, the control unit 900 can more effectively perform the image quality improvement process by performing the image quality improvement process by means of a plurality of learning models that undergo learning using training data composed of pairs of groups that are grouped for respective imaging conditions or for respective en-face image generation ranges.
Note that although an example in which the selection unit 925 selects the learning model based on imaging conditions such as an imaging portion or a generation range related to the OCTA image is described in the present embodiment, a configuration in which the learning model is changed based on conditions other than the above-described conditions may be adopted. For example, the selection unit 925 may select the learning model according to a projection method (maximum intensity projection method or average intensity projection method) at the time of generating the OCTA image or the intensity en-face image or whether an artifact removal process of removing an artifact caused by a blood vessel shadow is performed. In this case, a learning model that has been learned using the respective kinds of training data according to the projection method and whether or not the artifact removal processing has been performed may be prepared.
(Modification 3)
In embodiment 2, the selection unit 925 automatically selects an appropriate learning model according to the imaging conditions, the generation range of the en-face image, or the like. In this regard, there are also cases where an operator wishes to manually select an image quality improvement process to be applied to an image. Thus, the selection unit 925 can select the learning model according to an instruction of the operator.
Further, there are also cases where an operator wishes to change the image quality improvement process applied to an image. Accordingly, the selection unit 925 can change the learning model according to an instruction of the operator to change the image quality improvement process to be applied to the image.
Hereinafter, an operation method when the image quality improvement process to be applied to an image is manually changed is described with reference to fig. 11A and 11B. Fig. 11A and 11B each show an example of a report screen displayed switching between images before and after the image quality improvement processing. On the report screen 1100 shown in fig. 11A, a tomographic image 1111 and an OCTA image 1101 to which the image quality improvement process using the automatically selected learning model has been applied are shown. On the report screen 1100 shown in fig. 11B, a tomographic image 1111 and an OCTA image 1102 to which an image quality improvement process using a learning model according to an instruction of an operator has been applied are shown. Further, on the report screen 1100 shown in fig. 11A and 11B, a process specification section 1120 for changing the image quality improvement process applied to the OCTA image is shown.
The OCTA image 1101 displayed on the report screen 1100 shown in fig. 11A is an OCTA image in which deep capillaries in the macular region are visualized. On the other hand, the image quality improvement processing applied to the OCTA image using the learning model automatically selected by the selection unit 925 is processing suitable for Radial Papillary Capillaries (RPCs). Therefore, with respect to the OCTA image 1101 displayed on the report screen 1100 shown in fig. 11A, the image quality improvement processing that has been applied to the OCTA image is not optimal processing for blood vessels extracted in the OCTA image.
Accordingly, the operator selects "deep capillary" in the process specification section 1120 through the input unit 260. In response to a selection instruction from the operator, the selection unit 925 changes the learning model used by the image quality improvement unit 224 to a learning model that has been learned using an OCTA image related to deep capillaries of the macular region as training data.
The image quality improvement unit 224 performs image quality improvement processing again on the OCTA image using the learning model changed by the selection unit 925. As shown in fig. 11B, the display control unit 250 causes the high-quality OCTA image 1102 newly generated by the image quality improvement unit 224 to be displayed on the display unit 270.
Therefore, by configuring the selection unit 925 to change the learning model in response to an instruction of the operator, the operator can re-specify the appropriate image quality improvement process to be performed for the same OCTA image. Further, the designation of the image quality improvement process may be performed any number of times.
Here, an example has been shown in which the control unit 900 is configured so that the image quality improvement process to be applied to the OCTA image can be manually changed. In this regard, the control unit 900 may also be configured so that the image quality improvement process to be applied to the tomographic image or the intensity en-face image or the like can be manually changed.
Further, although the report screen shown in fig. 11A and 11B is in a form in which display is switched between images before and after the image quality improvement processing, the report screen may be in a form in which images before and after the image quality improvement processing are displayed in a juxtaposed manner or in a superimposed manner. In addition, the form of the process specifying portion 1120 is not limited to that shown in fig. 11A and 11B, and may be any form that allows an instruction concerning image quality improvement processing or learning models to be issued. Further, the kind of image quality improvement processing shown in fig. 11A and 11B is one example, and other kinds of image quality improvement processing according to training data for learning a model may also be included.
Further, similarly to modification 2, a plurality of images to which the image quality improvement process is applied may be displayed at the same time. At this time, a configuration may also be adopted that makes it possible to make a designation regarding the image quality improvement processing to be applied. Examples of the report screen in this case are shown in fig. 12A and 12B.
Fig. 12A and 12B each show an example of a report screen displayed being switched between a plurality of images before and after the image quality improvement processing. On the report screen 1200 shown in fig. 12A, an OCTA image 1201 before the image quality improvement processing is shown. On the report screen 1200 shown in fig. 12B, an OCTA image 1202 to which the image quality improvement process has been applied according to an instruction of the operator is shown. Further, on the report screen 1200 shown in fig. 12A and 12B, a process specification section 1220 for changing the image quality improvement process applied to the OCTA image is shown.
In this case, the selection unit 925 selects a learning model according to the image quality improvement process for which the use process specification portion 1220 issues an instruction, as the learning model to be used by the image quality improvement unit 224. The image quality improvement unit 224 performs image quality improvement processing on the plurality of OCTA images 1201 using the learning model selected by the selection unit 925. The display control unit 250 causes the generated plurality of OCTA images 1202 having high image quality to be simultaneously displayed on the report screen 1200 as shown in fig. 12B.
Note that although the image quality improvement process for the OCTA image is described above, with respect to the image quality improvement process for the tomographic image or the intensity en-face image or the like, the learning model may also be selected and changed according to an instruction of the operator. Note that a plurality of images before and after the image quality improvement processing may also be displayed on the report screen in a juxtaposed manner or in a superimposed manner. In this case, a plurality of images to which the image quality improvement process is applied according to an instruction of the operator may also be displayed at the same time.
Example 3
In embodiments 1 and 2, the image quality improvement unit 224 automatically performs the image quality improvement process after the tomographic image or the OCTA image is imaged. However, it may sometimes take a long time to perform the image quality improvement processing using the learning model performed by the image quality improvement unit 224. In addition, it takes time to generate the motion contrast data by the motion contrast generation unit 222 and to generate the OCTA image by the en-face image generation unit 223. Therefore, in the case of displaying an image after waiting for the image quality improvement process to be completed after image capturing, it may take a long time to display an image after image capturing.
In this regard, when an eye to be inspected is imaged using the OCT apparatus, in some cases, imaging cannot be successfully performed due to blinking or unintentional movement of the eye to be inspected, or the like. Therefore, by allowing the operator to confirm whether or not photographing is successful in an early stage, the convenience of the OCT apparatus can be enhanced. Therefore, in embodiment 3, the OCT apparatus is configured such that before generating and displaying a high-quality OCTA image, the imaged image can be confirmed at an early stage by displaying an intensity en-face image or an OCTA image based on tomographic information obtained by imaging the eye to be examined.
Hereinafter, an OCT apparatus according to the present embodiment is described with reference to fig. 13. Note that, since the constitution of the OCT apparatus according to the present embodiment is similar to that of the OCT apparatus 1 according to embodiment 1, the same components as those shown in fig. 1 are denoted by the same reference numerals as those in embodiment 1, and a description thereof is omitted hereinafter. Hereinafter, the OCT apparatus according to the present embodiment will be described centering on the difference from the OCT apparatus 1 according to embodiment 1.
Fig. 13 is a flowchart of a series of image processing operations according to the present embodiment. First, in step S1301, the obtaining unit 210 obtains a plurality of pieces of three-dimensional tomographic information obtained by imaging the eye E to be inspected by the OCT imaging apparatus 100.
Since step S1302 is the same as step S502 according to embodiment 1, a description thereof is omitted here. When a three-dimensional tomographic image is generated in step S1302, the process advances to step S1303.
In step S1303, the en-face image generation unit 223 generates a front image (intensity en-face image) of the fundus by projecting the three-dimensional tomographic image generated in step S1302 on a two-dimensional plane. Thereafter, in step S1304, the display control unit 250 displays the generated intensity en-face image on the display unit 270.
Since step S1305 and step S1306 are the same as steps S503 and S504 according to embodiment 1, a description of these steps is omitted here. When the OCTA image is generated in step S1306, the process advances to step S1307. In step S1307, the display control unit 250 causes the display unit 270 to switch from displaying the intensity en-face image to displaying the OCTA image before the image quality improvement process generated in step S1306.
In step S1308, similar to step S505 according to embodiment 1, the image quality improvement unit 224 subjects the OCTA image generated in step S1306 to image quality improvement processing using a learning model, thereby generating a high-quality OCTA image. In step S1309, the display control unit 250 causes the display unit 270 to switch from displaying the OCTA image before the image quality improvement process to displaying the generated high-quality OCTA image.
As described above, before the obtaining unit 210 obtains the OCTA image, the display control unit 250 according to the present embodiment causes the display unit 270 to display an intensity en-face image (third image), which is a front image generated based on tomographic data obtained in the depth direction of the eye to be inspected. Further, the display control unit 250 immediately after obtaining the OCTA image causes the display unit 270 to switch from displaying the intensity en-face image to displaying the OCTA image. In addition, after the image quality improvement unit 224 generates the high-quality OCTA image, the display control unit 250 causes the display unit 270 to switch from displaying the OCTA image to displaying the high-quality OCTA image.
Therefore, the operator can check the front image of the eye to be inspected immediately after image capturing, and can immediately determine whether image capturing is successful. Further, since the OCTA image is displayed immediately after the OCTA image is generated, the operator can determine whether or not pieces of three-dimensional tomographic information for generating the motion contrast data have been properly obtained in an early stage.
Note that, regarding the tomographic image or the intensity en-face image or the like, by displaying the tomographic image or the intensity en-face image before performing the image quality improvement processing, the operator can determine whether or not the image capturing is successful at an early stage.
Although in the present embodiment, the process for generating motion contrast data (step S1305) starts after the process for displaying an intensity en-face image (step S1304), the timing of starting the process for generating motion contrast data is not limited thereto. For example, the motion contrast generation unit 222 may start the process for generating the motion contrast data simultaneously with the process for generating the intensity en-face image (step S1303) or the process for displaying the intensity en-face image (step S1304). Similarly, the image quality improvement unit 224 may start the image quality improvement process (step S1308) simultaneously with the process for displaying the OCTA image (step S1307).
Example 4
In embodiment 1, an example of switching the OCTA image before and after the display image quality improvement processing is described. In contrast, in embodiment 4, a comparison is made between images before and after the image quality improvement processing.
Hereinafter, the OCT apparatus according to the present embodiment is described with reference to fig. 14 and 15. Note that since the constitution of the OCT apparatus according to the present embodiment is the same as that of the OCT apparatus 1 according to embodiment 1 except for the control unit, the same components as those shown in fig. 1 are denoted by the same reference numerals as those in embodiment 1, and description of these components is omitted hereinafter. Hereinafter, the OCT apparatus according to the present embodiment will be described centering on the difference from the OCT apparatus 1 according to embodiment 1.
Fig. 14 is a diagram showing a schematic configuration of the control unit 1400 according to the present embodiment. Note that the components of the control unit 1400 according to the present embodiment are the same as those of the control unit 200 according to embodiment 1, except for the image processing unit 1420 and the comparing unit 1426. Accordingly, the same components as those shown in fig. 2 are denoted by the same reference numerals as those in embodiment 1, and description thereof is omitted hereinafter.
In addition to the tomographic image generation unit 221, the motion contrast generation unit 222, the en-face image generation unit 223, and the image quality improvement unit 224, a comparison unit 1426 is also provided in the image processing unit 1420 of the control unit 1400.
The comparing unit 1426 compares an image (original image) before the image quality improvement unit 224 performs the image quality improvement process with an image after the image quality improvement process. More specifically, the comparison unit 1426 compares images before and after the image quality improvement processing, and calculates differences between pixel values at corresponding pixel positions in the images before and after the image quality improvement processing.
Then, the comparing unit 1426 generates a color chart image having a color according to the magnitude of the difference. For example, in the case where the pixel value of the image after the image quality improvement processing is larger than the pixel value of the image before the image quality improvement processing, a warm (yellow to orange to red) tone is used, and in the case where the pixel value of the image after the image quality improvement processing is smaller, a cool (yellow to green to blue) tone is used. By using such a color scheme, it can be easily recognized that the position indicated by the warm color on the color map image is the tissue restored (or newly created) by the image quality improvement process. Similarly, it can be easily recognized that the position indicated by the cold color on the color map image is noise (or tissue that has been erased) that has been removed by the image quality improvement process.
Note that the color scheme of the color map image in question is one example. For example, the color scheme of the color map image may be arbitrarily set according to a desired configuration, for example, a color scheme of a tone different according to the magnitude of the pixel value in the image after the image quality improvement processing relative to the pixel value in the image before the image quality improvement processing is applied.
The display control unit 250 may superimpose the color map image generated by the comparing unit 1426 on the image before the image quality improvement process or the image after the image quality improvement process, and display the resultant superimposed image on the display unit 270.
Next, a series of image processing operations according to the present embodiment are described with reference to fig. 15. Note that since steps S1501 to S1505 are the same as steps S501 to S505 according to embodiment 1, a description of these steps is omitted here. When a high-quality OCTA image is generated by the image quality improving unit 224 in step S1505, the process advances to step S1506.
In step S1506, the comparison unit 1426 compares the OCTA image generated in step S1504 with the high-quality OCTA image generated in step S1505 to calculate the difference between the pixel values, and generates a color map image based on the difference between the pixel values. Note that, instead of the difference between the pixel values in the images before and after the image quality improvement processing, the comparison unit 1426 may perform comparison between the images using other methods such as by using the ratio of the pixel values or the correlation value between the images before and after the high image quality processing, and may generate a color map image based on the comparison result.
In step S1507, the display control unit 250 superimposes the color map image on the image before the image quality improvement process or on the image after the image quality improvement process, and displays the resultant superimposed image on the display unit 270. At this time, the display control unit 250 may set transparency for the color chart to ensure that the color chart image does not obscure the image on which the color chart image is superimposed, and cause the color chart image to be displayed on the target image in a superimposed manner.
Further, in the color map image, the display control unit 250 may set the transparency to a high value (the pixel value of the color map image is low) at a position where the difference between the images before and after the image quality improvement process is small, or may set the transparency so that a position where the difference is less than or equal to a predetermined value is completely transparent. By setting the transparency in this way, both the image displayed below the color map image and the color map image can be visually recognized in an advantageous manner. Note that, regarding the transparency of the color map image, the comparison unit 1426 may also generate a color map image including a transparency setting.
As described above, the control unit 1400 according to the present embodiment includes the comparing unit 1426, and the comparing unit 1426 compares the first image and the second image subjected to the image quality improvement processing. The comparing unit 1426 calculates a difference between the first image and the second image, and generates a colored color map image based on the difference. The display control unit 250 controls the display of the display unit 270 based on the comparison result obtained by the comparison unit 1426. More specifically, the display control unit 250 superimposes the color map image on the first image or the second image, and displays the resultant superimposed image on the display unit 270.
Therefore, by observing the color image superimposed on the images before and after the image quality improvement processing, it is possible to easily confirm the variation between the images caused by the image quality improvement processing. Therefore, even if a tissue that does not actually exist in an image is visualized by the image quality improvement process, or an originally existing tissue is erased from the image by the image quality improvement process, an operator can recognize such a tissue more easily, and the authenticity of the tissue can be determined more easily. Further, according to the color scheme of the color map image, the operator can easily recognize whether the position is a position newly visualized by the image quality improvement process or a position erased by the image quality improvement process.
Note that the display control unit 250 may enable or disable the superimposed display of the color chart images according to an instruction from the operator. An operation for turning on or off the superimposed display of the color map images may be simultaneously applied to a plurality of images displayed on the display unit 270. In this case, the comparison unit 1426 may generate a color map image for each of the corresponding images before and after the image quality improvement processing, and the display control unit 250 may superimpose and display the color map image on the corresponding image before the image quality improvement processing or the image after the image quality improvement processing. Further, the display control unit 250 may cause an image before the image quality improvement process or an image after the image quality improvement process to be displayed on the display unit 270 before the color map image is displayed.
Note that although the present embodiment is described taking an OCTA image as an example, in the case of performing image quality improvement processing on a tomographic image or an intensity en-face image or the like, similar processing may be performed. Further, the comparison processing according to the present embodiment and the processing for displaying a color chart can also be applied to the OCT apparatuses according to embodiment 2 and embodiment 3.
(Modification 4)
Further, the comparing unit 1426 may compare images before and after the image quality improvement process, and the display control unit 250 may display a warning on the display unit 270 according to the comparison result of the comparing unit 1426. Specifically, in the case where the difference between the pixel values in the images before and after the image quality improvement processing calculated by the comparing unit 1426 is greater than a predetermined value, the display control unit 250 displays a warning on the display unit 270. According to such a configuration, in a case where a tissue that does not exist in reality is generated by a learning model in the generated high-quality image or an originally existing tissue is erased, the operator can be attracted to notice the fact. Note that the comparison between the difference and the predetermined value may be performed by the comparison unit 1426, or may be performed by the display control unit 250. Further, instead of the differences, statistical values such as average values of the differences may be compared with predetermined values.
In addition, the display control unit 250 may be configured such that, in the case where the difference between the images before and after the image quality improvement processing is greater than a predetermined value, the image after the image quality improvement processing is performed is not displayed on the display unit 270. In this case, in the generated high-quality image, if a tissue that does not exist in reality is generated or an originally existing tissue is erased by a learning model, misdiagnosis based on the high-quality image in question can be suppressed. Note that the comparison between the difference and the predetermined value may be performed by the comparison unit 1426, or may be performed by the display control unit 250. Further, instead of the difference, a statistical value such as an average value of the difference may be compared with a predetermined value.
Example 5
Next, an image processing apparatus (control unit 200) according to embodiment 5 is described with reference to fig. 20A and 20B. In the present embodiment, an example is described in which the display control unit 250 displays the processing result of the image quality improvement unit 224 on the display unit 270. Note that although the present embodiment is described using fig. 20A and 20B, the display screen is not limited to the example shown in fig. 20A and 20B. As in the case of the subsequent observation, the image quality improvement processing can be similarly applied to display screens in which a plurality of images obtained at different dates and times are displayed side by side. Further, as in the case of the image capturing confirmation screen, the image quality improvement process may be similarly applied to a display screen in which the inspector confirms whether image capturing is successful immediately after image capturing. The display control unit 250 may cause the display unit 270 to display the plurality of high-quality images generated by the image quality improvement unit 224 or the low-quality image with no image quality improvement. Accordingly, the display control unit 250 can output a low-quality image and a high-quality image, respectively, according to an instruction of the inspector.
Hereinafter, one example of the interface 3400 discussed is described with reference to fig. 20A and 20B. Reference numeral 3400 denotes the entire screen, reference numeral 3401 denotes a "patient" label, reference numeral 3402 denotes a "camera" label, reference numeral 3403 denotes a "report" label, and reference numeral 3404 denotes a "set" label. Further, the diagonal line in the "report" tab 3403 indicates the activation state of the report screen. In this embodiment, an example of displaying a report screen will be described. Reference numeral Im3405 denotes an SLO image, reference numeral Im3406 denotes an image in which an OCTA en-face image denoted by reference numeral Im3407 is displayed in a superimposed manner on the SLO image Im 3405. Here, the term "SLO image" refers to a front image of the fundus obtained by an SLO (scanning laser ophthalmoscope) optical system (not shown). Reference numerals Im3407 and Im3408 each denote an OCTA en-face image, reference numeral Im3409 denotes an intensity en-face image, and reference numerals Im3411 and Im3412 each denote a tomographic image. Reference numerals 3413 and 3414 denote boundary lines of upper and lower ranges of the OCTA en-face images denoted by Im3407 and Im3408, respectively, which are displayed in a superimposed manner on the corresponding tomographic images. The button 3420 is a button for designating execution of the image quality improvement process. Naturally, as described later, the button 3420 may be a button for inputting an instruction to display a high-quality image.
In the present embodiment, execution of the image quality improvement processing is performed when the button 3420 is designated, or whether to execute the image quality improvement processing is determined based on information stored (saved) in the database. First, an example of switching between display of a high-quality image and display of a low-quality image by designating a button 3420 according to an instruction from an inspector will be described. Note that the OCTA en-face image is described as a target image of the image quality improvement process.
When the inspector operates to designate a "report" tab 3403 to transition to a report screen, low-quality OCTA en-face images Im3407 and Im3408 are displayed. Thereafter, when the inspector performs an operation of the designation button 3420, the image quality improvement unit 224 performs image quality improvement processing on the images Im3407 and Im3408 displayed on the screen. After the image quality improvement process is completed, the display control unit 250 displays the high-quality image generated by the image quality improvement unit 224 on the report screen. Note that since the image denoted by reference numeral Im3406 is an image obtained by displaying the image Im3407 on the SLO image Im3405 in a superimposed manner, the image Im3406 is also an image subjected to image quality improvement processing. Then, the display of the button 3420 is changed to the activated state to provide a display that can be understood as performing the image quality improvement processing.
In this case, execution of the processing by the image quality improvement unit 224 is not necessarily limited to the timing at which the inspector performs the operation of the specification button 3420. Since the kinds of the OCTA en-face images Im3407 and Im3408 to be displayed when the report screen is opened are known in advance, the image quality improvement process can be performed when transitioning to the report screen. Subsequently, at the timing of pressing the button 3420, the display control unit 250 may display a high-quality image on the report screen. In addition, the number of kinds of images subjected to the image quality improvement processing in response to an instruction from an inspector or when transitioning to a report screen does not have to be two. Such a configuration may also be adopted to process an image that is highly likely to be being displayed, for example, a plurality of OCTA en-face images such as the surface layer (Im 2910), the deep layer (Im 2920), the outer layer (Im 2930), and the choroidal vascular network (Im 2940) shown in fig. 19A and 19B may be processed. In this case, the image obtained by performing the image quality improvement process may be temporarily stored in the memory or may be stored in the database.
Next, a case where the image quality improvement processing is performed based on the information stored (held) in the database will be described. In the case where the state in which the execution of the image quality improvement processing is to be performed is stored in the database, a high-quality image obtained by executing the image quality improvement processing is displayed by default when the display transitions to the report screen. Further, a configuration may be adopted in which: the default button 3420 is caused to be displayed in an activated state so that the inspector can know that a high-quality image obtained by performing the image quality improvement process is being displayed. If the inspector wishes to display a low-quality image in a state before the image quality improvement process, the inspector can display the low-quality image by performing an operation of the designation button 3420 to thereby release the activated state. If the inspector wishes to return to a high quality image, the inspector designates a button 3420.
It is assumed that whether or not to perform the image quality improvement processing on the data stored in the database can be commonly specified for all the data stored in the database, and for respective categories of data of respective groups (for respective examinations) such as image capturing data. For example, in the case where a state in which the image quality improvement process is to be performed on the entire database has been stored, if the inspector stores a state in which the image quality improvement process is not performed for a single item (single inspection) of the image pickup data, the image pickup data will be displayed in a state in which the image quality improvement process has not been performed on the image pickup data the next time the relevant image pickup data is displayed. A user interface (not shown) (e.g., a "store" button) may be used to store a state in which image quality improvement processing has been performed for each item of image capturing data (for each inspection). In addition, upon transition to other imaging data (other examination) or other patient data (for example, change to a display screen other than the report screen in accordance with an instruction from the examiner), a state in which execution of the image quality improvement process is to be performed may be stored based on the display state (for example, the state of the button 3420). In this way, in the case where whether or not the image quality improvement processing has been performed in the unit of image capturing data (inspection unit) has not been specified, the processing can be performed based on the information specified for the entire database, whereas in the case where the image quality improvement processing has been specified in the unit of image capturing data (inspection unit), the processing can be performed separately based on the information in question.
Although an example in which the images Im3407 and Im3408 are displayed as the OCTA en-face image is shown in the present embodiment, the OCTA en-face image to be displayed may be changed according to the designation of the inspector. Therefore, description will now be made regarding changing an image when execution of the image quality improvement process (button 3420 is in an activated state) has been designated.
The change of the image is performed using a user interface (not shown) (e.g., combo box). For example, when the examiner changes the kind of image from the surface layer image to the choroidal-blood-vessel network image, the image quality improvement unit 224 performs image quality improvement processing on the choroidal-blood-vessel network image, and the display control unit 250 displays the high-quality image generated by the image quality improvement unit 224 on the report screen. In other words, in response to an instruction from the inspector, the display control unit 250 may change the display of the high-quality image of the first depth range to the display of the high-quality image of the second depth range at least partially different from the first depth range. At this time, the display control unit 250 may change the display of the high-quality image of the first depth range to the display of the high-quality image of the second depth range by changing the first depth range to the second depth range in response to an instruction from the inspector. Note that, in the case where a high-quality image has been generated for an image having a high possibility of being displayed at the time of transition to the report screen as described above, the display control unit 250 may display the high-quality image that has been generated.
Note that the method for changing the kind of image is not limited to the above method, and an OCTA en-face image in which different depth ranges are set may also be generated by changing the layer and offset value used as references. In this case, when the layer or the offset value used as a reference is changed, the image quality improvement unit 224 performs image quality improvement processing for any of the OCTA en-face images, and the display control unit 250 displays a high quality image on the report screen. A user interface (not shown) (e.g., combo box or text box) may be used to make changes to the layer or offset value used as a reference. Further, the range for generating the OCTA en-face image may be changed by dragging any one of the boundary lines 3413 and 3414 (moving layer boundary) displayed in a superimposed manner on the tomographic images Im3411 and Im 3412.
In the case of changing the boundary line by dragging, an execution command for the image quality improvement process is continuously issued. Accordingly, the image quality improving unit 224 may always perform processing for the execution command, or may be configured to perform processing after changing the layer boundary by dragging. Or although execution of the image quality improvement process is continuously issued, the image quality improvement unit 224 may be configured to cancel the previous command at the point in time when the next command arrives, and execute the latest command.
Note that in some cases, the image quality improvement processing takes a relatively long time. Therefore, even when the command is executed at any of the above timings, it may take a relatively long time until a high-quality image is displayed. Accordingly, during a period from when a depth range for generating an OCTA en-face image is set in response to an instruction from an inspector to when a high-quality image is displayed, an OCTA en-face image (low-quality image) corresponding to the set depth range can be displayed. In other words, the following configuration may be adopted: when the above depth range is set, an OCTA en-face image (low quality image) corresponding to the set depth range is displayed, and when the image quality improvement process is completed, the display of the relevant OCTA en-face image (low quality image) is changed to the display of a high quality image. Further, information indicating that the image quality improvement process is being performed may be displayed during a period from setting the above depth range to displaying a high-quality image. Note that the foregoing can be applied not only to a case where the assumed state is a state in which the execution of the image quality improvement processing has been designated (the button 3420 is in the activated state), but also to a period from when the execution of the image quality improvement processing is instructed according to an instruction of an inspector until a high-quality image is displayed, for example.
Although the following examples are shown in the present embodiment: different layers are displayed as images Im3407 and Im3408 as an OCTA en-face image, and a low quality image and a high quality image are displayed by switching between them, but the present invention is not limited thereto. For example, a low quality OCTA en-face image as image Im3407 and a high quality OCTA en-face image as image Im3408 may be displayed side by side. In the case of displaying images by switching between them, since the images are switched in the same place, it is easy to compare portions having variations, whereas in the case of displaying images side by side, since the images can be displayed at the same time, it is easy to compare the entire images.
Next, fig. 20A and 20B will be used to describe execution of the image quality improvement processing in the case of a screen transition. Fig. 20B is an example of a screen on which the OCTA en-face image Im3407 shown in fig. 20A is displayed in an enlarged manner. Also in fig. 20B, similar to fig. 20A, a button 3420 is displayed. For example, a screen transition from the screen shown in fig. 20A to the screen shown in fig. 20B is designated by double-clicking the OCTA en-face image Im3407, and a screen transition from the screen shown in fig. 20B to the screen shown in fig. 20A is designated by clicking the "off" button 3430. Note that, regarding screen transition, a method for transitioning from one screen to another is not limited to the method described herein, and a user interface (not shown) may also be used.
In the case where execution of the image quality improvement process has been designated at the time of screen transition (button 3420 is in the activated state), this state is also maintained at the time of screen transition. In other words, in the case where the screen shown in fig. 20B is shifted to a state in which a high-quality image is displayed on the screen shown in fig. 20A, the high-quality image is also displayed on the screen shown in fig. 20B. In addition, the button 3420 is placed in an activated state. The same applies to the case of transition from the screen shown in fig. 20B to the screen shown in fig. 20A. On the screen shown in fig. 20B, the display can also be switched to a low-quality image by the designation button 3420.
Regarding the screen transition, the screen transition is not limited to the screen described herein, and may be performed while maintaining the display state of a high-quality image as long as it is transitioned to a screen displaying the same image pickup data, for example, a display screen for subsequent observation or a display screen for a panoramic image. In other words, on the display screen after the transition, an image corresponding to the state of the button 3420 on the display screen before the transition is displayed. For example, if the button 3420 is in an activated state on the display screen before the transition, a high-quality image is displayed on the display screen after the transition. Further, for example, if the activated state of the button 3420 is released on the display screen before the transition, a low-quality image is displayed on the display screen after the transition. Note that the following configuration may be adopted: if the button 3420 is in an activated state on the display screen for subsequent observation, a plurality of images obtained at different dates and times (different inspection dates) displayed side by side on the display screen for subsequent observation are switched to high-quality images. In other words, the following configuration may be adopted: if the button 3420 is in an activated state on the display screen for subsequent observation, switching to a high-quality image is commonly performed for a plurality of images obtained at different dates and times.
An example of a display screen for subsequent observation is shown in fig. 18. When the tab 3801 is selected in response to an instruction from the inspector, a display screen for subsequent observation as shown in fig. 18 is displayed. At this time, the depth range of the en-face image may be changed by the inspector performing an operation of selecting from the predefined depth range sets (3802 and 3803) displayed in the list box. For example, surface capillaries are selected in list box 3802, and deep capillaries are selected in list box 3803. The analysis results of the en-face image of the surface capillaries are shown in the upper display area, and the analysis results of the en-face image of the deep capillaries are shown in the lower display area. In other words, when a depth range is selected, a plurality of images obtained at different dates and times are collectively changed to display in parallel the analysis results of a plurality of en-face images within the selected depth range.
At this time, if the display of the analysis result is placed in the unselected state, the display may be changed collectively to display the analysis results of a plurality of en-face images obtained at different dates and times in parallel. Further, if the button 3420 is designated according to an instruction from the inspector, the display of the plurality of en-face images is collectively changed to the display of a plurality of high-quality images.
In addition, in the case where the display of the analysis result is in the selected state, if the button 3420 is designated according to an instruction from the inspector, the display of the analysis result of the plurality of en-face images is collectively changed to the display of the analysis result of the plurality of high-quality images. Here, the display of the analysis results may be performed such that the analysis results are displayed in any transparency on the image in a superimposed manner. At this time, the display changed to the analysis result may be, for example, a state in which the analysis result is superimposed on the image being displayed with any transparency. Further, the display changed to the analysis result may be, for example, a display changed to each analysis result and image (for example, two-dimensional map) obtained by subjecting each analysis result and image to fusion processing with any transparency.
Further, such layer boundaries and offset locations for specifying depth ranges may each be commonly changed from a user interface such as that identified by reference numerals 3805 and 3806. Note that the depth ranges of a plurality of en-face images obtained at different dates and times may be collectively changed by causing a tomographic image to be displayed together therewith and moving layer boundary data superimposed on the tomographic image in accordance with an instruction from an inspector. At this time, a plurality of tomographic images obtained at different dates and times may be displayed side by side, and when the above-described movement is performed on one tomographic image, the layer boundary data may be similarly moved on the other tomographic images.
For example, the image projection method may be changed by selection from a user interface such as a context menu, and whether projection artifact removal processing is to be performed.
Further, the selection button 3807 may be selected to display a selection screen, and an image selected from a list of images displayed on the selection screen may be displayed. Note that an arrow 3804 displayed at the upper part of the screen shown in fig. 18 is a mark indicating a currently selected check, and a reference check (baseline) is a check selected at the time of subsequent imaging (leftmost image of fig. 18). Naturally, a mark indicating the reference check may be displayed on the display unit.
Further, in the case where the "show difference" check box 3808 is designated, a measured value distribution (map or sector map) with respect to the reference image is displayed on the reference image. In addition, in this case, in the region corresponding to the inspection date other than the inspection date of the reference image, a differential measurement value map showing the difference between the measurement value distribution calculated for the reference image and the measurement distribution value calculated for the image displaying the relevant region is displayed. As a measurement result, a trend graph (a graph of measured values of images on each inspection date obtained by measuring a change with time) may be displayed on the report screen. In other words, time-series data (e.g., time-series diagrams) of a plurality of analysis results corresponding to a plurality of images obtained at different dates and times may be displayed. At this time, as for the analysis results regarding the dates and times other than the plurality of dates and times corresponding to the plurality of images being displayed, the analysis results may be displayed as time-series data in a state in which the analysis results can be distinguished from the plurality of analysis results corresponding to the plurality of images being displayed (for example, the colors of the respective points on the time chart differ depending on whether or not the corresponding images are displayed). In addition, regression lines (curves) of the trend graph and corresponding mathematical formulas may be displayed on the report screen.
Although a description has been given in the present embodiment with respect to the OCTA en-face image, the present invention is not limited thereto. The image related to the processing for displaying an image, improving image quality, analyzing an image, and the like according to the present embodiment may be an intensity en-face image. In addition, the kind of image is not limited to the en-face image, and may be a different kind of image such as a tomographic image, an SLO image, a fundus image, or a fluorescent fundus image. In this case, the user interface for performing the image quality improvement process may be a user interface for instructing to perform the image quality improvement process for a plurality of images of different kinds, or may be a user interface for selecting any image from a plurality of images of different kinds and instructing to perform the image quality improvement process.
According to the foregoing configuration, the display control unit 250 can display the image processed by the image quality improvement unit 224 according to the present embodiment on the display unit 270. At this time, as described above, in a state where at least one condition is selected among a plurality of conditions concerning the display of a high-quality image, the display of an analysis result, the depth range of a front image to be displayed, and the like, even if the display screen is changed to another display screen, the selected state can be maintained.
Further, as described above, in the case where the state of at least one condition among the plurality of conditions is selected, even if the other conditions are changed to the selected state, the state of at least one condition can be maintained. For example, in a case where the display of the analysis result is in a selected state, in response to an instruction from the inspector (for example, when the button 3420 is specified), the display control unit 250 may change the display of the analysis result for the low-quality image to the display of the analysis result for the high-quality image. Further, in a case where the display of the analysis result is in the selected state, in response to an instruction from the inspector (for example, when the designation of the button 3420 is released), the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image.
Also, in a case where the display of the high-quality image is in an unselected state, in response to an instruction from the inspector (for example, when the specification of the display of the analysis result is released), the display control unit 250 may change the display of the analysis result of the low-quality image to the display of the low-quality image. In addition, in a case where the display of the high-quality image is in an unselected state, in response to an instruction from the inspector (for example, when the display of the analysis result is specified), the display control unit 250 may change the display of the low-quality image to the display of the analysis result of the low-quality image. Further, in a case where the display of the high-quality image is in the selected state, in response to an instruction from the inspector (for example, when the specification of the display of the analysis result is released), the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the high-quality image. Further, in a case where the display of the high-quality image is in the selected state, in response to an instruction from the inspector (for example, when the display of the analysis result is specified), the display control unit 250 may change the display of the high-quality image to the display of the analysis result of the high-quality image.
Further, let us consider a case where the display of the high-quality image is in an unselected state and the display of the first analysis result is in a selected state. In this case, in response to an instruction from the inspector (for example, when the display of the second analysis result is specified), the display control unit 250 may change the display of the first analysis result of the low-quality image to the display of the second analysis result of the low-quality image. Further, let us consider a case where the display of the high-quality image is in a selected state and the display of the first analysis result is in a selected state. In this case, in response to an instruction from the inspector (for example, when the display of the second analysis result is specified), the display control unit 250 may change the display of the first analysis result of the high-quality image to the display of the second analysis result of the high-quality image.
Note that the following configuration may be adopted: on the display screen for subsequent observation, as described above, changes in these displays are collectively reflected for a plurality of images obtained at different dates and times. Here, the display of the analysis results may be performed such that the analysis results are displayed in any transparency on the image in a superimposed manner. At this time, the display changed to the analysis result may be, for example, a state in which the analysis result is superimposed on the image being displayed with any transparency. Further, the display changed to the analysis result may be, for example, a display changed to each analysis result and image (for example, two-dimensional map) obtained by subjecting each analysis result and image to fusion processing with any transparency.
(Modification 5)
In the above-described various embodiments and modifications, the display control unit 224 may cause the display unit 270 to display an image selected from among the high-quality image and the input image generated by the image quality improvement unit 404 according to an instruction from the inspector. Further, in response to an instruction from the inspector, the display control unit 250 may switch the image displayed on the display unit 270 from a captured image (input image) to a high-quality image. In other words, the display control unit 250 may change the display of the low-quality image to the display of the high-quality image in response to an instruction from the inspector. Further, the display control unit 250 may change the display of the high-quality image to the display of the low-quality image in response to an instruction from the inspector.
In addition, the image quality improvement unit 224 may start image quality improvement processing (inputting an image into the image quality improvement engine) by the image quality improvement engine (learning model for improving image quality) in response to an instruction from the inspector, and the display control unit 224 may cause the display unit 270 to display the high-quality image generated by the image quality improvement unit 224. In contrast, when the image pickup apparatus (OCT image pickup unit 100) picks up an input image, the image quality improvement engine may automatically generate a high-quality image based on the input image, and the display control unit 405 may cause the display unit 270 to display the high-quality image in response to an instruction from the inspector. Here, the term "image quality improvement engine" includes a learning model that performs the above-described image quality improvement processing.
Note that these processing operations may also be similarly performed with respect to the output of the analysis result. In other words, the display control unit 250 may change the display of the analysis result of the low-quality image to the display of the analysis result of the high-quality image in response to an instruction from the inspector. Further, the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the analysis result of the low-quality image in response to an instruction from the inspector. Naturally, in response to an instruction from the inspector, the display control unit 250 may change the display of the analysis result of the low-quality image to the display of the low-quality image. Further, in response to an instruction from the inspector, the display control unit 250 may change the display of the low-quality image to the display of the analysis result of the low-quality image. Also, the display control unit 250 may change the display of the analysis result of the high-quality image to the display of the high-quality image in response to an instruction from the inspector. Further, in response to an instruction from the inspector, the display control unit 250 may change the display of the high-quality image to the display of the analysis result of the high-quality image.
In addition, the display control unit 250 may change the display of the analysis result of the low-quality image to the display of a different kind of analysis result of the low-quality image in response to an instruction from the inspector. Further, the display control unit 250 may change the display of the analysis result of the high-quality image to the display of a different kind of analysis result of the high-quality image in response to an instruction from the inspector.
In this case, the display of the analysis result of the high-quality image may be performed such that the analysis result of the high-quality image is displayed in any transparency in a superimposed manner on the high-quality image. Further, the display of the analysis results of the low-quality images may be performed such that the analysis results of the low-quality images are displayed in a superimposed manner on the low-quality images with any transparency. At this time, the change to display of the analysis result may be, for example, a state in which the analysis result is superimposed on the image being displayed with any transparency. Further, the display changed to the analysis result may be, for example, a display changed to an image (for example, a two-dimensional map) obtained by subjecting the analysis result and the image to fusion processing with any transparency.
(Modification 6)
Analysis results such as the thickness of a desired layer or various blood vessel densities may be displayed on the report screen described in the above-described various embodiments and modifications. Further, as the analysis result, parameter values (distributions) related to a site of interest including at least one of an optical nerve head, a macular region, a vascular region, a nerve bundle, a vitreous region, a macular region, a choroidal region, a sclera region, a lamina cribosa region, a retinal layer boundary edge, photoreceptor cells, blood cells, a vascular wall, a vascular inner wall boundary, a vascular outer boundary, ganglion cells, a cornea region, an angular region, schlemm's canal, and the like may be displayed. At this time, for example, an accurate analysis result may be displayed by analyzing the medical image subjected to various artifact removal processes. Note that the artifact may be, for example, a false image region caused by light absorption of a blood vessel region or the like, a projection artifact, or a banding artifact in a front image that appears in the main scanning direction of measurement light due to the state of the eye to be inspected (movement or blinking or the like). Further, the artifact may be of any kind as long as it is, for example, an imaging failure area that randomly occurs on a medical image of a predetermined portion of the subject at each imaging. Further, a value (distribution) of a parameter related to an area (imaging failure area) including at least one of the types of the above-described artifacts may be displayed as an analysis result. Further, a value (distribution) of a parameter related to a region including at least one abnormal site such as drusen, a neovascular site, white blood cells (hard exudates), a pseudo acne, or the like may be displayed as an analysis result.
The analysis result may be displayed using an analysis chart or using a sector indicating a statistical value corresponding to each divided area or the like. Note that the analysis result may be generated using a learning model (analysis result generation engine, or a learning model for generating analysis results) obtained by learning the analysis result of the medical image as the training data. At this time, the learning model may be a model obtained by learning using the following training data: training data including medical images and analysis results of the medical images, or training data including medical images and analysis results of medical images different in kind from related medical images, and the like. Further, the learning model may be a model obtained by learning using training data including input data in which a plurality of medical images of different kinds of predetermined portions such as an intensity front image and a motion contrast front image are taken as a set. Here, the intensity front image corresponds to the intensity En-face image, and the motion contrast front image corresponds to the OCTA En-face image. Further, the following configuration may be adopted: so that an analysis result obtained using a high-quality image generated by a learning model for improving image quality is displayed.
In addition, the input data included in the training data may be a high-quality image generated by a learning model for improving image quality, or may be a set composed of a low-quality image and a high-quality image. Further, the training data may be, for example, data obtained by labeling input data for which information including at least one information among an analysis value (for example, an average value or an intermediate value) obtained by analyzing an analysis region, a table including the analysis value, an analysis chart, and a position of the analysis region such as a sector in an image, or the like is employed as correct answer data (for supervised learning). Note that the following configuration may be adopted: so that the analysis result obtained by the learning model for analysis result generation is displayed in response to an instruction from the inspector.
Further, in the above-described various embodiments and modifications, various diagnostic results, such as results related to glaucoma or age-related macular degeneration, may be displayed on the report screen. At this time, for example, an accurate diagnosis result may be displayed by analyzing the medical image subjected to various artifact removal processes as described above. In addition, in the diagnosis result, the position of the specified abnormal portion or the like may be displayed on the image, and the state of the abnormal portion or the like may be displayed using a character or the like. Further, a classification result (for example, classification of keting) for an abnormal site may be displayed as a diagnosis result.
Note that the diagnostic result may be a result generated using a learning model (a diagnostic result generation engine, or a learning model for generating a diagnostic result) obtained by learning using a diagnostic result of a medical image as training data. Further, the learning model may be a model obtained by learning using the following training data: training data including medical images and diagnostic results for the medical images, or training data including medical images and diagnostic results for medical images of a different kind than the relevant medical images, etc. Further, a configuration may be adopted so as to display a diagnosis result obtained using a high-quality image generated by a learning model for improving image quality.
In addition, the input data included in the training data may be a high-quality image generated by a learning model for improving image quality, or may be a set composed of a low-quality image and a high-quality image. Further, the training data may be, for example, data obtained by labeling input data for which information including at least one of diagnosis, the kind or state (degree) of a lesion (abnormal portion), the position of the lesion in an image, the position of the lesion with respect to a region of interest, a discovery result (interpretation discovery result, etc.), the basis of diagnosis (affirmative medical support information, etc.), and the basis of negative diagnosis (negative medical support information) is employed as (supervised learning) correct answer data. Note that the following configuration may be adopted: so that the diagnosis result obtained by the learning model for diagnosis result generation is displayed in response to an instruction from the examiner.
Further, in the above-described various embodiments and modifications, the object recognition result (object detection result) or the division result regarding the region of interest, the artifact, the abnormal region, and the like as described above may be displayed on the report screen. At this time, for example, a rectangular frame or the like may be displayed around the object superimposed on the image. Further, for example, colors or the like may be displayed superimposed on an object on an image. Note that the object recognition result or the segmentation result may be a result generated using a learning model obtained by learning using training data in which information indicating object recognition or segmentation is labeled as correct answer data on a medical image. Note that the above-described analysis result generation or diagnosis result generation may be realized by using the above-described object recognition result or segmentation result. For example, processing for generating an analysis result or for generating a diagnosis result may be performed for a region of interest obtained by the object recognition processing or the segmentation processing.
The learning model may be a learning model obtained by learning using training data including input data in which a plurality of medical images of different kinds as images of a predetermined portion of a subject are taken as a set. At this time, for example, it is conceivable to take the motion contrast front image and the intensity front image (or intensity tomographic image) of the fundus as input data of a set as input data included in the training data. Further, for example, it is conceivable to take a tomographic image (B-scan image) of the fundus and a color fundus image (or a fluorescent fundus image) as input data of a set as input data included in the training data. In addition, the plurality of medical images of different kinds may be of any kind as long as the medical images are obtained by different modalities, different optical systems, different principles, or the like.
The learning model may be a learning model obtained by learning using training data including input data in which a plurality of medical images of different parts of a subject are taken as a set. At this time, for example, it is conceivable to take a tomographic image of the fundus (B-scan image) and a tomographic image of the anterior ocular segment (B-scan image) as input data of a set as input data included in the training data. Further, for example, it is also conceivable to take a three-dimensional OCT image (three-dimensional tomographic image) of the fundus macula and a tomographic image obtained by circular scanning (or raster scanning) of the fundus optical nerve head as input data of a set as input data included in the training data.
Note that the input data included in the training data may be a plurality of medical images of different parts and different kinds of subjects. At this time, for example, it is conceivable to take a tomographic image of the anterior ocular segment and a color fundus image as input data of a set as input data included in the training data. Further, the learning model described above may be a learning model obtained by learning using training data including input data in which a plurality of medical images of different imaging perspectives as images of a predetermined portion of a subject are taken as a set. Further, the input data included in the training data may be data obtained by joining together a plurality of medical images obtained by temporally dividing a predetermined portion into a plurality of areas as in the case of a panoramic image, for example. At this time, by using a wide-angle image such as a panoramic image as training data, it is possible to acquire the feature value of the image with high accuracy due to facts such as the information amount being larger than in the case of a narrow-angle image, and thus the result of each process can be enhanced. Further, the input data included in the training data may be input data in which a plurality of medical images obtained at different dates and times of a predetermined portion of the subject are taken as a set.
Further, a display screen on which at least one of the analysis result, the diagnosis result, the object recognition result, and the division result described above is to be displayed is not limited to the report screen. Such a display screen may be, for example, at least one display screen among an image capturing confirmation screen, a display screen for subsequent observation, a preview screen for making various adjustments before image capturing (a display screen on which various real-time moving images are displayed), and the like. For example, by displaying the aforementioned at least one result obtained by using the learning model described above on the image capturing confirmation screen, the inspector can inspect an accurate result even immediately after image capturing. Further, changing the display between the low-quality image and the high-quality image described above may be, for example, changing the display between the analysis result for the low-quality image and the analysis result for the high-quality image.
The various learning models described above may be obtained by machine learning using training data. For example, deep learning composed of a multi-level neural network is a machine learning. Further, for example, a Convolutional Neural Network (CNN) may be used for at least a portion of the multi-level neural network. Additionally, techniques related to automatic encoders may be used for at least a portion of the multi-level neural network. Furthermore, a technique related to back propagation (error back propagation method) may be used for learning. However, the machine learning is not limited to deep learning, and any model may be used as long as the model itself can extract (represent) feature values of training data such as images by learning.
Also, the image quality improvement engine (learning model for improving image quality) may be a learning model obtained by incremental learning using training data including at least one high-quality image generated by the image quality improvement engine. At this time, the following configuration may be adopted: it is possible to select by an instruction from the inspector as to whether or not to use a high-quality image as training data for incremental learning to be performed.
(Modification 7)
The following configuration may be employed: such that the above-described learning model is used for each at least one frame of the real-time moving image on the preview screen in the above-described various embodiments and modifications. In this case, the following structure may be adopted: in the case where a plurality of real-time moving images of different locations or different types are displayed on a preview screen, a learning model corresponding to each real-time moving image is used. In this way, for example, since the processing time can be shortened even for a real-time moving image, the inspector can obtain highly accurate information before starting image capturing. Therefore, for example, since a malfunction such as re-imaging can be reduced, the accuracy and efficiency of diagnosis can be improved. Note that the plurality of real-time moving images may include, for example, a moving image of an anterior segment of an eye for alignment in XYZ directions, and a front moving image of a fundus for focusing or OCT focusing of a fundus observation optical system. Further, the plurality of real-time moving images may also include, for example, tomographic moving images of the fundus for coherence gate adjustment (adjustment of an optical path length difference between a measurement optical path length and a reference optical path length) in OCT, and the like.
Further, the moving image to which the above learning model can be applied is not limited to a real-time moving image, and for example, the moving image may be a moving image stored (saved) in a storage unit. At this time, for example, a moving image obtained by aligning for every at least one frame of a tomographic moving image of the fundus stored (saved) in the storage unit may be displayed on the display screen. For example, in the case where it is desired to appropriately observe a vitreous body, first, the frame may be selected based on conditions such as that the vitreous body exists as much as possible in the reference frame. At this time, each frame is a tomographic image (B-scan image) in the X-Z direction. Subsequently, a moving image in which other frames have been aligned in the X-Z direction with respect to the selected reference frame may be displayed on the display screen. At this time, for example, a configuration may be adopted such that high-quality images (high-quality image frames) generated in order for each at least one frame of a moving image by a learning model for improving image quality are continuously displayed.
Note that as the above-described method for alignment between frames, the same method may be applied, or the applied methods may be different, with respect to the method for alignment in the X direction and the method for alignment in the Z direction (depth direction). In addition, the alignment in the same direction may be performed multiple times by different methods. For example, a coarse alignment may be performed, after which a fine alignment may be performed. Further, the alignment method includes, for example, alignment using (coarse Z direction) of a retinal layer boundary obtained by subjecting a tomographic image (B scan image) to a segmentation process, alignment using (fine X direction or Z direction) of correlation information (similarity) between a plurality of regions obtained by segmenting the tomographic image and a reference image, alignment using (X direction) of one-dimensional projection images generated for the respective tomographic images (B scan images), and alignment using (X direction) of a two-dimensional front image. Further, a configuration may be adopted such that after coarse alignment is performed in units of pixels, fine alignment is performed in units of sub-pixels.
In this case, it is possible that an imaging target such as the retina of the eye to be inspected has not been successfully imaged during various adjustments. Therefore, since there is a large difference between the medical image input to the learning model and the medical image used as the training data, it is possible that a high quality image cannot be accurately obtained. Accordingly, the following configuration may also be employed: so that when an evaluation value such as a value obtained when evaluating the image quality of a tomographic image (B scan) exceeds a threshold value, display of a high-quality moving image (continuous display of high-image quality frames) is automatically started. Further, the following configuration may be adopted: so that when an evaluation value such as a value obtained when evaluating the image quality of a tomographic image (B scan) exceeds a threshold value, the image quality improvement button is changed to a state (activated state) in which the button can be selected by the inspector.
Further, the following configuration may be adopted: different learning models for improving image quality are prepared for respective imaging modes different in scanning pattern and the like, and a learning model for improving image quality corresponding to the selected imaging mode is selected. Further, a learning model for improving image quality obtained by learning using training data including various medical images obtained in different imaging modes may be used.
(Modification 8)
In the respective embodiments and modifications described above, in the case where the learning model is undergoing incremental learning, it is possible that it will be difficult to output (infer/predict) using the learning model itself undergoing incremental learning. Thus, the medical image can be prohibited from being input into the learning model that is undergoing incremental learning. Further, the same learning model as the learning model undergoing incremental learning is prepared as a further auxiliary learning model. In this case, the following structure may be adopted: so that the input of the medical image to the auxiliary learning model in which the incremental learning is being performed can be performed. Subsequently, after the incremental learning is completed, the learning model that has undergone the additional learning is evaluated, and if there is no problem, it is sufficient to switch from the auxiliary learning model to the learning model that has undergone the additional learning. Further, the following configuration may be adopted: so that if there is a problem, an auxiliary learning model is used.
Further, the following configuration may be adopted: so that a learning model obtained by learning for each imaging portion can be selectively utilized. Specifically, a plurality of learning models may be prepared, which include a first learning model obtained using training data including a first imaging region (lung, eye to be examined, etc.), and a second learning model obtained using training data including a second imaging region different from the first imaging region. Further, the control unit 200 may have a selection unit for selecting any one of the plurality of learning models. At this time, the control unit 200 may have a control unit for performing incremental learning with respect to the selected learning model. In response to an instruction from the examiner, the control unit may retrieve, as incremental learning for the selected learning model, data in which an imaging portion corresponding to the selected learning model and an image obtained by imaging the relevant imaging portion form a pair, and perform learning using the retrieved and obtained data as training data. Note that the imaging part corresponding to the selected learning model may be a part obtained based on the header information of the data, or a part manually input by the inspector. In addition, the retrieval of data may be performed from a server or the like of an external facility such as a hospital or laboratory, for example, through a network. In this way, by using an image obtained by imaging an imaging portion corresponding to the learning model, incremental learning can be effectively performed for each imaging portion.
Note that the selection unit and the control unit may be constituted by software modules executed by a processor such as a CPU or MPU of the control unit 200. Further, the selection unit and the control unit may be constituted by a circuit having a specific function such as an ASIC or by a separate device or the like.
Further, when training data for incremental learning is obtained from a server or the like of an external facility such as a hospital or laboratory through a network, it is desirable to reduce reliability degradation due to tampering or system failure during incremental learning or the like. Thus, the correctness of the training data for incremental learning can be detected via consistency confirmation by digital signature or hash. In this way, training data for incremental learning can be protected. At this time, in the case where the correctness of the training data for the incremental learning cannot be detected as a result of confirming the consistency by digital signature or hash, a warning of this fact is given, and the incremental learning is performed without using the training data in question. Note that the server may be any form of server, such as a cloud server, a fog server, or an edge server, regardless of its installation location.
(Modification 9)
In the above-described various embodiments and modifications, the instruction from the inspector may be a voice instruction or the like, in addition to a manual instruction (for example, an instruction using a user interface or the like). At this time, for example, a machine learning model including a speech recognition model (a speech recognition engine or a learning model for speech recognition) obtained by machine learning may be used. Further, the manual instruction may be an instruction input through characters using a keyboard, a touch panel, or the like. At this time, for example, a machine learning model including a character recognition model (character recognition engine, learning model for character recognition) obtained by machine learning may be used. Further, the instruction from the inspector may be an instruction by a gesture or the like. At this time, a machine learning model including a gesture recognition model (gesture recognition engine, learning model for gesture recognition) obtained by machine learning may be used.
Further, the instruction from the inspector may be a result of detecting the line of sight of the inspector on a monitor or the like. The line-of-sight detection result may be, for example, a pupil detection result using a moving image of the examiner obtained by imaging from around the monitor. At this time, the object recognition engine as described above may be used according to pupil detection of the moving image. The instruction from the examiner may be an instruction by brain waves, weak electric signals flowing through the body, or the like.
In this case, for example, the training data may be the following training data: character data or voice data (waveform data) or the like indicating an instruction to display the result obtained by the processing of the various learning models described above is employed as input data, and an execution command for actually causing the result obtained by the processing of the various learning models to be actually displayed on the display unit is employed as correct answer data. Further, the training data may be the following training data: for example, character data or voice data or the like indicating an instruction to display a high-quality image obtained by a learning model for improving image quality is employed as input data, and an execution command for displaying a high-quality image and an execution command for changing the button 3420 to an activated state are employed as correct answer data. Of course, any kind of training data may be used as long as instruction content and execution command content indicated by the character data, voice data, or the like, for example, correspond to each other. Further, the voice data may be converted into character data using an acoustic model, a language model, or the like. Further, processing for reducing noise data superimposed on voice data may be performed using waveform data obtained with a plurality of microphones. Further, the following configuration may be adopted: so that it is possible to select between an instruction issued by a character or voice or the like and an instruction input using a mouse or touch panel or the like, in accordance with an instruction from an inspector. In addition, the following configuration may be adopted: so that it is possible to select to turn on or off an instruction by a character or voice or the like according to an instruction from the inspector.
In this case, the machine learning includes deep learning as described above, and, for example, a Recurrent Neural Network (RNN) may be used as at least a part of the multi-layer neural network. Here, as an example of a machine learning model according to the present modification, RNN which is a neural network that processes time series information will be described with reference to fig. 16A and 16B. Further, a long-short-term memory (hereinafter referred to as "LSTM") as one RNN will be described with reference to fig. 17A and 17B.
Fig. 16A shows the structure of an RNN as a machine learning model. The RNN 3520 has a loop structure in a network, and inputs data x t 3510 at time t and outputs data h t 3530. Since the RNN 3520 has a loop function in the network, the state of the current time can be taken over to the next state, and thus timing information can be processed. Fig. 16B shows an example of input/output of the parameter vector at time t. The data x t 3510 includes N data (parameter ms1 to parameter msN). Further, the data h t 3530 outputted by the RNN 3520 includes N pieces of data (parameter ms1 to parameter msN) corresponding to the inputted data.
However, since RNNs cannot process long-time information during back propagation, LSTM may be used. LSTM may learn long-term information by providing forget gates, input gates, and output gates. Fig. 17A shows the structure of LSTM. In LSTM 3540, the information that the network takes over at the next time t is the internal state of the network, c t-1, called the cell, and the output data h t-1. Note that the lower case letter (c, h, x) in the figure represents a vector.
Next, LSTM 3540 is shown in detail in fig. 17B. In fig. 17B, reference numeral FG denotes a forgetting gate network, reference numeral IG denotes an input gate network, reference numeral OG denotes an output gate network, and each of these networks is an S-shaped layer. Thus, a vector in which each element has a value from 0 to 1 is output. The forget gate network FG determines how much past information is retained and the input gate network IG determines which value to update. Reference CU denotes a cell update candidate network, which is an activation function tanh layer. This will create a vector of new candidate values to be added to the cell. The output gate network OG selects elements of cell candidates and selects how much information to send at the next time.
Note that the above LSTM model is a basic form, and the present invention is not limited to the network shown here. The coupling between the networks may be changed. QRNN (quasi-recurrent neural network) may be used instead of LSTM. Further, the machine learning model is not limited to the neural network, and Boosting (lifting) or Support Vector Machine (support vector machine) or the like may be used. Further, in the case of inputting an instruction from an inspector through characters, voice, or the like, a technique related to natural language processing (for example, sequence to sequence) may be applied. In addition, a dialog engine (dialog model or learning model for dialog) that responds to the inspector with an output such as text or voice may be applied.
(Modification 10)
In the above-described various embodiments and modifications, a high-quality image or the like may be stored in the memory according to an instruction from the inspector. At this time, after the inspector instructs to save a high-quality image or the like, when a file name is registered, the file name may be displayed as a recommended file name including, at any part (e.g., the first part or the last part) of the file name, information (e.g., characters) indicating that the image is an image generated by processing using a learning model for improving image quality (image quality improvement processing) in a state where the file name may be edited according to an instruction from the inspector.
Further, when the display unit is caused to display a high-quality image on various display screens such as a report screen, a display indicating that the image being displayed is a high-quality image generated by processing using a learning model for improving image quality may be displayed together with the high-quality image. In this case, since the user can easily recognize by the related display that the displayed high-quality image is not an actual image obtained by image capturing, misdiagnosis can be reduced, and diagnosis efficiency can be improved. Note that the display indicating that a high-quality image is generated by the process using the learning model for improving the image quality may be of any form as long as it is a display that can distinguish an input image from a high-quality image generated by the correlation process. Further, also for the process using the various learning models described above, not just the process using the learning model for improving the image quality, a display indicating that the result being displayed is generated by the process using the learning model of the relevant kind may be displayed together with the relevant result.
At this time, a display screen such as a report screen may be stored in the storage unit in accordance with an instruction from the inspector. For example, the report screen may be stored in the storage unit as a single image in which a high-quality image or the like and a display indicating that these images are high-quality images generated by processing using a learning model for improving image quality are displayed side by side.
Further, regarding the display instructing to generate a high-quality image by processing using the learning model for improving the image quality, the display instructing which training data is used by the learning model for improving the image quality at the time of learning may be displayed on the display unit. The display of the discussion may include a description of the kind of input data and correct answer data of the training data, or any display related to the correct answer data such as a camera location included in the input data and the correct answer data. Note that, also regarding the processing using the various learning models described above, not only the processing using the learning model for improving the image quality, but also the display of which training data to use for instructing the learning model of the relevant type to perform learning may be displayed on the display unit.
Further, the following configuration may be adopted: so that information (e.g., characters) indicating that an image is generated by processing using a learning model for improving image quality can be displayed or stored in a state where the information is superimposed on a high-quality image or the like. At this time, the position where the information is superimposed on the image may be any position as long as the position is in a region (for example, an edge of the image) that does not overlap with a region where a region of interest or the like as an imaging target is displayed. Further, a non-overlapping region may be determined, and the information may be superimposed in the determined region.
Further, the following configuration may be adopted: so that in the case where the default setting is set to cause the button 3420 to enter the activated state (the image quality improvement process is set to "on") as the initial display screen of the report screen, the report image corresponding to the report screen including the high-quality image or the like is transmitted to the server in accordance with the instruction from the inspector. Further, the following configuration may be adopted: so that when the inspection is ended (for example, in the case where the image confirmation screen or the preview screen is changed to the report screen according to an instruction from the inspector) in the case where the default setting is set so that the button 3420 is brought into the activated state, a report image corresponding to the report screen including a high-quality image or the like is (automatically) transmitted to the server. At this time, the following configuration may be adopted: causing a report image generated based on various settings of default settings (e.g., settings related to at least one of a depth range on an initial display screen of the report screen for generating an en-face image, whether an analysis chart is superimposed, whether the image is a high quality image, and whether a display screen for subsequent observation is shown, etc.) to be transmitted to a server.
(Modification 11)
In the above-described various embodiments and modifications, among the above-described various learning models, an image (for example, a high-quality image, an image showing an analysis result such as an analysis chart, an image showing an object recognition result, or an image showing a segmentation result) obtained using a first kind of learning model may be input to a second kind of learning model different from the first kind. At this time, the following configuration may be adopted: such that a result (e.g., an analysis result, a diagnosis result, an object recognition result, or a segmentation result) is generated through the processing of the second kind of learning model.
Further, among the above-described various learning models, an image to be input into a learning model of a second type different from the learning model of the first type may be generated from an image input into the learning model of the first type by using a result (e.g., an analysis result, a diagnosis result, an object recognition result, or a segmentation result) obtained by the processing of the learning model of the first type. At this time, there is a high possibility that the generated image is an image suitable as an image for the process of the learning model of the second kind. Thus, the accuracy of an image (e.g., a high-quality image, an image showing an analysis result such as an analysis chart, an image showing an object recognition result, or an image showing a segmentation result) obtained when the generated image is input to the second kind of learning model is enhanced.
In addition, using the analysis result, the diagnosis result, or the like obtained by the processing of the learning model as a search key, retrieval of a similar image using an external database stored in a server or the like can be performed. Note that in the case where the plurality of images stored in the database have been managed in a state where the respective feature values of the plurality of images have been attached as supplementary information by machine learning or the like, a similar image search engine (a similar image search model or a learning model for similar image search) that uses the images themselves as search keywords may be used.
(Modification 12)
Note that the processing for generating motion contrast data in the foregoing embodiments and modifications is not limited to the configuration of processing based on the intensity value of the tomographic image. The above-described various processes can be applied to an interference signal obtained with the OCT imaging unit 100, a signal obtained by subjecting the interference signal to fourier transform, a signal obtained by subjecting the relevant signal to any process, and tomographic data including tomographic images based on these signals, and the like. Also in these cases, effects similar to those of the foregoing configuration can be obtained.
Although an optical fiber system using a coupler as a separation unit is used, a spatial optical system using a collimator and a beam splitter may be used. Further, the configuration of the OCT imaging unit 100 is not limited to the above-described configuration, and some of the components included in the OCT imaging unit 100 may be provided as separate components from the OCT imaging unit 100.
Further, although in the foregoing embodiments and modifications, the configuration of the mach-zehnder interferometer is used as the configuration of the interference optical system of the OCT imaging apparatus 100, the configuration of the interference optical system is not limited thereto. For example, the interference optical system of the OCT apparatus 1 may have a configuration of a michelson interferometer.
In addition, although the spectral domain OCT (SD-OCT) apparatus using the SLD as a light source is described as the OCT apparatus in the above-described embodiments and modifications, the configuration of the OCT apparatus according to the present invention is not limited thereto. For example, the present invention can also be applied to a swept source OCT (SS-OCT) apparatus using a wavelength swept light source capable of sweeping the wavelength of emitted light or any other kind of OCT apparatus. Furthermore, the present invention can also be applied to a line-OCT apparatus using line light.
Further, in the foregoing embodiments and modifications, the obtaining unit 210 obtains an interference signal obtained by the OCT imaging unit 100 or a three-dimensional tomographic image generated by the image processing unit 220. However, the configuration in which the obtaining unit 210 obtains these signals or images is not limited to the above-described configuration. For example, the obtaining unit 210 may obtain these signals from a server or an image pickup apparatus connected to the control unit through a LAN, WAN, the internet, or the like.
Note that a learning model may be provided in the control unit 200, 900, or 1400 as an image processing apparatus. The learning model may be constituted, for example, by a software module executed by a processor such as a CPU. Further, the learning model may be provided in a separate server connected to the control unit 200, 900 or 1400. In this case, the control unit 200, 900 or 1400 may perform the image quality improvement process using the learning model by connecting to a server including the learning model via any network such as the internet.
(Modification 13)
Further, the image to be processed by the image processing apparatus or the image processing method according to the above-described various embodiments and modifications includes a medical image obtained using an arbitrary modality (image capturing apparatus or image capturing method). The medical image to be processed may include a medical image obtained by any image capturing apparatus or the like, and an image created by the image processing apparatus or the image processing method according to the above-described embodiments and modifications.
In addition, the medical image to be processed is an image of a predetermined portion of the subject (subject), and the image of the predetermined portion includes at least a part of the predetermined portion of the subject. The medical image may also include other parts of the subject. The medical image may be a still image or a moving image, and may be a black-and-white image or a color image. In addition, the medical image may be an image representing a structure (form) of the predetermined portion, or may be an image representing a function of the predetermined portion. For example, images representing functions include images representing hemodynamics (blood flow, blood flow velocity, etc.) such as an OCTA image, a doppler OCT image, an fMRI image, and an ultrasound doppler image. Note that predetermined portions of a subject may be determined according to an imaging target, and include organs such as a human eye (eye to be examined), brain, lung, intestine, heart, pancreas, kidney, and liver, and any portion such as head, chest, leg, and arm.
Further, the medical image may be a tomographic image of the subject, or may be a frontal image. Examples of the front image include a front image of the fundus, a front image of the anterior segment of the eye, a fundus image obtained by fluorescence imaging, and an en-face image generated using data of at least a partial range of an imaging target in the depth direction for data obtained by OCT (three-dimensional OCT data). The en-face image may be an OCTA en-face image (motion contrast frontal image) generated using data of at least a partial range of the imaging target in the depth direction for three-dimensional OCTA data (three-dimensional motion contrast data). Further, three-dimensional OCT data or three-dimensional motion contrast data is an example of three-dimensional medical image data.
In addition, the term "imaging apparatus" refers to an apparatus for performing imaging to obtain an image to be used for diagnosis. Examples of the imaging apparatus include an apparatus that obtains an image of a predetermined portion of a subject by irradiating the predetermined portion with light, radiation such as X-rays, electromagnetic waves, or ultrasonic waves, and an apparatus that obtains an image of a predetermined portion by detecting radiation emitted from a subject. More specifically, examples of the imaging apparatus according to the above-described various embodiments and modifications include at least an X-ray imaging apparatus, a CT apparatus, an MRI apparatus, a PET apparatus, a SPECT apparatus, an SLO apparatus, an OCT apparatus, an OCTA apparatus, a fundus camera, and an endoscope.
Note that a time domain OCT (TD-OCT) apparatus and a fourier domain OCT (FD-OCT) apparatus may be included as examples of the OCT apparatus. Further, examples of fourier domain OCT apparatuses may include a spectrum domain OCT (SD-OCT) apparatus and a swept source OCT (SS-OCT) apparatus. Further, an adaptive optical SLO (AO-SLO) apparatus and an adaptive optical OCT (AO-OCT) apparatus using an adaptive optical system or the like may be included as examples of the SLO apparatus or the OCT apparatus, respectively. Further, a polarization-sensitive SLO (PS-SLO) apparatus and a polarization-sensitive OCT (PS-OCT) apparatus or the like for visualizing information on a polarization phase difference or depolarization may be included as examples of the SLO apparatus or OCT apparatus, respectively.
According to one of the embodiments and modifications of the present invention described above, an image more suitable for image diagnosis than an image generated according to the conventional technique can be generated.
(Other embodiments)
The present invention can also be realized by a process in which a program realizing one or more functions according to the above-described embodiments is supplied to a system or an apparatus via a network or a storage medium, and the program is read and executed by one or more processors in a computer of the system or the apparatus. In addition, the present invention may be implemented by circuitry (e.g., an ASIC) that implements one or more functions.
The present invention is not limited to the above-described embodiments, and various changes and modifications may be made without departing from the spirit and scope of the present invention. Accordingly, to apprise the public of the scope of the present invention, the following claims are made.
The present application claims the priority benefits of japanese patent application No. 2018-166817 filed on month 9 and 6 of 2018 and japanese patent application No. 2019-068663 filed on month 3 and 29 of 2019, which are incorporated herein by reference in their entirety.
[ List of reference numerals ]
200. Control unit (image processing device)
224. Image quality improving unit
250. Display control unit
Claims (11)
1. An image processing apparatus comprising:
an obtaining unit configured to obtain a plurality of first images of an eye to be inspected at different dates and times;
An image quality improvement unit configured to generate each of a plurality of second images having at least one of lower noise and higher contrast than each of the plurality of first images obtained using each of the plurality of first images obtained as input data of a learning model; and
A display control unit configured to control the display unit to switch, according to a first instruction from an operator, display between display of the obtained plurality of first images and display of the generated plurality of second images at a plurality of display areas, among which the obtained plurality of display areas of the first images and the generated plurality of display areas of the second images correspond to each other,
Wherein the display control unit controls the display unit to change the display from the display of the generated plurality of second images corresponding to the first depth range of the eye to the display of the generated plurality of second images corresponding to the second depth range, the second depth range being at least partially different from the first depth range, by changing the depth range from the first depth range to the second depth range according to a second instruction from the operator,
The depth range refers to the depth range where the data of projection or integration is located when the second image is generated.
2. The image processing apparatus according to claim 1, the image processing apparatus further comprising:
A selection unit configured to select a learning model to be used by the image quality improvement unit from a plurality of learning models based on imaging conditions of the obtained first image.
3. The image processing apparatus according to claim 1,
Wherein the first image is a front image generated based on information of the eye to be inspected in a range of a depth direction,
The image processing apparatus further includes a selection unit configured to select, from among a plurality of learning models, a learning model corresponding to a range in a depth direction for generating the obtained first image as the learning model to be used by the image quality improvement unit.
4. The image processing apparatus according to claim 3, wherein:
When the range in the depth direction for generating the obtained first image is changed in accordance with an instruction from the operator, the display control unit changes from displaying the first image and the second image being displayed in a juxtaposed manner on the display unit to displaying the first image based on the changed range in the depth direction and the second image generated from the first image on the display unit.
5. The image processing apparatus according to claim 1, wherein:
In accordance with an instruction from an operator, the display control unit enlarges and commonly displays the obtained first image and the generated second image being displayed in a juxtaposed manner on the display unit.
6. The image processing apparatus according to claim 1, wherein:
the display control unit causes the obtained first image and the generated second image to be switched and displayed on the display unit in accordance with an instruction from an operator.
7. The image processing apparatus according to claim 6, wherein:
an image quality improvement unit generates a plurality of second images from the plurality of obtained first images; and
In accordance with an instruction from an operator, a display control unit causes the plurality of obtained first images and the plurality of generated second images to be switched and displayed in common on a display unit.
8. The image processing apparatus according to claim 1, wherein:
the display control unit sets transparency for at least one of the obtained first image and the generated second image, and causes the obtained first image and the generated second image to be displayed in a superimposed manner on the display unit.
9. The image processing apparatus according to claim 1, the image processing apparatus further comprising:
A comparing unit configured to compare the obtained first image and the generated second image, and generate a colored color map image based on a comparison result,
Wherein the display control unit causes the color map image to be displayed in a superimposed manner on the obtained first image or the generated second image on the display unit.
10. An image processing method, comprising:
Obtaining a plurality of first images of the eye to be inspected at different dates and times;
Generating each of a plurality of second images having at least one of lower noise and higher contrast than each of the plurality of first images obtained using each of the plurality of first images obtained as input data of a learning model; and
According to a first instruction from an operator, a control display unit switches display between display of the obtained plurality of first images and display of the generated plurality of second images at a plurality of display areas, in which the obtained plurality of display areas of the first images and the generated plurality of display areas of the second images correspond to each other,
Wherein the control display unit changes the display from the display of the generated plurality of second images corresponding to the first depth range of the eye to the display of the generated plurality of second images corresponding to the second depth range, which is at least partially different from the first depth range, by changing the depth range from the first depth range to the second depth range according to a second instruction from the operator,
The depth range refers to the depth range where the data of projection or integration is located when the second image is generated.
11. A non-transitory computer readable medium having a program stored thereon, which when executed by a processor causes the processor to perform the steps of the image processing method according to claim 10.
Applications Claiming Priority (5)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2018-166817 | 2018-09-06 | ||
| JP2018166817 | 2018-09-06 | ||
| JP2019-068663 | 2019-03-29 | ||
| JP2019068663A JP7305401B2 (en) | 2018-09-06 | 2019-03-29 | Image processing device, method of operating image processing device, and program |
| PCT/JP2019/023650 WO2020049828A1 (en) | 2018-09-06 | 2019-06-14 | Image processing apparatus, image processing method, and program |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112638234A CN112638234A (en) | 2021-04-09 |
| CN112638234B true CN112638234B (en) | 2024-11-05 |
Family
ID=
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011013334A (en) * | 2009-06-30 | 2011-01-20 | Yamaha Corp | Image display device |
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011013334A (en) * | 2009-06-30 | 2011-01-20 | Yamaha Corp | Image display device |
Non-Patent Citations (1)
| Title |
|---|
| Sheet, D ; Karri, SPK等.DEEP LEARNING OF TISSUE SPECIFIC SPECKLE REPRESENTATIONS IN OPTICAL COHERENCE TOMOGRAPHY AND DEEPER EXPLORATION FOR IN SITU HISTOLOGY.《IEEE International Symposium on Biomedical Imaging》.2015,第777-780页. * |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113226153B (en) | Image processing apparatus, image processing method, and computer-readable storage medium | |
| US11935241B2 (en) | Image processing apparatus, image processing method and computer-readable medium for improving image quality | |
| US12100154B2 (en) | Medical image processing apparatus, medical image processing method, computer-readable medium, and learned model | |
| US12039704B2 (en) | Image processing apparatus, image processing method and computer-readable medium | |
| JP7269413B2 (en) | MEDICAL IMAGE PROCESSING APPARATUS, MEDICAL IMAGE PROCESSING SYSTEM, MEDICAL IMAGE PROCESSING METHOD AND PROGRAM | |
| GB2589250A (en) | Medical image processing device, medical image processing method and program | |
| US11887288B2 (en) | Image processing apparatus, image processing method, and storage medium | |
| CN113543695B (en) | Image processing apparatus and image processing method | |
| WO2020075719A1 (en) | Image processing device, image processing method, and program | |
| JP2021037239A (en) | Area classification method | |
| WO2020138128A1 (en) | Image processing device, image processing method, and program | |
| JP2021122559A (en) | Image processing device, image processing method, and program | |
| JP2021164535A (en) | Image processing device, image processing method and program | |
| WO2021100694A1 (en) | Image processing device, image processing method, and program | |
| CN112638234B (en) | Image processing apparatus, image processing method, and computer readable medium | |
| JP7488934B2 (en) | IMAGE PROCESSING APPARATUS, OPERATION METHOD OF IMAGE PROCESSING APPARATUS, AND PROGRAM | |
| JP2021069667A (en) | Image processing device, image processing method and program | |
| JP7579372B2 (en) | Medical image processing device, medical image processing system, medical image processing method and program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |