JP6719104B2 - Image output device, image transmission device, image reception device, image output method, and recording medium - Google Patents

Description

The present disclosure relates to an image output device, an image transmission device, an image reception device, and the like.

Conventionally, an optical microscope has been used to observe a microstructure in a living tissue or the like. The optical microscope uses light that has passed through or is reflected by the observation target. The observer observes the image magnified by the lens. A digital microscope is also known in which an image magnified by a lens of a microscope is photographed and displayed on a display. By using a digital microscope, it is possible for multiple people to observe at the same time and to observe at a remote place.

In recent years, a technique for observing a microstructure by a CIS (Contact Image Sensing) method has been attracting attention. In the case of the CIS method, the observation target is placed close to the imaging surface of the image sensor. As the image sensor, generally, a two-dimensional image sensor in which a large number of photoelectric conversion units are arranged in rows and columns in the imaging surface is used. The photoelectric conversion unit is typically a photodiode formed on a semiconductor layer or a semiconductor substrate, and receives incident light to generate charges.

The image acquired by the image sensor is defined by a large number of pixels. Each pixel is partitioned by a unit area including one photoelectric conversion unit. Therefore, the resolution in the two-dimensional image sensor usually depends on the arrangement pitch of the photoelectric conversion units on the imaging surface. In this specification, the resolution determined by the arrangement pitch of the photoelectric conversion units may be referred to as the “specific resolution” of the image sensor. Since the array pitch of the individual photoelectric conversion units is as short as the wavelength of visible light, it is difficult to further improve the specific resolution.

Techniques have been proposed for realizing a resolution (that is, high resolution) that exceeds the inherent resolution of the image sensor. Patent Document 1 discloses a technique of forming an image of the subject (that is, a high-resolution image) by using a plurality of images obtained by shifting the imaging position of the subject.

JP 62-137037 A

However, a high-resolution image has a large data size, and there is a problem in that handling such as transmission/reception or storage of the image is burdensome.

Therefore, the present disclosure provides an image output device, an image transmission device, an image reception device, and the like that reduce the burden of handling such as transmission/reception or storage of high-resolution images.

An image output apparatus according to an aspect of the present disclosure includes at least one processor and a non-transitory recording medium that holds executable instructions, and the instructions are for a subject obtained based on photographing by a digital microscope. Obtaining a first resolution image that is an image, an image having a higher resolution than the first resolution image, and obtaining a second resolution image that is an image of the subject obtained based on photographing by the digital microscope, An enlargement ratio for the first resolution image displayed on the display device is accepted, and an evaluation value based on the received enlargement ratio is determined whether or not it is higher than a predetermined value, and the evaluation value is the predetermined value. If it is determined that the second resolution image is higher than the predetermined value, the second resolution image is transmitted to an external device. Cause the at least one processor to do not send to an external device.

Note that these comprehensive or specific aspects may be realized by a recording medium such as a system, a method, an integrated circuit, a computer program or a computer-readable CD-ROM, and the system, the method, the integrated circuit, the computer program. Alternatively, it may be realized by any combination of recording media.

According to the present disclosure, it is possible to reduce the burden of handling such as transmission/reception or storage of high-resolution images.

FIG. 1 is a configuration diagram showing an example of an image processing system including an image output device according to a first embodiment. FIG. 6 is a diagram showing an example of a display screen displayed by the display device in the first embodiment. 6 is a flowchart showing an example of processing operation of the image output apparatus according to the first embodiment. 7 is a flowchart showing another example of the processing operation of the image output device in the first embodiment. FIG. 6 is a configuration diagram showing an example of an image processing system including an image output device according to a modified example of the first embodiment. FIG. 7 is a configuration diagram showing an example of an image processing system including an image output device according to a second embodiment. 9 is a flowchart showing processing operations of the image transmitting apparatus and the image receiving apparatus according to the second embodiment. FIG. 13 is a configuration diagram showing an example of an image processing system including an image output device according to a first modification of the second embodiment. 9 is a flowchart showing processing operations of the image transmitting apparatus and the image receiving apparatus according to the first modification of the second embodiment. FIG. 14 is a configuration diagram showing an example of an image processing system including an image output device according to a second modification of the second embodiment. 13 is a flowchart showing processing operations of the image transmitting device and the image receiving device according to the second modification of the second embodiment. FIG. 16 is a configuration diagram showing an example of an image processing system including an image output device according to a third embodiment. FIG. 16 is a configuration diagram showing an example of an image processing system including an image output device according to a first modification of the third embodiment. FIG. 16 is a configuration diagram showing another example of an image processing system including an image output device according to a second modification of the third embodiment. FIG. 3 is a plan view schematically showing a part of a subject 2. FIG. 15B is a plan view schematically showing the photodiodes involved in the imaging of the area shown in FIG. 15A as extracted. It is sectional drawing which shows typically the direction of the light ray which permeate|transmits the to-be-photographed object 2 and injects into the photodiode 4p. It is sectional drawing which shows typically the direction of the light ray which permeate|transmits the to-be-photographed object 2 and injects into the photodiode 4p. It is a figure which shows typically six pixels Pa acquired by six photodiodes 4p. FIG. 17 is a cross-sectional view schematically showing a state in which light rays are incident from an irradiation direction different from the irradiation directions shown in FIGS. 16A and 16B. FIG. 17 is a cross-sectional view schematically showing a state in which light rays are incident from an irradiation direction different from the irradiation directions shown in FIGS. 16A and 16B. FIG. 18 is a diagram schematically showing six pixels Pb acquired under the irradiation directions shown in FIGS. 17A and 17B. FIG. 17B is a cross-sectional view schematically showing a state in which light rays are incident from an irradiation direction different from the irradiation directions shown in FIGS. 16A and 16B and the irradiation directions shown in FIGS. 17A and 17B. FIG. 17B is a cross-sectional view schematically showing a state in which light rays are incident from an irradiation direction different from the irradiation directions shown in FIGS. 16A and 16B and the irradiation directions shown in FIGS. 17A and 17B. FIG. 19 is a diagram schematically showing six pixels Pc acquired under the irradiation directions shown in FIGS. 18A and 18B. FIG. 16A is a cross-sectional view schematically showing a state in which light rays are incident from an irradiation direction shown in FIGS. 16A and 16B, an irradiation direction shown in FIGS. 17A and 17B, and an irradiation direction different from the irradiation directions shown in FIGS. 18A and 18B. is there. FIG. 19B is a diagram schematically showing the six pixels Pd acquired under the irradiation direction shown in FIG. 19A. It is a figure which shows the high-resolution image HR synthesize|combined from four sub-images Sa, Sb, Sc, and Sd. FIG. 5 is a cross-sectional view schematically showing irradiation directions adjusted so that light rays that have passed through two adjacent regions of the subject 2 are incident on different photodiodes. It is a figure which shows an example of the cross-section of a module typically. FIG. 22B is a plan view showing an example of the external appearance of the module 10 shown in FIG. 22A when viewed from the image sensor 4 side. It is a figure for demonstrating an example of the manufacturing method of a module. It is sectional drawing which shows the example of the irradiation angle at the time of acquisition of a sub-image. It is sectional drawing which shows the example of the method of irradiating a to-be-photographed object with an irradiation angle different from the irradiation angle shown to FIG. 24A. 1 is a schematic diagram showing an example of a configuration of an image acquisition device according to an embodiment of the present disclosure. It is a perspective view showing an example appearance of image acquisition device 100a. It is a perspective view which shows the state which closed the cover part 120 in the image acquisition apparatus 100a shown to FIG. 26A. It is a figure which shows typically an example of the mounting method of the socket 130 with respect to the stage 32 of the image acquisition device 100a. It is a figure which shows typically an example of the method of changing an irradiation direction. It is a figure which shows typically the change of the direction of the light ray which injects into a to-be-photographed object, when the stage 32 is inclined by the angle (theta) with respect to a reference plane. It is a figure which shows the cross-section of a CCD image sensor, and the example of distribution of the relative transmittance Td of a to-be-photographed object. It is a figure which shows the cross-section of a backside illumination type CMOS image sensor, and the example of distribution of the relative transmittance Td of a to-be-photographed object. It is a figure which shows the cross-section of a backside illumination type CMOS image sensor, and the example of distribution of the relative transmittance Td of a to-be-photographed object. It is a figure which shows the cross-section of a photoelectric conversion film laminated|stacked image sensor, and the example of distribution of the relative transmittance Td of a to-be-photographed object.

(Findings that form the basis of the present invention)
The present inventor has found that the following problems occur with respect to the high resolution technology described in the “Background Art” section.

In a pathological examination using a microscope, the magnification of a lens used varies depending on the condition of each specimen (also referred to as a pathological specimen) to be observed. For example, in a cancer examination, first, observation is performed with a lens having a 10-fold objective. For specimens that do not have a region with a high suspicion of cancer, the observation is completed only by observing with a 10× objective lens, but if there is a region with a high suspicion of cancer, the objective lens is switched to a 40× lens for examination. .. In this way, the magnification of the lens is changed according to the sample. Since it is not known before the examination whether a high magnification is required, the specimen is imaged at each predetermined magnification from a high magnification to a low magnification.

When an image of a sample is taken with a digital microscope, the taking time and the image size increase as the resolution of the image increases. If the size of the image corresponding to the objective 10 times obtained by the digital microscope is 100 Mbytes, the size of the image corresponding to the objective 40 times is 1.6 Gbytes.

Moreover, while the number of pathological specimens that must be examined increases, the number of pathologists has not kept up. Therefore, in the future, it is expected that the pathological image, which is an image obtained by a digital microscope, will be transmitted to a remote place and pathologists at the remote place will perform a pathological diagnosis. Further, with the advancement of medical treatment as a whole, it has become difficult for one pathologist to diagnose all organs or symptoms. Therefore, remote diagnosis is performed by transmitting an image of a sample that is difficult to be diagnosed by a pathologist at the site to a pathologist having remote expertise at a remote location. However, there is a problem that it takes time to transmit a high-resolution pathological image.

In order to solve such a problem, an image output apparatus according to an aspect of the present invention includes an image acquisition unit that acquires a first resolution image that is an image of a subject obtained based on photographing by a digital microscope, A high-resolution image acquisition unit that acquires a second resolution image that is an image of a higher resolution than a one-resolution image and that is an image of the subject that is obtained based on photographing by the digital microscope, and that is displayed on a display device. An enlargement accepting unit that receives an enlargement ratio for the first resolution image, a determination unit that determines whether or not an evaluation value based on the enlargement ratio accepted by the enlargement accepting unit is higher than a predetermined value, and the determination unit When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to an external device, and the evaluation unit determines that the evaluation value is not higher than the predetermined value. In this case, the transmission unit does not transmit the second resolution image to the external device.

As a result, the second resolution image is transmitted only when the evaluation value is high. In other words, only the sample (subject) for which sufficient diagnosis is necessary, the image of the sample can be transmitted with high resolution, for example, to a remote place. Therefore, for example, in order to transmit a high-resolution image of a sample of low importance that can be diagnosed in a short time, it is possible to prevent delay in transmission of a high-resolution image of a sample that requires sufficient diagnosis. Further, since the high resolution image of the sample having a low evaluation value is not transmitted, the burden of handling the high resolution image can be reduced.

Further, the transmitting unit may transmit the first resolution image to the external device when the determining unit determines that the evaluation value is not higher than the predetermined value.

Thereby, for example, a pathologist at a remote place can also diagnose a sample using the transmitted first resolution image which is a low resolution image.

In addition, the image output device further outputs the second resolution image to a recording medium and saves it when the evaluation value is determined by the determination unit to be higher than the predetermined value. A first output unit may be provided that does not output the second resolution image to the recording medium when the unit determines that the evaluation value is not higher than the predetermined value.

The image size of the high-resolution image (second resolution image) is large, and in the pathological examination, the image of the specimen used for the examination is stored every 10 years. In addition, as the technology for acquiring specimens from the human body is improving, the number of specimens to be imaged is expected to increase in the future. Therefore, storage of images taken with a digital microscope, especially high-resolution images, becomes an important issue.

Therefore, in one aspect of the present invention, the second resolution image is stored only when the evaluation value is high. That is, it is possible to store the image of the sample in high resolution only for the sample (subject) for which sufficient diagnosis is necessary. Therefore, for example, it is possible to prevent the free space of the recording medium from being limited in order to store a high-resolution image of a less important sample that can be diagnosed in a short time. Further, since the high resolution image of the sample having a low evaluation value is not stored, the burden of handling the high resolution image can be reduced.

Further, for example, the determination unit derives, as the evaluation value, the maximum enlargement ratio among the one or more enlargement ratios accepted by the enlargement accepting unit. Alternatively, the determination unit may display the image of the subject on the display device as the number of times a high magnification rate, which is a magnification rate higher than a threshold value, is received by the enlargement acceptance section. The higher the evaluation time, the higher the evaluation value may be derived. Alternatively, the determination unit may calculate the evaluation value that is higher as the area of the subject displayed on the display device is larger at the enlargement ratio higher than a threshold value.

This makes it possible to derive an appropriate evaluation value according to the intended use of the image of the subject, and to transmit only the optimum high resolution image (second resolution image).

An image output device according to another aspect of the present invention is an image output device having an image transmission device and an image reception device connected to the image transmission device via a communication line. Is an image acquisition unit that acquires a first resolution image, which is an image of a subject obtained based on photographing by a digital microscope, and an image having a higher resolution than the first resolution image, based on photographing by the digital microscope. Whether the evaluation value indicated by the high-resolution image acquisition unit that acquires the second resolution image, which is the image of the subject obtained by the above, and the evaluation value-related information notified from the image receiving device is higher than a predetermined value. If a determination unit that determines whether or not the evaluation value is higher than the predetermined value by the determination unit, the second resolution image is transmitted to the image receiving device, the determination unit When it is determined that the evaluation value is not higher than the predetermined value, the image receiving device includes a transmitting unit that transmits the first resolution image to the image receiving device, and the image receiving device is displayed on a display device. An enlargement accepting unit that receives an enlargement factor for the first resolution image, and the first resolution image or the second resolution image from the image transmission device, and the image is enlarged based on the enlargement factor accepted by the enlargement accepting unit. A display output unit for displaying the first resolution image or the second resolution image on the display device, wherein the display output unit is further based on the enlargement ratio accepted by the enlargement accepting unit. The evaluation value related information is transmitted to the image transmitting device.

Alternatively, an image output device according to another aspect of the present invention is an image output device having an image transmission device and an image reception device connected to the image transmission device via a communication line. Is an image acquisition unit that acquires a first resolution image, which is an image of a subject obtained based on photographing by a digital microscope, and an image having a higher resolution than the first resolution image, based on photographing by the digital microscope. A high-resolution image acquisition unit that acquires a second resolution image that is the image of the subject obtained by the above; and a transmission unit that transmits the first resolution image or the second resolution image to the image reception device, The image receiving device acquires an enlargement accepting unit that receives an enlargement ratio for the first resolution image displayed on a display device, and acquires the first resolution image or the second resolution image from the image transmitting device, and the enlargement accepting unit. A display output unit for displaying the first resolution image or the second resolution image enlarged on the basis of the enlargement ratio accepted by the display device, and an evaluation based on the enlargement ratio accepted by the enlargement accepting unit. A determination unit that determines whether or not the value is higher than a predetermined value and notifies the image transmission device of the determination result, wherein the transmission unit determines that the evaluation value is higher than the predetermined value. In the case of the determination result showing, the second resolution image is transmitted to the image receiving device, and in the case of the determination result showing that the evaluation value is not higher than the predetermined value, The one-resolution image is transmitted to the image receiving device. The evaluation value related information is information including the enlargement ratio received by the enlargement receiving unit, the number of times the enlargement ratio is received, the time during which the image of the subject is displayed, the area of the displayed subject, and the like. ..

As a result, for example, even if there is no pathologist at the point where the sample, which is the subject, is placed, the pathologist at the remote location can use the image receiving apparatus to display the image of the sample transmitted from the image transmitting apparatus installed at that point. It is possible to obtain and diagnose the sample based on the image. Even at this time, an unnecessary high-resolution image is not transmitted to the image receiving device, so that the burden of handling the high-resolution image can be reduced.

Further, the image transmitting device further outputs the second resolution image to a recording medium and stores the second resolution image when the evaluation unit determines that the evaluation value is not higher than the predetermined value. An output unit may be provided.

As a result, even if the high-resolution image (second resolution image) is not transmitted to the pathologist at the remote location due to the low evaluation value, the high-resolution image is stored. Therefore, when the pathologist needs a high resolution image, the high resolution image can be quickly read from the recording medium and transmitted to the pathologist.

Note that each of the image output device, the image transmission device, and the image reception device according to one embodiment of the present invention includes at least one processor and a non-transitory recording medium holding an executable instruction. May be. In this case, each component provided in each of the above-described devices is realized by the instruction stored in the recording medium being executed by the at least one processor.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that each of the embodiments described below shows a comprehensive or specific example. Numerical values, shapes, materials, constituent elements, arrangement positions and connection forms of constituent elements, steps, order of steps, and the like shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the constituent elements in the following embodiments, constituent elements not described in the independent claim showing the highest concept are described as arbitrary constituent elements.

(Embodiment 1)
FIG. 1 is a configuration diagram showing an example of an image processing system including the image output device according to the first embodiment.

The image processing system 100S includes a digital microscope 1500, an image output device 1001, and a display device 1501.

The digital microscope 1500 is an image acquisition device described later, and photographs a pathological sample as a subject. Here, in the present specification, the “digital microscope” is a device that digitally images at least a part of a pathological tissue preparation sample. For example, the digital microscope is an image acquisition device, which will be described later, that captures the specimen by a contact image sensing (CIS) method, a virtual slide scanner, or the like. Here, the image acquisition device that captures images by the CIS method is a device that acquires a first resolution image and a second resolution image that are images of the subject by arranging the subject directly on an image sensor and performing imaging. .. Specifically, the image acquisition device generates a plurality of first resolution images by irradiating a subject with light from a plurality of different irradiation directions and photographing the subject. Then, the image acquisition device rearranges each pixel of the plurality of first resolution images to generate a second resolution image having a higher resolution than each of the first resolution images. Further, the digital microscope 1500 may be a digital microscope such as a virtual slide scanner that generates enlarged images with a plurality of resolutions, instead of an image acquisition device described later. The virtual slide scanner is a device that photographs a sample through a microscope. The operation of shifting the relative position of the slide with respect to the microscope and the operation of photographing are repeated, and a plurality of obtained images are connected to generate an image of a wide range of specimen. In the virtual slide scanner, the resolution of the image to be acquired is selected by changing the objective lens attached to the microscope. If an objective lens with a low magnification is used, the resolution will be low, but the range of the sample imaged in one image will be wide, so there will be few imaging points and the imaging time will be short. The image acquisition device will be described in detail in Embodiment 4.

The display device 1501 is a device that displays an image obtained based on imaging by a digital microscope to an operator (for example, a pathologist), and is, for example, a liquid crystal display or a projector.

The image output device 1001 acquires an image obtained by photographing with the digital microscope 1500 and displays the image on the display device 1501. Further, the image output device 1001 transmits the image to a device handled by a pathologist at a remote place, for example, via a communication line.

The image output device 1001 includes an image acquisition unit 1101, a high resolution image acquisition unit 1102, an enlargement reception unit 1201, a determination unit 1202, a display output unit 1203, and a transmission unit 1204.

The image acquisition unit 1101 acquires a first resolution image, which is an image of a subject obtained based on shooting by the digital microscope 1500. It should be noted that the image of the subject obtained by photographing with the digital microscope 1500 may be an image directly obtained by the photographing, or an image obtained by performing processing on the image obtained directly. May be Here, the image acquisition unit 1101 acquires, as the first resolution image, a sub-image obtained by, for example, photographing with a contact image sensor in the digital microscope 1500. The resolution of the first resolution image is called the first resolution.

The high-resolution image acquisition unit 1102 acquires a second-resolution image that is an image (high-resolution image) having a higher resolution than the first-resolution image and that is an image of a subject obtained based on shooting by the digital microscope 1500. The high resolution image acquisition unit 1102 may acquire the second resolution image by generating the second resolution image from the plurality of sub images, or may acquire the second resolution image from the digital microscope 1500. The resolution of the second resolution image is called the second resolution. Here, the first resolution has a pixel pitch of 0.9 μm, and the second resolution has a pixel pitch of 0.3 μm. When the ratio of the pixel pitch of the first resolution to the pixel pitch of the second resolution is the resolution ratio, the resolution ratio is 3.

The enlargement receiving unit 1201 receives the enlargement ratio for the first resolution image displayed on the display device 1501. Specifically, the enlargement accepting unit 1201 acquires the enlargement factor K as a numerical value, for example, by operating the keyboard. Further, the enlargement receiving unit 1201 may update the enlargement ratio K by using, for example, the number of rotations of a mouse having a wheel as a change value of the enlargement ratio.

The display output unit 1203 outputs, to the display device 1501, the first resolution image or the second resolution image enlarged based on the enlargement ratio received by the enlargement receiving unit 1201. As a result, the enlarged first resolution image or second resolution image is displayed on the display device 1501. Although the image output device 1001 according to the present embodiment includes the display output unit 1203, the display output unit 1203 may not be provided.

The determination unit 1202 determines whether the evaluation value based on the enlargement ratio received by the enlargement reception unit 1201 is higher than a predetermined value. Here, the predetermined value may be, for example, a value arbitrarily set by a pathologist or an operator or a predetermined value. Specifically, when the evaluation value is derived as the maximum enlargement ratio among the one or plural enlargement ratios accepted by the enlargement accepting unit 1201, the predetermined value is, for example, the resolution ratio (that is, the above-mentioned value). In the case of, the value may be equal to 3).

The predetermined value is a value greater than 1. However, if the predetermined value is set to a low value, the output described below will be frequently performed. In order to avoid this, it is desirable that the predetermined value is as high as possible. The output described below is transmission by the transmission unit 1204 or output by the first output unit 1205. Therefore, a configuration may be used in which a predetermined value is switched depending on the thickness of the communication line and the congestion status of the line. For example, when using a mobile telephone line with a thin line, the predetermined value is set to a high value of 10, and when using a thick optical line, the predetermined value is set to a low value of 3. If the line is busy, the predetermined value is set to a high value of 10, and if the line is idle, the predetermined value is set to a low value of 3. Further, a predetermined value may be switched depending on the size of the storage used for storage (for example, the recording medium 1502 shown in FIGS. 8 and 12 to 14 described later). For example, when the storage is a large-capacity HDD (Hard Disk Drive), the predetermined value is set to 3, and when the storage is a small-capacity SSD (Solid State Drive), the predetermined value is set to 10. Is also good.

When the determination unit 1202 determines that the evaluation value is higher than the predetermined value, the transmission unit 1204 transmits the second resolution image. On the other hand, when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value, the transmission unit 1204 does not transmit the second resolution image. At this time, that is, when the determination unit 1202 determines that the evaluation value is not high, the transmission unit 1204 transmits the first resolution image. The first resolution image or the second resolution image is transmitted to a device handled by a pathologist at a remote place, for example, via a communication line.

FIG. 2 is a diagram showing an example of a display screen displayed by the display device 1501.

The display device 1501 displays a display screen 601 including an end button 602, a bird's eye view image display unit 603, a finding input field 604, a read button 605, and an enlarged image display unit 606.

When the end button 602 is selected, the display device 1501 transmits to the image output device 1001 a signal informing that display of an image should be ended. The whole of the first resolution image or the second resolution image is displayed on the bird's-eye view image display unit 603. A finding by a pathologist, for example, is written in the finding input field 604. When the read button 605 is selected, the display device 1501 ends the display of the first resolution image and transmits a signal instructing the reading of the second resolution image to the image output device 1001, for example. In the enlarged image display unit 606, the first resolution image or the second resolution image is enlarged and displayed.

FIG. 3 is a flowchart showing an example of the processing operation of the image output device 1001.

First, the image acquisition unit 1101 of the image output device 1001 acquires a first resolution image from the digital microscope 1500 (step S11). Next, the enlargement accepting unit 1201 accepts the enlargement ratio of the image according to the operation on the keyboard or the like by the operator (step S12). Then, the display output unit 1203 enlarges the first resolution image at the accepted enlargement ratio and outputs it to the display device 1501. By the output, the display output unit 1203 causes the display device 1501 to display the first resolution image enlarged to the accepted enlargement ratio (step S13).

Next, the display output unit 1203 determines whether to end the display of the first resolution image based on the signal output from the display device 1501 when the end button 602 or the read button 605 is selected. (Step S14). If it is determined that the display should not be terminated (No in step S14), the enlargement receiving unit 1201 further receives a new enlargement ratio. That is, the operator who observes the image displayed on the display device 1501 changes the magnification ratio according to the sample and observes it.

On the other hand, when it is determined that the display should be ended (Yes in step S14), the determination unit 1202 derives the evaluation value based on the enlargement ratio accepted by the enlargement accepting unit 1201, and the evaluation value is greater than the predetermined value. Is also high (step S15).

The evaluation value is, for example, the expansion ratio received immediately before (immediate expansion ratio) or the maximum expansion ratio (maximum expansion ratio) of the one or more expansion ratios received in step S12. That is, the determination unit 1202 derives the immediately previous expansion rate or the maximum expansion rate as the evaluation value. Alternatively, the determination unit 1202 may derive the evaluation value based on other requirements. For example, other requirements are the number of times the magnification rate is received in step S13 (the number of times of reception), the time during which the image is displayed on the display device 1501 (display time), and the area of the image displayed on the display device 1501. It is at least one of (display area).

Further, the determination unit 1202 may derive the evaluation value by multiplying each of the maximum enlargement ratio, the number of receptions, the display time, and the display area by a coefficient, and integrating them.

Here, if it is determined that the evaluation value is not higher than the predetermined value (No in step S15), the transmission unit 1204 transmits the first resolution image without transmitting the second resolution image via the communication line. Yes (step S16). On the other hand, when it is determined that the evaluation value is higher than the predetermined value (Yes in step S15), the high resolution image acquisition unit 1102 acquires the second resolution image (step S17). Further, the transmission unit 1204 transmits the second resolution image acquired by the high resolution image acquisition unit 1102 via the communication line (step S18).

In the flowchart of FIG. 3, only the first resolution image is enlarged and displayed at the accepted enlargement ratio, but as shown in the flowchart of FIG. 4, the second resolution image is also enlarged at that enlargement ratio. It may be displayed.

FIG. 4 is a flowchart showing another example of the processing operation of the image output device 1001.

First, the image acquisition unit 1101 of the image output device 1001 acquires a first resolution image from the digital microscope 1500 (step S21). Then, the display output unit 1203 outputs the first resolution image to the display device 1501 to display the first resolution image on the display device 1501 (step S22).

Next, the enlargement accepting unit 1201 accepts the enlargement ratio of the image according to the operation on the keyboard or the like by the operator (step S23). The determination unit 1202 derives an evaluation value based on the enlargement ratio received by the enlargement reception unit 1201 and determines whether the evaluation value is higher than a predetermined value (step S24).

Here, if it is determined that the evaluation value is not higher than the predetermined value (No in step S24), the display output unit 1203 enlarges and displays the first resolution image at the enlargement ratio accepted in step S23. Output to the device 1501. By this output, the display output unit 1203 causes the display device 1501 to display the enlarged first resolution image (step S25). At this time, the second resolution image is not transmitted. Then, the display output unit 1203 determines whether to end the display of the first resolution image based on the signal output from the display device 1501 when the end button 602 or the read button 605 is selected ( Step S26). If it is determined that the display should be ended (Yes in step S26), the image output apparatus 1001 ends the process. On the other hand, when it is determined that the display should not be ended (No in step S26), the image output device 1001 repeatedly executes the processing from step S23.

When it is determined in step S24 that the evaluation value is higher than the predetermined value (Yes in step S24), the determination unit 1202 determines whether or not the determination is the first time for the sample (step). S27). Here, if it is determined that it is the first time (Yes in step S27), the high resolution image acquisition unit 1102 acquires the second resolution image (step S28). Further, the transmission unit 1204 transmits the second resolution image acquired by the high resolution image acquisition unit 1102 via the communication line (step S29).

After the second resolution image is transmitted in step S29, or when it is determined that it is not the first time in step S27 (No in step S27), the display output unit 1203 sets the enlargement ratio accepted in step S23 to the first. The two-resolution image is enlarged and output to the display device 1501. By the output, the display device 1501 displays the enlarged second resolution image on the display device 1501 (step S30). At this time, the second resolution image may be enlarged by a value obtained by dividing the enlargement ratio by the resolution ratio, instead of enlarging it by the enlargement ratio accepted in step S23. For example, when the received enlargement ratio is 3 and the resolution ratio is 3, the second resolution image is enlarged to 1 time.

Next, the display output unit 1203 determines whether to end the display of the second resolution image based on the signal output from the display device 1501 when the end button 602 is selected (step S31). .. If it is determined that the display should be ended (Yes in step S31), the image output apparatus 1001 ends the process. On the other hand, if it is determined that the display should not be ended (No in step S31), the image output device 1001 repeatedly executes the processing from step S23.

(effect)
As described above, in the present embodiment, the second resolution image is transmitted only when the evaluation value is high. In other words, only the sample (subject) for which sufficient diagnosis is necessary, the image of the sample can be transmitted with high resolution, for example, to a remote place. Therefore, for example, in order to transmit a high-resolution image of a sample of low importance that can be diagnosed in a short time, it is possible to prevent delay in transmission of a high-resolution image of a sample that requires sufficient diagnosis. Further, since the high resolution image of the sample having a low evaluation value is not transmitted, the burden of handling the high resolution image can be reduced.

Specifically, it is conceivable that the image output apparatus 1001 according to the present embodiment is used by being installed in a local hospital where there are only inexperienced pathologists or only surgery. In this case, the transmission unit 1204 can transmit the second resolution image of the sample that is difficult to diagnose to a remote place where a pathologist with abundant experience exists. First, the low resolution first resolution image is transmitted to the pathologist. Then, the high resolution second resolution image can be transmitted only when a high evaluation value is obtained, that is, when a more detailed diagnosis is required for the sample. When the first resolution image is 100 Mbytes (=800 Mbits), the second resolution image is 1.6 Gbytes, and the communication speed is 50 Mbps, the first resolution image can be transmitted in 16 seconds, but It takes 256 seconds to send. It takes time to send all images in high resolution. However, it is possible to determine that most of the biopsies performed in medical examinations are low-resolution images and that there are no abnormalities. Therefore, it is possible to improve the efficiency of the diagnosis by sending high-resolution images only to the necessary samples. it can. Further, the image output apparatus 1001 is used in a small hospital where there is only one pathologist, and important images are transmitted to a pathologist at a remote place to select a transmission image when performing a double check. You can

(Details of evaluation value)
Here, the above-mentioned evaluation value will be described in detail.

As described above, the determination unit 1202 may derive the evaluation value based on at least one of the number of receptions, the display time, and the display area.

The evaluation value based on the number of times of reception, display time, or display area is not essential for diagnosis, but is the sample observed at high magnification, or is the sample observed at high magnification because it is necessary for diagnosis? It is effective in determining. That is, in a diagnosis with a normal microscope, it takes a lot of time to replace the lens, and therefore, if it is known that the specimen is normal at the stage of observing at a low magnification, the observation at a high magnification is not performed. In the determination of whether the sample is normal, the sample is determined to be normal when the sample does not have a tumor or when there are no test items required for diagnosis. On the other hand, high-magnification enlargement of an image acquired by an electron microscope such as the digital microscope 1500 of this embodiment can be performed by a simple operation using only a mouse or a keyboard. Therefore, with the digital microscope 1500, there are very many cases in which observation is performed at high magnification even for a sample that has been observed only at low magnification using a normal microscope.

It is considered that this is because a diagnostician such as a pathologist, a doctor or a laboratory technician confirms whether or not the imaging of the sample is appropriately performed by looking at the image magnified at high magnification. Further, it is considered that the diagnostician wants to see a small region included in the normal sample as an image magnified at a high magnification even for a normal sample.

Parameters such as the number of times of reception, display time, or display area in observation at high magnification, which are not necessary for such diagnosis, differ depending on the preference or experience of the diagnostician. However, the numerical values of these parameters are extremely small compared to the parameters of observation at high magnification, which is essential for diagnosis. Therefore, in the present embodiment, by using the evaluation value based on the number of receptions, the display time, or the display area, it is possible to determine whether or not observation at high magnification is necessary for diagnosis. In order to effectively utilize the limited communication band, it is desirable not to transmit a high-resolution image when an observation at a high magnification, which is not necessary for such diagnosis, is performed.

Further, when the evaluation value is derived based on at least one of the number of receptions, the display time, and the display area and the high resolution image is transmitted, the transmission unit 1204 may add a tag to the high resolution image. Good. That is, the transmission unit 1204 adds a tag indicating the type of parameter (at least one of the number of times of reception, display time and display area) used for the evaluation value and the evaluation value to the high resolution image, Send the tagged high resolution image. As a result, the image receiving apparatus that receives the high-resolution image can extract the high-resolution image to which the tag indicating the number of times of acceptance, for example, as the type of parameter is added, from the plurality of high-resolution images. Further, if there are a plurality of extracted high resolution images, the image receiving apparatus can arrange those high resolution images in ascending order or descending order of the evaluation values based on the evaluation values indicated by the tags. This allows the diagnostician to easily select a high resolution image to be observed.

(Details of evaluation value: Number of receptions)
The number of receptions is, specifically, the number of times that the enlargement reception unit 1201 has received a high enlargement ratio that is a higher enlargement ratio than the threshold value. This threshold is, for example, the above-mentioned resolution ratio (specifically, 3). The determination unit 1202 derives a higher evaluation value as the number of times of acceptance increases. For example, the evaluation value may be the number of times of acceptance itself. In this case, the determination unit 1202 determines whether or not the number of times of acceptance, which is the evaluation value, is greater than one (predetermined value=1). That is, if the number of receptions is one, the display of the first resolution image enlarged at a high enlargement ratio is highly likely to be a display not necessary for diagnosis. In other words, there is a high possibility that the sample is determined to be normal by the first resolution image having a lower magnification than the high magnification. On the contrary, if the number of receptions is two or more, the display of the first resolution image enlarged at a high enlargement ratio is highly likely to be a display necessary for diagnosis. That is, there is a high possibility that the determination as to whether or not the sample is normal is doubtful. Therefore, it is possible to effectively utilize the limited communication band by determining whether or not the number of receptions is greater than one and transmitting the high resolution image when it is determined that the number of receptions is greater than one. it can.

Alternatively, the determination unit 1202 may determine whether or not the number of times of acceptance, which is the evaluation value, is greater than two times (predetermined value=2). Specifically, when diagnosing a malignant tumor, the diagnosis is often made three times or more in order to confirm the mode and size of the tumor. For example, the center, left edge, and right edge of a tumor are often diagnosed. Therefore, by setting the predetermined value to "2" as described above, it is possible to transmit only the high-resolution image of the sample having a high suspicion of a malignant tumor.

In this way, when the evaluation value is derived based on the number of times of acceptance, the high-resolution image is transmitted even though the sample observation at high magnification is performed only once or twice or less. It can be suppressed. That is, it is possible to suppress the transmission of the high-resolution image due to the observation of the sample at a high magnification which is not necessary for the diagnosis.

(Details of evaluation value: Display time)
Specifically, the display time is the time during which the image of the subject is displayed on the display device 1501 at a high magnification. The determination unit 1202 derives a higher evaluation value as the display time is longer. For example, the evaluation value may be the display time during which the image of the subject is displayed on the display device 1501 at a high magnification ratio of the above-described resolution ratio (specifically, 3) or more. In this case, the determination unit 1202 determines whether the display time, which is the evaluation value, is longer than, for example, 10 seconds (predetermined value=10 seconds). That is, if the display time is 10 seconds or less, it is highly possible that the display of the image enlarged at the high enlargement ratio is not necessary for the diagnosis. As a result, it is possible to prevent the high-resolution image from being transmitted due to the observation of the specimen at a high magnification that is not necessary for diagnosis.

Further, the time required for diagnosing a malignant tumor varies depending on the diagnostician or the site to be observed. Therefore, the predetermined value (10 seconds in the above example) may be changed depending on the diagnostician, the site to be observed, or the method for obtaining and creating the sample. For example, the predetermined value may be set to 10 seconds for a sample created by a normal method, and the predetermined value may be set to 30 seconds for a sample created in a short time for intraoperative rapid diagnosis.

Further, in the above example, the predetermined value is set to 10 seconds or 30 seconds, but a time that is m times (m is a real number larger than 1) the average display time required for observing a normal sample is set to the predetermined value. May be

Further, the determination unit 1202 multiplies the display time at each of the received enlargement ratios (that is, a high enlargement ratio of 3 or more) by a weight, and calculates the sum of the weighted display times as the evaluation value. May be derived as This weight has a larger value as the enlargement ratio is higher. Specifically, the determination unit 1202 treats the enlargement ratio itself as a weight, and derives the evaluation value from the evaluation value=Σ[enlargement ratio (k)×display time (k)]. The magnification rate (k) is the k-th accepted magnification rate of 3 or more, and the display time (k) is the display time at which the first resolution image magnified at the k-th magnification rate is displayed. is there. That is, when the enlargement ratio equal to or higher than the resolution ratio is received n times, the evaluation value is derived as the sum of the enlargement ratio (k) and the display time (k) when k is 1 to n. For example, the accepted enlargement factor Q and the display time T (second) at that time are (Q, T)=(3, 7), (4, 6), (5, 5), and (6. 4). In such a case, the determination unit 1202 derives the evaluation value by the evaluation value=(3×7)+(4×6)+(5×5)+(6×4). In the calculation of the evaluation value, “enlargement ratio (k)-resolution ratio (specifically 3)+1” may be used instead of the enlargement ratio (k). In this case, the determination unit 1202 derives the evaluation value by the evaluation value=(1×7)+(2×6)+(3×5)+(4×4). By using such an evaluation value, a higher evaluation value is derived as the observation time is longer and the enlargement ratio (magnification) at the time of observation is higher. Further, in this case, the determination unit 1202 may determine whether or not the evaluation value exceeds 30 (predetermined value=30).

Although the determination unit 1202 derives the evaluation value after calculating the display time at each magnification rate, the evaluation value at the time of observation may be updated at any time. That is, the determination unit 1202 acquires the magnification rate at the time of observation for each elapse of a certain time, and adds the multiplication result of the acquired magnification rate and the certain time to the evaluation value derived immediately before. , The evaluation value may be updated. As a result, a new evaluation value is derived every time a certain period of time elapses.

Further, the determination unit 1202 may multiply the display time by a value that monotonically increases with respect to the enlargement ratio, instead of multiplying the display time by the enlargement ratio. The value that monotonically increases with respect to the enlargement ratio is a value obtained by a function, and is, for example, the logarithm of the enlargement ratio. That is, the determination unit 1202 derives the evaluation value by the evaluation value=Σ[log expansion rate (k)×display time (k)].

Further, the determination unit 1202 may derive a value obtained by integrating the enlargement ratio or the logarithm of the enlargement ratio over the display time as the evaluation value, instead of deriving the evaluation value by the above-described sum total.

(Details of evaluation value: display area)
The display area is specifically the area of the subject displayed on the display device 1501 at the high magnification ratio. For example, the determination unit 1202 identifies a region in which a subject is shown in one frame by using edge detection, and calculates the area of the identified region as the area of the image of the subject. When a plurality of regions are detected in one frame by edge detection, the determination unit 1202 determines, for example, the area of the region including the place (coordinates) designated by the user with the mouse among the plurality of regions. May be the area of the image of the subject. Here, when the displayed image is shifted, the display area is enlarged by the area of the newly displayed image. That is, when the area including the location (that is, the coordinates) designated by the user with the mouse is continuous (that is, when there is no edge), when the user shifts the image, a new area that is continuous with the area appears. Therefore, the determination unit 1202 may add the area of the newly appearing area, which is continuous to the area including the location designated by the user with the mouse, to the area of the image of the subject. The area of the subject, which is the display area, is a value (quotient) obtained by dividing the area of the image of the subject calculated as described above by the actual enlargement ratio. The determination unit 1202 derives a higher evaluation value as the display area is wider.

For example, the evaluation value is the display area of the subject displayed at a high magnification ratio equal to or higher than the resolution ratio (specifically, 3). In this case, the determination unit 1202 determines whether or not the display area, which is the evaluation value, is wider than 10,000 μm ² . When the resolution ratio is 3, and the distance between the positions of the specimen corresponding to two adjacent pixels is 1/3 μm (=0.333 μm), the area of the specimen imaged by one pixel is 1/9 μm ² . , the size of the sample corresponding to 10,000 ² becomes ^{10,000μm 2 / (1 / 9μm 2} ) = 9 × 10 4 pixels. That is, the determination unit 1202 determines whether or not it is wider than 9×10 ⁴ pixels (predetermined value=9×10 ⁴ pixels). That is, if the display area is 9×10 ⁴ pixels or less, it is highly possible that the display of the first resolution image enlarged at a high enlargement ratio is not necessary for diagnosis. As a result, it is possible to prevent the high-resolution image from being transmitted due to the observation of the specimen at a high magnification that is not necessary for diagnosis.

Further, in the above example, the predetermined value is set to 10,000 μm ² (that is, 9×10 ⁴ pixels), but m times the average display area required for observing a normal sample (m is larger than 1). The area of (real number) may be set as a predetermined value.

Further, the determination unit 1202 multiplies the display area at each of the received enlargement ratios (that is, a high enlargement ratio of 3 or more) by a weight, and calculates the sum of the weighted display areas as an evaluation value. May be derived as This weight has a larger value as the enlargement ratio is higher. Specifically, the determination unit 1202 treats the square of the enlargement ratio as a weight, and the determination unit 1202 derives the evaluation value by the evaluation value=Σ[enlargement ratio (k)^2×display area (k)]. .. The magnification rate (k) is the k-th accepted magnification rate of 3 or more, and the display area (k) is the display area of the subject magnified at the k-th magnification rate, as described above. Note that A^B indicates the calculation of A raised to the B power. That is, when the enlargement ratio equal to or larger than the resolution ratio is received n times, the evaluation value is derived as the sum of the enlargement ratio (k)^2×display area (k) when k is 1 to n. For example, the enlargement factor Q and the display area S (1×10 ⁴ pixels) at that time are (Q, S)=(3, 2), (4, 0.5), (5, 1.5) , (6, 1). In such a case, the determination unit 1202 determines that the evaluation value=[(3^2)×2×10 ⁴ ]+[(4^2)×0.5×10 ⁴ ]+[(5^2)×1. The evaluation value is derived by 5×10 ⁴ ]+[(6^2)×1×10 ⁴ ]. In the calculation of the evaluation value, instead of the square of the enlargement ratio (k), the square of “enlargement ratio (k)-resolution ratio (specifically 3)+1” may be used as the weight. In this case, the determination unit 1202 determines that the evaluation value=[(1^2)×2×10 ⁴ ]+[(2^2)×0.5×10 ⁴ ]+[(3^2)×1. The evaluation value is derived by 5×10 ⁴ ]+[(4̂2)×1×10 ⁴ ]. If such an evaluation value is used, a higher evaluation value is derived as the area of the observed subject is wider and the magnification rate (magnification) at the time of observation is higher. Further, in this case, the determination unit 1202 may determine whether the evaluation value exceeds, for example, 8.1×10 ⁵ (predetermined value=8.1×10 ⁵ ).

It should be noted that the weight may have a larger value as the enlargement ratio is higher, and any function that calculates such a value is not limited to the square of the enlargement ratio, and any function may be used. For example, the determination unit 1202 may derive the evaluation value by the evaluation value=Σ[enlargement ratio (k)^3×display area (k)].

Further, the determining unit 1202 may derive a value obtained by integrating the exponentiation of the enlargement ratio with the display area as the evaluation value, instead of deriving the evaluation value by the above-described sum total.

(Modification)
Here, a modified example of the image output device according to the first embodiment will be described. When the evaluation value is high, the image output device according to the present modification transmits and saves the second resolution image.

FIG. 5 is a block diagram showing an example of an image processing system including an image output device according to this modification.

An image processing system 100Sa according to this modification includes a digital microscope 1500, an image output device 1001a, a display device 1501, and a recording medium 1502.

The recording medium 1502 holds the image output from the image output device 1001a. The recording medium 1502 is realized by a storage device such as a hard disk, a BD (Blu-ray (registered trademark) Disc), a DVD, an SD card (registered trademark), a RAM (Random Access Memory), or a cache.

The image output device 1001a further includes a first output unit 1205 in addition to the components included in the image output device 1001 according to the first embodiment.

When the determination unit 1202 determines that the evaluation value is higher than the predetermined value, the first output unit 1205 outputs the second resolution image to the recording medium 1502 and stores it. On the other hand, when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value, the first output unit 1205 does not output the second resolution image to the recording medium 1502.

In other words, the first output unit 1205 stores the image in the recording medium 1502 by outputting the image transmitted from the transmitting unit 1204 to the recording medium 1502. That is, the first output unit 1205 stores the first resolution image or the second resolution image transmitted in steps S16 and S18 of FIG. 3 or step S29 of FIG. 4 in the recording medium 1502.

(effect)
The image output apparatus 1001a according to the present modification can automatically save a high-resolution image (second resolution image) for a sample that requires high-resolution diagnosis. It is possible to provide a button for selecting whether or not to save the high-resolution image, but the pathologist may make a diagnosis of 60 or more samples in one hour, and it is possible to forget to operate the button. However, by using the image output device 1001a according to the present modification, it is possible to prevent forgetting the operation and reliably store an important high resolution image.

Further, when the image output apparatus 1001a according to the present modification is used, the high resolution image is stored and transmitted to the sample for which the diagnosis is performed with the high resolution image, and the diagnosis is performed with the low resolution image (first resolution image). The low-resolution image is stored and transmitted to the sample subjected to the. Accordingly, by setting the size of the image to be stored and transmitted according to the sample, the size of the image to be stored, the size of the image to be transmitted (transferred), and the transfer time can be reduced.

(Embodiment 2)
The image output device according to the present embodiment is composed of an image transmitting device and an image receiving device which are connected to each other via a communication line. It should be noted that, of the devices and their components in the present embodiment, the same devices and their components as in the first embodiment will be assigned the same reference numerals as in the first embodiment, and detailed explanations thereof will be omitted. ..

FIG. 6 is a configuration diagram showing an example of an image processing system including the image output device according to the second embodiment.

The image processing system 200S includes a digital microscope 1500, an image output device 2001, a display device 1501, a recording medium 1502, and a recording medium 1503.

The recording medium 1502 holds the image output from the image receiving device 1200 of the image output device 2001.

The recording medium 1503 holds the image output from the image transmission device 1100 of the image output device 2001. Like the recording medium 1502, the recording medium 1503 is realized by a storage device such as a hard disk, a BD (Blu-ray (registered trademark) Disc), a DVD, an SD card (registered trademark), a RAM, or a cache.

The image output device 2001 includes an image transmitting device 1100 and an image receiving device 1200 that are connected to each other via a communication line. The image output device 2001 has the same functions as the image output device 1001 of the first embodiment as a whole. Further, for example, the set including the digital microscope 1500, the image transmission device 1100, and the recording medium 1503 is placed in a facility such as a hospital without a pathologist. The set including the image receiving device 1200, the display device 1501, and the recording medium 1502 is arranged, for example, in a facility where a pathologist is located, far away from the hospital.

The image transmission device 1100 includes an image acquisition unit 1101, a high resolution image acquisition unit 1102, a transmission unit 1204, and a second output unit 1103.

The second output unit 1103 stores the second resolution image in the recording medium 1503 by outputting the second resolution image acquired by the high resolution image acquiring unit 1102 to the recording medium 1503. That is, when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value, the second output unit 1103 outputs the second resolution image to the recording medium 1503 and stores it.

The image receiving device 1200 includes an enlargement receiving unit 1201, a determination unit 1202, a display output unit 1203, and a first output unit 1205.

The display output unit 1203 acquires the first resolution image or the second resolution image from the image transmission device 1100 via the communication line. Then, as in the first embodiment, the display output unit 1203 outputs the first resolution image or the second resolution image enlarged based on the enlargement ratio received by the enlargement receiving unit 1201 to the display device 1501. As a result, the display output unit 1203 causes the display device 1501 to display the enlarged first resolution image or second resolution image.

The determination unit 1202 determines whether the evaluation value is higher than a predetermined value, as in the first embodiment. Then, the determination unit 1202 notifies the image transmission device 1100 of the determination result via the communication line.

When the determination result indicates that the evaluation value is high, the transmission unit 1204 of the image transmission device 1100 transmits the second resolution image to the image reception device 1200 via the communication line. On the other hand, if the determination result indicates that the evaluation value is not high, the transmission unit 1204 transmits the first resolution image instead of the second resolution image via the communication line.

FIG. 7 is a flowchart showing the processing operation of the image transmitting apparatus 1100 and the image receiving apparatus 1200.

The image acquisition unit 1101 of the image transmission device 1100 acquires the first resolution image from the digital microscope 1500 (step S41). The transmission unit 1204 transmits the first resolution image to the image reception device 1200 via the communication line (step S42).

The display output unit 1203 of the image receiving apparatus 1200 receives the first resolution image transmitted from the image transmitting apparatus 1100 via the communication line (step S51). Next, the enlargement accepting unit 1201 of the image receiving device 1200 accepts the enlargement ratio of the image according to the operation of the keyboard, for example, by the operator (step S52). Then, the display output unit 1203 enlarges the first resolution image at the accepted enlargement ratio and outputs it to the display device 1501. The display output unit 1203 causes the display device 1501 to display the first resolution image enlarged to the accepted enlargement ratio by the output (step S53).

Next, the display output unit 1203 of the image receiving apparatus 1200 should end the display of the first resolution image based on the signal output from the display apparatus 1501 when the end button 602 or the read button 605 is selected. It is determined whether or not (step S54). If it is determined that the display should not be terminated (No in step S54), the enlargement receiving unit 1201 further receives a new enlargement ratio. That is, the operator (pathologist) who observes the image displayed on the display device 1501 changes the magnification rate according to the sample and observes it.

On the other hand, when it is determined that the display should be ended (Yes in step S54), the determination unit 1202 derives an evaluation value based on the enlargement ratio accepted by the enlargement accepting unit 1201, and the evaluation value is greater than the predetermined value. Is also high (step S55).

Here, if it is determined that the evaluation value is not higher than the predetermined value (No in step S55), the first output unit 1205 of the image receiving device 1200 outputs the first resolution image to the recording medium 1502 and saves it. Yes (step S59). On the other hand, when it is determined that the evaluation value is higher than the predetermined value (Yes in step S55), the first output unit 1205 requests the second resolution image from the image transmission device 1100 via the communication line (step S56). ).

The transmitting unit 1204 of the image transmitting apparatus 1100 determines whether or not the second resolution image is requested from the image receiving apparatus 1200 (step S43). If it is determined that the request is not made (No in step S43), the high resolution image acquisition unit 1102 acquires the second resolution image (step S44). Then, the second output unit 1103 outputs the second resolution image acquired by the high resolution image acquisition unit 1102 to the recording medium 1503 and stores it (step S45). Therefore, when the determination unit 1202 determines that the evaluation value is not high, the second output unit 1103 outputs the second resolution image to the recording medium 1503 and stores it.

On the other hand, when determining that the second resolution image is requested (Yes in step S43), the high resolution image acquisition unit 1102 acquires the second resolution image (step S46). Then, the transmission unit 1204 transmits the second resolution image acquired by the high resolution image acquisition unit 1102 to the image reception device 1200 via the communication line (step S47). Therefore, like the first embodiment, when the determination unit 1202 determines that the evaluation value is high, the transmission unit 1204 transmits the second resolution image, and the determination unit 1202 determines that the evaluation value is not high. If it does, the second resolution image is not transmitted.

In the above example, the processes of steps S44 and S46 are performed after the determination of step S43, but they may be performed before the determination of step S43.

When the second resolution image is transmitted in step S47, the first output unit 1205 of the image receiving device 1200 receives the second resolution image (step S57). Then, the first output unit 1205 outputs the second resolution image to the recording medium 1502 and stores it (step S58).

(effect)
In the image output device 2001 according to the present embodiment as described above, the same effect as that of the first embodiment can be obtained even if the facility for acquiring the image of the subject and the facility for observing the image are separated.

(Modification 1)
Here, a first modification of the image output device according to the second embodiment will be described. In this modification, when it is determined that the evaluation values are high for the plurality of samples, the plurality of second resolution images are transmitted in order from the second resolution image corresponding to the sample having the high evaluation value.

FIG. 8 is a configuration diagram showing an example of an image processing system including an image output device according to this modification.

An image processing system 200Sa according to this modification includes a digital microscope 1500, an image output device 2001a, a display device 1501, a recording medium 1502, and a recording medium 1503.

The image output device 2001a includes an image transmitting device 1100a and an image receiving device 1200a which are connected to each other via a communication line.

The image receiving device 1200a further includes an evaluation value storage unit 1206 in addition to the components included in the image receiving device 1200 according to the second embodiment. The evaluation value storage unit 1206 stores the evaluation values derived for each sample (subject). In addition, the first output unit 1205 requests the image transmission device 1100a for the second resolution image corresponding to each evaluation value stored in the evaluation value storage unit 1206.

The image transmitting device 1100a includes a transmitting unit 1204a instead of the transmitting unit 1204 according to the second embodiment. The transmitting unit 1204a has not only the same function as the transmitting unit 1204 in the second embodiment, but also the second resolution corresponding to the sample having a high evaluation value when the request from the first output unit 1205 is received. A plurality of second resolution images are transmitted in order from the image.

FIG. 9 is a flowchart showing the processing operation of the image transmitting device 1100a and the image receiving device 1200a.

The image acquisition unit 1101 and the high resolution image acquisition unit 1102 of the image transmission device 1100a acquire the first resolution image and the second resolution image of the subject (step S71). The transmitting unit 1204a transmits the first resolution image to the image receiving device 1200a via the communication line (step S72). Then, the image acquisition unit 1101 and the high-resolution image acquisition unit 1102 determine whether or not there is a next subject (step S73). When it is determined that there is a next subject (step S73), the image transmission device 1100a repeats the processing from step S71.

The display output unit 1203 of the image receiving device 1200a receives the first resolution image transmitted from the image transmitting device 1100a via the communication line (step S81). Next, the enlargement accepting unit 1201 of the image receiving device 1200a accepts the enlargement ratio of the image according to the operation of the keyboard, etc. by the operator (step S82). Then, the display output unit 1203 enlarges the first resolution image at the accepted enlargement ratio and outputs it to the display device 1501. The output unit for display 1203 causes the display device 1501 to display the first resolution image magnified to the accepted magnification rate by the output (step S83).

Next, whether the display output unit 1203 of the image receiving device 1200a should end the display of the first resolution image based on the signal output from the display device 1501 when the end button 602 or the read button 605 is selected. It is determined whether or not (step S84). If it is determined that the display should not be ended (No in step S84), the enlargement receiving unit 1201 further receives a new enlargement ratio. That is, the operator (pathologist) who observes the image displayed on the display device 1501 changes the magnification rate according to the sample and observes it.

On the other hand, when it is determined that the display should be ended (Yes in step S84), the determination unit 1202 derives the evaluation value based on the enlargement ratio accepted by the enlargement accepting unit 1201, and the evaluation value is greater than the predetermined value. Is also high (step S85).

Here, if it is determined that the evaluation value is not higher than the predetermined value (No in step S85), the first output unit 1205 of the image receiving device 1200a outputs the first resolution image to the recording medium 1502 and saves it. Yes (step S87). On the other hand, when it is determined that the evaluation value is higher than the predetermined value (Yes in step S85), the determination unit 1202 stores the evaluation value higher than the predetermined value in the evaluation value storage unit 1206 in association with the subject. (Step S86).

Next, the display output unit 1203 of the image receiving device 1200a determines whether or not there is a next subject (step S88). Here, if it is determined that there is a next subject (Yes in step S88), the image receiving device 1200a repeatedly executes the processing from step S81. On the other hand, when it is determined that there is no next subject (No in step S88), the first output unit 1205 of the image receiving device 1200a refers to the evaluation value stored in the evaluation value storage unit 1206. Then, the first output unit 1205 requests the image transmission device 1100 via the communication line to transmit the second resolution images of the subject associated with the evaluation value in order from the highest evaluation value (step S89). ).

The transmission unit 1204a of the image transmission device 1100a determines whether the second resolution image is requested from the image reception device 1200a (step S74). Here, if it is determined that it is not requested (No in step S74), the second output unit 1103 outputs the second resolution image of each subject acquired in step S71 to the recording medium 1503 and stores it (step S74). Step S75). On the other hand, when the transmitting unit 1204a determines that the second resolution image is requested (Yes in step S74), the evaluation value higher than the predetermined value among the second resolution images of the subjects acquired in step S71. The second resolution image corresponding to is transmitted to the image receiving device 1200a. At this time, the transmission unit 1204a transmits those second resolution images in order from the second resolution image having a high evaluation value to the image reception device 1200a via the communication line (step S76).

The first output unit 1205 of the image receiving device 1200a receives each second resolution image corresponding to an evaluation value higher than a predetermined value in order from the second resolution image having a higher evaluation value, and outputs it to the recording medium 1502 ( Step S90). As a result, each second resolution image having an evaluation value higher than the predetermined value is stored in the recording medium 1502.

(effect)
As described above, in the present modification, when the second resolution images of a plurality of subjects determined to have a high evaluation value are collectively transmitted, the second resolution images having a high evaluation value are transmitted in order. Therefore, it is possible to quickly transmit the second resolution image having high importance. As a result, the pathologist can diagnose the specimen using the second resolution images in order from the second resolution image with the highest importance.

That is, it is conceivable that the image transmitting apparatus 1100a is used by being placed in a local hospital where only a pathologist with little experience or only surgery is used. In this case, the second resolution image of the sample that is difficult to diagnose is transmitted by the transmission unit 1204a to a large hospital with an experienced pathologist. Although the communication line is limited, by using the image output device 2001a according to the present modification, the second resolution image of the sample that is difficult to diagnose can be preferentially transmitted from the local hospital to the large hospital. Further, the image transmitting apparatus 1100a is used in a small hospital where there is only one pathologist, and important images are transmitted to a pathologist at a remote place to select a transmission image when performing a double check. You can

In the pathological examination, a plurality of specimens may be imaged from the same patient. For example, in the examination for cancer, a part of an organ may be acquired and sections of different cross sections of the part may be observed. That is, a plurality of two-dimensional images are acquired by imaging specimen sections at different positions in the vertical direction (referred to as different heights for convenience), and the three-dimensional cancer range is confirmed by these images. Here, when there is a suspicion of cancer in an image at any height, it is desirable to transmit or hold images at other heights including that image with high resolution.

Therefore, the high resolution image acquisition unit 1102 may give the same image group identifier to a plurality of second resolution images obtained from the same organ of the same patient. In this case, when transmitting the second resolution image, the transmission unit 1204a also transmits the other second resolution image associated with the image group identifier given to the second resolution image to the image reception device 1200a. Accordingly, a pathologist who handles the image receiving device 1200a can collectively acquire a plurality of second resolution images obtained for the same organ of the same patient without selecting each, and the range of cancer and the like can be obtained. You can easily check it three-dimensionally.

(Modification 2)
Although the determination unit 1202 is included in the image receiving apparatus in the second embodiment and the first modification thereof, it may be included in the image transmitting apparatus. The image transmitting apparatus in the second modified example includes a determination unit 1202.

FIG. 10 is a block diagram showing an example of an image processing system including an image output device according to this modification.

An image processing system 200Sb according to this modification includes a digital microscope 1500, an image output device 2001b, a display device 1501, a recording medium 1502, and a recording medium 1503.

The image output device 2001b includes an image transmitting device 1100b and an image receiving device 1200b that are connected to each other via a communication line.

The image transmission device 1100b further includes a determination unit 1202 in addition to the components included in the image transmission device 1100a according to the second embodiment and the first modification thereof.

The image reception device 1200b does not include the determination unit 1202, but includes an enlargement reception unit 1201, a display output unit 1203, and a first output unit 1205.

FIG. 11 is a flowchart showing the processing operation of the image transmitting device 1100b and the image receiving device 1200b.

The image acquisition unit 1101 of the image transmission device 1100b acquires the first resolution image from the digital microscope 1500 (step S101). The transmission unit 1204 transmits the first resolution image to the image reception device 1200 via the communication line (step S102).

The display output unit 1203 of the image receiving device 1200b receives the first resolution image transmitted from the image transmitting device 1100b via the communication line (step S121). Next, the enlargement accepting unit 1201 of the image receiving device 1200b accepts the enlargement ratio of the image according to the operation on the keyboard or the like by the operator (step S122). Then, the display output unit 1203 enlarges the first resolution image at the accepted enlargement ratio and outputs it to the display device 1501. The output unit for display 1203 causes the display device 1501 to display the first resolution image enlarged to the accepted enlargement ratio by the output (step S123).

Next, the display output unit 1203 of the image receiving device 1200b should end the display of the first resolution image based on the signal output from the display device 1501 when the end button 602 or the read button 605 is selected. It is determined whether or not (step S124). If it is determined that the display should not be ended (No in step S124), the enlargement receiving unit 1201 further receives a new enlargement ratio. That is, the operator (pathologist) who observes the image displayed on the display device 1501 changes the magnification rate according to the sample and observes it. On the other hand, when the display output unit 1203 determines that the display should be ended (Yes in step S124), the evaluation value related information necessary for deriving the evaluation value is transmitted to the image transmitting apparatus 1100b via the communication line. (Step S126). The evaluation value-related information is, for example, at least one of the immediately previous enlargement ratio, the maximum enlargement ratio, the display count, the display time, and the display area.

The determination unit 1202 of the image transmission device 1100b receives the evaluation value related information (step S104). Then, the determination unit 1202 derives the evaluation value based on the evaluation value related information. Further, the determination unit 1202 determines whether or not the derived evaluation value is higher than a predetermined value, and notifies the image reception device 1200b of the determination result via the communication line (step S105).

When the first output unit 1205 of the image receiving device 1200b receives the determination result notified from the image transmitting device 1100b via the communication line (step S127), it is determined whether the determination result is affirmative (Yes). Confirm (step S128). Here, if it is confirmed that the determination result is negative, that is, the evaluation value is not higher than the predetermined value (No in step S128), the first output unit 1205 of the image receiving device 1200b determines that The one-resolution image is output to the recording medium 1502 and stored (step S132). On the other hand, when it is confirmed that the determination result is affirmative, that is, the evaluation value is higher than the predetermined value (Yes in step S128), the first output unit 1205 of the image receiving device 1200b connects the communication line. The second resolution image is requested from the image transmission device 1100 via the image transmission device 1100 (step S129).

The transmission unit 1204 of the image transmission device 1100b determines whether the second resolution image is requested from the image reception device 1200b (step S106). If it is determined that the request is not made (No in step S106), the high resolution image acquisition unit 1102 acquires the second resolution image (step S107). Further, the second output unit 1103 outputs the second resolution image acquired by the high resolution image acquisition unit 1102 to the recording medium 1503 and stores it (step S108).

On the other hand, when it is determined that the second resolution image is requested (Yes in step S106), the high resolution image acquisition unit 1102 acquires the second resolution image (step S109). Then, the transmission unit 1204 transmits the second resolution image acquired by the high resolution image acquisition unit 1102 to the image reception device 1200b via the communication line (step S110).

In the above example, the processes of steps S107 and S109 are performed after the determination of step S106, but may be performed before the determination of step S106.

When the second resolution image is transmitted in step S110, the first output unit 1205 of the image receiving device 1200b receives the second resolution image (step S130). Then, the first output unit 1205 outputs the second resolution image to the recording medium 1502 and stores it (step S131).

(effect)
As described above, in the present modification, even if the determination unit 1202 is included in the image transmission device 1100b, it is possible to achieve the same effects as those of the first and second embodiments.

Also in the present modification example, similarly to the modification example 1, the second resolution images of the plurality of subjects associated with the evaluation value higher than the predetermined value are transmitted and received in order from the second resolution image with the highest evaluation value. You may.

(Embodiment 3)
In the first and second embodiments and their modifications, the second resolution image is transmitted or not transmitted depending on whether or not the evaluation value is higher than a predetermined value. In the present embodiment, depending on whether or not the evaluation value is higher than a predetermined value, the second resolution image is saved or not saved without switching between the transmission and the non-transmission. It should be noted that, of the devices and their components in the present embodiment, the same devices and their components as those in the first or second embodiment are designated by the same reference numerals as those in the first or second embodiment, and their detailed description is omitted. The description is omitted.

FIG. 12 is a configuration diagram showing an example of an image processing system including the image output device according to the third embodiment.

The image processing system 300S includes a digital microscope 1500, an image output device 3001, a display device 1501, and a recording medium 1502.

The image output device 3001 includes an image acquisition unit 1101, a high resolution image acquisition unit 1102, an enlargement reception unit 1201, a determination unit 1202, a display output unit 1203, and a first output unit 1205. That is, the image output apparatus 3001 according to the present embodiment includes the first output unit 1205 instead of the transmission unit 1204 among the constituent elements included in the image output apparatus 1001 according to the first embodiment. Although the image output device 3001 according to the present embodiment includes the display output unit 1203, the display output unit 1203 may not be provided as in the first embodiment.

When the determination unit 1202 determines that the evaluation value is higher than the predetermined value, the first output unit 1205 outputs the second resolution image to the recording medium 1502, as in the modification of the first embodiment. save. On the other hand, when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value, the first output unit 1205 does not output the second resolution image to the recording medium 1502.

(effect)
As a result, in the present embodiment, similarly to the modification of the first embodiment, a high resolution image (second resolution image) can be automatically saved for a sample that requires high resolution diagnosis. it can. Further, since a high resolution image is not stored for a sample that does not require diagnosis at high resolution, it is possible to prevent the free space of the recording medium 1502 from being limited.

(Modification 1)
Although the image output device 3001 in the third embodiment is configured as one device, it may be configured by two devices as in the second embodiment. The image output device according to this modification includes an image transmission device and an image reception device that are connected to each other via a communication line.

FIG. 13 is a configuration diagram showing an example of an image processing system including an image output device according to a first modification of the third embodiment.

The image processing system 300Sa includes a digital microscope 1500, an image output device 3001a, a display device 1501, and a recording medium 1502.

The image output device 3001a includes an image transmitting device 3100a and an image receiving device 3200a that are connected to each other via a communication line. The image output apparatus 3001a has the same functions as the image output apparatus 3001 of the third embodiment as a whole. Further, for example, the set including the digital microscope 1500 and the image transmission device 3100a is arranged in a facility such as a hospital without a pathologist. The set including the image receiving device 3200a, the display device 1501, and the recording medium 1502 is arranged, for example, in a facility where a pathologist is present, far away from the hospital.

The image transmission device 3100a includes an image acquisition unit 1101. The image acquisition unit 1101 acquires the first resolution image from the digital microscope 1500 and transmits the first resolution image to the image reception device 3200a via the communication line.

The image reception device 3200a includes a high resolution image acquisition unit 1102, an enlargement reception unit 1201, a determination unit 1202, a display output unit 1203, and a first output unit 1205. The high resolution image acquisition unit 1102 according to the present modification acquires a plurality of first resolution images from the image transmission device 3100a via a communication line in response to a request from the first output unit 1205. Then, the high resolution image acquisition unit 1102 acquires the second resolution image by generating the second resolution image based on those first resolution images.

When the determination unit 1202 determines that the evaluation value is higher than the predetermined value, the first output unit 1205 requests the high resolution image acquisition unit 1102 for the second resolution image, and the high resolution image acquisition unit 1102 outputs the second resolution image. Acquire a two-resolution image. Then, the first output unit 1205 outputs the second resolution image to the recording medium 1502 and stores it. On the other hand, the first output unit 1205 does not output the second resolution image to the recording medium 1502 when the determination unit 1202 determines that the evaluation value is not higher than the predetermined value.

(effect)
In the image output device 3001a according to the present modification as described above, the same effect as that of the third embodiment can be obtained even if the facility for acquiring the image of the subject and the facility for observing the image are separated.

(Modification 2)
In the first modification of the third embodiment described above, the determination unit 1202 is included in the image receiving device, but may be included in the image transmitting device. The image transmitting apparatus in the second modified example includes a determination unit 1202.

FIG. 14 is a configuration diagram showing another example of the image processing system including the image output device according to the second modification of the third embodiment.

The image processing system 300Sb includes a digital microscope 1500, an image output device 3001b, a display device 1501, and a recording medium 1502.

The image output device 3001b includes an image transmitting device 3100b and an image receiving device 3200b that are connected to each other via a communication line. The image output device 3001b has the same functions as the image output device 3001 of the third embodiment as a whole. Further, for example, the set including the digital microscope 1500 and the image transmission device 3100b is arranged in a facility such as a hospital without a pathologist. The set including the image receiving device 3200b, the display device 1501, and the recording medium 1502 is arranged, for example, in a facility where a pathologist is located, far away from the hospital.

The image transmission device 3100b includes an image acquisition unit 1101 and a determination unit 1202. The image acquisition unit 1101 acquires the first resolution image from the digital microscope 1500 and transmits the first resolution image to the image reception device 3200b via the communication line. The determination unit 1202 acquires the above-described evaluation value related information transmitted from the image receiving device 3200b via the communication line, and derives the evaluation value based on the evaluation value related information. Then, the determination unit 1202 determines whether or not the evaluation value is higher than a predetermined value, and notifies the image reception device 3200b of the determination result via the communication line.

The image receiving device 3200b includes a high resolution image acquisition unit 1102, an enlargement reception unit 1201, a display output unit 1203, and a first output unit 1205.

The high resolution image acquisition unit 1102 acquires a plurality of first resolution images from the image transmission device 3100b via a communication line in response to a request from the first output unit 1205. Then, the high resolution image acquisition unit 1102 acquires the second resolution image by generating the second resolution image based on those first resolution images.

The display output unit 1203 transmits the evaluation value related information to the image transmitting device 3100b via the communication line. As a result, the determination unit 1202 of the image transmission device 3100b makes the determination.

The first output unit 1205 tells the high-resolution image acquisition unit 1102 that the determination result notified from the image transmission device 3100b is positive, that is, the evaluation value is higher than a predetermined value. Request a second resolution image. Then, the first output unit 1205 acquires the second resolution image from the high resolution image acquisition unit 1102, outputs the second resolution image to the recording medium 1502, and stores it. On the other hand, the first output unit 1205 confirms that the determination result notified from the image transmitting device 3100b is negative, that is, the evaluation value is not higher than the predetermined value, and then the second resolution is set. The image is not output to the recording medium 1502.

(effect)
As described above, in the present modification, even if the determination unit 1202 is included in the image transmission device 3100b, it is possible to achieve the same effects as those of the third embodiment and the first modification thereof.

(Embodiment 4)
Here, the digital microscope 1500 in each of the above-described embodiments and its modifications will be described in detail below. The digital microscope 1500 is hereinafter referred to as an image acquisition device (digitizer).

<Principle of high resolution image formation>
In the present disclosure, by using a plurality of images obtained by changing the irradiation direction of the illumination light and performing a plurality of shootings, an image having a higher resolution (resolution) than each of the plurality of images (hereinafter, “high resolution”) is used. Resolution image" or "high resolution image"). The high resolution image corresponds to the second resolution image described above, and each of the plurality of images (sub-images) used to form the high resolution image corresponds to the first resolution image described above. First, the principle of high-resolution image formation will be described with reference to FIGS. 15A to 20. Here, a CCD (Charge Coupled Device) image sensor will be described as an example. In the following description, constituent elements having substantially the same function are designated by common reference numerals, and description thereof may be omitted.

Please refer to FIG. 15A and FIG. 15B. FIG. 15A is a plan view schematically showing a part of the subject. The subject 2 shown in FIG. 15A is, for example, a thin piece (typically, having a thickness of several tens μm or less) of biological tissue. When the image of the subject 2 is acquired, the subject 2 is arranged close to the imaging surface of the image sensor. The distance from the imaging surface of the image sensor to the subject 2 is typically 1 mm or less, and can be set to about 1 μm, for example.

FIG. 15B is a plan view schematically showing, out of the photodiodes of the image sensor, the photodiodes involved in imaging the region shown in FIG. 15A. In the example described here, of the photodiodes 4p formed in the image sensor 4, six photodiodes are shown. Note that, for reference, in FIG. 15B, arrows indicating the x direction, the y direction, and the z direction orthogonal to each other are illustrated. The z direction indicates the normal line direction of the imaging surface. In FIG. 15B, the arrow indicating the u direction, which is the direction rotated by 45° from the x axis to the y axis in the xy plane, is also shown. Also in other drawings, arrows indicating the x direction, the y direction, the z direction, or the u direction may be illustrated.

The components other than the photodiode 4p in the image sensor 4 are covered with the light shielding layer. In FIG. 15B, the hatched area indicates the area covered by the light shielding layer. The area (S2) of the light receiving surface of one photodiode on the image pickup surface of the CCD image sensor is smaller than the area (S1) of the unit region including the photodiode. The ratio of the light receiving area S2 to the pixel area S1 (S2/S1) is called the "aperture ratio". Here, description will be given assuming that the aperture ratio is 25%.

16A and 16B schematically show directions of light rays that pass through the subject 2 and are incident on the photodiode 4p. 16A and 16B show a state in which light rays are incident from a direction perpendicular to the image pickup surface. As schematically shown in FIGS. 16A and 16B, here, a lens for image formation is not arranged between the subject 2 and the image sensor 4, and the image of the subject 2 passes through the subject 2. Acquired using substantially parallel rays.

FIG. 16C schematically shows the image Sa (first sub-image Sa) acquired under the irradiation direction shown in FIGS. 16A and 16B. As shown in FIG. 16C, the first sub-image Sa is composed of 6 pixels Pa acquired by the 6 photodiodes 4p. Each of the pixels Pa has a value (pixel value) indicating the amount of light incident on the individual photodiode 4p.

As shown in FIGS. 16A and 16B, when the subject 2 is illuminated from the direction perpendicular to the image pickup surface, the light transmitted through the region of the entire subject 2 located immediately above the photodiode 4p is incident on the photodiode 4p. To do. In this example, the first sub-image Sa has information of areas A1, A2, A3, A4, A5 and A6 (see FIG. 15A) of the entire subject 2. It should be noted that the light transmitted through the region not directly above the photodiode 4p does not enter the photodiode 4p. Therefore, in the first sub-image Sa, the information of the areas other than the areas A1, A2, A3, A4, A5, and A6 in the entire subject 2 is missing.

17A and 17B show a state in which light rays are incident from an irradiation direction different from the irradiation directions shown in FIGS. 16A and 16B. The rays shown in FIGS. 17A and 17B are inclined in the x direction with respect to the z direction. At this time, light that has passed through a region of the entire subject 2 that is different from the region located immediately above the photodiode 4p is incident on the photodiode 4p.

FIG. 17C schematically shows the image Sb (second sub-image Sb) acquired under the irradiation direction shown in FIGS. 17A and 17B. As shown in FIG. 17C, the second sub-image Sb is also composed of 6 pixels acquired by 6 photodiodes 4p. However, the pixels Pb forming the second sub-image Sb are different from the regions A1, A2, A3, A4, A5 and A6 in the entire subject 2 in regions B1, B2, B3, B4, B5 and B6 ( 15A) (see FIG. 15A). In other words, the second sub-image Sb does not have information on the areas A1, A2, A3, A4, A5 and A6 of the entire subject 2, and instead the areas B1, B2, B3, B4 are included. , B5 and B6 information. Here, for example, the region B1 is a region adjacent to the right side of the region A1 in the subject 2 (see FIG. 15A).

As can be understood by comparing FIGS. 16A and 16B with FIGS. 17A and 17B, by appropriately changing the irradiation direction, the light rays transmitted through different regions of the subject 2 are made incident on the photodiode 4p. be able to. As a result, the first sub-image Sa and the second sub-image Sb can include pixel information corresponding to different positions in the subject 2.

18A and 18B show a state in which light rays are incident from an irradiation direction different from the irradiation directions shown in FIGS. 16A and 16B and the irradiation directions shown in FIGS. 17A and 17B. The light rays shown in FIGS. 18A and 18B are inclined in the y direction with respect to the z direction.

FIG. 18C schematically shows the image Sc (third sub-image Sc) acquired under the irradiation direction shown in FIGS. 18A and 18B. As shown in FIG. 18C, the third sub-image Sc is composed of 6 pixels Pc acquired by 6 photodiodes 4p. As shown in the figure, the third sub-image Sc has information of the areas C1, C2, C3, C4, C5 and C6 shown in FIG. 15A in the entire subject 2. Here, for example, the region C1 is a region adjacent to the upper side of the region A1 in the subject 2 (see FIG. 15A).

FIG. 19A shows a state in which light rays are incident from an irradiation direction shown in FIGS. 16A and 16B, an irradiation direction shown in FIGS. 17A and 17B, and an irradiation direction different from the irradiation directions shown in FIGS. 18A and 18B. .. The light ray shown in FIG. 19A is inclined with respect to the z direction in a direction forming an angle of 45° with the x axis in the xy plane.

FIG. 19B schematically shows the image Sd (fourth sub-image Sd) acquired under the irradiation direction shown in FIG. 19A. As shown in FIG. 19B, the fourth sub-image Sd is composed of six pixels Pd acquired by the six photodiodes 4p. The fourth sub-image Sd has information of the areas D1, D2, D3, D4, D5 and D6 shown in FIG. 15A in the entire subject 2. Here, for example, the region D1 is a region adjacent to the right side of the region C1 (see FIG. 15A). As described above, each of the sub-images Sa, Sb, Sc, and Sd includes an image composed of different parts of the subject 2.

FIG. 20 shows a high resolution image HR composed of four sub images Sa, Sb, Sc and Sd. As shown in FIG. 20, the number of pixels or the pixel density of the high resolution image HR is four times the number of pixels or the pixel density of each of the four sub-images Sa, Sb, Sc and Sd.

For example, focus on the blocks of the areas A1, B1, C1, and D1 shown in FIG. 15A in the subject 2. As can be seen from the above description, the pixel Pa1 of the sub-image Sa shown in FIG. 20 has information on only the area A1 and not the entire block described above. Therefore, it can be said that the sub-image Sa is an image in which the information of the areas B1, C1, and D1 is missing.

However, by using the sub-images Sb, Sc, and Sd having the pixel information corresponding to different positions in the subject 2, as shown in FIG. 20, the missing information in the sub-image Sa is complemented and the information of the entire block is provided. It is possible to form a high resolution image HR. While the resolution of each sub-image is equal to the native resolution of the image sensor 4, in this example, a resolution four times the native resolution of the image sensor 4 is obtained. The degree of resolution enhancement (super resolution) depends on the aperture ratio of the image sensor. In this example, since the aperture ratio of the image sensor 4 is 25%, a maximum of four times higher resolution is achieved by light irradiation from four different directions. When N is an integer of 2 or more, if the aperture ratio of the image sensor 4 is approximately equal to 1/N, the resolution can be increased up to N times.

In this way, by sequentially irradiating parallel light from a plurality of different irradiation directions with respect to the subject to image the subject, it is possible to increase pixel information that is “spatially” sampled from the subject. .. By combining the obtained plurality of sub-images, it is possible to form a high-resolution image having a higher resolution than each of the plurality of sub-images. Of course, the irradiation direction is not limited to the irradiation direction described with reference to FIGS. 16A to 19B.

Note that, in the above example, the sub-images Sa, Sb, Sc, and Sd shown in FIG. 20 have pixel information of different regions in the subject 2 and have no overlap. However, there may be overlap between different sub-images. Further, in the above example, the light rays that have passed through the two adjacent regions in the subject 2 are both incident on the same photodiode. However, the setting of the irradiation direction is not limited to this example. For example, as shown in FIG. 21, the irradiation direction may be adjusted so that the light rays that have passed through two adjacent areas of the subject 2 are incident on different photodiodes, respectively.

<Module>
In the formation of the high-resolution image based on the principle described with reference to FIGS. 15A to 20, the acquisition of the sub-image is executed in the state where the subject 2 is arranged close to the imaging surface of the image sensor 4. In the embodiment of the present disclosure, a sub image is acquired using a module having a structure in which the subject 2 and the image sensor 4 are integrated. Hereinafter, an example of a configuration of a module and an example of a method for manufacturing a module will be described with reference to the drawings.

FIG. 22A schematically shows an example of the cross-sectional structure of the module. In the module 10 shown in FIG. 22A, the subject 2 covered with the encapsulant 6 is placed on the imaging surface 4A of the image sensor 4. In the illustrated example, a transparent plate (typically a glass plate) 8 is arranged on the subject 2. That is, in the configuration illustrated in FIG. 22A, the subject 2 is sandwiched between the image sensor 4 and the transparent plate 8. The module 10 having the transparent plate 8 is advantageous because workability is improved. As the transparent plate 8, for example, a general slide glass can be used. Note that each element is schematically shown in the drawings, and the actual size and shape of each element do not necessarily match the size and shape shown in the drawing. The same applies to the other drawings referred to below.

In the configuration illustrated in FIG. 22A, the image sensor 4 is fixed to the package 5. 22B shows an example of the external appearance of the module 10 shown in FIG. 22A when viewed from the image sensor 4 side. As shown in FIGS. 22A and 22B, the package 5 has a back surface electrode 5B on the surface opposite to the transparent plate 8. The back surface electrode 5B is electrically connected to the image sensor 4 via a wiring pattern (not shown) formed on the package 5. That is, the output of the image sensor 4 can be taken out through the back surface electrode 5B. In this specification, a structure in which the package and the image sensor are integrated is referred to as an “imaging element”.

An example of a method for manufacturing the module 10 will be described with reference to FIG. Here, a thin piece (tissue section) of biological tissue is exemplified as the subject 2. The module 10 having a thin piece of biological tissue as the subject 2 can be used for pathological diagnosis.

First, as shown in FIG. 23, the tissue section A02 is placed on the transparent plate 8. The transparent plate 8 may be a slide glass used for observing a sample with an optical microscope. Below, a slide glass is illustrated as the transparent plate 8. Next, the tissue section A02 is immersed in the staining solution Ss together with the transparent plate 8 to stain the tissue section A02. Next, the mounting medium 6 is applied onto the transparent plate 8 to cover the subject 2 obtained by staining the tissue section A02 with the mounting medium 6. The mounting medium 6 has a function of protecting the subject 2. Next, the image pickup device 7 is arranged on the subject 2 such that the image pickup surface of the image sensor 4 faces the subject 2. In this way, the module 10 is obtained.

The module 10 is produced for each imaging target. For example, in the case of pathological diagnosis, a plurality of (for example, 5 to 20) tissue sections are prepared from one sample. Therefore, a plurality of modules 10 having the tissue sections obtained from the same specimen as the subject 2 can be manufactured. By obtaining a plurality of sub-images for each of the plurality of modules 10, it is possible to form a high-resolution image corresponding to each of the plurality of modules 10.

As shown in FIG. 22A, the module 10 includes an image sensor 4 that acquires an image of the subject 2 unlike a slide used for observation with an optical microscope. Such a module may be called an “electronic preparation”. As shown in FIG. 22A, by using the module 10 having a structure in which the subject 2 and the image sensor 7 are integrated, there is an advantage that the arrangement between the subject 2 and the image sensor 4 can be fixed.

When the image of the subject 2 is acquired using the module 10, the subject 2 is irradiated with illumination light through the transparent plate 8. The illumination light transmitted through the subject 2 enters the image sensor 4. Thereby, the image of the subject 2 is obtained. By sequentially performing imaging while changing the relative arrangement of the light source and the subject, it is possible to change the angle and obtain a plurality of different images during irradiation. For example, as shown in FIG. 24A, the light source 310 is arranged directly above the image sensor 4. Then, if imaging is performed in a state where the collimated light CL is applied to the subject 2 from the normal direction of the imaging surface 4A of the image sensor 4, a sub image similar to the sub image Sa shown in FIG. 26C is obtained. Further, as shown in FIG. 24B, if the subject 2 is irradiated with the collimated light CL with the module 10 tilted and an image is taken, the sub-image Sb shown in FIG. 17C (or the sub-image Sc shown in FIG. 18C). Similar sub-images are obtained. In this way, by sequentially performing imaging while changing the attitude of the module 10 with respect to the light source, it is possible to obtain a high-resolution image by applying the principle described with reference to FIGS. 15A to 20. ..

<Image acquisition device>
FIG. 25 shows the outline of an example of the configuration of the image acquisition device according to the embodiment of the present disclosure. The image acquisition device 100a illustrated in FIG. 25 includes an illumination system 30. In the configuration illustrated in FIG. 25, the illumination system 30 is configured such that a light source 31 that generates illumination light, a stage 32 configured such that the module 10 is detachably mounted, and a posture of the stage 32 can be changed. It also includes a stage drive mechanism 33. FIG. 25 schematically shows a state in which the module 10 is loaded on the stage 32. However, illustration of the encapsulant 6 and the transparent plate 8 in the module 10 is omitted. The module 10 is not an essential component of the image acquisition device 100a.

The module 10 has an arrangement such that the illumination light transmitted through the subject 2 is incident on the image sensor 7 when connected to the stage 32. The illumination system 30 changes the irradiation direction with respect to the subject 2 by changing the posture of the stage 32, for example. The change of the "posture" in this specification broadly includes a change of the inclination with respect to the reference plane, a change of the rotation angle with respect to the reference azimuth, and a change of the position with respect to the reference point. The subject 2 is sequentially illuminated with the illumination light generated by the light source 31 from a plurality of different irradiation directions with the subject 2 as a reference. Details of the configuration of the lighting system 30 and an example of operation will be described later. By irradiating the subject 2 while changing the irradiation direction, a plurality of different images (sub-images) are acquired by the image sensor 7 according to a plurality of different irradiation directions. It is possible to form a high resolution image using the obtained plurality of images.

The image acquisition device 100a shown in FIG. 25 has an irradiation direction determination unit 40a. The irradiation direction determination unit 40a determines a plurality of different irradiation directions when the imaging element 7 acquires a plurality of sub-images. In the embodiment of the present disclosure, the acquisition of the sub-image is performed under the plurality of different irradiation directions determined by the irradiation direction determination unit. In other words, the sub-image in the embodiment of the present disclosure is a plurality of different images corresponding to the plurality of different irradiation directions determined by the irradiation direction determination unit. A specific example of the configuration and operation of the irradiation direction determination unit 40a will be described later.

Next, with reference to FIGS. 26A to 27B, an example of a method of changing the irradiation direction of the illumination light with respect to the subject will be described.

26A and 26B show an exemplary external appearance of the image acquisition device 100a. In the configuration illustrated in FIG. 26A, the image acquisition device 100a includes a main body 110 including the light source 31 and the stage 32, and a lid 120 that is openably and closably connected to the main body 110. By closing the lid 120, a dark room can be formed inside the image acquisition device 100a (see FIG. 26B).

In the illustrated example, a socket 130 for holding the module 10 is connected on the stage 32. The socket 130 may be fixed to the stage 32, or may be detachably attached to the stage 32. Here, a configuration in which the socket 130 is detachably attached to the stage 32 is illustrated. The socket 130 includes, for example, a lower base material 132 on which the module 10 is detachably configured, and an upper base material 134 on which an opening Ap is formed. In the configuration illustrated in FIG. 26A, the socket 130 holds the module 10 by sandwiching the module 10 between the lower base material 132 and the upper base material 134.

The lower substrate 132 may have an electrical connection portion having an electrical contact for electrical connection with the image pickup device 7 of the module 10. When the image of the subject is acquired, the module 10 is placed on the lower base material 132 so that the image pickup surface of the image pickup element 7 faces the light source 31. At this time, the electrical contact of the electrical connection portion and the back surface electrode 5B of the image pickup element 7 (see FIGS. 22A and 22B) come into contact with each other, so that the image pickup element 7 of the module 10 and the electrical connection portion of the lower base material 132 are separated from each other. It is electrically connected.

FIG. 26C shows an example of a method of loading the socket 130 on the stage 32 of the image acquisition apparatus 100a. In the configuration illustrated in FIG. 26C, the socket 130 has an electrode 136 protruding from the bottom surface. This electrode 136 may be part of the electrical connection of the lower substrate 132. Further, in the example shown in FIG. 26C, the stage 32 of the image acquisition device 100a has a mounting portion 34 provided with a jack 36. As shown in FIG. 26C, for example, the socket 130 holding the module 10 is mounted on the stage 32 such that the electrode 136 of the socket 130 is inserted into the jack 36. Thereby, the electrical connection between the image pickup device 7 in the module 10 held in the socket 130 and the image acquisition device 100a is established. The stage 32 may have circuitry to receive the output of the image sensor 4 with the socket 130 holding the module 10 loaded. In the embodiment of the present disclosure, the image acquisition device 100a acquires information (image signal or image data) indicating the image of the subject 2 with the electrical connection portion of the socket 130 as an intermediary.

When imaging a plurality of subjects using a plurality of modules 10, the same number of sockets 130 as the modules 10 are prepared, and the objects to be imaged are changed by replacing the sockets 130 holding the modules 10. Good. Alternatively, the object of imaging may be changed by replacing the module 10 with the single socket 130 attached to the stage 32.

As shown in FIG. 26C, by mounting the socket 130 on the stage 32, the bottom surface of the socket 130 and the top surface of the mounting portion 34 can be brought into close contact with each other. As a result, the arrangement of the socket 130 with respect to the stage 32 is fixed. Therefore, the arrangement of the stage 32 and the module 10 held by the socket 130 can be kept constant before and after the change of the posture of the stage 32. Typically, when the stage 130 is loaded with the socket 130, the main surface of the transparent plate 8 of the module 10 and the stage 32 are substantially parallel to each other.

FIG. 27A shows an example of a method for changing the irradiation direction. As illustrated, the module 10 held in the socket 130 is irradiated with the illumination light CL emitted from the light source 31. The illumination light CL enters the subject of the module 10 via the opening Ap provided in the socket 130. The light transmitted through the subject is incident on the image pickup surface of the image pickup device 7 of the module 10.

The light emitted from the light source 31 is typically collimated light. However, when the light incident on the subject can be regarded as substantially parallel light, the light emitted from the light source 31 may not be the collimated light.

The light source 31 includes, for example, an LED chip. The light source 31 may include a plurality of LED chips each having a peak in a different wavelength band. For example, the light source 31 may include an LED chip that emits blue light, an LED chip that emits red light, and an LED chip that emits green light. When a plurality of light emitting elements are arranged close to each other (for example, about 100 μm), they can be regarded as point light sources.

It is possible to obtain a plurality of sub-images for each color by using a plurality of light emitting elements that emit light of different colors and, for example, by sequentially irradiating light of different colors in each irradiation direction. .. For example, a set of blue sub-images, a set of red sub-images, and a set of green sub-images may be acquired. The acquired set of sub-images can be used to form a high resolution color image. For example, in a pathological diagnosis scene, more useful information regarding the presence or absence of a lesion can be obtained by using a high-resolution color image. By using a white LED chip as the light source 31 and arranging a color filter on the optical path, illumination lights of different colors may be obtained time-sequentially. An image sensor for color image pickup may be used as the image sensor 4. However, from the viewpoint of suppressing the reduction of the amount of light incident on the photoelectric conversion unit of the image sensor 4, it is more advantageous not to arrange the color filter.

The light source 31 is not limited to the LED, and may be an incandescent lamp, a laser element, a fiber laser, a discharge tube, or the like. The light emitted from the light source 31 is not limited to visible light, and may be ultraviolet light, infrared light, or the like. The number and arrangement of the light emitting elements included in the light source 31 can be set arbitrarily.

As shown in FIGS. 25 and 27A, the image acquisition device 100 a has a stage drive mechanism 33. The stage drive mechanism 33 includes a gonio mechanism, a rotation mechanism, and the like, and changes the tilt angle of the stage 32 with respect to the main body 110 and/or the rotation angle about an axis passing through the center of the stage 32. The stage drive mechanism 33 may include a slide mechanism capable of translating the stage 32 in a reference plane (typically a horizontal plane).

By operating the stage driving mechanism 33, the posture of the stage 32 can be changed. Here, since the socket 130 holding the module 10 is attached to the stage 32, the attitude of the module 10 can be changed by changing the attitude of the stage 32. For example, it is assumed that the incident direction of the illumination light when the stage 32 is not tilted with respect to the reference plane is the normal direction of the image pickup surface of the image sensor. Here, the relationship (for example, parallel) between the inclination of the stage 32 with respect to the reference surface and the inclination of the module 10 with respect to the reference surface (may be referred to as the inclination of the transparent plate 8) is the change in the attitude of the stage 32. It is kept constant before and after. Therefore, as shown in FIG. 27B, when the stage 32 is tilted by the angle θ with respect to the reference plane, the direction of the light beam incident on the subject is also tilted by the angle θ. Note that, in FIG. 27B, the broken line N indicates the normal line of the imaging surface of the image sensor.

In this way, by changing the attitude of the module 10 together with the stage 32, it is possible to sequentially illuminate the subject with illumination light from a plurality of different irradiation directions with the subject 2 as a reference. Therefore, a plurality of images corresponding to a plurality of different irradiation directions with the subject 2 as a reference can be acquired by the image sensor 7 of the module 10. The irradiation direction with respect to the subject 2 is, for example, the angle formed by the normal line N of the image pickup surface of the image sensor and the incident light ray to the subject 2 (zenith angle θ shown in FIG. 27B), and the reference set on the image pickup surface. It can be represented by a set of angles (azimuth angle) formed by the azimuth and the projection of the incident light ray on the imaging surface.

It is possible to illuminate the subject 2 from a plurality of different irradiation directions by moving the light source 31 in the image acquisition device 100a or sequentially lighting a plurality of light sources arranged at different positions. Is. For example, the irradiation direction may be changed by moving the light source 31 along the direction connecting the light source 31 and the subject 2. The irradiation direction may be changed by combining the change of the attitude of the stage 32 and the movement of the light source 31.

<Image sensor used in module>
In the embodiment of the present disclosure, the image sensor 4 is not limited to the CCD image sensor, and may be a CMOS (Complementary Metal-Oxide Semiconductor) image sensor or another image sensor (as an example, a photoelectric conversion film laminated type described later). Image sensor). The CCD image sensor and the CMOS image sensor may be either a front side illumination type or a back side illumination type. Hereinafter, the relationship between the element structure of the image sensor and the light incident on the photodiode of the image sensor will be described.

FIG. 28 shows a sectional structure of the CCD image sensor and an example of the distribution of the relative transmittance Td of the subject. As shown in FIG. 28, the CCD image sensor roughly includes a substrate 80, an insulating layer 82 on the substrate 80, and a wiring 84 arranged in the insulating layer 82. A plurality of photodiodes 88 are formed on the substrate 80. A light shielding layer (not shown in FIG. 28) is formed on the wiring 84. Illustration of transistors and the like is omitted here. Also in the following drawings, illustration of transistors and the like is omitted. In general, the sectional structure near the photodiode in the front-illuminated CMOS image sensor is substantially the same as the sectional structure near the photodiode in the CCD image sensor. Therefore, the illustration and description of the cross-sectional structure of the front-illuminated CMOS image sensor are omitted here.

As shown in FIG. 28, when the illumination light is incident from the direction normal to the imaging surface, the irradiation light that has passed through the region R1 of the subject immediately above the photodiode 88 is incident on the photodiode 88. On the other hand, of the subject, the irradiation light that has passed through the region R2 immediately above the light shielding layer on the wiring 84 is incident on the light shielding region (region where the light shielding film is formed) of the image sensor. Therefore, when the light is emitted from the normal direction of the image pickup surface, an image showing the region R1 immediately above the photodiode 88 of the subject is obtained.

In order to obtain an image showing a region immediately above the light-shielding film, irradiation is performed from a direction inclined with respect to the normal line direction of the imaging surface so that the light transmitted through the region R2 enters the photodiode 88. Good. At this time, depending on the irradiation direction, part of the light transmitted through the region R2 may be blocked by the wiring 84. In the illustrated example, light rays passing through the hatched portion do not reach the photodiode 88. Therefore, in oblique incidence, the pixel value may decrease somewhat. However, not all of the transmitted light is blocked, so that it is possible to form a high resolution image using the sub image obtained at this time.

29A and 29B show a cross-sectional structure of a backside illumination CMOS image sensor and an example of a distribution of relative transmittance Td of a subject. As shown in FIG. 29A, in the back-illuminated CMOS image sensor, the transmitted light is not blocked by the wiring 84 even in the case of oblique incidence. However, when light that has passed through another area of the subject different from the area to be imaged (light schematically indicated by a thick arrow BA in FIG. 29A and FIG. 29B described later) is incident on the substrate 80, noise is generated. May occur and the quality of the sub-image may deteriorate. Such deterioration can be reduced by forming the light shielding layer 90 on a region other than the region where the photodiode is formed on the substrate, as shown in FIG. 29B.

FIG. 30 is a cross-sectional structure of an image sensor including a photoelectric conversion film formed of an organic material or an inorganic material (hereinafter, referred to as “photoelectric conversion film stacked image sensor”) and a distribution of relative transmittance Td of a subject. And an example of.

As shown in FIG. 30, the photoelectric conversion film stacked image sensor is roughly composed of a substrate 80, an insulating layer 82 provided with a plurality of pixel electrodes, a photoelectric conversion film 94 on the insulating layer 82, and a photoelectric conversion film. It has a transparent electrode 96 on the conversion film 94. As shown in the figure, in the photoelectric conversion film laminated image sensor, a photoelectric conversion film 94 for performing photoelectric conversion is formed on a substrate 80 (for example, a semiconductor substrate) instead of the photodiode formed on the semiconductor substrate. The photoelectric conversion film 94 and the transparent electrode 96 are typically formed over the entire imaging surface. Here, the illustration of a protective film for protecting the photoelectric conversion film 94 is omitted.

In the photoelectric conversion film stack type image sensor, charges (electrons or holes) generated by photoelectric conversion of incident light in the photoelectric conversion film 94 are collected by the pixel electrode 92. As a result, a value indicating the amount of light incident on the photoelectric conversion film 94 is obtained. Therefore, in the photoelectric conversion film laminated image sensor, it can be said that the unit area including one pixel electrode 92 corresponds to one pixel on the imaging surface. In the photoelectric conversion film stack type image sensor, the transmitted light is not blocked by the wiring even in the case of oblique incidence, as in the backside illumination CMOS image sensor.

As described with reference to FIGS. 15A to 20, in forming a high-resolution image, a plurality of sub-images showing images composed of different parts of the subject are used. However, in a typical photoelectric conversion film laminated type image sensor, the photoelectric conversion film 94 is formed over the entire image pickup surface, so that, for example, even in the case of vertical incidence, light is transmitted through a region other than a desired region of the subject. Photoelectric conversion may occur in the photoelectric conversion film 94 by the generated light. If extra electrons or holes generated at this time are drawn into the pixel electrode 92, an appropriate sub-image may not be obtained. Therefore, it is useful to selectively draw the electric charges generated in the region where the pixel electrode 92 and the transparent electrode 96 overlap (the shaded region in FIG. 30) to the pixel electrode 92.

In the configuration illustrated in FIG. 30, dummy electrodes 98 are provided in the pixels so as to correspond to the respective pixel electrodes 92. When acquiring the image of the subject, an appropriate potential difference is applied between the pixel electrode 92 and the dummy electrode 98. As a result, the charges generated in the region other than the region where the pixel electrode 92 and the transparent electrode 96 overlap each other are drawn into the dummy electrode 98, and the charges generated in the region where the pixel electrode 92 and the transparent electrode 96 overlap selectively. Can be pulled into. The same effect can be obtained by patterning the transparent electrode 96 or the photoelectric conversion film 94. In such a configuration, it can be said that the ratio (S3/S1) of the area S3 of the pixel electrode 92 to the area S1 of the pixel corresponds to the “aperture ratio”.

As described above, when N is an integer of 2 or more, if the aperture ratio of the image sensor 4 is approximately equal to 1/N, the resolution can be increased up to N times. In other words, a smaller aperture ratio is more advantageous for higher resolution. In the photoelectric conversion film stack type image sensor, it is possible to adjust the ratio (S3/S1) corresponding to the aperture ratio by adjusting the area S3 of the pixel electrode 92. The ratio (S3/S1) is set in the range of 10% to 50%, for example. The photoelectric conversion film laminated image sensor having the ratio (S3/S1) within the above range can be used for super-resolution.

As can be seen from FIGS. 28 and 29B, the surface of the CCD image sensor and the front-illuminated CMOS image sensor facing the subject is not flat. For example, a CCD image sensor has a step on its surface. Further, in the back-illuminated CMOS image sensor, in order to obtain a sub-image for forming a high-resolution image, a patterned light-shielding layer needs to be provided on the imaging surface, and the surface facing the subject is flat. is not.

On the other hand, the image pickup surface of the photoelectric conversion film stack type image sensor is a substantially flat surface as can be seen from FIG. Therefore, even when the subject is arranged on the image pickup surface, deformation of the subject due to the shape of the image pickup surface hardly occurs. In other words, by acquiring the sub-image using the photoelectric conversion film stack type image sensor, a more detailed structure of the subject can be observed.

Although the image output device according to one or more aspects has been described above based on some embodiments, the present invention is not limited to these embodiments. Without departing from the spirit of the present invention, the present disclosure may include various modifications that can be conceived by those skilled in the art to the present embodiment, and embodiments configured by combining components in different embodiments. ..

In addition, in each of the above-described embodiments, each component may be configured by dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program (that is, an instruction) recorded in a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the image output device of each of the above-described embodiments is a program that causes a computer to execute each step in the flowcharts of FIGS. 3, 4, 7, 9, or 11. The program execution unit may be composed of one processor or plural processors.

Further, in the present disclosure, all or part of a unit or device, or all or part of the functional blocks of the block diagrams illustrated in FIGS. 1, 5, 6, 8, 10, and 12 to 14. Is a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration)
May be implemented by one or more electronic circuits including The LSI or IC may be integrated on one chip, or may be configured by combining a plurality of chips. For example, the functional blocks other than the memory element may be integrated in one chip. Although referred to as an LSI or an IC here, they may be called system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration) depending on the degree of integration. A Field Programmable Gate Array (FPGA) that is programmed after the manufacture of the LSI, or a reconfigurable logic device that can reconfigure the bonding relation inside the LSI or set up circuit sections inside the LSI can be used for the same purpose.

Furthermore, all or some of the functions or operations of a unit, a device, or a part of a device can be performed by software processing. In this case, the software may be recorded on a non-transitory recording medium such as one or more ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the software Causes a processor and peripheral devices to perform specific functions within the software. The system or apparatus may include one or more non-transitory recording media on which software is recorded, a processor, and required hardware devices, such as interfaces.

The present disclosure has the effect of reducing the burden of handling high-resolution images, and can be applied to, for example, an image output device that handles high-resolution images of pathological samples.

100S, 100Sa, 200S, 200Sa, 200Sb Image processing system 1001, 1001a, 2001, 2001a Image output device 1100, 1100a, 1100b Image transmission device 1101 Image acquisition unit 1102 High resolution image acquisition unit 1103 Second output unit 1200, 1200a, 1200b Image receiving apparatus 1201 Enlargement receiving section 1202 Judgment section 1203 Display output section 1204, 1204a Transmission section 1205 First output section 1206 Evaluation value storage section 1500 Digital microscope 1501 Display apparatus 1502, 1503 Recording medium

Claims (11)

An image output device connected to a display device and connected to an external device via a communication line,
At least one processor,
And a non-transitory recording medium holding executable instructions,
The command is
Acquire the first resolution image, which is the image of the subject obtained based on the shooting with a digital microscope,
An image having a higher resolution than the first resolution image, and obtaining a second resolution image that is an image of the subject obtained based on photographing by the digital microscope,
Displaying the first resolution image on the display device,
Accepting the magnification for said first resolution image displayed on the display device,
An evaluation value based on the received enlargement ratio is determined whether it is higher than a predetermined value,
If the evaluation value is when it is determined to be higher than the predetermined value, it transmits the second resolution image to the external device, wherein the evaluation value is determined to not higher than the predetermined value Does not send the second resolution image to the external device,
In the judgment of the evaluation value,
(A) In the reception of the enlargement ratio, the larger the number of times the high enlargement ratio, which is an enlargement ratio higher than the threshold, is received, (b) the first resolution image is displayed on the display device at the high enlargement ratio. The longer the time is, or (c) the larger the area of the first resolution image displayed on the display device at the high magnification is, the higher the evaluation value is derived.
An image output device that causes the at least one processor to perform the above. The instructions further include
When it is determined that the evaluation value is not higher than the predetermined value, the first resolution image is transmitted to the external device,
The image output apparatus according to claim 1, wherein the image output apparatus causes the at least one processor to execute the following. The instructions further include
When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is output to a recording medium and saved, and it is determined that the evaluation value is not higher than the predetermined value. In that case, the second resolution image is not output to the recording medium,
The image output apparatus according to claim 1, wherein the image output apparatus causes the at least one processor to execute the following. An image output device having an image transmitting device and an image receiving device connected to the image transmitting device via a communication line,
The image transmission device,
At least one transmitting processor,
And a non-transitory transmission recording medium that holds an executable transmission command,
The transmission instruction is
Acquiring a first resolution image, which is an image of a subject obtained based on photographing by a digital microscope, and transmitting it to the image receiving device,
An image having a higher resolution than the first resolution image, and obtaining a second resolution image that is an image of the subject obtained based on photographing by the digital microscope,
The evaluation value derived by the evaluation value related information notified from the image receiving device, it is determined whether or not higher than a predetermined value,
When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device, and it is determined that the evaluation value is not higher than the predetermined value. Does not transmit the second resolution image to the image receiving device,
And causing the at least one transmitting processor to:
The image receiving device is a device connected to a display device,
At least one receiving processor;
A non-transitory recording medium for holding an executable receiving command,
The receiving instruction is
Wherein the image transmitting apparatus to obtain the first-resolution image is displayed on the display device,
Accepting a magnification rate for the first resolution image displayed on the display device,
Enlarging the first resolution image displayed on the display device based on the accepted enlargement ratio,
Transmitting the evaluation value related information based on the received enlargement ratio to the image transmitting device,
And causing the at least one receiving processor to:
The evaluation value related information is
(A) The number of times a high magnification rate, which is a magnification rate higher than a threshold value, is accepted in the acceptance of the magnification rate, and (b) the time during which the first resolution image is displayed on the display device at the high magnification rate. Or (c) shows the area of the first resolution image displayed on the display device at the high magnification ratio,
In the judgment of the evaluation value,
(A) When the evaluation value related information indicates the number of times, the greater the number of times, (b) When the evaluation value related information indicates the time, the longer the time, or (c ) When the evaluation value related information indicates the area, the higher the area, the higher the evaluation value is derived.
Image output device. An image output device having an image transmitting device and an image receiving device connected to the image transmitting device via a communication line,
The image transmission device,
At least one transmitting processor,
And a non-transitory transmission recording medium that holds an executable transmission command,
The transmission instruction is
Acquiring a first resolution image, which is an image of a subject obtained based on photographing by a digital microscope, and transmitting it to the image receiving device,
Acquiring a second resolution image, which is an image having a higher resolution than the first resolution image, and which is an image of the subject obtained based on photographing by the digital microscope,
And causing the at least one transmitting processor to:
The image receiving device is a device connected to a display device,
At least one receiving processor;
A non-transitory recording medium for holding an executable receiving command,
The receiving instruction is
Wherein the image transmitting apparatus to obtain the first-resolution image is displayed on the display device,
Accepting the magnification for said first resolution image displayed on the display device,
Enlarging the first resolution image displayed on the display device based on the accepted enlargement ratio,
An evaluation value based on the received enlargement ratio is determined whether or not it is higher than a predetermined value, and the determination result is notified to the image transmission device,
In the judgment of the evaluation value,
(A) In the reception of the enlargement ratio, the larger the number of times the high enlargement ratio, which is an enlargement ratio higher than the threshold, is received, (b) the first resolution image is displayed on the display device at the high enlargement ratio. The longer the time is, or (c) the larger the area of the first resolution image displayed on the display device at the high magnification is, the higher the evaluation value is derived.
And causing the at least one receiving processor to:
The transmission instruction of the image transmission device further includes
When the determination result notified from the image receiving device indicates that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device, and the evaluation value is If it indicates that the second resolution image is not higher than a predetermined value, the second resolution image is not transmitted to the image receiving device,
An image output device that causes the at least one transmission processor to perform the above. The transmission instruction further comprises:
In the determination of the evaluation value, if it is determined that the evaluation value is not higher than the predetermined value, the second resolution image is output and stored on a recording medium,
The image output apparatus according to claim 4 or 5 to be executed by the at least one sending processor that. An image output method performed by an image output device connected to a display device and connected to an external device via a communication line,
Acquire the first resolution image, which is the image of the subject obtained based on the shooting with a digital microscope,
An image having a higher resolution than the first resolution image, and obtaining a second resolution image that is an image of the subject obtained based on photographing by the digital microscope,
Displaying the first resolution image on the display device,
Accepting the magnification for said first resolution image displayed on the display device,
An evaluation value based on the received enlargement ratio is determined whether it is higher than a predetermined value,
If the evaluation value is when it is determined to be higher than the predetermined value, it transmits the second resolution image to the external device, wherein the evaluation value is determined to not higher than the predetermined value Does not send the second resolution image to the external device ,
In the judgment of the evaluation value,
(A) In the reception of the enlargement ratio, the larger the number of times the high enlargement ratio, which is an enlargement ratio higher than the threshold, is received, (b) the first resolution image is displayed on the display device at the high enlargement ratio. The longer the time is, or (c) the larger the area of the first resolution image displayed on the display device at the high magnification is, the higher the evaluation value is derived.
Image output method. A computer-readable non-transitory recording medium in which a program for a computer of an image output device connected to a display device and connected to an external device via a communication line is recorded.
The program is
Acquire the first resolution image, which is the image of the subject obtained based on the shooting with a digital microscope,
An image having a higher resolution than the first resolution image, and obtaining a second resolution image that is an image of the subject obtained based on photographing by the digital microscope,
Displaying the first resolution image on the display device,
Accepting the magnification for said first resolution image displayed on the display device,
An evaluation value based on the received enlargement ratio is determined whether it is higher than a predetermined value,
If the evaluation value is when it is determined to be higher than the predetermined value, it transmits the second resolution image to the external device, wherein the evaluation value is determined to not higher than the predetermined value Does not send the second resolution image to the external device ,
In the judgment of the evaluation value,
(A) In the reception of the enlargement ratio, the larger the number of times the high enlargement ratio, which is an enlargement ratio higher than the threshold, is received, (b) the first resolution image is displayed on the display device at the high enlargement ratio. The longer the time is, or (c) the larger the area of the first resolution image displayed on the display device at the high magnification is, the higher the evaluation value is derived.
A recording medium that causes a computer to execute the above. An image receiving device connected to a display device and connected to an image transmitting device via a communication line,
The image transmission device,
At least one transmitting processor,
And a non-transitory transmission recording medium that holds an executable transmission command,
The transmission instruction is
Acquiring a first resolution image, which is an image of a subject obtained based on photographing by a digital microscope, and transmitting the image to the image receiving device,
An image having a higher resolution than the first resolution image, and obtaining a second resolution image that is an image of the subject obtained based on photographing by the digital microscope,
The evaluation value derived from the evaluation value related information notified from the image receiving device determines whether or not it is higher than a predetermined value,
When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device, and it is determined that the evaluation value is not higher than the predetermined value. Does not transmit the second resolution image to the image receiving device,
And causing the at least one transmitting processor to:
The image receiving device,
At least one receiving processor;
A non-transitory recording medium for holding an executable receiving command,
The receiving instruction is
Wherein the image transmitting apparatus to obtain the first-resolution image is displayed on the display device,
Accepting a magnification rate for the first resolution image displayed on the display device,
Enlarging the first resolution image displayed on the display device based on the accepted enlargement ratio,
Transmitting the evaluation value related information based on the received enlargement ratio to the image transmitting device,
And causing the at least one receiving processor to :
The evaluation value related information is
(A) In the acceptance of the enlargement ratio, the number of times a high enlargement ratio, which is an enlargement ratio higher than a threshold, is received, and (b) the time during which the first resolution image is displayed on the display device at the high enlargement ratio. Or (c) indicates the area of the first resolution image displayed on the display device at the high magnification ratio,
In the judgment of the evaluation value,
(A) When the evaluation value related information indicates the number of times, the greater the number of times, (b) When the evaluation value related information indicates the time, the longer the time, or (c ) When the evaluation value related information indicates the area, the larger the area is, the higher the evaluation value is derived.
Image receiving device. An image receiving device connected to a display device and connected to an image transmitting device via a communication line,
The image transmission device,
At least one transmitting processor,
And a non-transitory transmission recording medium that holds an executable transmission command,
The transmission instruction is
Acquiring a first resolution image, which is an image of a subject obtained based on photographing by a digital microscope, and transmitting the image to the image receiving device,
Acquiring a second resolution image, which is an image having a higher resolution than the first resolution image, which is an image of the subject obtained based on photographing by the digital microscope,
And causing the at least one transmitting processor to:
The image receiving device,
At least one receiving processor;
A non-transitory recording medium for holding an executable receiving command,
The receiving instruction is
Wherein the image transmitting apparatus to obtain the first-resolution image is displayed on the display device,
Accepting the magnification for said first resolution image displayed on the display device,
Enlarging the first resolution image displayed on the display device based on the accepted enlargement ratio,
An evaluation value based on the received enlargement ratio is determined whether or not it is higher than a predetermined value, and the determination result is notified to the image transmission device ,
In the judgment of the evaluation value,
(A) In the reception of the enlargement ratio, the larger the number of times the high enlargement ratio, which is an enlargement ratio higher than the threshold, is received, (b) the first resolution image is displayed on the display device at the high enlargement ratio. The longer the time is, or (c) the larger the area of the first resolution image displayed on the display device at the high magnification is, the higher the evaluation value is derived.
And causing the at least one receiving processor to:
The transmission instruction of the image transmission device further includes
When the determination result notified from the image receiving device indicates that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device, and the evaluation value is If the second resolution image is not higher than a predetermined value, the second resolution image is not transmitted to the image receiving device,
And causing the at least one transmitting processor to perform the above.
Image receiving device. An image receiving device connected to a display device, and an image transmitting device connected via a communication line,
The image receiving device,
At least one receiving processor;
A non-transitory recording medium for holding an executable receiving command,
The receiving instruction is
The first resolution image of a subject obtained based on the photographing by a digital microscope, is displayed on the display device is obtained from the image transmitting apparatus,
Accepting a magnification rate for the first resolution image displayed on the display device,
Enlarging the first resolution image displayed on the display device based on the accepted enlargement ratio,
Transmitting evaluation value related information based on the received enlargement ratio to the image transmitting device,
And causing the at least one receiving processor to:
The image transmission device,
At least one transmitting processor,
And a non-transitory transmission recording medium that holds an executable transmission command,
The transmission instruction is
Acquiring the first resolution image and transmitting it to the image receiving device,
An image having a higher resolution than the first resolution image, and obtaining a second resolution image that is an image of the subject obtained based on photographing by the digital microscope,
Evaluation value derived by the evaluation value related information notified from the image receiving device, determines whether or not higher than a predetermined value,
When it is determined that the evaluation value is higher than the predetermined value, the second resolution image is transmitted to the image receiving device, and it is determined that the evaluation value is not higher than the predetermined value. Does not transmit the second resolution image to the image receiving device,
And causing the at least one transmitting processor to :
The evaluation value related information is
(A) The number of times a high magnification rate, which is a magnification rate higher than a threshold value, is accepted in the acceptance of the magnification rate, and (b) the time during which the first resolution image is displayed on the display device at the high magnification rate. Or (c) indicates the area of the first resolution image displayed on the display device at the high magnification ratio,
In the judgment of the evaluation value,
(A) When the evaluation value related information indicates the number of times, the greater the number of times, (b) When the evaluation value related information indicates the time, the longer the time, or (c ) When the evaluation value related information indicates the area, the larger the area is, the higher the evaluation value is derived.
Image transmission device.

Images