JP4709571B2 - Visual information processing system and visual information processing method thereof - Google Patents

Description

  The present invention relates to a visual information processing system and a visual information processing method thereof, and more particularly to a visual information processing system using a spatial light modulation element and a high-speed imaging element and a visual information processing method thereof.

  Today, imaging devices and cameras using solid-state imaging devices such as CMOS image sensors are spreading their application fields in various forms regardless of consumer use or industrial use.

  In particular, in the field of industrial imaging devices, application forms in which physical fluctuations of an object are extracted from image data and the extracted information is used as input values and feedback values for various control systems have become widespread.

  For example, a system that extracts a physical quantity such as the position of the center of gravity of a moving object and the posture angle of the moving object by imaging a moving object, and performs various controls, for example, control of a robot arm, based on the physical quantity or a change amount thereof. is there. In an imaging device used in such a control system, although it depends on the moving speed of an object, in general, it is possible to realize a much higher resolution and a higher frame rate in real time than a consumer application. It becomes necessary.

  The imaging camera device disclosed in Patent Document 1 discloses a technology that achieves both high spatial resolution and high-speed real-time performance. FIG. 15 illustrates a configuration of an imaging camera device 200 disclosed in Patent Document 1, and includes an imager (solid-state imaging device) 20 having a high spatial resolution, a calculation unit 30, and a coordinate conversion unit 40. .

  Image data (image data at the current frame time) of the partial image (m × n) specified by the coordinate conversion unit 40 is cut out from the entire image (M × N) of the imager 20 to the arithmetic unit 30. Is input. The calculation unit 30 performs a calculation for extracting the image feature amount ξ from the partial image data. The image feature amount ξ is, for example, the size, position, posture angle, etc. of the object included in the partial image data, or the amount of change thereof. The coordinate conversion unit 40 calculates the coordinates (m ′ × n ′) of the partial image to be cut out at the next frame time based on the extracted image feature amount ξ. For example, when the object is moving in parallel or rotating, the coordinates (m × n) of the partial image of the previous frame are subjected to coordinate conversion for parallel movement or rotation by affine transformation or the like, and the next frame The coordinates (m ′ × n ′) of the partial image are calculated.

  The calculated coordinates (m ′ × n ′) of the next frame are sequentially read from all the images of the imager, and image data of a new partial image (m ′ × n ′) is input to the arithmetic unit 30.

  In the imaging camera device 200 disclosed in Patent Document 1, the image feature amount ξ calculated by the calculation unit 30 is fed back to read out a necessary partial image instead of reading out all image data of the imager. This makes it possible to achieve high-speed real-time performance while maintaining high spatial resolution.

On the other hand, a technique for imaging an imaging object not only as a two-dimensional image but also as a three-dimensional image is being developed today. For example, Non-Patent Document 1 discloses a method for generating a three-dimensional image based on the triangulation principle.
JP 2003-319262 A “Outline of Image Processing / Experiment”, [online], [Search April 1, 2005], Internet <URL: https://www.image.esys.tsukuba.ac.jp/range_finder/gaiyo.html>

  By the way, recently, a device called a light spatial modulation element has been developed and put into practical use, and has been widely used in projector apparatuses and the like. There are two types of spatial light modulators: a transmissive type such as a liquid crystal light valve, and a reflective type called DMD (Digital Micro-mirror Device).

  In a projector apparatus using a transmissive light spatial modulation element, an image signal is applied to a liquid crystal light valve, and light from a light source is transmitted through the liquid crystal light valve to project an image on a front screen or the like.

  On the other hand, a projector apparatus using a reflective spatial modulation element projects an image on a front screen or the like after reflecting light from a light source to a DMD. The DMD has a large number of minute mirrors corresponding to the number of pixels. By changing the angle of the mirror for each pixel, the amount of light projected on the screen is turned on and off for each pixel, and a two-dimensional image is displayed. To form. Although the change in the mirror angle changes in a binary manner depending on the on / off signal, the continuous light intensity is expressed for the human eye by setting the on / off period by pulse width modulation. Can do. Further, by changing the duty ratio of the pulse width modulation for each pixel and each frame, a continuous moving image can be formed both spatially and temporally.

  Conventionally, light spatial modulation elements such as liquid crystal light valves and DMDs are often used exclusively for projector apparatuses. With the development of high-brightness and high-speed system technology, it is not only used as a projector for presentations, but is also widely used as a projector for large-sized TVs or theater movies. Is used as an image forming element of a projector apparatus.

  The visual information processing system according to the present invention does not use these light spatial modulation elements as conventional image forming elements.

  Image formation (imaging) itself is performed by a solid-state image sensor such as a CMOS image sensor. At the same time, spatial and temporal modulation is performed on the light from the object incident on the solid-state imaging device by using an optical spatial modulation device such as a liquid crystal light valve or DMD.

  An object of the visual information processing system according to the present invention is to realize new visual information processing that has not been obtained in the past by combining a light spatial modulation element and a solid-state imaging element.

  In particular, an image feature amount is extracted from image data picked up by an imager (solid-state image pickup device) as in the technique disclosed in Patent Document 1, and an optical spatial modulation device is provided in a feedback loop based on the image feature amount. Thus, the object is to realize new visual information processing adapted to the movement and change of the object.

  Specifically, one of the objects of the present invention is to combine a spatial light modulation element and an imager (solid-state image sensor), thereby feeding back a three-dimensional image of an imaging object including a moving object based on an image feature amount. An object of the present invention is to provide a visual information processing system and a visual information processing method thereof that can be generated at high speed and with high efficiency by control.

  Another object of the present invention is to combine a light spatial modulation element and an imager so that the difference between brightness and darkness is very large, and it is also suitable for an object whose difference changes temporally and spatially. An object of the present invention is to provide a visual information processing system and a visual information processing method thereof that can ensure a dynamic range of brightness.

  Another object of the present invention is to provide a specific fixed or moving object while providing high-sensitivity image data by sacrificing resolution to some extent as a whole image by combining a spatial light modulator and an imager. Is to provide a visual information processing system and its visual information processing method capable of realizing high resolution.

  Another object of the present invention is to provide a visual information processing system capable of extracting a predetermined luminance variation for each pixel and generating a luminance variation spectrum image by combining a spatial light modulator and an imager, and the visual information It is to provide an information processing method.

  Furthermore, another object of the present invention is to provide visual information that can perform spatial filter processing such as edge enhancement on an entire image or a specific fixed or moving object by combining a spatial light modulator and an imager. To provide a processing system and a visual information processing method thereof.

In addition, the visual information processing system according to the present invention groups pixels in a specified area into a plurality of groups, and turns on / off the plurality of pixel signals in each group at different frequencies, thereby making incident light from an imaging target object. A spatial light modulator that reflects or transmits light, and incident light from the imaging object output from the spatial light modulator is captured by an imager having the same resolution as the spatial light modulator, An image feature extraction unit that extracts a feature amount of the imaging target, wherein the image feature extraction unit designates a region based on the feature amount, and changes luminance variation of the imaging target in the region at the different frequency. The frequency decomposition is performed in response to the above, and a spectrum image corresponding to the different frequency is generated. According to the visual information processing system of this configuration, it is possible to extract a predetermined luminance variation for each pixel and generate a luminance variation spectrum image.

In addition, the visual information processing method according to the present invention performs imaging by turning on and off a plurality of pixel signals in the group at different frequencies by a spatial light modulation element in which pixels in a specified region are grouped into a plurality of groups. Reflecting or transmitting incident light from the object and outputting the incident light from the imaging object output from the light spatial modulation element with an imager having the same resolution as the resolution of the light spatial modulation element And extracting the feature amount of the imaging target, wherein the step of extracting the feature amount specifies a region based on the feature amount, and the luminance variation of the imaging target is determined in the region. It includes a process of generating a spectrum image corresponding to the different frequency by performing frequency decomposition corresponding to the different frequency. According to the visual information processing method according to the present method, it is possible to extract a predetermined luminance variation for each pixel and generate a luminance variation spectrum image.

  According to the visual information processing system and the visual information processing method according to the present invention, by combining the light spatial modulation element and the imager, it is possible to realize new visual information processing that has not been obtained conventionally, In particular, by providing an optical spatial modulation element in the feedback loop based on the image feature amount, it is possible to realize new visual information processing adapted to the movement and change of the object.

  Embodiments of a visual information processing system and a visual information processing method according to the present invention will be described with reference to the accompanying drawings.

(1) 1st Embodiment FIG. 1: is a figure which shows the system configuration example of the visual information processing system 1 which concerns on the 1st Embodiment of this invention. The visual information processing system 1 according to the first embodiment generates a three-dimensional image of an imaging object at high speed and with high efficiency.

  The visual information processing system 1 includes an image feature extraction unit 2 and a light pattern irradiation unit 3.

  Among these, the image feature extraction unit 2 has the same configuration as that of the imaging camera device 200 shown in FIG. 15, and includes an imager (solid-state imaging device) 20 having a high spatial resolution, a calculation unit 30, and coordinate conversion. Part 40. The imager 20 is a CMOS image sensor having a predetermined number of pixels, for example.

  Image data (image data at the current frame time) of the partial image (m × n) designated by the coordinate conversion unit 40 is extracted from the entire image (M × N) of the imager 20 in the calculation unit 30. Is input. The calculation unit 30 performs a calculation for extracting the image feature amount ξ from the partial image data. The image feature amount ξ is, for example, the size, position, posture angle, etc. of the object included in the partial image data, or the amount of change thereof. The coordinate conversion unit 40 calculates the coordinates (m ′ × n ′) of the partial image to be cut out at the next frame time based on the extracted image feature amount ξ. For example, when the object is moving in parallel or rotating, the coordinates (m × n) of the partial image of the previous frame are subjected to coordinate conversion for parallel movement or rotation by affine transformation or the like, and the next frame The coordinates (m ′ × n ′) of the partial image are calculated.

  The calculated coordinates (m ′ × n ′) of the next frame are sequentially read from all the images of the imager, and image data of a new partial image (m ′ × n ′) is input to the arithmetic unit 30.

  In the imaging camera device 200 disclosed in Patent Document 1, the image feature amount ξ calculated by the calculation unit 30 is fed back to read out a necessary partial image instead of reading out all image data of the imager. This makes it possible to achieve high-speed real-time performance while maintaining high spatial resolution.

  On the other hand, the light pattern irradiation unit 3 includes a light spatial modulation element 10, a light source 12, and a light pattern generation unit 11.

  The light pattern generation unit 11 generates various light pattern signals based on the image feature quantity ξ fed back from the image feature extraction unit 2, and outputs the light pattern signal to the light spatial modulation element 10.

  The spatial light modulator 10 generates various light patterns by modulating the radiated light from the light source 12 based on the light pattern signal. The spatial light modulator 10 is roughly divided into a reflective spatial light modulator 10a and a transmissive spatial light modulator 10b. Examples of the reflective spatial light modulator 10a include a digital micromirror device (hereinafter referred to as DMD). Further, as the transmissive light spatial modulation element 10b, for example, there is a transmissive liquid crystal light valve.

  The spatial light modulation element 10 of either the reflection type or the transmission type has a large number of pixels on the reflection surface or transmission surface, and the intensity of the reflection light or transmission light can be controlled for each pixel. Also, any of the spatial light modulators 10 can change the light intensity control for each pixel at a high speed in terms of time. In this sense, the spatial light modulator 10 can also be called a spatiotemporal modulator.

  An operation of generating a three-dimensional image of the visual information processing system 1 configured as described above will be described.

  FIG. 2 is a first diagram illustrating the operation concept of the three-dimensional image generation of the visual information processing system 1. First, the emitted light from the light source 12 is spatially modulated by the light spatial modulation element 10 to generate a predetermined light pattern, and the imaging object 100 is irradiated with this light pattern via the optical system.

  When the light spatial modulation element 10 is a reflection type, the light source 12 may be disposed on the front side of the light spatial modulation element 10, and when the light spatial modulation element 10 is a transmission type, the light source 12 may be disposed on the rear side of the light spatial modulation element 10. .

  The light pattern illustrated in FIG. 2 shows a state in which the imaging object 100 is irradiated with a vertical stripe pattern composed of a plurality of vertical stripes.

  An image from the imaging object 100 is incident on the imager 20 via the optical system. At this time, an image in which the imaging object 100 and the light pattern are superimposed is incident as incident light. Image data captured by the imager 20 is input to the calculation unit 30 of the image feature extraction unit 2.

  The computing unit 30 generates a three-dimensional image from a plurality of image data on which light patterns are superimposed. At this time, the image data extracted from the imager 20 may be the entire pixel area of the imager 20 or image data obtained by extracting a partial area of the imager 20 as described above. When extracting image data of a partial area, it is possible to perform processing at a higher speed (high frame rate) than when extracting image data of all pixel areas.

  In the calculation unit 30, an image feature amount ξ is further extracted. The image feature amount ξ is, for example, the position (center of gravity position), size, posture angle, etc. of the imaging object. In addition, these are the amount of change (speed).

  By feeding back the image feature quantity ξ and adaptively controlling the type and number of light patterns, a three-dimensional image can be generated at a higher speed and with higher accuracy than the conventional three-dimensional image generation method.

  FIG. 3 is a diagram for explaining a conventional method for generating a three-dimensional image. The details are described in, for example, Non-Patent Document 1 and the like, so only a brief description is given here.

  First, in the example of FIG. 3, k types and k different light patterns 300 are prepared as one set, and these light patterns are sequentially irradiated onto the imaging object 100. The imager 20 acquires k pieces of image data in which the light pattern and the imaging object are superimposed. One k-dimensional image (not shown) is obtained by processing the k pieces of image data.

  In the case of continuous imaging, the above process is repeated. In the conventional method, the same k types and k light patterns are irradiated to the imaging target for each frame.

  The types and number of light patterns depend on the size and resolution (spatial resolution) of the imaging object. For example, the set of light patterns shown in FIG. 3 is a striped pattern having the two most vertical stripes, which is the coarsest interval (this is called the coarsest interval), and the most dense interval (this is called the closest interval). In this pattern, the vertical stripes are successively halved.

  In this case, the coarsest interval needs to substantially match the size of the object to be imaged. On the other hand, the closest interval substantially matches the spatial resolution. Therefore, when imaging an imaging object of the same size, if a high spatial resolution is to be obtained, a light pattern with a small close interval is required, resulting in an increase in the types and number of light patterns. Even in the case of the same resolution, the larger the imaging target or the wider the imaging target range, the larger the coarsest interval needs to be increased. In this case also, the types and number of light patterns increase. become.

  In the conventional 3D image generation method, a uniform light pattern is generated. Therefore, when the information of the imaging target is unknown, or when the imaging target is changed, it is optimal. It was difficult to obtain an optical pattern.

  In contrast, the visual information processing system 1 according to the present embodiment is configured to be able to extract the image feature quantity ξ as shown in FIG. It is possible to adaptively generate (adaptive control) a light pattern that optimizes the evaluation function.

  FIG. 4 schematically shows the concept of light pattern generation by the visual information processing system 1 according to the present embodiment. For example, the imager 20 generates a plurality of image data 301 based on the light pattern 300 of the Nth frame, and generates a three-dimensional image of the Nth frame based on this.

  Based on this three-dimensional image, the image feature quantity ξ is extracted by the image feature extraction unit 2. A light pattern that optimizes the evaluation function can be generated using the image feature amount ξ.

  For example, when the processing time is set as the evaluation function and the processing time is minimized, the type and number of light patterns may be minimized within a range in which a predetermined resolution is maintained. At this time, by feeding back the position and size of the imaging object as the image feature amount ξ, it becomes possible to reduce the coarsest interval of the light pattern from a wide range to the size range of the imaging object, As a result, the type and number of light patterns are reduced, and the processing time is shortened.

  Further, in the conventional three-dimensional image generation method, since there are many types and numbers of light patterns, it is difficult to generate an accurate three-dimensional image due to changes in the frame when the imaging target moves. On the other hand, in the visual information processing system 1 according to the present embodiment, as described above, the type and number of light patterns are reduced and the frame time is shortened. A three-dimensional image can be generated or three-dimensional information can be extracted.

  Furthermore, in the visual information processing system 1 according to the present embodiment, it is possible to detect the presence / absence of movement of the imaging target, the moving speed, etc. as the image feature amount ξ. Flexible processing is possible.

  FIG. 5 shows an example of processing for a dynamic imaging object. FIG. 5A shows a case where there is no movement in the entire image. In this case, even if the frame time is somewhat long, there is little error due to the movement of the object, so that it is possible to use a light pattern with high spatial resolution in which the closest interval is reduced.

  On the other hand, FIG. 5B shows a situation where the moving object 400 appears in a part of the imaging range. The image feature extraction unit 2 can extract the position, size, speed, and the like of the moving object 400 as the image feature amount ξ. By feeding back these image feature amounts ξ to the light pattern generation unit 3, it is possible to generate a light pattern having a different pattern only in the region 401 (region with motion) around the moving object 400. For example, the light pattern is changed to a light pattern in which the density of vertical stripes in a region with movement is made coarser than the surrounding region.

  In the light pattern before the change, since the interval between the vertical stripes is too large compared to the movement amount, it is difficult to continuously generate a three-dimensional image of the moving object 400. On the other hand, in the light pattern after the change, the spatial resolution is slightly reduced by roughening the interval between the vertical stripes in the region 401 around the moving object 400, but three-dimensional measurement in continuous frames is possible. In addition, the high spatial resolution can be maintained for the peripheral area where there is no movement, as before the change.

  In generation of a light pattern using the light spatial modulation element 10, a light pattern having an arbitrary shape can be formed in principle on a two-dimensional pixel region. Therefore, it is possible to form not only a vertical stripe pattern that is uniform in the entire area as in the prior art but also a pattern of one or a plurality of partial areas in a form different from the surrounding pattern. It is also possible to follow the movement and move it.

(2) Second Embodiment FIG. 6 is a diagram showing a relationship between an optical spatial modulation element and an imager commonly used in visual information processing systems 1a to 1d according to second to fifth embodiments of the present invention. It is. The spatial light modulator in the first embodiment is used as a form for spatially modulating the emitted light from the light source. On the other hand, the spatial light modulators in the second to fifth embodiments are configured to spatially modulate the incident light from the imaging object as shown in FIG. Also in this case, as in the first embodiment, two forms of a reflection type and a transmission type can be taken. FIG. 6A shows a form using a reflective spatial light modulator 10a, for example, DMD. FIG. 6B shows a form using a transmissive spatial light modulator 10b, for example, a transmissive liquid crystal light valve. In any form, incident light from the imaging object 100 is modulated (spatial modulation) for each pixel by the spatial light modulation elements 10a and 10b, and then output to the imager 20.

  In the following description, a transmissive spatial light modulator is used in the drawings, but this is for avoiding duplication of description, and does not exclude a reflective spatial light modulator.

  FIG. 7 is a diagram illustrating a configuration example of the visual information processing system 1a according to the second embodiment. The visual information processing system 1a according to the second embodiment is configured to be able to control the exposure time of incident light from the reflecting object 100 for each pixel.

  The visual information processing system 1a includes a light spatial modulation element 10, an image feature extraction unit 2a, and an exposure time control unit 15.

  The spatial light modulator 10 controls the exposure time for each pixel based on a control signal from the exposure time control unit 15, and outputs reflected light or transmitted light to the imager 20 of the image feature extraction unit 2a. In the case of the reflection type spatial light modulator 10a, for example, DMD, the exposure time is controlled for each pixel by controlling the on-time, that is, the time for which the reflection surface of the micromirror is directed toward the imager 20 for each pixel. can do. In the case of the transmission type spatial light modulator 10b, the exposure time can be equivalently controlled for each pixel by controlling the amount of transmission for each pixel by controlling the voltage applied to the pixel.

  The basic configuration of the image feature extraction unit 2a is the same as that of the image feature extraction unit 2 according to the first embodiment. However, in the second embodiment, it is assumed that the image feature amount ξ obtained by the calculation unit 30 includes luminance.

  FIG. 8 is a diagram showing an operation concept of exposure time control by the visual information processing system 1a. The imager 20 alone has a limit on the dynamic range with respect to brightness due to problems such as element accuracy, and it is generally difficult to capture an object with extremely high luminance and an object with very low luminance simultaneously with a clear image. . Although it is possible in principle to provide a sensitivity control circuit or the like for each pixel of the imager 20, the number of pixels and the pixel area are reduced due to mounting restrictions of the sensitivity control circuit and the like. The performance of the imager is sacrificed.

  On the other hand, in the visual information processing system 1a according to the present embodiment, the exposure time is controlled by the light spatial modulation element 10, and reflected light or transmitted light whose exposure time is controlled is output to the conventional imager 20. It is. The brightness of the image data captured by the imager 20 is extracted as an image feature amount ξ for each pixel and fed back to the light spatial modulation element 10 via the exposure time control unit 15. If the brightness of a specific area of the image data captured by the imager 20 is too high, the exposure time of the pixels in that area is decreased. Conversely, if the brightness of another specific area is too low, the image exposure in that area is exposed. Increase time. As described above, by adaptively controlling the luminance of the image observed by the imager 20 in units of pixels, it is possible to realize a substantially wide dynamic range.

  In this embodiment, since the imager 20 itself can be used as it is, the resolution and sensitivity inherent in the imager 20 are not sacrificed.

  As a modification of the visual information processing system 1a according to the second embodiment, a visual information processing system 1a having an automatic spatial sensitivity correction function can be used.

  In general, the imager 20 does not necessarily have completely uniform sensitivity over the entire range of the pixel region, and there is a spatial difference in sensitivity. Similarly, color filter arrays such as RGB and CMY used for color imaging also have spatial sensitivity (transmittance) non-uniformity.

  In the visual information processing system 1a according to this embodiment, since the exposure time for each pixel can be controlled by the light spatial modulation element 10, the spatial nonuniformity of the imager 20 and the color filter array can be easily corrected. can do.

  Further, as another modification example of the visual information processing system 1a according to the second embodiment, a form for correcting luminance non-uniformity due to a deviation in the position of illumination light, which occurs in the case of indoor imaging or the like, is adopted. You can also. When the luminance gradually decreases from a region close to the illumination light toward a region far from the illumination light, it is necessary to correct the luminance to be uniform especially when generating a binary image. In the visual information processing system 1a according to this embodiment, the luminance information of the imager 20 is monitored, and the exposure time of the light spatial modulation element 10 is adaptively controlled, so that the entire pixel has a uniform luminance regardless of the position of the illumination light. Image data can be generated.

(3) Third Embodiment FIG. 9 is a diagram illustrating a configuration example of a visual information processing system 1b according to a third embodiment. The visual information processing system 1b according to the third embodiment is configured to perform processing for improving the resolution of the imager 20 by combining the imager 20 and the spatial light modulator 10.

  The visual information processing system 1b includes a light spatial modulation element 10 and an image feature extraction unit 2b.

  The spatial light modulator 10 according to the present embodiment sequentially switches pixels that reflect or transmit incident light within a predetermined group to the imager 20 based on a control signal from the pixel switching unit 16 of the image feature extraction unit 2b. Output.

  The basic configuration of the image feature extraction unit 2b is the same as that of the image feature extraction unit 2 according to the first embodiment. However, in the third embodiment, it is assumed that the image feature amount ξ obtained by the calculation unit 30 includes area information of a specific imaging target.

  FIG. 10 is a diagram illustrating an operation concept of high-resolution processing by the visual information processing system 1b.

  In the visual information processing system 1b according to the third embodiment, the resolution of the spatial light modulator 10 is configured to be higher than the resolution of the imager 20. In the example of FIG. 10, the spatial light modulator 10 has 16 × 16 (256) pixels, the imager 20 has 4 × 4 (16) pixels, and the spatial light modulator 10 has a resolution 16 times that of the imager 20. is doing.

  For example, when imaging an object installed in a dark place without illumination or the like, an imager 20 with higher sensitivity is required. In general, increasing the pixel area is the most effective for improving the sensitivity of the imager 20, and many high-sensitivity imagers have a larger pixel area than a normal imager. For this reason, in general, a high-sensitivity imager is limited in the number of pixels in a predetermined area, and the resolution is often lower than that of a normal imager.

  In the visual information processing system 1b according to the third embodiment, even such a high-sensitivity and low-resolution imager is combined with the light spatial modulation element 10 to achieve high resolution.

  The resolution of the imager 20 illustrated in FIG. 10 is 1/16 compared to the resolution of the spatial light modulator 10. That is, in this example, 16 pixels of the spatial light modulator 10 correspond to one pixel of the imager 20.

  First, the pixels of the spatial light modulation element 10 are grouped every 16 (4 × 4) elements. Then, only one pixel in the group is sequentially turned on. In the example of FIG. 10, the pixel is sequentially switched (scanned) and turned on from the upper left pixel of the (4 × 4) element group. For other groups, the pixels are turned on in the same switching sequence.

  With such a pixel switching method, the number of pixels corresponding to the number of groups is output from each group every time one pixel is switched. In the example of FIG. 10, one pixel from each of the 16 groups of the spatial light modulators 10 is turned on, and a total of 16 pixels are output to the imager 20. The imager 20 first generates one piece of low-resolution image data using these 16 pixels. In the spatial light modulator 10, the pixels in the group are sequentially switched on, and the imager 20 sequentially generates low-resolution image data. When switching of all the pixels in the group is completed, the imager 20 generates 16 low-resolution images.

  Next, by combining these 16 pieces of low resolution image data, one piece of high resolution image data is generated. Since each of the 16 pieces of low-resolution image data and the pixel positions in the group have a one-to-one correspondence, it is possible to generate one piece of high-resolution image data by combining the 16 pieces of low-resolution image data. . In this example, the resolution can be increased to 16 times the resolution of the imager 20.

  In the example of FIG. 10, a mode is shown in which 16 pixels in the group are turned on one by one. On the other hand, for example, 4 (2 × 2) pixels of 16 pixels may be turned on. In this case, the imager 20 generates four pieces of low-resolution image data. By combining these four pieces, it is possible to generate image data with a resolution four times higher. Compared to the previous example, the resolution of the image data after synthesis is 1/4, but the number of image data to be synthesized is also 1/4, and the processing time is shortened.

  Thus, in the visual information processing system 1b according to this embodiment, the resolution of the imager 20 can be improved, and the resolution to be improved can be easily changed as described above.

  As shown in FIG. 9, in the visual information processing system 1b according to the present embodiment, the image feature extraction unit 2b extracts a specific region of the imaging target. Therefore, by feeding back this specific area, the above-described high resolution processing can be performed only for the range of the predetermined pixel area.

  For example, the area of the moving object is extracted as the image feature amount ξ by the image feature extraction unit 2b. Then, the high resolution processing is performed on the moving object region, and the other regions remain at low resolution. By such adaptive processing, it is possible to achieve a visual information processing system 1b that realizes high resolution for a moving object to be watched and suppresses a decrease in sensitivity and an increase in processing time as an entire image. It becomes.

(4) Fourth Embodiment FIG. 11 is a diagram illustrating a configuration example of a visual information processing system 1c according to a fourth embodiment.

The visual information processing system 1c according to the fourth embodiment combines the imager 20 and the spatial light modulation element 10 to extract a spectrum image obtained by extracting frequency components such as luminance fluctuations of the imaging target for each pixel and converting it into an image. It is configured so that it can be generated.

  The visual information processing system 1c includes a light spatial modulation element 10 and an image feature extraction unit 2c.

  The spatial light modulator 10 according to the present embodiment is turned on / off by a predetermined periodic signal for each pixel based on a control signal from the pixel on / off control unit 17 of the image feature extraction unit 2c, and reflects incident light. The light is transmitted and output to the imager 20.

  The basic configuration of the image feature extraction unit 2c is the same as that of the image feature extraction unit 2b according to the third embodiment. The image feature extraction unit 2c includes region information of a specific imaging target as an image feature amount ξ obtained by the calculation unit 30. It is supposed to be.

  FIG. 12 is a diagram showing an operation concept of spectrum image generation processing by the visual information processing system 1c.

  In the fourth embodiment, the spatial light modulator 10 and the imager 20 are basically configured with the same number of pixels.

  The spatial light modulator 10 is divided into a plurality of groups having a plurality of pixels. In the example of FIG. 12, grouping is performed every 4 (2 × 2) pixels.

  An on / off control signal having a different frequency is applied to each of the four pixels in the group. For example, a control signal for switching on / off at frequencies 1 to 4 is applied to the pixels 1 to 4, respectively.

  When the luminance of the imaging object fluctuates at a predetermined frequency, when the fluctuation frequency matches the frequency of the on / off control signal applied to the spatial light modulator 10, the imager 20 holds the luminance of the corresponding pixel. Is done. On the other hand, when the frequency of the luminance fluctuation of the imaging object and the frequency of the on / off control signal applied to the spatial light modulator 10 do not match, the imager 20 suppresses the luminance of the corresponding pixel. That is, since the light spatial modulation element 10 functions as a correlator and the integrator is configured by the capacitance component included in the imager 20, it is possible to extract the fluctuation frequency of the imaging object for each pixel.

  For example, when a fluorescent lamp is lit with a commercial power supply of 50 Hz in a complicated background, in the visual information processing system 1c according to the present embodiment, by matching one of the on / off control frequencies to 50 Hz, A spectrum image in which only the fluorescent lamp portion is emphasized can be generated.

  The on / off control frequency in each group can be arbitrarily set, and can be appropriately set according to the physical condition of the imaging object and the frequency to be measured.

  For example, when detecting a crack or the like of a structure, if it is known in advance that the crack or the like fluctuates at a specific resonance frequency, the crack or the like can be obtained by matching one of the on / off control frequencies with the resonance frequency. It is possible to generate a spectrum image corresponding to the position. As a result, it is possible to visualize a crack or the like by extracting the resonance frequency and imaging it even for a fine crack or the like that is difficult to visually recognize.

  Conversely, even when the resonance frequency of the observation object is unknown, if the frequency of a specific part in the image is different from the frequency in the surrounding range, an image (extracted from the specific part) (Spectrum image) can also be generated.

  For example, a printed wiring board is vibrated at a known frequency, and a part having a frequency component different from the vibration frequency is extracted as an image, so that it is possible to detect a soldering failure portion of a mounted component as image information. .

  Further, by feeding back specific area information and specific frequency components as the image feature amount ξ from the image feature extraction unit 2c, a spectrum image centered on the specific frequency components is obtained for a part of all the pixel areas. Can be generated. In this form, even if the region where the spectrum image is to be generated and the center frequency thereof are continuously changed, it is possible to track the change by adaptive control.

(5) Fifth Embodiment FIG. 13 is a diagram illustrating a configuration example of a visual information processing system 1d according to a fifth embodiment. The visual information processing system 1d according to the fifth embodiment is configured such that spatial filtering processing such as edge enhancement can be realized by simple arithmetic processing by combining the imager 20 and the spatial light modulator 10. .

  The visual information processing system 1d includes a light spatial modulation element 10 and an image feature extraction unit 2d.

  The spatial light modulator 10 according to the present embodiment is turned on / off for each pixel based on a control signal from the spatial pattern generation control unit 18 of the image feature extraction unit 2d, and reflects or transmits incident light to the imager 20. Output.

  The image feature extraction unit 2d has a basic configuration similar to that of the image feature extraction unit 2b according to the third embodiment, and includes region information of a specific imaging target as an image feature amount ξ obtained by the calculation unit 30. It is supposed to be.

  FIG. 14 is a diagram illustrating an operation concept of spatial filtering processing by the visual information processing system 1d.

  In the fifth embodiment, the number of pixels of the spatial light modulator 10 is configured to be larger than the number of pixels of the imager 20. In the example of FIG. 14, the number of pixels of the spatial light modulator 10 is configured to be four times the number of pixels of the imager 20.

  The spatial light modulator 10 is divided into a plurality of groups having a plurality of pixels. In the example of FIG. 14, grouping is performed every 4 (2 × 2) pixels.

  For the four pixels in the group, an on / off control signal is applied to the corresponding pixel based on a predetermined spatial pattern. For example, as illustrated in FIG. 14, in the spatial pattern 1, all four pixels in the group are turned on. In the space pattern 2, two pixels in the right half are turned on and two pixels in the left half are turned off. On the other hand, in the space pattern 3, the two pixels in the left half are turned on and the two pixels in the right half are turned off.

  In the visual information processing system 1d according to the present embodiment, edges in the X direction and the Y direction are obtained only by subtracting from the image data of the imager 20 obtained when the spatial light modulator 10 is set to the spatial patterns 1, 2, and 3. Spatial filtering processing for emphasis processing can be realized.

  Specifically, as shown in the equation (1) in FIG. 14, the image of the imager 20 obtained in the case of the space pattern 2, I (x, y, t + Δt), is obtained in the case of the space pattern 1. By subtracting 1/2 of I (x, y, t) of the image of the imager 20 to be generated, image data in which the edge in the X direction is emphasized can be generated. Strictly speaking, the image data obtained from the spatial pattern 1 and the image data obtained from the spatial pattern 2 have different times, but by switching the pixels of the spatial light modulator 10 at high speed, the equation (1) in FIG. The arithmetic expression on the right side can be approximated on the left side.

  Similarly, as shown in the equation (2) of FIG. 14, the image of the imager 20 obtained at the time of the spatial pattern 3, I (x, y, t + 2Δt), is obtained at the time of the spatial pattern 1. By subtracting 1/2 of I (x, y, t) of the image of the imager 20, image data in which the edge in the Y direction is emphasized can be generated.

  According to the visual information processing system 1d according to the fifth embodiment, a plurality of spatial patterns are formed using the light spatial modulation element 10, and the image data of the imager 20 obtained according to each spatial pattern is subtracted. The spatial filtering process can be performed by a simple operation such as the above.

  Also, as shown in FIG. 13, it is possible to perform spatial filtering processing only on a specific region by feeding back an image feature amount ξ indicating a predetermined region. Furthermore, different types of spatial filter processing can be applied to a plurality of different regions.

  In addition, a spatial matching process or the like can be performed by generating an image that matches a wider area, such as a template image, as a spatial pattern by the light spatial modulation element 10. Further, not only the image prepared in advance but also the modulation pattern based on the image information of the previous frame is prepared, so that processing such as block matching can be performed.

  According to the visual information processing system and the visual information processing according to each of the above-described embodiments, a new visual information processing that has not been obtained conventionally can be performed by combining a spatial light modulation element and an imager (solid-state imaging element). Can be realized.

  In particular, by extracting image features from image data captured by an imager and providing a spatial light modulation element in the feedback loop based on these image features, new visual information processing adapted to the movement and change of the object Can be realized.

  Note that the present invention is not limited to the above-described embodiments as they are, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, the constituent elements over different embodiments may be appropriately combined.

The figure which shows the system configuration example of 1st Embodiment of the visual information processing system which concerns on this invention. The 1st figure explaining operation | movement of the three-dimensional image generation in 1st Embodiment of the visual information processing system which concerns on this invention. FIG. 7 is a second diagram illustrating the operation of generating a three-dimensional image in the first embodiment of the visual information processing system according to the present invention. The 3rd figure explaining operation | movement of the three-dimensional image generation in 1st Embodiment of the visual information processing system which concerns on this invention. FIG. 9 is a fourth diagram illustrating the operation of generating a three-dimensional image in the first embodiment of the visual information processing system according to the present invention. The figure which shows the relationship between the light spatial modulation element and imager in 2nd thru | or 5th embodiment of the visual information processing system which concerns on this invention. The figure which shows the system configuration example of 2nd Embodiment of the visual information processing system which concerns on this invention. The figure explaining the operation | movement concept of exposure time control in 2nd Embodiment of the visual information processing system which concerns on this invention. The figure which shows the system configuration example of 3rd Embodiment of the visual information processing system which concerns on this invention. The figure explaining the operation | movement concept of the high resolution process in 3rd Embodiment of the visual information processing system which concerns on this invention. The figure which shows the system configuration example of 4th Embodiment of the visual information processing system which concerns on this invention. The figure explaining the operation | movement concept of the spectrum image generation process in 4th Embodiment of the visual information processing system which concerns on this invention. The figure which shows the system configuration example of 5th Embodiment of the visual information processing system which concerns on this invention. The figure explaining the operation | movement concept of the spectrum image generation process in 5th Embodiment of the visual information processing system which concerns on this invention. The figure which shows the structural example of the conventional imaging camera apparatus which performs extraction of an image feature-value, and a feedback process.

Explanation of symbols

1, 1a, 1b, 1c, 1d Visual information processing system 2, 2a, 2b, 2c, 2d Image feature extraction unit 3 Light pattern irradiation unit 10 Light spatial modulation element 10a Light spatial modulation element (reflection type)
10b Light spatial modulation element (transmission type)
DESCRIPTION OF SYMBOLS 11 Light pattern generation part 12 Light source 15 Exposure time control part 16 Pixel switching part 17 Pixel on / off control part 18 Spatial pattern generation part 20 Imager 30 Calculation part 30
40 Coordinate converter 50 High-resolution processor 60 Spectrum image generator 70 Spatial filtering processor 100 Imaging object ξ Image feature ρ Coordinate transformation parameter

Claims (8)

A spatial light modulation element that groups pixels in a specified area into a plurality of pixels, and turns on / off a plurality of pixel signals in each group at different frequencies to reflect or output incident light from an imaging object. When,
An image feature extraction unit for capturing incident light from the imaging object output from the light spatial modulation element with an imager having the same resolution as the resolution of the light spatial modulation element, and extracting a feature quantity of the imaging object; ,
With
The image feature extraction unit designates a region based on the feature amount, and in that region, the luminance variation of the imaging object is frequency-resolved corresponding to the different frequencies, and spectrum images corresponding to the different frequencies are obtained. A visual information processing system characterized by generating. The image feature extraction unit
An imager unit having a pixel area of a predetermined size and outputting image data of a partial area of the pixel area for each frame according to pixel position information;
The image data of the partial area is input, an image feature amount is calculated based on the input image data and a previously calculated coordinate conversion parameter, and the calculated image feature amount and the coordinate conversion parameter are A calculation unit for generating a coordinate conversion parameter for the next frame based on the
A coordinate converter that converts the pixel position of the partial area based on the coordinate conversion parameter for the next frame, and generates pixel position information of the partial area of the next frame;
The visual information processing system according to claim 1, comprising: The visual information processing system according to claim 1, wherein the light spatial modulation element is a digital micromirror device (DMD) or a liquid crystal light valve. Light that groups pixels in a specified area into a plurality of pixels, and sequentially switches a plurality of pixel signals in each group based on a plurality of predetermined spatial patterns to reflect or transmit incident light from an imaging object and output the light A spatial modulation element;
An image feature extraction unit that captures incident light from the imaging object output from the light spatial modulation element with an imager having the same number of pixels as the number of groups, and extracts a feature amount of the imaging object;
With
The image feature extraction unit designates a region based on the feature amount, calculates a plurality of signals sequentially output from the imager based on the plurality of spatial patterns in the region, and a plurality of pixels in the group A visual information processing system characterized by performing spatial filtering on an object. A spatial light modulation element in which pixels in a specified area are grouped into a plurality of pixels, and the plurality of pixel signals in the group are turned on / off at different frequencies to reflect or transmit incident light from the object to be imaged. Output step;
Capturing incident light from the imaging object output from the spatial light modulation element with an imager having the same resolution as the resolution of the spatial light modulation element, and extracting a feature amount of the imaging object;
With
The step of extracting the feature amount designates a region based on the feature amount, and in the region, the luminance variation of the imaging object is frequency-resolved corresponding to the different frequency, and the spectrum corresponding to the different frequency. A visual information processing method comprising processing for generating an image. The step of extracting the feature amount includes:
Outputting image data of a partial region of the pixel region for each frame from an imager having a pixel region of a predetermined size according to pixel position information;
The image data of the partial area is input, an image feature amount is calculated based on the input image data and a previously calculated coordinate conversion parameter, and the calculated image feature amount and the coordinate conversion parameter are Generating a coordinate transformation parameter for the next frame based on,
Transforming the pixel position of the partial area based on the coordinate conversion parameter for the next frame to generate pixel position information of the partial area of the next frame;
The visual information processing method according to claim 5, further comprising : 6. The visual information processing method according to claim 5, wherein the light spatial modulation element is a digital micromirror device (DMD) or a liquid crystal light valve. A light spatial modulation element in which pixels in a specified region are grouped into a plurality of pixels sequentially switches a plurality of pixel signals in the group based on a plurality of predetermined spatial patterns, and reflects incident light from an object to be imaged. A step of transmitting and outputting;
Capturing incident light from the imaging object output from the light spatial modulation element with an imager having the same number of pixels as the number of groups, and extracting a feature amount of the imaging object;
With
The step of extracting the feature amount designates a region based on the feature amount, calculates a plurality of signals sequentially output from the imager based on the plurality of spatial patterns in the region, and extracts a plurality of features in the group The visual information processing method characterized by including the process which performs the spatial filtering which makes this pixel the process target.

Images