JP2001157196A - System and method for security - Google Patents

Abstract

PROBLEM TO BE SOLVED: To display a close object and an object approaching or moving away on a display, while using a mobile camera. SOLUTION: As shown in figure 1, this security system has a mobile camera 1, visual system 2, display 3, loudspeaker 4, voice synthesizer 5, microphone 6, voice recognizing device 7 and central control unit 8. Also as shown in figure 29, the mobile camera 1 and the visual system 2 are installed at positions, where the back of a vehicle can be photographed, and the display 3, loudspeaker 4, voice synthesizer 5, microphone 6, voice recognizing device 7 and central control unit 8 are installed on a driver's seat so that a driver can find out the object at the back of the vehicle.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a security device which detects the position and size of an object in an image and warns a user that an object is nearby. The position and size of the object from the edge information of the object in the moving image composed of an arbitrary band among binary images, grayscale images, three primary color wavelengths, visible light wavelengths, infrared wavelengths, ultraviolet wavelengths, and all other electromagnetic waves. The present invention relates to an apparatus which detects a nearby object, an approaching object, and an distant object on a display and informs the type, position, and change of the position of the object by artificial voice.

[0002]

2. Description of the Related Art Conventionally, monitoring devices have been installed in banks, convenience stores, factories, offices and houses. However, many surveillance devices have a fixed surveillance camera and always photograph the same place (Japanese Patent Laid-Open No. 6-3).
51021, JP-A-9-52396). Therefore, a plurality of surveillance cameras are required to eliminate the blind spot from the surveillance range, and the user must view the images of these surveillance cameras in order. Some monitoring devices can be operated remotely, but must be manually operated by the user.
Therefore, even when the user operates one surveillance camera, the user must keep watching the video taken by the surveillance camera. In addition, when the user needs to operate a plurality of monitoring cameras, an excessive burden is imposed on the user.
For example, when a surveillance camera is installed in a vehicle, the driver must operate the direction of the surveillance camera while watching the image (Japanese Patent Application Laid-Open Nos. 63-54699 and 3-164434).
1). In order to solve such a problem, CD-R
A method of changing the direction of the surveillance camera with reference to a map stored in the OM has also been developed (Japanese Patent Laid-Open No. 5-254372).
reference). However, even with such a monitoring device, it is not possible to search for an object according to the situation, so that the driver must always operate the monitoring camera at all times. Monitoring devices such as an object tracking type area monitoring device (see JP-A-6-88876) have also been developed, but at present the usage range is limited.

On the other hand, in recent years, high-performance speech synthesizers (see JP-A-5-053595, JP-A-5-09196, and JP-A-5-108084) and high-performance speech recognizers (JP-A-05-002399, JP-A-05-002399) have been proposed. -011798,
Japanese Patent Application Laid-Open No. Hei 5-017877) has been developed. These are incorporated into a navigation device together with a speaker and a microphone, so that a user can easily use the navigation device via voice (Japanese Patent Application Laid-Open Nos. 6-20196, 8-35846, and 11-276516). reference).

In such a situation, the present inventor has developed a visual device capable of searching for an object indoors and outdoors using a moving camera (see Japanese Patent Application No. 11-253634). This visual device can search for a moving object and a stationary object, photograph at an appropriate magnification, and detect the type, position, and change in position of the object.

In consideration of the above, a moving camera controlled by a visual device captures an object, and a speech synthesizer converts the type, position, and change of the position of the object into artificial voice to notify the user, If the voice recognition device can recognize the user's voice and control the moving camera, it would be possible to manufacture a security device that can be easily operated by the user to search for objects around the user. Be expected.

[0006]

SUMMARY OF THE INVENTION Accordingly, an object of the present invention is to provide an image processing apparatus in which a visual camera detects a position and a size of an object from an image captured by the moving camera so that the moving camera has an appropriate orientation and magnification. Display the image of the object photographed on the display, the speaker notifies the type, position and position change of the object by artificial voice through the voice synthesizer, and the voice recognition device recognizes the user's voice captured by the microphone By doing so, it is an object for the user to control the direction and magnification of the mobile camera.

[0007]

According to the first aspect of the present invention, there is provided an information processing system comprising:
A security device for searching for objects within 00 meters,
Means for generating a moving image of the object using a moving camera, and using a visual device, detecting the position and size of the object from the moving image, recognizing the type of the object, and inputting a control command Means for controlling the orientation and magnification of the moving camera, means for displaying the moving image using a display, means for converting map text into artificial sound using a speech synthesizer, and using a speaker. Means for outputting the artificial voice using a microphone, means for inputting the user's voice using a microphone, means for recognizing the user's voice using a voice recognition device and converting the voice into a command text, and central control. Using a device, the type of the object, the position and the change in the position are converted into the map text, the command text is converted into the control command, and the map is converted into the moving image. By inserting the text and the instruction text, the moving camera, the display, the visual device, a warning device and means for mediating communication between the speech synthesis apparatus and the speech recognition device. The present invention generates the moving image by photographing the object with the moving camera, and executes the image processing on the moving image by the visual device to specify the position of the object, and Recognizing the type of the object, controlling the orientation and the magnification of the moving camera so that the object is enlarged at the center of the moving image. Further, the present invention displays the moving image captured by the moving camera by using the display, and notifies the user of the type, the position, and the change of the position of the object using a voice synthesis technology. . Further, in the present invention, while viewing the moving image in which the map text and the instruction text are inserted, the user can change the orientation and the magnification of the moving camera by the voice of the user, the position of the object and the position of the object. The size can be adjusted. Therefore, the present invention can realize the vigilance device that photographs the object near the moving camera, the object approaching the moving camera, and the object moving away from the moving camera. In addition, the problems related to the search for the object are preferably solved.

According to a second aspect of the present invention, a means for acquiring a frame image of the moving image for an object in the moving image captured by the moving camera, a means for sequentially storing the frame image as a digital image, Means for generating a coarse edge information image from the digital image, means for generating a formed edge information image from the coarse edge information image using the digital image, and processing the object area from the background using the formed edge information image. Means for separating, means for normalizing the object area to generate a normalized image, means for geometrically analyzing the formed edge information image, means for recognizing the type of the object area from the normalized image, Means for detecting the position and size of the object area divided by the coarse edge information image, and setting the direction and magnification of the moving camera to the position of environmental coordinates. Means for converting, the position and the size of the object area to the position of the environment coordinates, means for generating an environment map represented by the environment coordinates, and using the environment map Means for estimating the position of an object, means for selecting one of the environmental coordinates for the plurality of object regions, means for inputting a control command from outside, and vibration for vibrating the mobile camera A vigilance device, characterized by a visual device comprising: means for generating instructions; means for determining the position of the environmental coordinates to which the mobile camera is to move; and means for generating camera instructions for controlling the mobile camera. It is. First, the visual device adjusts the orientation and the magnification of the moving camera so that the object photographed by the moving camera is photographed at an appropriate size. A series of processes for generating edge information of the object from the moving image and degenerating the edge information are all realized by local processing. This allows the visual device to quickly image the object with an appropriate size without depending on the shape and size of the object. In addition, compared to the case where the object area is extracted by classification of color information, the edge information is less affected by illumination and the like, so that the application range of the visual device is wide. Second, a series of processes for generating the edge information of the object from the moving image and normalizing the object region separated from the background using the edge information are all realized by local processing. This allows the visual device to generate the normalized image at high speed without depending on the shape and size of the object. Moreover, since the normalized image does not include the background, the method of recognizing the normalized image is more efficient than the case of recognizing the object region surrounded by the background. The application range of the visual device is wide because it is hardly affected by the size and the like. Third, environmental data representing the type of the object and a position in the environmental coordinates is created from the result of recognition of the normalized image and the orientation and the magnification of the moving camera, so that the visual device is The environmental map, which is a set of environmental data, can be created. Since the environment map includes the environment data for a certain period, the distribution and the movement status can be recorded for each type of the object, and the application range of the visual device is wide. Fourth, using the environment map, the position of the object that has not yet been recognized in the environment coordinates can be obtained. Further, since the actual position of the object existing in a range where the mobile camera can capture an image can be estimated, the application range of the visual device is wide. Fifth, since the type of the object in the normalized image can be estimated by analyzing the rough shape of the edge information generated from the object, the means for recognizing the normalized image is faster. Thus, the object normalized image can be accurately recognized. Further, when the object in the normalized image is not a recognition target, the means for recognizing the normalized image can generate the environment map before generating the recognition result. Has a wide range of applications. Therefore, the problems relating to the search for the object are suitably solved.

According to a third aspect of the present invention, there is provided a central control device connected to a moving camera, a display, a visual device, a voice synthesizing device, and a voice recognizing device, and means for inputting an environment map generated by the visual device. Means for converting the type, position and change of the position of the object area described in the environmental map into map text, means for holding the map text output to the speech synthesizer, and output by the speech recognition device Means for inputting instruction text to be executed, means for converting the instruction text into control instructions for the visual device, means for holding the control instructions, means for inputting a video signal generated by the moving camera, Means for inserting the map text and the instruction text into a video signal;
Means for outputting, to the display, the video signal into which the map text and the command text have been inserted, the central control device comprising:
Since the type of the object region, the position and the change in the position are described in the environment map generated by the visual device, the user uses the voice synthesizer to set the type of the object region, The present invention converts the type of the object region, the position, and the change in the position into the map text so that the position and the change in the position can be known. In addition, the voice recognition device converts the voice of the user into the command text, so that the user can control the orientation and magnification of the moving camera using the visual device, Convert the instruction text into the control instruction. Further, since the video signal generated by the moving camera only shows the object currently being photographed by the moving camera, the present invention provides the user with the display so that the user can obtain information other than the object. Inserts the map text and the command text into the video signal. Therefore, the problems relating to the search for the object are suitably solved.

According to a fourth aspect of the present invention, there is provided a warning device according to the first aspect, wherein the warning device is 90 to 90 degrees clockwise from the front of the vehicle and the ship.
The moving camera and the visual device are installed at a position where a range of 270 degrees can be photographed, and the display, the microphone, the speaker, the central control device, and the voice synthesis are provided in a driver seat of the vehicle and a cockpit of the boat. By installing a device and the voice recognition device, searching for the object within 1000 meters from the moving camera, the object approaching the moving camera, and the object moving away from the moving camera. It is a vigilance method. For example, the vehicles such as automobiles, mini cars, buses, four-wheeled vehicles such as trucks and dump trucks, two-wheeled vehicles such as motorcycles and bicycles, wheelchairs (including electric type), three-wheeled electric vehicles, walking aids and the like, and personal watercraft, yachts, boats, For the vessels such as cruisers, fishing boats, tug boats, passenger ships and tankers, the present invention displays the object behind the vehicle and the vessel on the display, and displays the type, the position, and the position of the object. The change in position is converted to the artificial voice. Even if the driver's seat and the cockpit are equipped with a rearview mirror, a side mirror or the display for displaying an image captured by a surveillance camera, the driver and the driver are actually in a blind spot or approach from a distance. It is difficult to find the coming object. Even more, if the driver and the driver are distracted by the objects in front and side, the driver neglects to be alert to the object in the rear, and thus the present invention relates to the object in the rear. The driver and the operator can be alerted. Therefore, various problems related to searching for the object behind the vehicle and the ship are suitably solved.

According to a fifth aspect of the present invention, there is provided a warning device according to the first aspect, wherein the warning device is provided in a clockwise direction from 0 to 1 from the front of the vehicle and the ship.
The moving camera and the visual device are installed at a position capable of photographing an angle of 80 degrees or a range of 180 to 360 degrees, and the display, the microphone, the speakers, and the center are provided at a driver's seat of the vehicle and a cockpit of the boat. By installing a control device, the speech synthesis device and the speech recognition device, the object within 1000 meters from the moving camera, the object approaching the moving camera, and moving away from the moving camera It is an alert method to search for objects. For example, automobiles, mini cars, buses, four-wheeled vehicles such as trucks and dump trucks, motorcycles and two-wheeled vehicles such as bicycles, wheelchairs (including electric type),
The present invention relates to the electric tricycle, the vehicle such as a walking aid, and the boat such as a personal watercraft, a yacht, a boat, a cruiser, a fishing boat, a tug boat, a passenger boat, and a tanker. An object is displayed on the display, and the type, the position, and the change of the position of the object are converted into the artificial voice. Even if the driver's seat and the cockpit are rearview mirrors,
Even if the display for displaying an image captured by a side mirror or a surveillance camera is installed, it is difficult for a driver and a driver to actually find the object in a blind spot or approaching from a distance. Even more, if the driver and the driver are distracted by the objects in front and behind, the driver neglects to be cautious of the objects on the sides, and the present invention is directed to the objects on the sides. The driver and the operator can be alerted. Therefore, various problems relating to the search for the object on the side of the vehicle and the ship are suitably solved.

According to a sixth aspect of the present invention, there is provided a warning device according to the first aspect, wherein the number is 0 to 9 in a clockwise direction from the front of the vehicle and the boat.
The mobile camera and the visual device are installed at positions where the camera can capture images in the range of 0 degrees and 270 to 360 degrees, and the display, the microphone, the speakers, and the center are provided in the driver's seat of the vehicle and the cockpit of the boat. Control device,
By installing the speech synthesis device and the speech recognition device, the object within 1000 meters from the moving camera, the object approaching the moving camera, the object moving away from the moving camera,
Is a vigilant way to explore. For example, cars, mini cars,
The vehicles such as buses, trucks and dump trucks, such as four-wheeled vehicles, motorcycles and bicycles, two-wheeled vehicles, wheelchairs (including electric type), electric three-wheeled vehicles, walking aids, etc., and watercraft, yachts, boats, cruisers, fishing boats, tugboats For the ship such as a passenger ship and a tanker, the present invention displays the object in front of the vehicle and the ship on the display, and displays the type of the object, the position, and the change in the position. Convert to the artificial voice. Even if the driver's seat and the cockpit are equipped with a display for displaying an image captured by a telescope or a surveillance camera, the driver and the driver are actually in a blind spot or the object approaching from a distance. It is difficult to find. Even more, if the driver and the driver are distracted by the objects behind and on the sides, they neglect to be vigilant against the objects ahead, and the present invention relates to the above-mentioned objects with respect to the objects ahead. The driver and the operator can be alerted. Therefore, various problems relating to the search for the object in front of the vehicle and the ship are suitably solved.

According to a seventh aspect of the present invention, there is provided a security device according to the first aspect, wherein 2 to 200 of the moving cameras and the visual devices and 1 to 200 of the displays are used. This is a warning device that searches for ~ 200 objects at the same time. The present invention uses 2 to 200 moving cameras and the visual device to simultaneously photograph two or more positions that cannot be photographed by one moving camera, or 1 to 200 of the moving cameras. Objects can be photographed simultaneously. Further, the present invention relates to 1 to 200
, The user can see 1 to 200 objects at the same time, and by selecting one of the displays with the voice of the user, the display is The moving camera that is capturing the displayed moving image can be controlled by the voice of the user. Therefore, various problems relating to the search for the object using the plurality of moving cameras are suitably solved.

According to an eighth aspect of the present invention, there is provided a security device according to the first and seventh aspects, wherein the movable camera is provided with a visible light image pickup device and an ultraviolet light image pickup device, so that the object coated with the ultraviolet reflection paint is removed. A method for selectively searching for the object on which an ultraviolet reflective film is stretched.
For example, applying a UV-reflective paint used in sunscreen creams, etc. to road signs, traffic lights, lane marks, markers, etc., and mixing with paint used for the road signs, the traffic lights, the lane marks, the markers, etc. And the surface of the road sign, the traffic light, the lane mark, the marker, etc.
By performing the cutting process, the present invention enables the alerting device to quickly and accurately find the road sign, the traffic light, the lane mark, the marker, and the like. Therefore, the problems relating to the search for the object are suitably solved.

According to a ninth aspect of the present invention, in the warning method according to the eighth aspect, the object coated with the ultraviolet reflecting paint is selectively searched for by emitting ultraviolet rays using an ultraviolet emitting device. It is a vigilance method. For example, at night, in the case of cloudy and rainy weather, the warning device can quickly and accurately make the road sign, the ultraviolet light emitting device such as an ultraviolet fluorescent lamp and a black light emit the ultraviolet light toward the object. The traffic light, the lane mark, the marker, and the like can be found. Therefore, the problems relating to the search for the object are suitably solved.

[0016]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing a security device according to an embodiment of the present invention;

First, the invention according to claim 1 is represented by a configuration as shown in FIG. Here, the manufacturer of the present invention has a mobile camera 1, a display 3, a speaker 4, a voice synthesizer 5,
Commercial products can be used for the microphone 6 and the voice recognition device 7. In particular, the high-performance car navigation system already has a display 3, a speaker 4, a voice synthesizer 5,
Since the microphone 6 and the voice recognition device 7 are incorporated, the manufacturer can use these at low cost.
Therefore, hereinafter, the visual device 2 and the central control device 8 will be mainly described.
Will be described.

As shown in FIG. 1, an image captured by the moving camera 1 is transmitted to the visual device 2 as an analog moving image or a digital moving image without deteriorating the image quality of the frame image. The video taken by the mobile camera 1 is NTS
The video signal is converted into a video signal of C, PAL, HDTV, VGA, MPEG format or the like and transmitted to the central control unit 8. In addition, the mobile camera 1 sends a signal representing the current state of the mobile camera 1 to the visual device 2, while the visual device 2
Transmits a signal for controlling the direction and magnification of the mobile camera 1 to the mobile camera 1. Further, as described in claim 2, the visual device 2 transmits the environment map to the central control device 8. The central control device 8 analyzes the environment map received from the visual device 2 and transmits it to the voice synthesizing device 5, and transmits the user's command received from the voice recognition device 7 to the visual device 2. At the same time, the central control device 8 receives the image captured by the mobile camera 1, inserts the environment map and the user's command, and transmits the image to the display 3. Therefore, hereinafter, the embodiment of the visual control device 2 will be described, and then the embodiment of the central control device 8 will be described.

First, it is assumed that the mobile camera 1 has a mechanism for changing the direction in the horizontal direction and the vertical direction, and is capable of panning and tilting by inputting a command for controlling the moving angle from the outside. The mobile camera 1 has a mechanism for changing the magnification of a captured image, and can perform zooming by inputting a command for controlling the magnification from outside. Therefore, the mobile camera 1 can finely vibrate the camera itself according to an external command. As a result, the frame image of the moving image captured by the mobile camera 1 is blurred, and the object in the frame image is captured as if it were vibrating. Therefore, the stationary object is photographed as if it is moving in the frame image.

Next, it is assumed that the mobile camera 1 can output the current direction and magnification, which have been moved by movement commands such as pan, tilt, and zoom, as necessary, and whether the camera is currently moving or stopped. Mobile camera 1
Can also be output as needed. At this time, if the visual device 2 can control the pan, tilt, and zoom of the moving camera 1, the visual device 2 is used so that the moving camera 1 can always capture a moving object and a stationary object in appropriate sizes. The direction and magnification of 1 can be changed.

Therefore, in order for the moving camera 1 to always photograph a moving object and a stationary object with an appropriate size, the visual device 2 has three coordinate systems according to functions, namely, a camera coordinate system, an image coordinate system, and an environment coordinate system. We use properly. First, the camera coordinate system is a three-dimensional spherical coordinate system inside the camera which is used by the moving camera 1 to control pan, tilt and zoom in each minimum control unit. Generally, the origin of the camera coordinate system is a position unique to the mobile camera 1 called a home position. The camera coordinate system is the only coordinate system that can represent the physical positions of moving and stationary objects. Nevertheless, since the camera coordinate system differs for each model of the mobile camera 1, the camera coordinate system has no versatility. Therefore, the visual device 2 needs to keep the camera coordinate system secret from other means. Second, the image coordinate system is a two-dimensional coordinate system having a pixel as a unit and having an origin at the center of a frame image captured by the moving camera 1. This is used to indicate at which pixel in the frame image the moving object and the stationary object are located. Therefore, the image coordinate system is suitable for distinguishing fine positions of a moving object and a plurality of stationary objects in a frame image, but the image coordinate system alone represents the physical positions of these moving objects and a stationary object. I can't. Third, the environment coordinate system is a three-dimensional spherical coordinate system used by the visual device 2 to logically unify the positions of the moving object and the stationary object. The environment coordinate system uses angles in radians in the horizontal and vertical directions, and uses real numbers in units of 1.0 in the distance direction to represent the product of the size of the object and the distance to the object. In general, since the size of an object does not change extremely in a short time, the distance to the object and the magnification of the moving camera 1 may be regarded as proportional. The origin of the environment coordinate system is arbitrary. That is, the environment coordinate system is used in principle to represent the relative coordinates of any two points on the environment coordinate system. The visual device 2 can distinguish a plurality of moving objects and stationary objects by projecting the moving objects and stationary objects in the environment that can be photographed by the moving camera 1 onto the environment coordinate system.

Therefore, the camera coordinate system and the image coordinate system need to mutually perform coordinate conversion with the environment coordinate system. The camera / environmental coordinate conversion means 2 which fulfills the role is
0, image / environmental coordinate conversion means 21 and motion control means 23
(See FIG. 2). These units obtain the respective units of the camera coordinate system and the image coordinate system from the specifications of the mobile camera 1 and the image acquisition unit 11 (see FIG. 2), and calculate a matrix for converting the units into the environment coordinate system. Also, by calculating the inverse matrix of the transformation matrix from the camera coordinate system to the environment coordinate system, the transformation matrix from the environment coordinate system to the camera coordinate system can be obtained. However, since the origin of the camera coordinate system is the home position of the mobile camera 1, the position of the environment coordinate system converted from the camera coordinate system is a relative position from the home position of the mobile camera 1 on the environment coordinate system. On the other hand, the position of the camera coordinate system converted from the environment coordinate system is a relative position from the current position of the moving camera 1 on the camera coordinate system. in addition,
Since the image coordinate system is a two-dimensional coordinate system, it cannot be converted to the environment coordinate system using only the image coordinate system. Therefore, the image / environmental coordinate conversion means 21 calculates the conversion matrix as needed using the orientation and magnification of the mobile camera 1 expressed in the environment coordinate system and the size of the area of the moving object and the stationary object in the frame image. Thus, the image coordinate system can be converted to the environment coordinate system. Note that the position of the environment coordinate system converted from the image coordinate system is a relative position from the center of the image.

An embodiment of the visual device 2 will be described using the above-described camera coordinate system, image coordinate system, and environment coordinate system.
As shown in FIG. 2, an embodiment of the visual device 2 includes an image acquisition unit 11 that receives an image signal (frame image) of the mobile camera 1 and converts the image signal into a digital image 111 having an appropriate format and size; , An edge information generating unit 14 for generating rough edge information 112 of a moving object from two digital images 111, and forming the rough edge information 112 into more accurate and clear formed edge information 114. Edge information forming means 1
5, an object / background separation unit 16 for separating an area divided by the formed edge information 114, an area normalizing unit 27 for normalizing the separated area to an image size, and a moving object using the formed edge information 114. , An image recognition unit 29 that recognizes the type of the moving object from the normalized moving object region and the geometric analysis result, Position / size detection means 17 for detecting the position and size, and camera / environment coordinates for converting the camera coordinate system representing the direction and magnification of the mobile camera 1 into an environment coordinate system which is a basic coordinate system of the visual device 2. The position of the moving object detected by the position / size detection unit 17 is converted into the environment coordinate system using the conversion unit 20 and the orientation and magnification of the mobile camera 1 expressed in the environment coordinate system. Image / environmental coordinate conversion means 21, position selection means 22 for selecting one of a plurality of moving objects detected by position / size detection means 17, time measurement means 32 for measuring time in units of 1 millisecond, The orientation and magnification of the mobile camera 1 represented in the environment coordinate system, the positions of a plurality of moving objects represented in the environment coordinate system, the types of the moving objects, the geometric analysis results,
An environment understanding means 31 for creating an environment map from the time and the time;
Environmental map holding means 33 for holding an environmental map until output to an output destination; object position estimating means 34 for estimating the position of a moving object to be recognized next using the environmental map; A control command input unit 24 for inputting a control command for external control, a vibration command generation unit 25 for generating a command for vibrating the mobile camera 1, and a direction and a magnification of the mobile camera 1 expressed in an environmental coordinate system. make use of,
Motion control means for determining the position of the mobile camera 1 represented in the camera coordinate system with reference to the four positions input from the position selection means 22, the control command input means 24, the vibration command generation means 25, and the object position estimation means 34 23, a camera command generating means 2 for generating a command for moving the mobile camera 1 to the position of the camera coordinate system determined by the motion control means 23
6 is used.

When the image acquisition means 11 inputs a frame image of a moving image from the moving camera 1, if the moving image is an analog signal, the frame image is converted into a digital signal by A / D conversion and converted into a digital image 111. I do. Note that especially C
If it is possible to directly input the voltage of a CD image sensor, a photodiode image sensor, or the like, it is only necessary to convert the digital signal into an appropriate number of bits by A / D conversion. If the moving image is a digital signal, it is expanded if it is compressed, and is input as it is if it is not compressed. With these, an arbitrary frame image can be cut out from the moving image. Since the converted digital image 111 has an arbitrary image size according to an appropriate format, the image obtaining unit 11 converts the image data into a format in which the image data can be referred to in pixel units, and The minute is cut out and output as a digital image 111.
If the image acquisition means 11 can output all the pixels of the digital image 111 in parallel,
Can be performed in parallel for each pixel.

When the image storage unit 12 receives the digital image 111 from the image acquisition unit 11, the digital image 111 is stored for a certain period of time in accordance with the time resolution of the visual device 2 or the calculation capability of each unit. That is, even if the digital image 111 is input during this fixed time, the image storage unit 12 does not change the stored image, so that each subsequent unit can input the same digital image 111 at a different timing. Moreover, the image storage means 12 stores the digital image 1
11 is not subjected to image processing, the digital image 1
All two pixels are stored while maintaining a two-dimensional phase relationship. If the image storage means 12 can output all the pixels of the digital image 111 in parallel,
Communication from the image storage unit 12 to the edge information generation unit 14 can be performed in parallel for each pixel.

The edge information generating means 14 is used by the image storing means 1
When the digital image 111 is input from Step 2, a coarse edge information image 113 of the moving object is generated by comparing the digital image 111 with the digital image 111 input immediately before. The edge information generation means 14 performs only the neighborhood processing for each pixel and executes the coarse edge information image 1
13 can be generated, which is suitable for parallelization. If the edge information generating means 14 determines that the coarse edge information image 1
If all 13 pixels can be output in parallel, communication from the edge information generating means 14 to the edge information forming means 15 and the position / size detecting means 17 can be performed in parallel for each pixel.

When the edge information forming means 15 receives the coarse edge information image 113 from the edge information generating means 14, the edge information forming means 15 refers to the digital image 111 stored in the image storage means 12 and is more accurate and clear than the coarse edge information image 113. The generated edge information image 115 of the moving object is generated. Since the edge information forming unit 15 can generate the formed edge information image 115 only by the neighborhood process for each pixel, it is suitable for parallelization. If the edge information forming unit 15 can output all the pixels of the formed edge information image 115 in parallel, the communication from the edge information forming unit 15 to the object / background separating unit 16 and the geometric analysis unit 37 is performed for each pixel. Can be done in parallel.

The object / background separating means 16 receives the formed edge information image 115 from the edge information forming means 15,
Pixels included in the object region 141 of the moving object and pixels included in the background are separated into different groups, and the separation results are sequentially output in group units. When the object areas 141 are clearly distinguished by the formed edge information 114 even though they are adjacent to each other, the object / background separation unit 16 can separate these object areas 141 into different groups. Therefore, the number of groups may be three or more. The object / background separation means 16 is suitable for parallelization because the object region 141 and the background can be separated by only neighborhood processing for each pixel. If the object / background separation means 16 can output all the pixels of the object area image 142 in parallel,
Communication to 7 can be performed in parallel for each pixel.

When the area normalizing means 27 receives the object area image 142 and the digital image 111 from the object / background separating means 16 and the image acquiring means 11, respectively, it cuts out the separated object area 143 of the moving object from the digital image 111, Digital image 1 while deforming area 143
A normalized image 145 is generated by complementing and enlarging as much as possible according to the image size of No. 11. The region normalizing means 27 can normalize the separated object region 143 only by the neighborhood processing for each pixel, and is suitable for parallelization. If the area normalizing means 27 can output all the pixels of the normalized image 145 in parallel, the communication from the area normalizing means 27 to the image recognizing means 29 can be performed for each pixel in parallel.

The geometric analysis means 37 is the edge information forming means 1
5, a geometric analysis such as a stroke extraction method, a Fourier transform, an affine transform, and a Hough transform is performed to estimate the shape of a moving object in a frame image, and generate a geometric analysis result. . In conventional image recognition, line segments are extracted directly from the image itself,
Recognition of an object in an image has been performed by using various transformations that are not affected by the position and size of the moving object. For this reason, despite the simplicity of the processing, the number of combinations became enormous, and as a result an enormous amount of calculation was required, and it took a long time to obtain a desired result, or only a low-quality recognition result was obtained. . However, in the visual device 2, the object / background separation unit 16, the region normalization unit 27, and the image recognition unit 29 recognize the pattern of the moving object in the frame image. Therefore, the geometric analysis means 37 generates the formed edge information image 1
15 using the simple method to roughly analyze only a figure such as a contour of a moving object that the object / background separation means 16, the area normalization means 27 and the image recognition means 29 are not good at. 29, the environment understanding means 31 can create an environment map accurately and at high speed.
7 itself can also be reduced. The geometric analysis result is output from the geometric analysis means 37 to the image recognition means 29 and the environment understanding means 31.

When the image recognition means 29 receives the normalized image 145 from the area normalizing means 27, the image recognition means 29 recognizes the normalized area 144 of the moving object in the normalized image 145 by using an appropriate pattern recognition method. That is, the type of the moving object is output. Normalized image 1 input to image recognition means 29
In 45, since the shape of the moving object is deformed by the area normalizing means 27, the image recognizing means 29 performs a geometric analysis using a method resistant to positional displacement such as a stroke extraction method, a Fourier transform, an affine transform, and a Hough transform. Rather, the input normalized image 145 and template image 14
It is more suitable to perform pattern matching for comparing No. 6. As the pattern matching method, a neural network such as a perceptron capable of learning the template image 146 by an error back propagation method (back propagation) can be used. When a neural network is used, parallelization and speeding up are possible by using an accelerator dedicated to the neural network. The recognition result of the normalized image 145, that is, the type of the moving object is determined by the image recognition unit 29 to the environment understanding unit 3.
1 is output. However, when the image recognizing unit 29 receives the result of the geometric analysis from the geometric analyzing unit 37, it first determines whether or not the image is a figure to be recognized by the image recognizing unit 29. If the result of the geometric analysis is not a target figure, the image recognition means 29 does not operate. If the result of the geometric analysis is the target graphic, the image recognition means 29 performs pattern matching using the template image 146 for the target graphic. For example, in the case of a perceptron that learns by the error backpropagation method, the learning data is created by learning the perceptron for each target figure, and then the learning data is selected based on the result of the geometric analysis, so that pattern matching is efficiently performed on the target figure. To do. In other words, if the geometric analysis result is circular, use only learning data for circular objects such as human faces and balls, and if the geometric analysis result is triangular, use only learning data for triangular objects such as road signs and mountain ranges. I do. In this case, not only can the patterns of many objects be recognized by a small-scale perceptron, but also the perceptron itself can be efficiently learned.

When the position / size detecting means 17 inputs the rough edge information image 113 from the edge information generating means 14,
The position and size of the area of the moving object pointed by the rough edge information 112 are detected. The position / size detection unit 17 can generate the overlapping information image 132 indicating the detection result of the position and size of the area of the moving object only by the neighborhood processing for each pixel, and is suitable for parallelization. If the position / size detection means 17 can output all the pixels of the overlapping information image 132 in parallel, the communication from the position / size detection means 17 to the image / environment coordinate conversion means 21
It can be performed in parallel for each pixel.

The camera / environmental coordinate conversion means 20 and the image /
The environment coordinate conversion unit 21 converts the position of the area of the moving object represented by the overlap information image 132 generated by the position / size detection unit 17 into a position in the environment coordinate system. At this time, if the total number of positions of the area of the moving object in the frame image is two or more, two or more positions exist on the environment coordinate system. Therefore, in order to control the pan, tilt, and zoom of the moving camera 1 to adjust the direction and the magnification of the moving camera 1 to one of the moving objects, it is necessary to select one from a plurality of positions on the environment coordinate system. There is. Position selection means 2
2 selects one position on the environment coordinate system in accordance with a predetermined criterion. The criteria used here are mainly as follows. First, of the positions of the moving object on the environment coordinate system, the one closest to the moving camera 1 (or the largest one) is selected. This is because when the moving object is far (or small), there is a possibility that the edge information generating means 14 has generated noise. Therefore, the probability that the moving object exists from a plurality of positions on the environment coordinate system is small. But choose the higher one. Second, when a plurality of positions are concentrated within a certain range on the environment coordinate system, one of them is selected. This has two possibilities. 1
One is that the edge information generating means 14 has generated coarse edge information 112 dispersed for one moving object,
The other is the possibility that a plurality of moving objects actually exist. Third, when a large number of positions on the environment coordinate system have substantially the same distance (or substantially the same size), the position closest to the origin, that is, the position closest to the direction of the mobile camera 1 is selected. By appropriately combining these in accordance with the application and the situation, the position selecting means 22 can select one from a plurality of positions on the environment coordinate system.

The timer 32 outputs the current time in 1 millisecond units by a timer circuit. The current time is continuously output from the clock means 32 to the environment understanding means 31.

When the environment understanding means 31 receives the recognition result of the object area 141 of the moving object from the image recognition means 29, it creates environment data including the recognition result, the position of the mobile camera 1 on the environment coordinate system, and the current time. I do. When the environment understanding unit 31 inputs the positions of all the object regions 141 in the frame image on the environment coordinate system from the image / environment coordinate conversion unit 21, the recognition result including null data indicates that the mobile camera 1
The environment data including the position on the environment coordinate system and the position obtained by adding the position on the environment coordinate system of one object region 141 in the frame image and the current time are created by the number of object regions 141. At this time, the environment map is a set of environment data created a predetermined time before the current time, and the position in the environment data is represented by an environment coordinate system having the home position of the mobile camera 1 as an origin. The environment understanding means 31 adds and deletes environment data to and from the environment map as time elapses. The environment understanding means 31 deletes the duplicated environmental data by replacing the environmental data with a change in the position, etc., and sets the position in the environmental data whose recognition result is null data to the position in other environmental data whose recognition result is not null data. , The environment data whose recognition result is null data is deleted. In addition, if the position in the environment data where the recognition result is not null data is close to the position in other environment data where the recognition result is not null data, if the recognition results in these environment data match, the former environment Delete data. The accuracy of the environmental map is determined by the recording time of the environmental data and the range near the position in the environmental data.
After the current position of the mobile camera 1 on the environment coordinate system is added to the environment map, the environment map is output from the environment understanding unit 31 to the environment map holding unit 33. However, when the environment understanding means 31 receives the result of the geometric analysis from the geometric analysis means 37, it first determines whether or not the image is a figure to be recognized by the image recognition means 29. If the result of the geometric analysis is a figure that is not a target, the environment understanding means 31 immediately deletes the environment data near the position on the environment coordinate system of the mobile camera 1 and whose recognition result is null data from the environment map. . As a result, unnecessary environment data is deleted from the environment map, and the object position estimating means 34 does not need to output unnecessary object estimated positions. If the geometric analysis result is the target graphic, the process waits until the recognition result is input from the image recognition unit 29.

The environment map holding means 33 is used as the environment understanding means 31.
If the output destination of the environmental map requires a signal in an appropriate format, the environment map is converted into the format required by the output destination of the environmental map.
Thereafter, the environmental map holding means 33 stores the environmental map for a certain period of time until the environmental map is reliably transmitted to the output destination of the environmental map.

The object position estimating means 34 is the environment understanding means 31
When an environmental map is input from the user, one environmental data whose recognition result is null data is selected, and a position in the environmental data is extracted. From this position, camera / environmental coordinate conversion means 20
By subtracting the current position of the mobile camera 1 on the environment coordinate system calculated by the above, the object position estimating means 34 sets the current position of the mobile camera 1 on the environment coordinate system as the origin.
The relative position on the environment coordinate system of a moving object for which a recognition result has not yet been obtained despite the generation of the rough edge information 112 in the past can be obtained. However, a moving object is not always present at this position. Therefore, this position is an estimated position of the moving object where the moving object may exist. On the other hand, if there is no environment data whose recognition result is null data in the environment map, a position on the appropriate environment coordinate system is generated within a range in which the mobile camera 1 can move.
At this time, the following can be considered as a reference for generating an appropriate position. First, an arbitrary position is generated by a pseudo random number. Second, a position having a low density of positions in the environmental data in the environmental map is calculated. Third, positions within the movable range of the mobile camera 1 are sequentially generated in an appropriate order. For example, the position is generated in order from the upper left position to the right in the movable range of the mobile camera 1, and when it reaches the right end, it descends one step down and then generates the position in order to the left,
When it reaches the left end, it descends one step and repeats generating positions sequentially to the right. By combining these criteria according to the application and the situation, it is possible to efficiently estimate the estimated position of the moving object that has not been photographed by the mobile camera 1 yet. The object estimation position is output from the object position estimation means 34 to the motion control means 23.

Now, the visual device 2 has a position selecting means 22.
Camera 1 moves to the following position in addition to the position selected in
Is required to be panned, tilted and zoomed. First, in order for the moving camera 1 to take an image as if a stationary object is moving, the visual device 2 must vibrate the moving camera 1. Therefore, the vibration command generation unit 25 specifies a position to which the mobile camera 1 should move as a position on the environment coordinate system. The position specified by the vibration command generation means 25 is determined by a pseudo random number or the like within a range where the mobile camera 1 does not vibrate extremely. In addition, the visual device 2 is required to pan, tilt, and zoom the mobile camera 1 by inputting a control command from the outside. In general use, panning, tilting and zooming are performed based on the current orientation and magnification of the mobile camera 1. Therefore, the control command input means 24 is used to temporarily store the control command and then store the current position of the mobile camera 1. Is calculated on the environment coordinate system with the origin as. By improving the control command input means 24, the mobile camera 1 can be easily moved to a specific position.

Therefore, in order to control the pan, tilt and zoom of the mobile camera 1, the motion control means 23 selects one of the four positions on the environment coordinate system described above. At that time, the motion control means 23 includes the control command input means 24, the object position estimating means 34, the position selecting means 22, the vibration command generating means 2
The input position is selected from the means having the input position in the order of 5. The selected position is converted from the environment coordinate system to the camera coordinate system by the motion control means 23. Then, the camera command generating means 26
The command is replaced with a command recognizable by the mobile camera 1 and transmitted to the mobile camera 1. This allows the visual device 2 to control the pan, tilt, and zoom of the mobile camera 1.

Depending on the model of the mobile camera 1, if a specific command is not sent to the mobile camera 1, the state of the mobile camera 1, that is, the direction and magnification of the mobile camera 1, the mobile camera 1 is moving or stopped. Some information cannot be obtained, such as whether it is inside. Then, the motion control unit 23 selects one of the four positions described above and outputs the selected one to the camera command generation unit 26, and then transmits to the mobile camera 1 a command for inquiring information as to whether or not the mobile camera 1 is moving. Thus, the motion control unit 23 instructs the camera command generation unit 26 to wait until receiving this information from the mobile camera 1 via the camera / environmental coordinate conversion unit 20. If the received information indicates that the mobile camera 1 is moving, the motion control means 23 instructs the camera command generation means 26 again to transmit a command for inquiring information such as whether the mobile camera 1 is moving. . If the received information indicates that the mobile camera 1 is stopped, the motion control unit 23 instructs the camera command generation unit 26 so that the mobile camera 1 transmits a command for inquiring the current orientation and magnification to the mobile camera 1. During this period, the motion control means 23 does not select four positions. The camera command generation means 26 transmits a corresponding command to the mobile camera 1 according to the instruction from the motion control means 23. The camera / environmental coordinate conversion means 20 transmits the information as to whether the mobile camera 1 is moving or not to the motion control means 23 as it is, and also converts the current orientation and magnification of the mobile camera 1 from the camera coordinate system to the position in the environmental coordinate system. Convert to This allows the visual device 2 to move the mobile camera 1
The pan, tilt and zoom of the mobile camera 1 can be controlled while sequentially examining the state of the camera.

Finally, in the visual device 2, each of the edge information generating means 14, the edge information forming means 15, the object / background separating means 16, the area normalizing means 27 and the image recognizing means 29 is It is not always necessary to match the resolution or image size of the input image. For example, in the visual device 2, the edge information generating unit 14 is caused to input a low resolution digital image 116 in which the resolution of the digital image 111 is reduced, while the edge information forming unit 15
Input a coarse edge information image 113 obtained by enlarging the image size of the low-resolution coarse edge information image 117 generated by the edge information generating means 14 to the image size of the digital image 111 by an appropriate method. By inputting the digital image 111 to the area normalizing means 27, the load on the edge information generating means 14 can be reduced. That is, the pan, tilt, and zoom of the mobile camera 1 after the position / size detection unit 17 can be controlled at a higher speed without substantially changing the quality of the normalized image 145 generated by the edge information forming unit 15 or later. it can. Therefore, when this method is further advanced, the edge information forming unit 15 stores the coarse edge information 11 in the low-resolution coarse edge information image 117 generated by the edge information generating unit 14.
2 is an extracted coarse edge information image obtained by cutting out an area where 2 exists.
19 is input to the object / background separation means 16 and the area normalization means 27 by inputting a cutout digital image 120 obtained by cutting out a region at the same position as the cutout rough edge information image 119 from the digital image 111. The load of generating the normalized image 145 after the edge information forming unit 15 can be reduced. If the visual camera 2 allows the moving camera 1 to shoot the moving object at an appropriate size in the center of the frame image, the cut coarse edge information image 119 and the cut digital image 120 of the digital image 111 are cut. The exit area can be determined in advance. With this method, the visual device 2 can achieve the same performance as an object search device using a wide-angle camera and a high-resolution camera by using one moving camera 1.

The image storage means 12, the edge information generation means 14, used in the visual device 2 according to the second aspect,
The edge information forming means 15, the object / background separating means 16, the area normalizing means 27, the position / size detecting means 17, and the image recognizing means 29 are composed of an array operation unit 40 (ARRAY OPERAT).
ION UNIT) can be implemented by using the data processing device 110 composed of the ION UNIT. So in the following,
Data processing device 110 using array operation unit 40
The visual device 2 will be described with reference to the drawings.

First, the array operation unit 40 generates one pixel of the output image by using one pixel of the input image and its neighboring pixels. Therefore, as shown in FIG. 3, by using the data processing device 110 in which the array operation units 40 are arranged in a lattice in accordance with the size of the input image, the data processing device 110 can generate an output image from the input image. it can. In FIG. 3, the array operation unit 40 is abbreviated as AOU. Next, the array operation unit 40 may be implemented by dedicated hardware, or may be implemented by software on a general-purpose computer. That is, as long as the output image can be generated from the input image, the mounting means is not limited. Therefore, the image processing of the data processing device 110 can be indicated by indicating the algorithm of the array operation unit 40. In order to show the algorithm of the array operation unit 40, the image storage unit 12, the edge information generation unit 14, the edge information formation unit 15, the area normalization unit 27, the position / size detection unit 17, and the image shown in FIG. Recognition means 2
The mathematical formula used in No. 9 will be described.

Assuming that x , y , and w are arbitrary 2 ⁿ gradation images having a width w, a height h, and the number of bands b, x , y , and w are respectively located at positions p.
Expressions 1, 2, and 3 are expressed using band pixel values x _ijk , y _ijk , and w _ijk of (i, j, k). The underlined characters indicate vectors. N is a non-negative integer, w, h, b, i, j, and k are natural numbers.

[0045]

(Equation 1)

[0046]

(Equation 2)

[0047]

(Equation 3)

First, a function relating to point processing for each band pixel value of the image will be described below.

When converting the image x into a binary image, the following expression 4 is used.
, The band pixel value is binarized.

[0050]

(Equation 4)

ImagexIs converted to a band maximum value image,
The maximum value among the values of each band of the pixel in the i-th row and the j-th column in accordance with Expression 5
Select a value. The band maximum value image is a single band image.
Therefore, for convenience, it can be handled as the image with 1 band.
And Therefore function B _ij1(x) Has the first subscript 1
It has become.

[0052]

(Equation 5)

[0053] As the image x is a binary image, when reversing the image x, is calculated according to Equation 6.

[0054]

(Equation 6)

The logarithmic transformation at the position p (i, j, k) of the image x is performed according to the following equation (7). Here, e is an offset, and is used to make the value output by the natural logarithmic function fall within the effective range, so that e = 1 is generally sufficient. By this logarithmic conversion, division between band pixel values can be subtracted. Further, ^assuming that the image x is a digital image 111 of 2 ⁿ gradations, if a look-up table including 2 ⁿ elements is stored in the memory 42 regardless of the number of bands, it is necessary to calculate the natural logarithmic function every time. No need to have a standard log table.

[0056]

(Equation 7)

A set P _ijk (q) of positions near q at the position p (i, j, k) of the image is represented by Expression 8. Where q is 4, 8, 24, 48, 80, 12
0, (2r + 1) ² −1, and a series of numbers, where r is a natural number. The position beyond the image size is set P _ijk
If included in (q), position p unless otherwise specified
(I, j, k) shall be substituted. In other cases, a fictitious position whose pixel value corresponds to 0 and which is not included in the image is substituted according to the designation. Thereby, the edge processing is automatically performed. Therefore, the number N _ijk of elements of the set P _ijk (q) is always q.

[0058]

(Equation 8)

Next, functions and operators relating to neighborhood processing of up to eight neighborhoods for each band pixel value of an image will be described below.

The smoothing at the position p (i, j, k) of the image x is performed according to equation (9). However, int (v) means that the real number v is truncated below the decimal point. If If a band-pixel value of the image x is an integer value, twice the right shift instruction based on the sum of x _LMK when N _ijk = 4 in hardware implementation, the sum of x _LMK when N _ijk = 8 The circuit for executing the division can be omitted by changing the circuit to execute the right shift instruction three times.

[0061]

(Equation 9)

The Laplacian calculation is performed by using the following equation.
As shown in FIG. 10, it is a simple second-order difference operator. 8
The nearer part captures subtle changes in noise, and
The number of intersections increases, which is suitable for the present invention. Where N
_ijkIs 4 or 8, so if the hardware implementation
N_ijkX = 4_ijkTo the left shift instruction twice, N
_ijkX = 8_ijkExecute left shift instruction three times for
Perform multiplication of real numbers by changing the circuit to
Circuits to be used can be omitted.

[0063]

(Equation 10)

As a method of finding a zero point from the value obtained by Laplacian, a pixel that changes from positive to negative has been conventionally found. , Find a pixel passing through a zero point, such as negative to zero or zero to positive, or where zero continues. In the present invention, the zero point found by Equation 11 is not a place where there is an edge, but a place where there is noise,
That is, there is no edge. In addition, binarization of real values is performed at the same time according to Expression 11.

[0065]

[Equation 11]

Assuming that the image x is an arbitrary binary image, when filling a pixel having a hole in the image x , the calculation is performed in accordance with Expression 12. Here, f is a parameter indicating the size of the hole to be filled, and generally f = 1 is sufficient. Note that a diagonal line cannot be detected due to its nature in the vicinity of 4; therefore, it is better to set it as close to 8 as possible.

[0067]

(Equation 12)

Assuming that the image x is an arbitrary binary image, when an isolated point or an isolated hole is to be deleted from the image x , the calculation is performed in accordance with Expression 13. Note that a diagonal line cannot be detected due to its nature in the vicinity of 4; therefore, it is better to set it as close to 8 as possible.

[0069]

(Equation 13)

Assuming that the image x is an arbitrary binary image, in order to detect a pixel having a line width of 1 in the image x , calculation is performed using four neighboring pixels in accordance with Expression 14.

[0071]

[Equation 14]

[0072] Two image x, y is any binary image. When the image y is assumed to be an image obtained by detecting the pixel out line width is 1 image x, the inner line width of the image x is 1 In order to extend the line width of a certain pixel, Equation 1 is calculated using four neighboring pixels.
Calculate according to 5.

[0073]

(Equation 15)

Therefore, by using the line width detection of Expression 14 and the line width expansion of Expression 15, complementation of the line width of the binary image can be easily described according to Expression 16.

[0075]

(Equation 16)

Next, a function and an operator relating to the neighborhood processing for each band pixel value of the image will be described below.

If there are two images x and y , the maximum value image of these images is calculated according to equation (17).

[0078]

[Equation 17]

If there are two images x and y , the difference between these images is calculated according to equation (18).

[0080]

(Equation 18)

Here, Laplacian of Expression 10 and Expression 18
By using the difference of, the sharpening of the image can be easily described in accordance with Expression 19.

[0082]

[Equation 19]

If there are two images x and y , and the image y is a single-band binary image, each band pixel value of the image x can be masked using the band pixel value of the image y according to Equation (20). .

[0084]

(Equation 20)

[0085] Two image x, there is y, if the image x and y are binary images, in accordance with Equation 21, the image based on the image x
y can be shaped.

[0086]

(Equation 21)

[0087] Two image x, there is y, if the image y is binary image in accordance with Equation 22, a band-pixel value of the image x not specified by the image y, in the vicinity of the band-pixel value of the image x The interpolation is performed using the average value of the band pixel values of the image x specified by the image y . However, int (v) means that the real number v is truncated below the decimal point.

[0088]

(Equation 22)

In the present invention, the processing is simplified by treating the position and the movement amount of the pixel as image data. This is called position imaging. In the following, some functions and operators relating to imaging will be described.

First, the operator for converting each value of l, m, and o at the position p (l, m, o) into band pixel values as image data is #, and the converted band pixel values are #p (l, m). ,
o). Next, consider a case where the band pixel value moves from the position p (i, j, k) to the position p (i + 1, j + m, k + o). At this time, the shift amount of the band pixel value is the position p (l,
m, o). That is, the movement amount can be regarded as a vector from a certain position. Finally, the operator that extracts the position from the band pixel value is # ^-1 .
Therefore, # ^-1 #p (l, m, o) = p (l, m, o)
Becomes

Therefore, according to Equation 23, the movement amount p (i,
j, k) in a plane represented by the width direction and the height direction by 180
Can be turned in the opposite direction.

[0092]

(Equation 23)

[0093] There are image x, when the image x is a single band binary image, the moving amount of the center of gravity position in the position of the image x p (i, j, 1 ) is calculated in accordance with Equation 24. Note that division must be performed when calculating the center of gravity, but division is canceled out when calculating the amount of movement into the vicinity of 8, so division is omitted in Expression 24.

[0094]

(Equation 24)

From the amount of movement p (i, j, k), the amount of movement into the vicinity of 8 can be calculated according to equations 25 and 26, and can be converted into an image of the amount of movement. The expression 26 is used only when the expression 25 cannot be used due to discretization of an image.

[0096]

(Equation 25)

[0097]

(Equation 26)

Therefore, using Equations 24, 25 and 26, the band pixel value of the displacement image of the single-band binary image x in the direction of the center of gravity can be simply described according to Equations 27 and 28. Note that the number of bands of the moving amount image is one.

[0099]

[Equation 27]

[0100]

[Equation 28]

On the other hand, since the position opposite to the position of the center of gravity can be obtained by using Expression 23, the band pixel value of the moving amount image in the direction opposite to the center of gravity of the single-band binary image x is simply described according to Expression 29. can do. Note that the number of bands of the moving amount image is one.

[0102]

(Equation 29)

If there are two images x and y , and the image y is a movement amount image, after moving the band pixel value of the image x to the movement position indicated by the image y according to Expression 30, the same band pixel The sum of the band pixel values that have been moved to the grayscale image can be obtained.

[0104]

[Equation 30]

Then, by using Equations 4, 27, 28 and 30, the total of the band pixel values moved to the same band pixel after moving the single band gray-scale image x in the direction of the center of gravity in the vicinity according to Equation 31 or Equation 32 Can be easily described.

[0106]

(Equation 31)

[0107]

(Equation 32)

If there are two images x and y , and the image x is a binary image and the image y is a moving amount image, the position of the destination of each band pixel value of the image x can be obtained. Band pixel values with overlapping destinations can be found. Therefore, the destination of each band pixel value of the image x does not overlap,
Moreover, the band pixel value of the movable image indicating that there is a moving band pixel value is generated according to Equation 33.
Note that the number of bands of the movable image is one.

[0109]

[Equation 33]

When there are three images x , y , and w , the image y is a movable image, and the image w is a movement amount image, the band pixel value of the image x can be moved according to Expression 34.

[0111]

(Equation 34)

Therefore, using Equations 29, 33 and 34, the band pixels of the image x obtained by moving the band pixels of the image x in the direction opposite to the position of the center of gravity calculated from the binary image y according to Equation 35 Value can be described easily.

[0113]

(Equation 35)

Therefore, by using Equations 1 to 35, the image storage unit 12, the edge information generation unit 14, the edge information formation unit 15, the area normalization unit 27, and the position / size detection unit shown in FIG. 17 can be described in all the array operation units 40 of the data processing device 110. In the following, the image storage unit 12, the edge information generation unit 14, the edge information formation unit 15, the area normalization unit 27, using an algorithm of an arbitrary array operation unit 40 in the data processing device 110.
And the position / size detecting means 17 will be described.

Since the image storage means 12 realized by the data processing device 110 stores the digital image 111, the array operation units 40 arranged in a lattice form operate in synchronization and in parallel. When the lattice i and a row j sequence arithmetic unit 40 arranged in columns and AOU _ij, Algorithm of AOU _ij is shown in FIG.

In step 1201, AOU _ij is arranged at the i-th row and the j-th column on the lattice. This is necessary to determine the neighborhood of AOU _ij , whether logical or physical.

In step 1202, the neighborhood of AOU _ij and initial values of variables are set.

At step 1203, it is determined whether or not the digital image 111 to be sequentially input has disappeared. If there is no digital image 111 (step 1203: YE
S), end the algorithm. If digital image 11
If there is 1 (step 1203: NO), step 12
Move to 04. However, when the array operation unit 40 is implemented only for a specific image size, an infinite loop may be used.

At step 1204, the digital image 111
Wait for input until is ready.

At step 1205, the digital image 111
Of pixels in the i-th row and the j-th column are input for the number of bands. For this reason, AOU
_ij requires a memory 42 for storing at least image data for the number of bands.

In step 1206, the pixels in the i-th row and the j-th column of the digital image 111 are stored so as to be able to output while waiting for input.

At step 1207, the digital image 111
Is output. Thereafter, the flow returns to step 1203.

Thus, the image storage unit 12 can store the digital image 111 using the data processing device 110 including the array operation unit 40.

In order for the edge information generating means 14 realized by the data processing device 110 to generate the coarse edge information image 113 from the digital image 111, the array operation units 40 arranged in a lattice form operate synchronously and in parallel. .
The array operation unit 40 arranged at i-th row and j-th column on the
OU _ij , AO for the edge information generating means 14
The algorithm of U _ij is as shown in FIG.

In step 1401, AOU _ij is arranged at the i-th row and the j-th column on the grid. This is necessary to determine the neighborhood of AOU _ij , whether logical or physical.

In step 1402, the neighborhood of AOU _ij and initial values of variables are set. In setting the neighborhood, the neighborhood size q used in each of the functions may be individually determined to be 4 or 8, or the whole may be unified to 4 or 8. In order to increase the accuracy of the rough edge information 112 generated by the edge information generating means 14 of the present invention, it is desirable to set all the neighborhood sizes q to 8. However, the edge information generating means 14 can cope with the situation by appropriately changing the neighborhood size as needed, due to restrictions on the calculation time for generating the rough edge information 112, the number of bands of the digital image 111, and the like.

At step 1403, the digital image 111
Judge whether or not is completed. If the digital image 111
If there is no (step 1403: YES), the algorithm ends. If there is a digital image 111 (step 1403: NO), the algorithm ends. However, when the array operation unit 40 is implemented for a specific number of bands and an image size, an infinite loop may be used.

At step 1404, the digital image 111
Of pixels in the i-th row and the j-th column are input for the number of bands. This is AOU
_ij collectively processes the pixels in the i-th row and the j-th column of the digital image 111. Therefore, AOU _ij requires a memory 42 for storing at least image data for the number of bands.

In step 1405, AOU _ij communicates with the array operation unit 40 in the vicinity to obtain a function S for each band pixel value of the input digital image 111.
Performs smoothing according to _ijk ( x ). The smoothed band pixel values are treated as band pixel values of the smoothed image. Here, the function S _ijk ( x ) may be repeated several times as necessary.
In the case of a general color image, two times is sufficient.

In step 1406, logarithmic conversion is performed on each band pixel value of the smoothed image in accordance with the function _Lijk ( x ). The logarithmically converted band pixel value is treated as a band pixel value of the logarithmically converted image.

In step 1407, AOU _ij communicates with the neighboring array operation unit 40, thereby sharpening each band pixel value of the logarithmically converted image according to the function E _ijk ( x ). The sharpened band pixel value is treated as a band pixel value of the sharpened image.

In step 1408, each band pixel value of one input pre-sharpened image is subtracted from each band pixel value of the sharpened image according to the function D _ijk ( x , y ). The band pixel value for which the difference has been calculated is treated as a band pixel value of the time difference image.

At step 1409, each band pixel value of the sharpened image before one input is replaced with a corresponding band pixel value of the sharpened image.

At step 1410, AOU _ij communicates with the neighboring array operation unit 40 to calculate Laplacian for each band pixel value of the time difference image according to the operator ∇ ² _ijk x . The band pixel value for which the Laplacian has been calculated is treated as the band pixel value of the time difference Laplacian image.

At step 1411, AOU _ij communicates with the neighboring array operation unit 40 to obtain a function Z for each band pixel value of the time difference Laplacian image.
_The zero point is extracted according to _ijk ( x ). The band pixel value from which the zero point has been extracted is treated as the band pixel value of the time difference zero point image.

In step 1412, the maximum value among the band pixel values of each band pixel value of the time difference Laplacian image is detected according to the function B _{ij 1} ( x ). The detected maximum value band pixel value is treated as a band pixel value of the maximum value time difference zero point image. Note that the number of bands is 1 for convenience.

In step 1413, AOU _ij communicates with the neighboring array operation unit 40 to calculate Laplacian for each band pixel value of the sharpened image according to the operator ∇ ² _ijk x . The band pixel value for which the Laplacian has been calculated is treated as the band pixel value of the Laplacian image.

At step 1414, AOU _ij communicates with the neighboring array operation unit 40 to extract a zero point for each band pixel value of the Laplacian image according to the function Z _ijk ( x ). The band pixel value from which the zero point is extracted is treated as the band pixel value of the zero point image.

In step 1415, the maximum value among the band pixel values of each band pixel value of the Laplacian image is detected according to the function B _{ij 1} ( x ). The detected maximum band pixel value is treated as a band pixel value of the maximum zero point image. Note that the number of bands is 1 for convenience.

In step 1416, for each band pixel value of the Laplacian image and each band pixel value of the time-difference Laplacian image, the maximum of the band pixel values at the same position in each image according to the function M _ijk ( x , y ). Find the value. The detected maximum band pixel value is treated as a band pixel value of the hybrid zero point image. Note that the number of bands is 1 for convenience.

At step 1417, the AOU _ij communicates with the neighboring array operation unit 40 to remove holes according to the function F _ijk ( x ) for the band pixel value of the hybrid zero point image. The band pixel value from which the hole has been removed is treated as the band pixel value of the hole-removed hybrid zero-point image. Note that the number of bands is 1 for convenience. Here, the function F _ijk ( x ) may be repeated several times as needed. In the case of a general color image, one time is sufficient.

At step 1418, AOU _ij communicates with the neighboring array operation unit 40 to remove isolated points and isolated holes in accordance with the function A _ijk ( x ) for the band pixel values of the hole-removed hybrid zero-point image. . The band pixel value from which the isolated points and the isolated holes have been removed is treated as the band pixel value of the noise-removed hybrid zero-point image. Note that the number of bands is 1 for convenience.

At step 1419, noise removal hybrid zero
Function I for the band pixel value of the point image _ijk(x0)
And 1 are inverted. The inverted band pixel value is
It is treated as a band pixel value of the report image 113.

At step 1420, coarse edge information image 1
13 band pixel values are output. Then step 1403
Return to

Thus, the edge information generating means 14 can generate the coarse edge information image 113 from the digital image 111 using the data processing device 110 including the array operation unit 40.

[0146] As shown in FIG.
Edge information forming means 15 realized by
A rough edge information image 113 composed of information 112;
From digital image 111 and from formed edge information 114
In order to generate the formed formation information image 115
In addition, the array operation units 40 arranged in a lattice form
Work in parallel. Array performance arranged at i row and j column on the grid
Calculation unit 40 _ijThen, AOU_ijArgo
The rhythm is as shown in FIG.

At step 1501, AOU _ij is arranged at the i-th row and the j-th column on the grid. This is necessary to determine the neighborhood of AOU _ij , whether logical or physical.

In step 1502, the neighborhood of AOU _ij and initial values of variables are set. In setting the neighborhood, the neighborhood size q used in each of the functions may be individually determined to be 4 or 8, or the whole may be unified to 4 or 8. In order to increase the accuracy of the formed edge information 114 formed by the edge information forming means 15 of the present invention, it is desirable to set all the neighborhood sizes q to 8. However, the edge information forming means 15 can cope with the situation by appropriately changing the neighborhood size as necessary, due to restrictions on the calculation time for forming the rough edge information 112, the number of bands of the input digital image 111, and the like. .

In step 1503, it is determined whether or not the digital image 111 or the rough edge information image 113 that is sequentially input has disappeared. If there is no digital image 111 or coarse edge information image 113 (step 1503: YES), the algorithm ends. If there is either the digital image 111 or the rough edge information image 113 (step 1503: NO), the process proceeds to step 1504. However, when implementing the array operation unit 40 for a specific number of bands and image size,
It may be an infinite loop.

In step 1504, the digital image 111
Then, the pixels of the i-th row and the j-th column of the rough edge information image 113 are inputted for the number of bands. This is because AOU _ij collectively processes the pixels in the i-th row and the j-th column of the digital image 111 and the rough edge information image 113. Therefore, AOU _ij requires a memory 42 for storing at least image data for the number of bands.

At step 1505, the digital image 111
And the pixel at the i-th row and the j-th column of the coarse edge information image 113 are separated. This is because AOU _ij is a digital image 1
This is for processing the eleventh pixel in the i-th row and the j-th column and the pixel in the i-th row and the j-th column of the rough edge information image 113 as pixels of an independent image. If the pixel at the i-th row and the j-th column of the digital image 111 and the pixel at the i-th row and the j-th column of the rough edge information image 113 are separated and input from the beginning, nothing is performed.

In step 1506, the AOU _ij communicates with the neighboring array operation unit 40, and the function S is applied to each band pixel value of the input digital image 111.
Performs smoothing according to _ijk ( x ). The smoothed band pixel values are treated as band pixel values of the smoothed image. Here, the function S _ijk ( x ) may be repeated several times as necessary.
In the case of a general color image, two times is sufficient.

At step 1507, logarithmic conversion is performed on each band pixel of the smoothed image in accordance with the function _Lijk ( x ).
The logarithmically converted band pixel value is treated as a band pixel value of the logarithmically converted image.

In step 1508, AOU _ij communicates with the neighboring array operation unit 40 to _sharpen each band pixel value of the logarithmically converted image according to the function E _ijk ( x ). The sharpened band pixel value is treated as a band pixel value of the sharpened image.

In step 1509, AOU _ij communicates with the neighboring array operation unit 40 to calculate Laplacian for each band pixel value of the sharpened image in accordance with the operator ∇ ² _ijk x . The band pixel value for which the Laplacian has been calculated is treated as the band pixel value of the Laplacian image.

In step 1510, AOU _ij communicates with the neighboring array operation unit 40 to extract a zero point for each band pixel value of the Laplacian image according to the function Z _ijk ( x ). The band pixel value from which the zero point is extracted is treated as the band pixel value of the zero point image.

At step 1511, the maximum value among the band pixel values is detected according to the function B _{ij 1} ( x ) for each band pixel value of the zero point image. The detected maximum band pixel value is treated as a band pixel value of the maximum zero point image. Note that the number of bands is 1 for convenience.

In step 1512, 0 and 1 are inverted with respect to the band pixel value of the maximum value zero point image according to the function I _ijk ( x ). The inverted band pixel value is treated as a band pixel value of the basic edge information image.

In step 1513, the input band pixel value of the coarse edge information image 113 is first treated as the band pixel value of the shaped coarse edge information image, and the AOU _ij communicates with the neighboring array operation unit 40 to form the basic edge image. Using the band pixel value of the information image, the band pixel value of the shaped coarse edge information image is shaped according to the function Q _ijk ( x , y ). The shaped band pixel value is treated again as a band pixel value of the shaped rough edge information image. Where the function Q _ijk ( x ,
y ) is repeated until the band pixel value of the originally shaped rough edge information image does not change. However, due to restrictions on calculation time, the quality of the input rough edge information image 113, the quality required for the formed edge information image 115 to be formed, and the like,
It is better that the shaping process be terminated at an appropriate number of repetitions.

In step 1514, AOU _ij communicates with the neighboring array operation unit 40 to perform line width complementation on the band pixel values of the shaped rough edge information image according to the function C _ijk ( x ). The complemented band pixel values are treated as band pixel values of the formed edge information image 115.

At step 1515, the band pixel value of the formed edge information image 115 is output. Then step 150
Return to 3.

Thus, the edge information forming means 15 can form the rough edge information image 113 on the formed edge information image 115 using the data processing device 110 including the array operation unit 40.

Here, the formation of the rough edge information image 113 into the formed edge information image 115 means that a high-resolution digital image of the same scene is taken from edge information generated from a low-resolution digital image 111 of a certain scene. Image 1
11 to estimate the edge information to be generated. Therefore, as shown in FIG. 8, the resolution of the digital image 111 is 1 / n with respect to the natural number n.
The low-resolution coarse edge information image 11 is obtained from the low-resolution digital image 116
7, the coarse edge information image 113 can be generated by expanding the low-resolution coarse edge information image 117 by n times. However, in order to reduce the resolution of the digital image 111 to 1 / n, it is sufficient to simply extract one of n consecutive digital images 111 in the horizontal and vertical directions. Further, in order to enlarge the low-resolution coarse edge information image 117 by n times, pixels having a band pixel value of 0 between consecutive pixels of the low-resolution coarse edge information image 117 in the horizontal and vertical directions are simply set to n. -1 should be filled.
At this time, if n is not too large, the data processing device 110 that realizes the edge information forming unit 15 can generate the coarse edge information image 11 obtained by enlarging the low resolution coarse edge information image 117.
3 is almost the same as the formed edge information image 115 in which the data processing device 110 that realizes the edge information forming means 15 forms the coarse edge information image 113 generated from the digital image 111. become. The reason for this is to refer to which edge information among the edge information internally generated by the edge information forming means 15 using the digital image 111,
This is because the edge information forming means 15 only uses the rough edge information image 113. Therefore, when the coarse edge information image 113 obtained by enlarging the low resolution coarse edge information image 117 is input to the edge information forming unit 15, the edge information generating unit 14 that generates the low resolution coarse edge information image 117 from the low resolution digital image 116 is realized. The data processing device 110 can reduce the amount of calculation or the amount of hardware.

When this method is further developed, as shown in FIG. 9, in the low resolution coarse edge information image 117 generated from the low resolution digital image , A low-resolution cut-out coarse edge information image 118 can be generated. This low-resolution cut coarse edge information image 1
When a cut coarse edge information image 119 obtained by enlarging an image 18 and a cut digital image 120 obtained by cutting the same region of the digital image 111 are input to the data processing device 110 that realizes the edge information forming means 15, the cut formation edge information An image 121 can be generated. In this case, the data processing device 110 that implements the edge information forming unit 15 can reduce the amount of hardware. The visual device 2
It can also be considered that the direction and the magnification of the moving camera 1 are changed so that the cut-out digital image 120 is always at the center of the digital image 111.

As shown in FIG. 10, the data processing device 11
0, the region normalizing means 27 realizes the object region 1
In order to generate a normalized image 145 including a normalized region 144 from an object region image 142 including the object region 41 and a digital image 111 including a separated object region 143 overlapping the object region 141, an array operation unit 40 arranged in a grid pattern Work synchronously and in parallel. When the lattice i and a row j sequence arithmetic unit 40 arranged in columns and AOU _ij, Algorithm of AOU _ij is shown in Figure 11.

At step 2701, AOU _ij is arranged at the i-th row and the j-th column on the lattice. This is necessary to determine the neighborhood of AOU _ij , whether logical or physical.

At step 2702, the neighborhood of AOU _ij and initial values of variables are set. In the setting of the neighborhood, the neighborhood size q used in each of the functions may be determined individually, or all may be unified. In order to increase the accuracy of the normalized image 145 generated by the area normalizing means 27 of the present invention, it is desirable to set all the neighborhood sizes q to large values. However, depending on the restriction of the calculation time for normalizing the separated object region 143, the size of the input digital image 111, etc., the region normalizing means 27 can cope with the situation by appropriately changing the neighborhood size as needed. .

In step 2703, it is determined whether or not the sequentially input object region image 142 or digital image 111 has disappeared. If there is no object area image 142 or digital image 111 (step 2703: YES),
End the algorithm. If there is the object area image 142 or the digital image 111 (step 2703: N
O), and proceed to step 2704. However, when the array operation unit 40 is implemented only for a specific number of bands and an image size, an infinite loop may be used.

At step 2704, the object area image 142
Of the i-th row and j-th column for one band, and the pixels of the i-th row and j-th column of the digital image 111 for the number of bands. This is AOU
_ij is to collectively process the pixel at the i-th row and the j-th column of the object area image 142 and the pixel at the i-th row and the j-th column of the digital image 111. Therefore, AOU _ij requires a memory 42 for storing at least image data for the total number of bands.

At step 2705, the object area image 142
And the pixel at the i-th row and the j-th column of the digital image 111 are separated. This is because AOU _ij is the object area image 142
Of the i-th row and the j-th column of the digital image 111 are processed as pixels of the independent image. If the pixel of the i-th row and the j-th column of the object area image 142 and the pixel of the i-th row and the j-th column of the digital image 111 are separated and input from the beginning, nothing is performed. Object area image 142
And the digital image 111 are copied to the updated object area image and the updated digital image, respectively.

In step 2706, the AOU _ij communicates with the neighboring array operation unit 40 to calculate the movement amount for each band pixel value of the updated object region image according to the function R _ij1 ( x ). The band pixel value obtained by imaging the movement amount is treated as a band pixel value of the movement amount image.

In step 2707, the AOU _ij communicates with the neighboring array operation unit 40, and the destination band pixel that can move according to the function H _ijk ( x , y ) for each band pixel value of the updated object region image. You can find the value. The value indicating whether the destination is a movable destination is treated as a band pixel value of the movable image.

In step 2708, AOU _ij communicates with the neighboring array operation unit 40 to move each band pixel value of the updated object region image to a movable destination according to the function U _ijk ( x , y ). The shifted band pixel value is newly treated as a band pixel value of the updated object region image.

In step 2709, the AOU _ij communicates with the neighboring array operation unit 40 to move each band pixel value of the updated digital image to a movable destination according to the function U _ijk ( x , y ). The shifted band pixel value is newly treated as a band pixel value of the updated digital image.

In step 2710, it is determined whether or not the number of movements representing the number of repetitions from step 2706 to step 2709 has reached the designated number. If the number of times of movement has not reached the specified number of times (step 2710: N
O), returning to step 2706; If the number of times of movement has reached the specified number of times (step 2710: YES), the process moves to step 2711. The designated number is determined by the size of the digital image 111, the size of the separated object region 143 of the digital image 111, and the size q of the neighborhood. If appropriate parameters are set according to the purpose of use, it does not matter if the specified number is overestimated, but if the specified number is too large, the time required for normalization becomes longer.

In step 2711, AOU _ij communicates with the neighboring array operation unit 40, and a function V is applied to each band pixel value of the updated object region image that has completed the movement.
_{According to ijk} ( x , y ), interpolation is performed using the average value in the vicinity. Note that x
And y are both updated object region images. The band pixel value filled with the average value is treated as a band pixel value of the normalized updated object region image.

At step 2712, AOU _ij communicates with the neighboring array operation unit 40, and the function V is applied to each band pixel value of the updated digital image that has completed the movement.
Fill with the average value of the neighborhood according to _ijk ( x , y ). Note that x is an updated digital image, and y is an updated object area image. The band pixel value filled with the average value is treated as a band pixel value of the normalized updated digital image.

In step 2713, it is determined whether or not the number of interpolations representing the number of repetitions from step 2711 to step 2712 has reached the designated number. If the number of interpolations has not reached the designated number (step 2713: N
O), and return to step 2711. If the number of times of interpolation has reached the designated number of times (step 2713: YES), the process moves to step 2714. Generally, the number of times of interpolation is about half of the neighborhood size q is sufficient.

At step 2714, it is determined whether the number of continuations representing the number of repetitions from step 2706 to step 2713 has reached the designated number. If the continuation number has not reached the specified number (step 2714: N
O), returning to step 2706; If the continuation number has reached the specified number (step 2714: YES), the process moves to step 2715. The designated number is determined by the size of the digital image 111, the size of the separated object region 143 of the digital image 111, and the size q of the neighborhood. If appropriate parameters are set according to the purpose of use, it does not matter if the specified number is overestimated, but if the specified number is too large, the time required for normalization becomes longer.

At step 2715, the band pixel value of the updated digital image is output as the band pixel value of the normalized image 145. Thereafter, the flow returns to step 2703.

Thus, the area normalizing means 27 can generate the normalized image 145 from the object area image 142 and the digital image 111 using the data processing device 110 including the array operation unit 40.

As shown in FIG. 12, the data processing device 11
The position / size detection means 17 realized by 0 is arranged in a grid pattern in order to generate a duplicate information image 132 having pixels of the duplicate information 131 from the coarse edge information image 113 having the coarse edge information 112 as pixels. Array operation unit 4
0 operates synchronously and in parallel. If the array operation unit 40 arranged at the i-th row and the j-th column on the lattice is AOU _ij , AOU _ij
The algorithm of _ij is as shown in FIG.

At step 1701, AOU _ij is arranged at the i-th row and the j-th column on the lattice. This is necessary to determine the neighborhood of AOU _ij , whether logical or physical.

At step 1702, the neighborhood of AOU _ij and initial values of variables are set. In the setting of the neighborhood, the neighborhood size q used in each of the functions may be determined individually, or all may be unified. In order to increase the accuracy of the duplicated information image 132 generated by the data processing device 110 of the present invention, it is desirable to set all the neighborhood sizes q to large values.
However, the position / size detecting means 1 depends on the calculation time for calculating the center of gravity of the rough edge information 112 of the object and the size of the input rough edge information image 113.
7 can be dealt with by changing the neighborhood size as needed.

At step 1703, it is determined whether or not the sequentially input rough edge information image 113 has disappeared. If there is no rough edge information image 113 (step 170)
3: YES), end the algorithm. If there is a rough edge information image 113 (step 1703: NO),
Move to step 1704. However, when the array operation unit 40 is implemented only for a specific image size, an infinite loop may be used.

At step 1704, the rough edge information image 1
Thirteen pixels in row i and column j are input for one band. Therefore A
OU _ij requires a memory 42 for storing at least one band of image data.

At step 1705, the rough edge information image 1
The 13 rough edge information 112 is converted into the overlap information 131 of the overlap information image 132. The overlap information 131 is a band pixel value corresponding to 1 or 0.

At step 1706, AOU _ij communicates with the neighboring array operation unit 40 to calculate the amount of movement for each band pixel value of the overlap information image 132 according to the function Δ _ij1 ( x ). The band pixel value obtained by imaging the movement amount is treated as a band pixel value of the movement amount image.

At step 1707, AOU _ij communicates with the neighboring array operation unit 40 to move each band pixel value of the overlapping information image 132 according to the function Λ _ij1 ( x ). The shifted band pixel value is newly treated as a band pixel value of the overlapping information image 132.

At step 1708, it is determined whether or not the number of movements representing the number of repetitions from step 1705 to step 1707 has reached the designated number. If the number of times of movement has not reached the specified number of times (step 1708: N
O), returning to step 1705; If the number of times of movement has reached the specified number of times (step 1708: YES), the process moves to step 1709. Note that the designated number is determined by the size of the formed edge information image 115, the size of the object represented by the formed edge information 114, and the size q of the neighborhood. If appropriate parameters are set according to the purpose of use, it does not matter if the designated number is overestimated, but if the designated number is too large, the time required to detect the position and size increases.

At step 1709, AOU _ij communicates with the neighboring array operation unit 40 to calculate the movement amount for each band pixel value of the overlapping information image 132 according to the function Δ ′ _ij1 ( x ). The band pixel value obtained by imaging the movement amount is treated as a band pixel value of the movement amount image.

At step 1710, AOU _ij communicates with the neighboring array operation unit 40 to move each band pixel value of the overlap information image 132 according to the function Λ ′ _ij1 ( x ). The shifted band pixel value is newly treated as a band pixel value of the overlapping information image 132.

At step 1711, the duplicate information image 132
Is output. Thereafter, the flow returns to step 1703.

[0194] Each duplicate information 13 of the duplicate information image 132
1 is the coarse edge information 11 around the position
Since it represents the total number of 2, it means the size of the object centered on its position as a result.

Thus, the position / size detecting means 17 can generate the overlapping information image 132 from the rough edge information image 113 by using the data processing device 110 including the array operation unit 40.

The image storage means 12, the edge information generation means 14, the edge information formation means 15, the object /
The algorithm of the background separating means 16, the area normalizing means 27 and the position / size detecting means 17 has been described. Since all of these means can be realized by neighborhood processing, these means could use the data processing device 110 composed of the array operation units 40 arranged in a lattice.
However, it is difficult to realize the image recognizing means 29 only by the neighborhood processing. Therefore, in order to examine whether the data processing device 110 including the array operation units 40 arranged in a lattice pattern can be used in the image recognition unit 29, the processing of the image recognition unit 29 is classified in detail.

In the image recognition means 29, the normalized image 145
(See FIG. 10) It is required to identify a moving object or a stationary object enlarged over the whole of the candidates prepared in advance and generate a recognition result. The most basic method for identifying a moving object or a stationary object is to prepare as many template images 146 (see FIG. 14) of the moving object or the stationary object as possible and to convert the normalized image 145 to the template image 1.
By comparing with 46, the template image 146 most similar to the normalized image 145 is found. However, the image recognizing means 29 uses the normalized image 14
Since the template image 146 most similar to the normalized image 145 cannot be found only by extracting and comparing arbitrary pixels from the template image 146 and the template image 146, all pixels of the normalized image 145 and the template image 146 are compared. There is a need to. Therefore, the image recognition means 29
Requires global processing such as least squares and neural networks. Since the data processing device 110 has a structure suitable for neighborhood processing, it is difficult to realize the image recognition unit 29 using only the data processing device 110.

By the way, the image recognizing means 29 outputs the normalized image 1
There is no need to perform global processing throughout the process of generating a recognition result from 45. That is, in the process of generating a recognition result from the result of extracting and comparing arbitrary pixels from the normalized image 145 and the template image 146, the image recognition unit 2
9 requires global processing, but in the process of extracting and comparing arbitrary pixels from the normalized image 145 and the template image 146, the image recognition means 29 does not necessarily require global processing. Here, the process of extracting and comparing arbitrary pixels from the normalized image 145 and the template image 146 is the most basic pattern matching. If this pattern matching can be realized by neighborhood processing, the recognition result is obtained from the result of the pattern matching. Can be realized by a general-purpose processor that executes a simple numerical calculation such as a majority decision. Therefore, a method of implementing pattern matching by the data processing device 110 will be described below.

[0199] First, the normalized image 145 and x, n pieces of template image 146 and ^{y 1, y 2 ,,, y h} ,,, y n. Using the natural number g, the matching result image 147
The matching result δ _ij1 of the i-th row and the j-th column of the normalized image 145 is compared with the pixel of the i-th row and the j-th column of the template image 146 according to Expression 36. 146. Since the matching result image 147 is a single band image, it is handled as an image having one band for convenience. Therefore, the third subscript of the matching result δ _ij1 is 1
It has become.

[0200]

[Equation 36]

Here, the matching result δ _ij1 generated according to Expression 36 is not necessarily unified in the entire matching result image 147. Template image 146
If there are many, the matching result image 147 is more likely to be mosaic. Therefore, the data processing device 1
A method 10 calculates a histogram for the matching result δ _ij1 and the matching result in the vicinity of q, and converges the matching result δ _ij1 will be described below.

When an arbitrary single-band image x is the matching result image 147, using a natural number g and real numbers u and v,
The matching result image 147 is updated according to Equations 37 and 38. Since the matching result image 147 is a single band image, it is handled as an image having one band for convenience. Therefore, the third subscript of the function Ψ _ij1 ( x ) is 1
It has become.

[0203]

(37)

[0204]

(38)

The data processing apparatus 110 repeatedly calculates Equations 37 and 38 until the matching result image 147 no longer changes, so that the matching result of the entire matching result image 147 can be converged. At this time, the matching result converges as follows by the combination of the normalized image 145 and the template image 146. If about half of the pixels in the normalized image 145 are most similar to the pixels in the particular template image 146, most of the matching results in the matching result image 147 will converge to this particular template image 146 number. However, if some clusters of pixels in the normalized image 145 are similar to clusters of pixels in several different template images 146, then
In the matching result image 147, clusters of numbers of several template images 146 surrounded by 0 are formed. Further, if the normalized image 145 has no correlation with the set of template images 146, the matching result of the matching result image 147 becomes almost zero. Therefore, the pattern matching realized by the data processing device 110 is based on the normalized image 1
Although it is difficult to specify the template image 146 most similar to 45, it is considered that some similar template images 146 can be selected from the template images 146. Therefore, in the process of generating the recognition result from the result of the pattern matching, it is only necessary to select one of the most influential candidates from the similar candidates of the template images 146 enumerated by the matching result image 147 generated by the pattern matching.

[0206] As shown in FIG.
The pattern matching realized by 0 is arranged in a lattice pattern to generate a matching result image 147 including a matching result indicating the number of the image most similar to the normalized image 145 among the template images 146. The array operation units 40 operate synchronously and in parallel.
The array operation unit 40 arranged at i-th row and j-th column on the
If OU _ij is used, the algorithm of AOU _ij is as shown in FIG.

At step 2901, AOU _ij is arranged at the i-th row and the j-th column on the lattice. This is necessary to determine the neighborhood of AOU _ij , whether logical or physical.

At step 2902, the neighborhood of AOU _ij and initial values of variables are set. In the setting of the neighborhood, the neighborhood size q used in each of the functions may be determined individually, or all may be unified. In order to increase the accuracy of the matching result image 147 generated by the data processing device 110 of the present invention, it is desirable to set all the neighborhood sizes q to large values. However, the pattern matching can be dealt with by appropriately changing the neighborhood size as necessary, due to restrictions on the calculation time for updating the matching result, the size of the input normalized image 145, and the like.

At step 2903, it is determined whether or not the template images 146 sequentially inputted have disappeared. If there is no template image 146 (step 290)
3: YES), proceed to step 2905. If there is a template image 146 (step 2903: N
O), proceed to step 2904.

At step 2904, template image 1
Forty-six pixels in row i and column j are input for the number of bands. Therefore A
OU _ij requires a memory 42 for storing at least image data obtained by multiplying the number of bands by the number of template images 146. Thereafter, the flow returns to step 2903.

At step 2905, it is determined whether or not the normalized image 145 sequentially inputted has disappeared. If there is no normalized image 145 (step 2905: YE
S), end the algorithm. If the normalized image 145
If there is (step 2905: NO), step 290
Move to 6. However, when the array operation unit 40 is implemented only for a specific image size, an infinite loop may be used.

At step 2906, the pixels of the i-th row and the j-th column of the normalized image 145 are inputted for the number of bands. Therefore, AOU _ij
Is a memory 4 for storing image data of at least the number of bands.
2 required.

At step 2907, a matching result image 147 is obtained from the normalized image 145 and the template image 146.
Δ _ij1 is calculated. The matching result is a band pixel value indicating the number of the template image 146 closest to the normalized image 145.

At step 2908, AOU _ij communicates with the neighboring array operation unit 40, and the function Ψ is applied to each band pixel value of the matching result image 147.
Update the matching result according to _ij1 ( x ). The updated band pixel value is treated as a band pixel value of the matching result image 147 again. Here, the function Ψ _ij1 ( x ) is repeated until the band pixel value of the matching result image 147 no longer changes. However, it is better to terminate the update process at an appropriate number of repetitions due to the restrictions on the calculation time, the quality of the input normalized image 145, the quality required for the updated matching result image 147, and the like.

At step 2909, the band pixel value of the matching result image 147 is output. Then step 290
Return to 5.

Thus, the pattern matching in the image recognizing means 29 can generate a matching result image 147 from the normalized image 145 by using the data processing device 110 including the array operation unit 40.

Up to this point, a method of performing image processing including only neighborhood processing using the data processing device 110 including the array operation unit 40 has been described. Hereinafter, the object / background separation unit 16 will be described using only the neighborhood processing using the data processing device 110 including the array operation unit 40.

First, a nonlinear oscillator generally causes a pull-in phenomenon. The pull-in phenomenon is a phenomenon in which nonlinear oscillators having different periods interact with each other in a periodic behavior such as a limit cycle or an attractor so that they are restricted so as to oscillate at a period of a simple constant ratio. At this time, when the vibration of one nonlinear vibrator is changed, the vibrations of the other nonlinear vibrators also change, so that these nonlinear vibrators are synchronized. In addition, by adjusting the interaction of the nonlinear oscillators, the phase difference between the vibrations can be made as small or large as possible. By manipulating this interaction, a group of nonlinear oscillators can be divided into a plurality of groups having different phases. The object / background separation means 16 uses such a nonlinear oscillator pull-in phenomenon to separate the object and the background so that the edge information in the edge information image is used as a boundary, and an object area image representing the object area 141. 142 is generated. Here, a case where Van der Pol is used as the nonlinear oscillator will be described as an example.

First, in a non-linear oscillator network composed of non-linear oscillators arranged in a lattice, i-th row j
Assuming that the nonlinear oscillator in the column is ω _ij , the nonlinear oscillator ω
The set Ω _ij (q) of non-linear oscillators near q of _ij is represented by Expression 39. Where q is 4, 8, 24, 4
It is a sequence of numbers that follows 8, 80, 120, (2r + 1) ² -1, and r is a natural number. When a nonlinear oscillator that exceeds the network size is included in the neighborhood set Ω _ij (q), the nonlinear oscillator ω _ij is substituted. Thereby, the edge processing is automatically performed. Therefore, the number of elements of the neighborhood set Ω _ij (q) is always q. As can be seen from this, the nonlinear oscillator network is treated the same as a single-band image. For simplicity of expression, in the nonlinear oscillator network, only two subscripts are used: the width direction and the height direction.

[0220]

[Equation 39]

[0221] Next, the nonlinear oscillator is coupled by a coupling value tau _ijkl which are calculated in accordance with Equation 40 with the nonlinear oscillators in a neighbor set Ω _ij (q _a) contained in the vicinity of q _a. When the logarithmic table is not used, approximation by Expression 41 is also possible. Μ and ν are appropriate positive constants.

[0222]

(Equation 40)

[0223]

[Equation 41]

If all the non-linear oscillators of the non-linear oscillator network are completely synchronized in phase, the processor 41
As long as the calculation is made, the nonlinear oscillator ω _ij continues to operate forever with the same phase. Therefore, if a disturbance ρ _ij is given, such a state can be avoided. Although a pseudo random number can be used as the disturbance, a simple expression such as Expression 42 is sufficient. Note that ζ _ij represents the presence / absence of edge information on the i-th row and the j-th column of the edge information image. If there is edge information, it is set to 1, otherwise 0. Κ is an appropriate positive constant.

[0225]

(Equation 42)

When the nonlinear oscillator ω _ij is in the neighborhood set Ω _ij (q _a )
In order to synchronize with the nonlinear oscillator ω _kl , the neighboring input sum σ _ij is calculated in accordance with Expression 43.

[0227]

[Equation 43]

The Van der Pol nonlinear oscillator ω_ijTo
Two parameters φ to configure_ijAnd ψ _ijEquations 44 and 4
Calculated according to 5. Note that γ and ε are appropriate positive constants
is there.

[0229]

[Equation 44]

[0230]

[Equation 45]

[0231] The non-linear oscillator is divided into the object area 141 and the background area.
In order to separate the nonlinear oscillator into
It is necessary to calculate, but simply the object area 141 and the background area
Since only the two regions are separated, the parameter ψ_ijBut
The phase shift is calculated based on the threshold value θ or more. Object area
Output λ that outputs the result of separating the area 141 from the background area _ij
Is obtained by Expression 46. Θ is an appropriate positive constant
Is a number.

[0232]

[Equation 46]

If the edge information is not enough to separate the object from the background, the edge information must be interpolated. For this purpose, it is necessary to find out how many non-linear oscillators are out of phase in the set of non-linear oscillators Ω _ij (q _b ) near q _b of the non-linear oscillator ω _ij . Therefore, the contour parameter η _ij is calculated by Expression 47.

[0234]

[Equation 47]

Based on the result, the boundary parameter _{ij ij} indicating the interpolation ratio of the edge information is calculated by Expression 48. Note that α, β, η _min , and η _max are appropriate positive constants.

[0236]

[Equation 48]

Here, the case of van der Pol as a nonlinear oscillator has been described. In addition, a nonlinear oscillator that is stable in a limit cycle such as a brasserator,
Any non-linear oscillator that causes a pull-in phenomenon, such as a chaotic oscillator that generates a Lorenz attractor or a wrestler equation attractor, can be operated. In this case, the parameters φ _ij and _{ij ij} may be replaced or added with the parameters of the respective nonlinear oscillators. At that time, it is only _necessary to add the neighborhood input sum σ _ij and the disturbance ρ _ij to appropriate parameters.
However, in the case of a chaotic oscillator, no disturbance ρ _ij is required.

By using Expressions 39 to 48, it is possible to describe the algorithms of all the array operation units 40 of the data processing device 110 on which the object / background separation means 16 can be mounted. Hereinafter, the object / background separation unit 16 will be described using an algorithm of an arbitrary array operation unit 40 in the data processing device 110.

[0239] As shown in FIG.
0, the object / background separation means 16 uses the triangle edge information 151
The array operation units 40 arranged in a lattice form operate synchronously and in parallel in order to separate the data into the outer region 153 of the triangle. Array operation unit 4 arranged at i row and j column on the lattice
If AOU _ij is 0, the algorithm of AOU _ij is shown in FIG.
It looks like 7.

At step 1601, AOU _ij is arranged at the i-th row and the j-th column on the grid.

At step 1602, the neighbors ω _ij and ω _kl are connected by a joint value τ _ijkl based on equations 40 and 41.

In step 1603, appropriate initial values are set for the parameters φ _ij and ψ _ij of the nonlinear oscillator.

In step 1604, it is determined whether or not there is no formed edge information image 115 input sequentially.
If there is no formed edge information image 115 (step 1
604: YES), end the algorithm. If the formed edge information image 115 exists (step 1604: N
O), proceed to step 1605; However, when the array operation unit 40 is implemented only for a specific number of bands and an image size, an infinite loop may be used.

At step 1605, the formed edge information 11
Input ζ _ij of 4.

In step 1606, a disturbance ρ _{ij is obtained} from ζ _ij of the formed edge information 114 input immediately before according to the equation (42).
Is calculated.

In step 1607, the neighborhood set Ω
ζ _kl , ξ _kl , ψ _kl are input from AOU _{kl of the} array operation unit 40 where the nonlinear oscillator ω _kl in _ij (q _a ) is located, and the total value σ _ij is calculated in accordance with Equation 43.

In step 1608, the parameters φ _ij and ψ _ij of the nonlinear oscillator are calculated according to equations (44) and (45). That is, the differential equations shown in these equations are solved by the Runge-Kutta method.

At step 1609, the output λ _ij of the non-linear oscillator is calculated according to equation (46). Here, λ _ij = 1 if ψ _ij ≧ θ, and λ _ij = 0 otherwise.

At step 1610, the neighborhood set Ω
λ _kl is input from AOU _{kl of the} array operation unit 40 where the non-linear oscillator ω _kl in _ij (q _b ) is located, and the contour parameter η _ij is calculated according to Equation 47.

At step 1611, the boundary parameter ξ _ij
Is calculated according to Equation 48. That is, the differential equation represented by this equation is solved by the difference method or the Runge-Kutta method.

At step 1612, it is determined whether or not the number of separations representing the number of repetitions from step 1606 to step 1611 has reached the designated number. If the number of separations has not reached the specified number of times (step 1612: N
O), returning to step 1606; If the number of separations has reached the specified number of times (step 1612: YES), the flow shifts to step 1613.

At step 1613, the object area image 142
The output λ _ij of the non-linear oscillator having the pixel value of the band is output. Thereafter, the flow returns to step 1604.

The following method can be used to determine the number of separations in step 1612. object/
In the background separating means 16, if the image size is constant, the separation ends in a certain fixed time in almost all formed edge information 114 regardless of the initial state of the nonlinear oscillator. The number of repetitions from 1606 to step 1611 may be obtained. This is because if the initial state of the nonlinear oscillator is within a certain range, there is not much difference in the time until the nonlinear oscillator is synchronized by the pull-in phenomenon.

It is a property of the nonlinear oscillator that it is possible to separate the inner region 152 of the triangle and the outer region 153 of the triangle using the edge information 151 of the triangle only by calculating the nonlinear oscillator. This is because the pull-in phenomenon is used. That is, when two nonlinear oscillators are coupled with a positive coupling value, they tend to be in phase, and when coupled with a negative coupling value, the phase difference tends to be as large as possible. When this property is used, nonlinear oscillators that are not directly coupled to each other have the same phase by coupling the nonlinear oscillators arranged in a lattice with neighboring components with a positive coupling value. Further, when the nonlinear oscillators located at the pixels sandwiching the formed edge information 114 are connected with each other with a negative connection value, both sides of the edge information are shifted from each other in phase as much as possible. In this way, different sets of phases can be formed on the inside and outside of the triangular edge information 151 without connecting all the nonlinear oscillators. Therefore, the object / background separation means 16 separates the object into a triangular inner area 152 and a triangular outer area 153 as shown in FIG. At this time, the phase difference between the inner region 152 of the triangle and the outer region 153 of the triangle exceeds 90 degrees and is as small as possible 1
As it approaches 80 degrees, the triangle and the background area can be separated.

What is important here is that, in the present embodiment, each time the formed edge information 114 is obtained, the connection value is changed in a pseudo manner by the following method. First, as determined by Equations 40 and 41, a coupling value for coupling the nonlinear oscillator ω _kl to the nonlinear oscillator ω _ij is set to τ _ijkl (see Step 1602). Both ζ _ij and ζ _kl of the formed edge information 114 are 1 when there is an edge and 0 when there is no edge. Ζ _ij and ζ in the formed edge information 114
_{When kl} is input (see step 1605), the formation edge information 1 is transferred from AOU _kl to AOU _ij of the array operation unit 40.
14Zeta _kl is transferred, AOU _ij in binding value tau _ijkl (1-
ζ _kl ) is calculated and used as a substitute for the joint value τ _ijkl (see step 1607). The substitute value τ _ijkl (1-ζ
_kl ), the boundary parameter _{ij ij} acts as a scaling factor from 0 to 1 (see step 1607).

As shown in FIG. 18, the formed edge information 114
Becomes the triangle edge information 154 in a broken line state, it is necessary to interpolate the broken line. First, when the system is operated using the edge information 154 of the dashed triangle (see step 1605), the phase difference between the inside and the outside of the edge information 154 of the dashed triangle exceeds about 90 degrees. The boundary between the inside and outside of the triangle is unclear. Therefore, each AOU _ij calculates the output λ _ij of the nonlinear oscillator (see step 1609). This output λ
_{If ij} is 1, _let ω _kl be the nonlinear oscillator of which λ _kl is 1 among the nearby nonlinear oscillators, and let the parameters ψ _ij and ψ _kl
Are both equal to or greater than θ. That is, λ _ij and λ _kl are approximately in phase, and the phase difference does not exceed 90 degrees at worst if θ is a positive value. The maximum value of this phase difference is determined by the value of θ, and when θ is increased in a range where both λ _ij and λ _kl are 1, this phase difference approaches 0 degree. Therefore, when λ _ij and λ _kl are used, the contour parameter η _ij representing the number of neighboring non-linear oscillators having approximately the same phase is calculated according to Equation 47 (see step 1610).
Subsequently, if the contour parameter η _ij is approximately half of the entire neighborhood, the boundary parameter ξ _ij which is the magnification of the joint value
Is reduced according to Equation 48, otherwise Equation 4
8 (see step 1611). For example, in the case of the vicinity of 8, if the value is between 3 and 5, the boundary parameter may be reduced according to Expression 48. When this process is repeatedly performed, when the edge information 154 of the dashed triangle shown in FIG. 18 is given, the process is separated into the inner region 155 of the dashed triangle and the outer region 156 of the dashed triangle.

As shown in FIG. 19, when two triangles overlap each other, edge information 157 of the front triangle and edge information 158 of the rear triangle are obtained. At this time, the inside area 159 of the front triangle and the inside area 1 of the back triangle
The non-linear oscillators in the three regions 60 and the double triangular background region 161 are separated from each other by shifting the phase of each other. Also, as shown in FIG. 20, even if the two overlapping circular edge information 162 are broken lines, they are separated into three parts: a front circular inner area 163, a rear circular inner area 164, and a double circular background area 165. You.

Thus, the object / background separation means 16 separates the object area 141 from the background using the formation edge information 114 of the formation edge information image 115 as a boundary by using the data processing device 110 composed of the array operation unit 40. be able to.

The visual device 2 has been described so far. Naturally, these visual devices 2 can be implemented by a general-purpose computer, but when a moving object is to be searched, it is necessary to execute each of the means at high speed depending on the moving speed of the moving object. . In particular, when the image size or resolution of the frame image is increased, the image storage unit 12, the edge information generation unit 14, the edge information formation unit 15, the object / background separation unit 16, the area normalization unit 27, The amount of calculation for the position / size detection unit 17 and the image recognition unit 29 increases in each of the width direction and the height direction in proportion to the image size or the resolution. Therefore, the visual device 2 may not be able to achieve the desired performance in some applications.

The image storage means 12, the edge information generation means 14, the edge information formation means 1
5. In order to implement the object / background separation means 16, the area normalization means 27, the position / size detection means 17, and the image recognition means 29, the array operation unit 40
3, the array operation units 40 are wired so as to be able to communicate only with the adjacent array operation units 40 in the data processing device 110. That is, the four neighborhoods are directly wired. As a result, as compared with the case where the eight neighborhoods are wired, it is possible to operate at the same high speed with a small number of electronic components and the amount of wiring, and to easily have expandability even when the neighborhood size is expanded in the future.

As shown in FIG. 21, the array operation unit 40 includes a processor (PROCESSOR) 41 for calculating a mathematical expression in image processing, and a memory for storing all parameters, constants, functions and operators used in the mathematical expression. MEMORY) 42 and a controller (CONTROLER) 43 for communicating with the array operation unit 40 in the vicinity. The processor 41 uses the address (ADDRESS) specified by the address bus 51 to control the memory 42 and the controller 4.
Three arbitrary memory elements and registers can be selected. The processor 41 is bidirectionally communicable with the memory 42 and the controller 43 via the data bus 52, and can access data (DATA) of any memory element and register specified by the address bus 51. Previous input data group (FRONT INPUT DATA SET) in which the array operation unit 40 is composed of one or more input pixels
Is input, the controller 43 causes the memory 42 to store the previous input data group. Further, the controller 43 transmits the calculation data in the memory 42 created by the function to the adjacent array operation unit 40, and transmits the calculation data received from the adjacent array operation unit 40 to the memory 42.
And, if necessary, transfer it to the array operation unit 40 other than the input. Finally the controller 43
Converts the image data of the output image to the result data (RESULT DAT
Output as A).

The reason why the controller 43 is mounted on each array operation unit 40 is as follows.
Since the processor 41 can operate while communicating with each other, the processor 41 can calculate even during the waiting time due to communication and realize high-speed processing. However, since there is no need to change the hardware, the controller 43 can automatically perform the edge processing of the image, that is, the exception processing for the edge pixels in the image, so that the program of the processor 41 needs to perform the edge processing. Is very simple.

As the processor 41 and the memory 42, general-purpose digital circuits can be used. A specific circuit diagram of the controller 43 is as shown in FIG. The address buffer (ADDRESS BUFFER) 53 is connected to the address bus (ADDRES).
S (BUS) 51 from the processor 41 via the address (ADD)
RESS) and an address decoder (ADDRESS DECODE)
R) 54 selects each register and other functional blocks. The data buffer (DATA BUFFER) 55 receives data (DATA) from the processor 41 via the data bus (DATA BUS) 52 and exclusively communicates with the register selected by the address decoder 54 via the internal data bus 56. The communication direction is specified by reading (READ). Address is flag register (FLAG REGISTER) 5
When 7 is specified, the data is stored in the flag register 57, decoded by the flag decoder (FLAG DECODER) 58, and transmitted to the adjacent array operation unit 40 as a plurality of signals (SIGNALS). A plurality of signals are received by a flag encoder (FLAG ENCODER) 59, and after being analyzed, a status register (STATUS REGISTER) 60
And is returned to the source array operation unit 40 as reception (RECEIVE). The reception is received by the flag encoder 59 of the transmission source of the plurality of signals, and as a result, completion of transmission of the plurality of signals is confirmed. When the status register 60 is selected by the address, the contents of the status register 60 are transmitted to the processor 41 as data via the data bus 52. One or more input images (INPUT
IMAGE) one or more front input delivery (FRONT INPUT)
SEND) received by the flag encoder 59, the previous input data group (FRONT INPUT DATA) including one or more input images.
SET) is read into the front input data register (FRONT INPUT DATA REGISTER) 61 prepared for the required storage capacity. When the previous input data register 61 is selected by the address, the contents of the previous input data register 61 are transmitted to the processor 41 as data. When the processor 41 completes the calculation, the result data register (RESULT DATA REGISTER) 62 is selected according to the address, and the result data register 62 converts the image data of the output image into the result data (RE
(SULT DATA). At the same time, the flag encoder 59 transmits a result transmission (RESULT SEND).

When the data necessary for calculation is obtained from the neighboring array operation unit 40, the output data register (OUTPUT DATA REGISTER) 63 is selected as an address, and the data to be transmitted to the nearby array operation unit 40 is calculated. CALCURATION DATA) as output data register 63
Read in. After that, it is transmitted as calculation data to all adjacent array operation units 40. When a plurality of signals (SIGNALS) are received from the upper array operation unit 40, the calculation data is transferred to the upper input data register (UPPER INPUT DATA REG).
ISTER) 64. Thereafter, when the upper input data register 64 is selected by the address, the contents of the upper input data register 64 are transmitted as calculation data. The same applies to the case where a plurality of signals are received from the lower, left and right array operation units 40.
5, the left input data register 66 and the right input data register 67 operate similarly.

Each block of various buffers, various registers, and the address decoder 54 is a general-purpose electronic circuit. Specifically, the flag decoder 58 and the flag encoder 59 have input / output signals as shown in FIGS. Type is OUTPUT DATA REGISTE
R) The type of content read into 63 is represented by 5 bits.
This number of bits is a value sufficient to distinguish all calculation data to be transmitted and received by the array operation unit 40. The count-X (COUNT-X) and the count-Y (COUNT-Y) each represent a 4-bit unsigned integer, and the array operation unit 40
Indicates the number of transfers during the period. When the array operation unit 40 transmits the calculation data, each count becomes 0. When the calculation data transmitted from the left and right array operation units 40 is transmitted again, 1 is added to the count -X of the flag encoder 59. And the upper and lower array operation units 4
When the calculation data transmitted from 0 is transmitted again, it becomes a value obtained by adding 1 to the count -Y of the flag encoder 59. After the processor 41 specifies in which direction the contents of the output data register 63 are to be transmitted, up, down, left and right, in the transmission flag (SEND FLAG) of the flag register 57, the central data of the address decoder 54 that specifies the output data register 63 is specified. When the coding (CENTRAL DECODING) is received by the flag decoder 58, the flag decoder 58 transmits (SE)
ND) according to the direction specified by the delivery flag. The delivery flag is represented by 4 bits, and when the calculation data of the array operation unit 40 is transmitted to the array operation units 40 on the four sides, the processor 41 sets it to 1111 and the calculation data transmitted from the array operation unit 40 on the right is When transferring to the left side, the processor 41 sets 1110; when transferring from the left side to the upper and lower right sides, it is set to 1101; when transferring from the lower side to the upper side, setting is 1000; and when transferring from the upper side to the lower side. Is set to 0100. As a result, the transfer can be efficiently performed without duplication, and the rules for determining the transfer direction are clear. Therefore, by combining the type, the count-X and the count-Y, the It is possible to determine which type of calculation data has been transmitted from the unit 40. At the same time that the calculation data is read into the result data register 62 as the result data, the flag decoder 58 receives the result decoding (RESULT DECODING) and transmits the result delivery (RESULT SEND).

When the flag encoder 59 receives the delivery in any one of the four directions, the flag encoder 59 determines the type of the receiving direction and counts.
X and count-Y are received, and the contents of the status register 60 of that part are updated. At the same time as this update, the reception is set to 1 in the receiving direction and transmitted. The flag encoder 59 of the array operation unit 40 of the transmission source receives the data at the moment when the reception becomes 1, and updates the reception status (RECEIVE STATUS) of the status register 60. Accordingly, in each array operation unit 40, the processor 41 can determine which input data register stores valid calculation data only by checking the reception status of the status register 60. Therefore, for example, if the calculation data is read into the upper input data register 64, the processor 4
1 can read the data from the upper input data register 64 by specifying the address, but at the same time, the upper decoder (UPPER DECO
DING) is transmitted to the flag encoder 59, the upper part of the reception status is returned to 0, and the reception directed upward is 0.
Sent as The same operation is performed on the lower left and right sides.
When at least one of the flag encoders 59 receives the previous input delivery for the input image, the status register 60 sets the front input delivery status (FRONT INPUT SEND STATUS) for the input image corresponding to the received previous input delivery to 1. In addition, the processor 41 is provided with a previous input data register 61 for an input image.
When data is read from the FRONT DECODI
NG), and sets the pre-input delivery status corresponding to the received pre-input delivery to 0. By reading the contents of the status register 60, the processor 41 can determine whether or not the latest input image is stored in the previous input data register 61.

FIG. 25 shows an algorithm when the processor 41 transmits calculation data to the array operation units 40 on the four sides via the controller 43. FIG. 25 shows the program control by the processor 41 and the flag decoder 58.
And a process by the flag encoder 59 for mixing with hardware logic. 25, in step 71, the processor 41 reads the contents of the status register 60. In step 72, it is determined whether or not the reception status of the read contents is all zero. If NO, the process ends. If YES, the process moves to step 73. In step 73, the processor 4
1 determines the type of data to be transmitted to the adjacent array operation unit 40, the counter, and the transmission direction, and writes the content to the flag register 57. In step 74, the processor 41 writes data to be transmitted to the adjacent array operation unit 40 in the output data register 63. Step 75
Then, the contents of the output data register 63 are transmitted to adjacent array operation units 40 as calculation data. In step 76, the transmission is set to 1 only in the direction specified by the transmission flag of the flag register 57 and transmitted. Thus, one transmission algorithm of the processor 41 ends. The processor 41 starts this transmission algorithm each time data to be transmitted is updated in the memory 42.

FIG. 26 shows an algorithm when the controller 43 receives calculation data from the upper array operation unit 40. FIG. 26 shows processing by hardware logic by the flag decoder 58 and the flag encoder 59. 26, in step 81, the flag encoder 59 inputs the delivery. In step 82, the flag encoder 5 determines whether the delivery is 1 or not.
9 makes a decision. If NO, the process ends. If YES, the process moves to step 83. In step 83, the upper input data register 64 reads the calculation data transmitted from the upper side. In step 84, the flag encoder 59 sets the reception status for the upper side of the status register 60 to 1
At the same time, the reception is set to 1 and transmitted to the upper array operation unit 40. The same applies to the lower left and right sides. Thus, one reception algorithm of the controller 43 ends. The controller 43 constantly monitors the delivery from the array operation units 40 in the up, down, left, and right directions, and starts the receiving algorithm each time the delivery is received.

The processor 41 sets the upper input data register 6
FIG. 27 shows an algorithm when data is received from No.4. FIG. 27 shows processing by hybridizing program control by the processor 41 and hardware logic by the flag decoder 58 and the flag encoder 59. 27, in step 91, the processor 41 reads the contents of the status register 60. In step 92, it is determined whether or not the reception status for the upper side of the read contents is 1. If NO, the process ends. If YES, the process moves to step 93. In step 93, the processor 41 sets the upper input data register 6
Read the data from 4. In step 94, the flag encoder 59 sets the reception status for the upper side of the status register 60 to 0, and simultaneously sets the reception to 0 and transmits it to the upper array operation unit 40. The same applies to the lower left and right sides. Thus, the one-time reception algorithm of the processor 41 ends. The processor 41 monitors the contents of the status register 60 at regular intervals, and starts this receiving algorithm every time the reception status of one of the upper, lower, left, and right is 1. Even if the processor 41 does not monitor the contents of the status register 60 at regular intervals, it can be implemented by interrupt processing.

The array operation unit 40 mainly includes
Although the description has been made on the assumption that one output image is generated from one or more input images, it is necessary to change the circuit so that calculation data in the middle of calculation can be output depending on the application. At this time, the program is simply changed so that the result delivery of the flag decoder 58 is increased by the number of calculation data to be output, and only the result delivery corresponding to the calculation data read into the result data register 62 is set to 1. good.

The visual device 2 has been described so far. In the following, the central control device 8 is divided into three groups, namely, a group for converting the environmental map output from the visual device 2 into a map text output to the voice synthesizing device 5, and a command text output from the voice recognizing device 7. A group for converting to a control command output to the device 2 and a mobile camera 1
And a group for inserting the map text and the instruction text into the video signal output from the group.

In the first group, the environment map input means 30
1, a map / text conversion unit 302 and a map text holding unit 303 are included. First, the type of the object area 141,
Every time an environmental map composed of environmental data representing the position and the change of the position is output from the visual device 2, the environmental map input means 301 inputs this environmental map and sequentially stores it in a buffer. When the map / text conversion unit 302 becomes capable of inputting, the environment map input unit 301 outputs each environment data according to the input order. When the environment data is input, the map / text converter 302 converts the type, position, and change in position of the object area 141 into a character string (map text).
At this time, the map / text conversion means 302 converts the position of the object and the change in the position into a coordinate that is based on the user's constant viewing direction, that is, the front, with the user's position as the origin. Change to words that are easy for the user to understand, such as front, back, right, left, up, down, near, far, approaching, retreating, crossing, passing, passing, etc. Further, by using an escape sequence, special symbols, and the like, the map / text conversion unit 302 converts the position based on the orientation and magnification of the mobile camera 1 included in the environmental map into coordinates using the origin as the origin, and converts the coordinates into the aforementioned words. Add to map text after making changes. Thus, the map text can represent the position of the same object area 141 with different coordinates. The map text holding unit 303 sequentially stores the map text converted by the map / text conversion unit 302 in a buffer and outputs the map text to the speech synthesizer 5.

The second group includes a command text input means 304, a text / command conversion means 305, and a control command holding means 306. First, a character string (instruction text) that is the recognition result of the instruction given by the user, such as front, back, right, left, up, down, near, far, more, and a little more.
Every time is output from the voice recognition device 7, the instruction text input means 304 inputs this instruction text and stores it sequentially in the buffer. When the text / command conversion means 305 becomes capable of inputting, the command text input means 304 outputs each command text according to the input order. Upon input of the command text, the text / command conversion unit 305 converts the command text into a control command that can be input to the visual device 2. At this time, the text / command conversion means 305 converts the command text into coordinates using the orientation of the moving camera 1, that is, the center of the display 3 and the position based on the magnification of the moving camera 1 as the origin. Is calculated relative to the current position. This allows the user to display 3
The user can easily control the mobile camera 1 while watching. The control command holding unit 306 sequentially stores the control commands converted by the text / command conversion unit 305 in a buffer and outputs the control commands to the visual device 2.

The third group includes video signal input means 3
07, a text insertion means 308 and a video signal output means 309. First, the video signal input means 307 receives the NTSC, PAL, HDTV, V
A video signal of a GA, MPEG format or the like is input, and a frame image is sequentially cut out and output to the text inserting unit 308. The text inserting means 308 has a map /
Each time a map text is input from the text conversion means 302 or a command text is input from the command text input means 304, a map text and a command text are input using a commercially available superimpose circuit or software for realizing a superimpose function. Is inserted into the frame image. The text inserting unit 308 outputs the frame image to the video signal output unit 309 regardless of the presence or absence of the map text and the instruction text. When the video signal output unit 309 receives the frame image from the text insertion unit 308,
The video signal is converted into a video signal of NTSC, PAL, HDTV, VGA, MPEG format or the like, and output to the display 3.

The central controller 8 has been described so far. Hereinafter, the installation positions of the mobile camera 1 and the visual device 2 using the automobile 321 as an example will be described with reference to the drawings.

As shown in FIG. 29, when the mobile camera 1 and the visual device 2 are installed at the center 322 of the rear bumper of the automobile 321 according to claim 4, the driver can check the place where the blind spot is present. That is, the security device automatically pans, tilts, and zooms the moving camera 1 to set the shooting range 3
By changing 26, the driver can find a pedestrian 324, a bicycle, a motorcycle 325, and another car 321 crossing the back of the car 321, and a child playing near the trunk of the car 321. Therefore, the driver can safely move backward in a parking lot or a garage having many blind spots. When the mobile camera 1 and the visual device 2 are installed at the rear center 323 of the ceiling of the automobile 321 according to claim 4, the driver does not twist the body in places (especially distant places) where it is difficult for the driver to check with the rearview mirror and the side mirror. And where you can't pay attention. That is, the alert device automatically pans, tilts, and zooms the moving camera 1 to change the shooting range 326, so that the driver can approach the pedestrian 324, the bicycle, the motorcycle 325, and another vehicle approaching from behind the automobile 321. 321 can be found. Therefore, the driver can safely open the door and drive.

In FIG. 29, the moving camera 1 according to claim 4
Although the case where the visual device 2 is installed at the rear of the automobile 321 has been described, the same applies to the side and front of the automobile 321. When the mobile camera 1 and the visual device 2 are installed on the door and the ceiling side of the car 321 according to claim 5, the driver can check the child playing near the door and the roadside zone. Also, according to claim 6, the vehicle 32
When the moving camera 1 and the visual device 2 are installed in the center of the front bumper and the front of the ceiling of the vehicle 321, the pedestrian 324, the bicycle, the motorcycle 325, and another vehicle 321 crossing the front of the vehicle 321, and the vehicle 321. You can find a child playing near the hood. Therefore, the driver can safely move forward at the intersection of a parking lot, a garage, and a narrow alley with many blind spots.

In addition, as shown in FIG.
The photographing unit 332 in which the moving camera 1 and the visual device 2 are integrated into one is installed behind the back of the seat 331 of the wheelchair, and the display 3, the speaker 4, the voice synthesizing device 5,
Microphone 6, voice recognition device 7, and central control device 8
By installing the interface unit 333 in which the information is integrated into the handrail of the seat 331 of the wheelchair, the user can check an object approaching from behind the wheelchair without twisting the body. In particular, it is difficult for the user of the wheelchair to check his / her back, for example, he / she cannot move his / her body freely, or the posture of the wheelchair leans forward to move the wheelchair. However, a user of a wheelchair can move safely by using the present invention. Further, according to the fifth and sixth aspects of the present invention, by installing the photographing unit 332 above the seat 331 or in front of the seat 331, the mobile camera 1 can photograph a traffic light, a road sign, and the like in a wider range. Also, the interface unit 333 is installed above the seat 331 or the interface unit 333 is installed.
By using a wearable computer as
Even a user with low vision can see the image on the display 3, a user with low hearing can hear the sound of the speaker 4, and even a user with low voice has a camera orientation and magnification toward the microphone 6. Can be indicated.

Up to this point, the position of the security device and the range of the security have been described using a vehicle as an example, but the same applies to a ship. Hereinafter, a security device that displays images captured by a plurality of mobile cameras 1 on a plurality of displays 3 will be described with reference to the drawings.

As an example, as shown in FIG. 31, it is assumed that images taken by four mobile cameras 1 are displayed on three displays 3. The exchange 311 is built in the central controller 8 by digital technology such as a crossbar switch used in a telephone exchange and a computer bus, and sequentially switches three of the images captured by the four mobile cameras 1. Select and send to display 3. However, FIG.
1, a speaker 4, a speech synthesizer 5, a microphone 6, and a speech recognizer 7, a map / text converter 302, a map text holder 303 of the central controller 8,
Instruction text input means 304, text / command conversion means 305, control command holding means 306, video signal input means 307, and video signal output means 309 are omitted.

If the map text converted by the map / text conversion means 302 describes that there is an object near the position where the mobile camera 1 is currently photographing, the exchange 311 will use this mobile camera. The switch is changed so that the image taken by the camera 1 can be displayed on the display 3. Such a description is stored in the map / text conversion means 302.
Can be easily added to the map text. Further, when the map / text conversion unit 302 adds the number of the mobile camera 1 to the map text, the number of the mobile camera 1 is displayed on the display 3 by the text insertion unit 308. In other words, the user can easily know which mobile camera 1 has captured the video currently displayed on the display 3 and can select the mobile camera 1 that captures a specific video. When only a smaller number of mobile cameras 1 than the number of displays 3 are shooting an object, the exchange 311 can transmit an image of the appropriate mobile camera 1 to the display 3 using an empty line. When a larger number of mobile cameras 1 than the number of displays 3 are shooting an object, the exchange 311 cuts off the currently connected line after a certain period of time, so that another mobile camera 1 shoots. Display video 3
Can be displayed. However, when a user gives an instruction by voice to keep displaying the image of the mobile camera 1 on the display 3, the exchanger 311 follows the instruction. The voice synthesizing device 5 includes all map / text conversion means 30.
2, a voice signal is sequentially transmitted to the speaker 4, and when the user specifies some of the numbers of the mobile cameras 1 by voice, the voice synthesizer 5 corresponds to the specified number. Only map text can be processed.

On the other hand, the voice recognition device 7 recognizes the number of the mobile camera 1 and words such as all, all, next, fixed, release, stop, etc., and adds the number to the instruction text. The mobile camera 1 can be selected. When the text / command conversion unit 305 extracts a number from the command text, the text / command conversion unit 305 sends a control command to the visual device 2 of the currently selected mobile camera 1 until the next number is extracted. I do. Further, the text / command conversion unit 305 transmits a signal corresponding to the number of the currently selected mobile camera 1 and words such as “fixed”, “release”, and “stop” to the speech synthesizer 5, so that the speech synthesizer 5 The operations described above can be performed.

Up to this point, the security device has a plurality of mobile cameras 1
Has described the case where the captured video is displayed on the plurality of displays 3. Hereinafter, a method for searching for an object coated with an ultraviolet reflective paint and an object coated with an ultraviolet reflective film will be described.

In general, many roads and sidewalks are provided with road signs, traffic lights, lane marks, markers and the like for ensuring safe traffic. However, besides these, pedestrians 324, bicycles,
Cars 321, billboards, buildings, trees and the like are everywhere. Therefore, a driver of a vehicle such as a car 321 or a bus must pay attention to a pedestrian 324, a bicycle, a car 321, a billboard, a building, a tree, and the like while driving. And may overlook markers and the like. Moreover, it is very difficult for a user of a vehicle having a low height such as a wheelchair, an electric tricycle, a walking assist device, and a minicar to see road signs, traffic lights, lane marks, and markers. In order for the warning device to efficiently find road signs, traffic lights, lane marks, markers, and the like that the driver tends to overlook, these objects are coated with an ultraviolet reflective paint or a UV reflective film is applied to these objects in accordance with claim 8 to move. Camera 1
Incorporates a visible light image sensor and an ultraviolet image sensor such as a Bee camera. Some UV-reflective paints are transparent, such as sunscreen creams, and can reflect UV light contained in sunlight. The same applies to the ultraviolet reflection film. Therefore, when an object coated with an ultraviolet-reflective paint and an object covered with an ultraviolet-reflective film are exposed to sunlight, these objects reflect ultraviolet light, so that the ultraviolet image pickup device captures the ultraviolet light reflected from these objects. , The orientation and the magnification of the moving camera 1 are controlled so that the visual device 2 can photograph these objects. At night, in the case of cloudy and rainy weather, the headlights are turned on, and according to claim 9, the visual device 2 is illuminated with ultraviolet light emitting devices such as ultraviolet fluorescent lamps and black lights. The direction and magnification of the mobile camera 1 are controlled so that these objects can be photographed. Therefore, drivers and users of vehicles such as the automobile 321, buses, wheelchairs, electric tricycles, walking aids, and minicars are less likely to miss road signs, traffic lights, lane marks, markers, and the like by using the warning device.

Although the present embodiment has been described above, the present invention is not limited to the above-described embodiment, and those skilled in the art can implement various aspects, and understand the technical idea of the present invention. It goes without saying that the configuration of the present invention can be appropriately modified without departing from the scope of the present invention, and such modifications also belong to the technical scope of the present invention.

[0286]

According to the first aspect of the present invention, the user of the present invention has an object at a long distance from the current position, an object not directly visible due to an obstacle, and a user's body fixed. An object that cannot be seen through the camera and an object that can only partially be seen indirectly using a mirror can be clearly seen using the moving camera 1 and the display 3. Moreover, the present invention automatically finds these objects using the visual device 2 and informs the user of the type, position and change in position of these objects using the speaker 4 and the speech synthesizer 5, so that the user can It is possible to obtain information on an object that is different from the object currently being viewed directly, thereby preventing an unexpected accident. Furthermore, since the present invention can control the direction and magnification of the mobile camera 1 using the microphone 6 and the voice recognition device 7, the visual device 2 cannot find an object different from the object selected as the imaging target by the visual device 2. Moving object 1 and the object that the user wants to keep watching
Can be taken at an appropriate magnification. For example, by installing the mobile camera 1 at the entrance of a house, a visitor or a thief can be confirmed inside the house. In particular, since the direction and the magnification of the mobile camera 1 can be controlled by voice, even a bedridden elderly person or a physically handicapped person can use the present invention.

According to the invention of claim 2, the visual device 2
Can search for moving and stationary objects not only indoors but also outdoors. Further, when a user's instruction is input from the voice recognition device 7, the visual device 2 can preferentially control the direction and magnification of the mobile camera 1 in accordance with the instruction, so that the user can view the image without using his hand. You can continue to see the object you want at the size you want. Visual device 2
Since parallelization and hardware implementation are easy, the present invention can search for an object in real time even if the image size of a moving image increases.

According to the third aspect of the present invention, the user always changes the direction of the viewing direction through the speaker 4 connected to the voice synthesizing device 5, that is, the position of the object converted into coordinates having the front as the origin. You can know. Thus, the user can intuitively know the position of the object without looking at the display 3. At the same time, the user can know, via the display 3, not only the object being photographed by the mobile camera 1, but also the type, position, and change in position of other objects, and the command given by the user. In particular, by converting the position represented by the environmental map into coordinates using the current orientation and magnification of the mobile camera 1 as the origin and adding the coordinates to the map text, the user can make a relative movement from the object being captured by the mobile camera 1. direction,
That is, words such as front, rear, right, left, more, and a little more can be instructed to the alert device. Therefore, the user may instruct the security device to issue an appropriate command to view an object different from the object being captured by the mobile camera 1 or to allow the mobile camera 1 to capture a position that needs to be monitored in the future. it can.

According to the invention described in claim 4, the driver of the vehicle and the operator of the ship can quickly find the vehicle and the ship approaching from behind the vehicle and the ship, so that the lane change and turning can be performed safely. Can be. Especially when the present invention is installed in a vehicle, the driver can find a child playing behind the rear bumper, notice a vehicle approaching from behind before opening the door, or leave the parking lot backward. The present invention is effective for traffic safety, for example, it is possible to confirm the traffic of persons and vehicles. Further, when the present invention is installed in a wheelchair (including an electric type) and a minicar, the user's body is fixed and the rearview mirror is very small, so that the user can hardly see the rear. Therefore, the present invention searches for a person and a vehicle approaching from behind, thereby assisting the user to move safely.

According to the fifth aspect of the present invention, the driver of the vehicle and the operator of the ship can quickly find the vehicle and the ship running in parallel with the vehicle and the ship, so that the lane change and turning can be performed safely. Can be. In particular, when the present invention is installed in a vehicle, the driver can find a child playing between the front wheel and the rear wheel, and prevent a tangling accident at the time of right turn and left turn. It is effective for traffic safety and accident prevention.

According to the invention as set forth in claim 6, the driver of the vehicle and the operator of the ship can quickly find the vehicle and the ship approaching from the front of the vehicle and the ship, so that the vehicle can safely turn right, left, and turn left. You can make a turn.
In particular, when the present invention is installed in a vehicle, a driver can find a child playing behind a front bumper, or check the traffic of a person or a vehicle when moving forward at an alley or an intersection. The present invention is effective for traffic safety and accident prevention. Further, when the present invention is installed in a wheelchair (including an electric type) and a minicar, since the user's body is fixed, the user can hardly see the upper front part. Therefore, the present invention assists the user in traveling safely by searching for a traffic light and a sign ahead.

According to the invention described in claim 7, the user of the present invention can simultaneously find a plurality of objects at different positions. When the present invention is installed in a house, the intrusion of a thief can be monitored. When the present invention is installed in a store, the manager can check the positions and operations of the customer and the clerk. In particular, the present invention is useful for preventing theft and shoplifting. When the present invention is installed in the automobile 321, the driver can drive while checking surrounding conditions that cannot be seen at the same time. When the present invention is installed in a large ship,
The pilot and the captain can navigate while checking the positions of small boats such as tug boats and yachts. When the present invention is installed in a railway vehicle, a driver and a conductor can find a person sandwiched between doors and a person close enough to get caught in the railway vehicle before departure. When the present invention is installed in an aircraft and a helicopter so that the mobile camera 1 can photograph the front, rear, left, right, up and down, the pilot and the passenger can visually recognize the guide man, the towing vehicle, the ramp, etc., and form a formation. You can see the location of another aircraft, or find suspicious aircraft or flying objects nearby. In particular, the invention is useful for finding victims on snowy mountains and drifters on the sea. When the present invention is installed in a submersible craft and a deep sea exploration boat, the pilot and the captain can visually recognize tugs, docks, cranes, etc., and can find creatures swimming nearby and floating objects. it can. In particular, the invention is useful for finding creatures hiding on the sea floor, wreckage of shipwrecks and remnants of crashed airplanes. When the present invention is installed in a spacecraft and a space station so that the mobile camera 1 can photograph the front, rear, left, right, up and down, the pilot and the passenger can visually recognize other spacecraft and space stations, artificial satellites and floating objects. it can. In particular, the present invention monitors the space where a large number of astronauts and robots are working, and the present invention is useful for quickly finding astronauts, robots, building materials, etc., which are floating due to the cutting of a lifeline. .

According to the eighth and ninth aspects of the present invention, the warning device can quickly and accurately find a road sign, a traffic light, a lane mark, a marker, and the like irrespective of daytime, cloudy weather, or rainy weather. Therefore, the present invention is effective for traffic safety and accident prevention, as it is difficult for a user of the present invention to overlook road signs, traffic lights, lane marks, markers, and the like. In particular, a road sign, a traffic light, a lane mark, a marker, and the like can be easily seen by a user of a wheelchair, an electric tricycle, a walking assist device, and a minicar having a low vehicle height.

[Brief description of the drawings]

FIG. 1 is a block diagram of a security device.

FIG. 2 is a block diagram of a visual device that controls a moving camera to generate an environment map.

FIG. 3 is a block diagram of an array operation unit arranged in a lattice.

FIG. 4 is a flowchart illustrating an algorithm of an image storage unit of the embodiment.

FIG. 5 is a flowchart illustrating an algorithm of an edge information generating unit according to the embodiment.

FIG. 6 is an explanatory diagram in a case where rough edge information is formed into formed edge information using a digital image.

FIG. 7 is a flowchart illustrating an algorithm of an edge information forming unit according to the embodiment.

FIG. 8 is an explanatory diagram in a case where low-resolution coarse edge information generated from a low-resolution digital image is formed as formed edge information.

FIG. 9 is an explanatory diagram of a case where an area of low-resolution coarse edge information generated from a low-resolution digital image is cut out and then formed as formed edge information.

FIG. 10 is an explanatory diagram in a case where a cut-out area of a digital image is normalized.

FIG. 11 is a flowchart showing an algorithm of a region normalizing means of the present embodiment.

FIG. 12 is an explanatory diagram for detecting the position and size of an object in an edge information image.

FIG. 13 is a flowchart illustrating an algorithm of a position / size detection unit according to the present embodiment.

FIG. 14 is an explanatory diagram of a case where pattern matching is performed on a normalized image from among template images.

FIG. 15 is a flowchart showing an algorithm of pattern matching in the image recognition means of the embodiment.

FIG. 16 is an explanatory diagram showing a state where edge information of a triangle is separated into an inner area and an outer area of the triangle;

FIG. 17 is a flowchart illustrating an algorithm of an object / background separation unit according to the present embodiment.

FIG. 18 is an explanatory diagram illustrating a state where edge information of a triangle in a broken line is separated into an inner region and an outer region of the broken triangle;

FIG. 19 is an explanatory diagram showing a state where edge information obtained by overlapping two triangles is separated into two triangle regions and a background region.

FIG. 20 is an explanatory diagram showing a state in which edge information in a broken line state when two circular object regions are superimposed is separated into two circular regions and a background region.

FIG. 21 is a block diagram of the internal structure of the array operation unit.

FIG. 22 is a block diagram of a controller.

FIG. 23 is an explanatory diagram showing input / output signals of the flag decoder.

FIG. 24 is an explanatory diagram showing input / output signals of a flag encoder.

FIG. 25 is a flowchart illustrating an algorithm by which a processor transmits data to an adjacent array operation unit via a controller.

FIG. 26 is a flowchart illustrating an algorithm in which a controller receives data from an adjacent array operation unit.

FIG. 27 is a flowchart showing an algorithm by which a processor receives data from an upper input register.

FIG. 28 is a block diagram of a central controller communicating with a mobile camera, a visual device, a display, a speech synthesizer, and a speech recognizer.

FIG. 29 is an explanatory diagram in the case where a moving camera and a visual device are installed at the rear of an automobile.

FIG. 30 is an explanatory diagram in the case where a moving camera and a visual device are installed at a rear portion of a wheelchair.

FIG. 31 is a block diagram of a security device in a case where images captured by four mobile cameras are displayed on three displays.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Moving camera 2 Visual device 3 Display 4 Speaker 5 Speech synthesizer 6 Microphone 7 Speech recognition device 8 Central control device 11 Image acquisition means 12 Image storage means 14 Edge information generation means 15 Edge information formation means 16 Object / background separation means 17 Position / Size detection means 20 camera / environment coordinate conversion means 21 image / environment coordinate conversion means 22 position selection means 23 movement control means 24 control command input means 25 vibration command generation means 26 camera command generation means 27 area normalization means 29 image recognition Means 31 Environmental understanding means 32 Clocking means 33 Environment map holding means 34 Object position estimating means 37 Geometric analysis means 40 Array operation unit 41 Processor 42 Memory 43 Controller 51 Address bus 52 Data bus 53 Address buffer 54 Address data Coder 55 Data buffer 56 Internal data bus 57 Flag register 58 Flag decoder 59 Flag encoder 60 Status register 61 Previous input data register 62 Result data register 63 Output data register 64 Upper input data register 65 Lower input data register 66 Left input data register 67 Right input data register 110 Data processing device 111 Digital image 112 Coarse edge information 113 Coarse edge information image 114 Formed edge information 115 Formed edge information image 116 Low resolution digital image 117 Low resolution coarse edge information image 118 Low resolution cut out coarse edge information image 119 cut-out coarse edge information image 120 cut-out digital image 121 cut-out formed edge information image 131 overlap information 132 overlap information image 141 object area 142 object Area image 143 Separated object area 144 Normalized area 145 Normalized image 146 Template image 147 Matching result image 151 Triangular edge information 152 Triangular inner area 153 Triangular outer area 154 Triangular edge information in broken line state 155 Broken triangular inner area 156 Outside area of broken line triangle 157 Front triangle edge information 158 Back triangle edge information 159 Front triangle inside area 160 Back triangle inside area 161 Double triangle background area 162 Two overlapping circular edge information 163 Front Circular inner area 164 Back circular inner area 165 Double circle background area 301 Environmental map input means 302 Map / text conversion means 303 Map text holding means 304 Command text input means 305 Text / command conversion means 3 06 Control command holding means 307 Video signal input means 308 Text insertion means 309 Video signal output means 311 Exchanger 321 Car 322 Center of rear bumper 323 Center of ceiling rear 324 Pedestrian 325 Motorcycle 326 Shooting range 331 Seat 332 Shooting section 333 Interface section

 ──────────────────────────────────────────────────続 き Continued on the front page F-term (reference) DD16 DD17 DD24 DD49 EE05 EE14 EE18 FF01 FF02 FF19 FF20 GG02 GG11 GG12 GG18 GG20 GG21 GG23 GG29 GG30 GG32 GG54 GG66 5L096 BA02 BA04 CA04 DA03 DA04 EA06 FA06 FA69 GA05 GA08 HA04 H15HH GG

Claims (9)

[Claims] 1. An alerting device for searching for an object within 1000 meters, comprising: a means for generating a moving image of the object using a moving camera; and a position and the position of the object from the moving image using a visual device. Means for controlling the direction and magnification of the moving camera by detecting the size, recognizing the type of the object, and inputting a control command; means for displaying the moving image using a display; Means for converting map text into artificial voice using a synthesizer, means for outputting the artificial voice using a speaker, means for inputting a user's voice using a microphone, and Means for recognizing the user's voice and converting it to a command text; and using a central controller, the type of the object, the position, and the change in the position are displayed on the map text. By converting the command text into the control command, and inserting the map text and the command text into the moving image, so that the moving camera, the display, the visual device, the voice synthesizer, and the voice Means for mediating communication between the recognition devices. 2. A means for acquiring a frame image of the moving image for an object in the moving image captured by a moving camera; a means for sequentially storing the frame image as a digital image; Means for generating an edge information image; means for generating a formed edge information image from the coarse edge information image using the digital image; means for separating the object region from the background using the formed edge information image; Means for normalizing the object area to generate a normalized image; means for geometrically analyzing the formed edge information image; means for recognizing the type of the object area from the normalized image; and the coarse edge information image. Means for detecting the position and size of the divided object area; and means for converting the orientation and magnification of the moving camera into environmental coordinate positions. Means for converting the position and the size of the object area into the position of the environment coordinates; means for generating an environment map represented by the environment coordinates; and Means for estimating a position; means for selecting one of the environmental coordinates with respect to the plurality of object regions; means for inputting a control command from outside; generation of a vibration command for vibrating the mobile camera An alerting device, comprising: means for determining the position of the environmental coordinates to which the mobile camera should move; and means for generating camera instructions for controlling the mobile camera. 3. A moving camera, a display, a visual device,
A central control device connected to a voice synthesis device and a voice recognition device, a unit configured to input an environment map generated by the visual device, and a type, a position, and a change in the position of an object region described in the environment map. Means for converting the command text into map text; means for holding the map text output to the speech synthesizer; means for inputting a command text output by the speech recognition device; and Means for converting to a control command; means for holding the control command; means for inputting a video signal generated by the mobile camera; means for inserting the map text and the command text into the video signal; and the map Means for outputting, to the display, the video signal into which the text and the instruction text have been inserted. Alert device to. 4. The vehicle according to claim 1, wherein the mobile camera and the visual device are installed at a position where a range of 90 to 270 degrees can be taken clockwise from the front of the vehicle and the ship. By installing the display, the microphone, the speaker, the central control device, the voice synthesizer, and the voice recognition device in the driver's seat and the cockpit of the ship, the object within 1000 meters from the moving camera An alerting method for searching for the object approaching the moving camera and the object moving away from the moving camera. 5. The security device according to claim 1, wherein the vehicle and the ship are clockwise from 0 to 180 degrees or 18 from the front.
The moving camera and the visual device are installed at a position capable of photographing a range of 0 to 360 degrees, and the display, the microphone, the speaker, the central control device, the central control device, and the driver's seat of the vehicle and the cockpit of the ship are installed. By installing a speech synthesis device and the speech recognition device, the object within 1000 meters from the moving camera, the object approaching the moving camera, and the object moving away from the moving camera A vigilant way to explore. 6. The security device according to claim 1, wherein the angle is 0 to 90 degrees clockwise and 270 degrees from the front of the vehicle and the ship.
The mobile camera and the visual device are installed at a position capable of photographing a range of up to 360 degrees, and the display, the microphone, and the driver's seat of the vehicle and the cockpit of the ship are installed.
By installing the speaker, the central control device, the speech synthesizer, and the speech recognition device, the object within 1000 meters from the moving camera, the object approaching the moving camera, An alerting method for searching for an object moving away from a moving camera. 7. The alerting device according to claim 1, wherein 2 to 200 of the moving cameras and the visual device, and 1 to 200 of the displays are used. A security device that searches for objects at the same time. 8. An object coated with an ultraviolet-reflective paint and an ultraviolet-reflective film by incorporating a visible light image pickup device and an ultraviolet image pickup device into the moving camera of the security device according to claim 1. An alerting method for selectively searching for the stretched object. 9. An alerting method according to claim 8, wherein the object which is coated with the ultraviolet reflecting paint is selectively searched for by emitting ultraviolet rays using an ultraviolet emitting device.