JP2004168304A - Vehicular travel guiding device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To certainly detect various solid objects existing in a traveling direction before a vehicle passes a narrow road, and to reduce a burden of a driver and ensure safety by informing the driver of a clearance between the detected solid object and an own vehicle. <P>SOLUTION: If a pair of right and left stereo images picked up by a stereo optical system 10 is processed by a stereo image processing means 15 to calculate a three-dimensional distance distribution over the whole of the image, a three-dimensional position of the solid abject on a road is detected from the distance distribution information by a solid object detecting means 100. The right and left closest distance between a detected front side wall or an edge of the solid abject and an extension of an own vehicle 1 side portion is calculated by a clearance distance calculating means 110 as a clearance distance. Information related to the right and left clearance distance is indicated on a display as an informing means 115, thereby informing that to the driver. Therefore, the driver can check whether safety passing is possible or not before passing the narrow road, so that the burden of the driver is reduced and a minor collision can be prevented. <P>COPYRIGHT: (C)2004,JPO

Description

  The present invention relates to a vehicle travel guide device that reduces the burden on a driver when passing through a gorge and ensures safety.

  2. Description of the Related Art Conventionally, in vehicles such as automobiles, Japanese Unexamined Utility Model Publication No. 5-68742 discloses a vehicle that supplements a driver's feeling when passing through a narrow road where many walls, guardrails, electric poles, and parked vehicles exist. Corner poles, tactile sensors consisting of contact switches that turn on when a bar-shaped member touches an obstacle, etc. The gap can be confirmed.

Recently, ultrasonic sensors have been installed on the sides and four corners of the vehicle to measure the distance by emitting ultrasonic waves and receiving reflected waves from obstacles. Techniques have been developed to reduce the burden of passing through a building.
JP-A-5-68742

  However, in the case where a mark that is a mark like the above-described corner pole is attached to the outside of the vehicle body, the driver's familiarity is required, and the effect of reducing the burden on the driver is small. Further, in the case of a contact type sensor such as a tactile sensor, the position cannot be confirmed until the contact with an object occurs, and a situation arises in which the steering wheel cannot be operated in time after the contact with an obstacle.

  Furthermore, in the technology using ultrasonic waves, the spatial resolution is poor, so that not only can the positional relationship of obstacles not be known, but also object-dependent that the ultrasonic waves emitted from pedestrians' clothes or smooth walls do not return. It is difficult to deal with various three-dimensional objects existing on the road.

  The present invention has been made in view of the above circumstances, and reliably detects various three-dimensional objects existing in the traveling direction before the vehicle passes through a narrow road, and detects a gap between the detected three-dimensional object and the own vehicle. An object of the present invention is to provide a traveling guide device for a vehicle that can reduce a burden on a driver and ensure safety by notifying the driver.

  In order to achieve the above object, a traveling guide device for a vehicle according to the present invention includes a three-dimensional object detection unit that detects a three-dimensional object based on a captured image outside the vehicle, and a gap distance between the three-dimensional object detected by the three-dimensional object detection unit. It is characterized by comprising: a gap distance calculating means for calculating; and a display device for displaying information relating to the gap distance calculated by the gap distance calculating means.

  The vehicle travel guide device of the present invention can reliably detect various three-dimensional objects existing in the traveling direction before the vehicle passes through a narrow road and inform the driver of a gap distance between the vehicle and the vehicle. Excellent effects are obtained, such as safety can be ensured by reducing the burden on the user.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. 1 is an overall configuration diagram of a travel guide device, FIG. 2 is a circuit block diagram of the travel guide device, FIG. 3 is an explanatory diagram showing a screen of a display, and FIG. 4 is a front view of a vehicle. FIG. 5, FIG. 5 is an explanatory diagram showing the relationship between the camera and the subject, FIG. 6 is a detailed circuit diagram of an image processor, FIG. 7 is an explanatory diagram of a city block distance calculation circuit, and FIG. 8 is a block diagram of a minimum value detection circuit. 9 is an explanatory view showing an example of an image captured by a vehicle-mounted CCD camera, FIG. 10 is an explanatory view showing an example of a distance image, FIG. 11 is a top view of the vehicle, FIG. 12 is a side view of the vehicle, and FIG. FIG. 14 is an explanatory diagram showing a method of dividing an image, FIG. 15 is an explanatory diagram showing a relationship between a detected object and a histogram, and FIG. 16 is an example of a detection result and a detection distance of an existing region of the object. FIG. 17 is an explanatory view showing an object detection 3 FIG. 18 is an explanatory diagram showing the shape of a two-dimensional window for object detection, FIG. 19 is an explanatory diagram showing an example of data constituting the contour of an object, and FIG. 20 is an outline image of the object. FIG. 21 is an explanatory diagram showing a shape of a search area in side wall detection, FIG. 22 is an explanatory view showing a side wall search area on an image, and FIG. 23 is a distribution of three-dimensional object data. FIG. 24 is an explanatory diagram showing the assumption of a straight line in the Hough transform, FIG. 25 is an explanatory diagram showing a voting region in the parameter space, FIG. 26 is an explanatory diagram showing a side wall candidate region, and FIG. FIG. 28 is an explanatory diagram showing the relationship of the existence range of the side wall, FIG. 28 is an explanatory diagram showing the detection result of the side wall, FIG. 29 is an explanatory diagram of the calculation of the gap distance, FIG. 30 is a flowchart showing the operation of the image processor, and FIG. No FIG. 32 is a timing chart showing the operation of the city block distance calculation circuit, FIG. 33 is a timing chart showing the operation of the deviation amount determination unit, FIG. 34 is a timing chart showing the entire operation of the image processor, and FIG. 36 and 37 are flowcharts of the object detection processing, FIGS. 37 and 38 are flowcharts of the side wall detection processing, and FIG. 39 is a flowchart of the gap distance calculation processing.

  In FIG. 1, reference numeral 1 denotes a vehicle such as an automobile. The vehicle 1 is equipped with a travel guide device 2 having one function of an active drive assist (ADA) system for performing advanced support control for a driver. The burden on the driver when passing through narrow roads where fences, guardrails, telephone poles, parked vehicles and the like exist is reduced, and safety is ensured.

  The travel guide device 2 processes a stereo optical system 10 composed of a pair of left and right cameras as an imaging system for capturing an object outside the vehicle, and processes a pair of left and right stereo images captured by the stereo optical system 10. A stereo image processing means 15 for calculating a three-dimensional distance distribution over the whole, and distance distribution information from the stereo image processing means 15 are input, and a continuous boundary serving as a road boundary such as a fence or a guardrail is input from the distance distribution information. Three-dimensional object detecting means 100 for detecting at high speed the three-dimensional position of a three-dimensional object on a road such as a side wall as a three-dimensional object, another vehicle, a pedestrian, a building, etc., and a front part detected by the three-dimensional object detecting means 100 A gap distance calculating means 110 for calculating, as a gap distance, left and right closest distances between the side wall or the edge of the three-dimensional object and an extension of the side of the own vehicle 1, and the gap distance calculating means 110 calculates the distance. And a notification unit 115 for notifying information related to the left and right gap distance to the driver.

  FIG. 2 is a circuit block diagram showing a hardware configuration of the travel guide device 2. The stereo optical system 10 uses a CCD camera using a solid-state imaging device such as a charge-coupled device (CCD), and will be described later. A pair of left and right CCD cameras 11a and 11b for short distance and one pair of left and right CCD cameras 12a and 12b for long distance are used as the stereo image processing means 15 in the stereo optical system 10. Is connected to the image processor 20 for realizing the above function.

  Further, a distance image processing computer 120 for realizing functions as the three-dimensional object detection means 100 and the gap distance calculation means 110 is connected to the image processor 20, and the distance image processing computer 120 is provided with the notification means. The display 3 as 115 is connected.

  The image processor 20 searches two stereo image pairs captured by the stereo optical system 10 for a portion where the same object is shown for each minute region, and calculates a corresponding positional shift amount to the object. , And a distance image memory 20b that stores distance information output from the distance detection circuit 20a.

  Further, the range image processing computer 120 mainly performs a process of detecting individual objects, a microprocessor 120a for mainly performing a process of detecting a side wall, and mainly performs a process of calculating a gap distance. Microprocessor 120c is connected in parallel via a system bus 121 to form a multi-microprocessor system configuration.

  The system bus 121 stores an interface circuit 122 connected to the distance image memory 20b, a ROM 123 for storing a control program, a RAM 124 for storing various parameters in the middle of calculation processing, and a parameter for a processing result. Output memory 125, display controller (DISP.CONT.) 126 for controlling display (DISP) 3, vehicle speed sensor 4, steering angle sensor 5 for detecting steering angle of steering, ADA support mode for driver And an interface circuit 127 for inputting signals from sensors and switches provided in the vehicle 1 such as a mode setting switch 6 for selecting a switch.

  In the distance image processing computer 120, a memory area used by each of the microprocessors 120a, 120b, and 120c is divided, and object detection processing and side wall detection processing are performed based on distance information from the image processor 20. When the mode setting switch 6 is operated by the driver and a prescribed signal is input, a gap distance calculation process is performed from the detected data of the three-dimensional object, and the result is displayed on the display 3.

  As shown in FIG. 3, on the screen of the display 3, a monitor 3a for displaying a scene in front of the vehicle in a state of a distance image to be described later is provided at a central portion, and on a side of the monitor 3a. , A mode display unit 3b for displaying each mode (cruise, guide, assist, check) of the ADA, and a corresponding display of the monitor unit 3a is turned on in response to an operation input of the mode setting switch 6. ing.

  Above the monitor 3a, a data display 3c and a position display 3d representing the front, rear, left and right of the vehicle in a shape obtained by dividing an ellipse into four parts are provided. In accordance with the mode, a mode data display section 3e is provided for displaying the calculated numerical values together with characters such as the following distance, speed, left gap distance, right gap distance, and the like.

  In the present invention, when the driver operates the mode setting switch 6 to select the guide mode when a situation has to pass through a narrow road where a large number of three-dimensional objects are present, the mode display unit The "guide" display portion 3b is turned on, and based on the distance image obtained by processing the stereo image pair captured by the stereo optical system 10, the edge of a three-dimensional object such as a side wall, a telephone pole, a parked vehicle, etc., around the road and the own vehicle are displayed. The left and right distances from the extension of one side are calculated, and the minimum value is determined as the closest distance, that is, the gap distance.

  Then, as shown in FIG. 3, a numerical value of the gap distance is displayed below the characters of the left gap distance and the right gap distance of the mode data display section 3e, and the left and right portions of the position display section 3d are lit. When the calculated gap distance is equal to or less than 0, the position display unit 3d turns on the corresponding side in red to warn of contact if the calculated gap distance is less than 0. If the gap distance is larger than 0 and less than about 20 cm, There is a risk that contact may occur due to inadvertent steering operation of the driver, so the corresponding side is lit in yellow, and if the gap distance is larger than about 20 cm, there is no The corresponding side is illuminated in green to indicate that it is low.

  That is, by operating the mode setting switch 6 before passing through the narrow road where the side wall, the telephone pole, the parked vehicle, and the like exist, the driver can confirm whether or not the vehicle can pass safely on the screen of the display 3. Therefore, the burden on the driver can be reduced and a collision accident or the like can be avoided. Further, since the driver can grasp the gap distance quantitatively, it is easy to learn the sense of vehicle position, and it is possible to promote safe driving.

  In addition, for simplicity, instead of the display 3, a display device such as a bar graph whose lighting portion changes according to the value of the gap distance may be employed. May be replaced by a means such as sound or voice instead of a visual display.

  Hereinafter, a function related to image processing by the image processor 20, a function related to three-dimensional object detection by the distance image processing computer 120, and a function related to gap distance calculation when the guide mode is selected by the mode setting switch 6 will be described. . Note that the calculation of the gap distance in the image processing computer 120 is performed when the detected three-dimensional object is not blocked as an obstacle in the traveling direction of the vehicle 1.

  As shown in FIG. 4, one set of left and right cameras constituting the stereo optical system 10 is composed of two CCD cameras 11a and 11b for short distance left and right images (typically referred to as CCD cameras 11). There are two CCD cameras 12a and 12b (represented in some cases as CCD cameras 12) for long-distance left and right images, respectively, which are attached at fixed intervals in front of the ceiling in the passenger compartment. Are attached at regular intervals outside the short distance CCD cameras 11a and 11b.

  When the stereo optical system 10 measures a distance from the nearest to, for example, 100 m away, if the mounting position of the CCD cameras 11, 12 in the vehicle interior is, for example, 2 m from the end of the hood of the vehicle 1, it is actually 2 m in front. It suffices if the position from 100 m to 100 m can be measured.

  Therefore, the short-range CCD cameras 11a and 11b measure the position from 2 m to 20 m in front, and the long-range CCD cameras 12a and 12b measure the position from 10 m to 100 m in front. The CCD camera 11 and the long distance CCD camera 12 have an overlap between 10 m and 20 m in front and can measure the entire range while ensuring reliability.

  In the following, a method of calculating the distance by the image processor 20 will be described by taking the short distance CCD camera 11 as an example. However, the distance is calculated for the long distance CCD camera 12 by the same processing. In the travel guide device 2 according to the present embodiment, the short distance CCD camera 11 is used to detect the three-dimensional object immediately before.

  That is, as shown in FIG. 5, when the mounting interval between the two short-range CCD cameras 11a and 11b is r, and a point P is located at a distance D from the installation surface of the two cameras 11a and 11b, Assuming that the focal lengths of the two cameras 11a and 11b are both f, the image of the point P is projected on a projection plane separated by f from the focal position for each camera.

At this time, the distance from the position of the image on the right CCD camera 11b to the position of the image on the left CCD camera 11a is r + x, and if this x is the shift amount, the distance D to the point P is equal to the shift amount x. It can be obtained by the following equation.
D = r × f / x (1)

  The shift amount x between the left and right images can be detected by extracting some feature such as an edge, a line segment, or a special shape, and finding out a portion where those features match, but in order to avoid a decrease in the amount of information, In the image processor 20, when finding the image of the same object in the left and right images, the image is divided into small regions, and the brightness or color pattern in each small region is compared with the left and right images to find a corresponding region. The distance distribution for each small area is obtained over the entire screen.

The evaluation of the degree of coincidence of the left and right images, that is, stereo matching, is as follows, assuming that the luminances (colors may be used) of the i-th pixel of the right image and the left image are Ai and Bi, respectively. When the minimum value of the city block distance H between the small regions of the left and right images satisfies a predetermined condition, it can be determined that the small regions correspond to each other.
H = Σ | Ai−Bi | (2)

  In the stereo matching based on the city block distance H, the amount of information is not reduced by subtracting the average value, and the calculation speed can be improved because there is no multiplication. However, the size of the small region to be divided is too large. Then, the possibility that a distant object and a nearby object are mixed in the area increases, and the detected distance becomes ambiguous. Therefore, it is better that the area is small in order to obtain the distance distribution of the image. However, if the area is too small, the amount of information for checking the degree of coincidence is insufficient. Therefore, for example, if the number of pixels divided into four is set to the maximum value of the area width so that a vehicle having a width of 1.7 m 100 m ahead is not included in the same area as a vehicle in an adjacent lane, There are four pixels for the stereo optical system 10. As a result of trying the optimum number of pixels in an actual image based on this value, the number of pixels becomes four in both the vertical and horizontal directions.

  In the following description, it is assumed that the image is divided into 4 × 4 small areas to check the degree of coincidence between the left and right images, and that the stereo optical system 10 is represented by short-range CCD cameras 11a and 11b.

  The circuit details of the image processor 20 are shown in FIG. 6. In this circuit, an image conversion unit 30 for converting an analog image captured by the stereo optical system 10 into a digital image is provided to a distance detection circuit 20a. A city block distance calculation unit 40 that sequentially calculates the city block distance H for determining the shift amount x of the left and right images with respect to the image data from 30 while shifting the pixels one by one, a minimum value HMIN of the city block distance H. And a minimum / maximum value detection unit 50 for detecting the maximum value HMAX, and checks whether the minimum value HMIN obtained by the minimum / maximum value detection unit 50 indicates the coincidence of the left and right small areas, and determines the amount of deviation. A shift amount determining unit 60 for determining x is provided, and a dual port memory 90 is employed as the distance image memory 20b.

  The image conversion unit 30 includes A / D converters 31a and 31b corresponding to the left and right image CCD cameras 11a and 11b, respectively. Each of the A / D converters 31a and 31b has a lookup table (LUT) as a data table. ) 32a, 32b and image memories 33a, 33b for storing images taken by the CCD cameras 11a, 11b, respectively. Note that the image memories 33a and 33b can be constituted by relatively low-speed memories because part of the image is repeatedly extracted and processed by the city block distance calculation unit 40 as described later, thereby reducing costs. be able to.

  Each of the A / D converters 31a and 31b has a resolution of, for example, 8 bits, and converts analog images from the left and right CCD cameras 11a and 11b into digital images having a predetermined luminance gradation. That is, when binarization of an image is performed for speeding up processing, information for calculating the degree of coincidence between the left and right images is significantly lost, so that the image is converted to, for example, a 256-level gray scale.

  The LUTs 32a and 32b are configured on a ROM. The LUTs 32a and 32b increase the contrast of a low-brightness portion of the image converted into a digital amount by the A / D converters 31a and 31b, and adjust the characteristics of the left and right CCD cameras 11a and 11b. Correct for differences. Then, the signals converted by the LUTs 32a and 32b are temporarily stored in the image memories 33a and 33b.

  In the city block distance calculator 40, two sets of input buffer memories 41a and 41b are connected to a left image memory 33a of the image converter 30 via a common bus 80, and a right image is stored. Two sets of input buffer memories 42a and 42b are connected to the memory 33b via a common bus 80.

  Two sets of, for example, eight-stage shift registers 43a and 43b are connected to the input buffer memories 41a and 41b for the left image, and the input buffer memories 42a and 42b for the right image similarly have 2 shift registers. A set of, for example, eight-stage shift registers 44a and 44b is connected. Further, a city block distance calculation circuit 45 for calculating a city block distance is connected to these four sets of shift registers 43a, 43b, 44a and 44b.

  The shift registers 44a and 44b for the right image are connected to two sets of ten-stage shift registers 64a and 64b of a shift amount determination unit 60, which will be described later, and the data transfer of the next small area starts. The old data for which the calculation of the city block distance H has been completed is sent to these shift registers 64a and 64b, and used for determining the shift amount x.

  Further, the city block distance calculation circuit 45 combines a high-speed CMOS type arithmetic unit 46 which is made into one chip by connecting an input / output latch to an adder / subtracter. As shown in detail in FIG. In a pipeline structure connected in a pyramid shape, calculation is performed by inputting, for example, eight pixels at the same time. The first stage of the pyramid structure has an absolute value calculator, the second to fourth stages constitute a first adder, a second adder, and a third adder, respectively, and the last stage is a sum adder. . In FIG. 7, only half of the absolute value calculation and the first and second adders are shown.

  Each of the input buffer memories 41a, 41b, 42a, and 42b is a high-speed type having a relatively small capacity corresponding to the speed of calculating the city block distance. , The address generated by the # 1 address controller 86 is commonly provided. The transfer to and from the four sets of shift registers 43a, 43b, 44a, 44b is controlled by a # 2 address controller 87.

  The minimum / maximum value detection unit 50 includes a minimum value detection circuit 51 that detects a minimum value HMIN of the city block distance H and a maximum value detection circuit 52 that detects a maximum value HMAX of the city block distance H, The configuration is such that two high-speed CMOS arithmetic units similar to the city block distance calculation circuit 45 are used for detecting the minimum value and the maximum value, and the output of the city block distance H is synchronized.

  As shown in FIG. 8, specifically, the minimum value detection circuit 51 includes a C latch 53 and a latch 54 in an arithmetic unit 46 including an A register 46a, a B register 46b, and an arithmetic and logic operation unit (ALU) 46c. , D latch 55, and the output from the city block distance calculation circuit 45 is input to the A register 46a and the B register 46b via the C latch 53, and the most significant bit (MSB) of the output of the ALU 46 is input. ) Is output to the latch 54. The output of the latch 54 is output to the B register 46b and the D latch 55, and the value in the middle of the calculation of the minimum value in the arithmetic unit 46 is stored in the B register 46b. 55. The maximum value detection circuit 52 has the same configuration as that of the minimum value detection circuit 51, except that the logic is reversed and the deviation amount δ is not stored.

  As described above, the city block distance H is sequentially calculated while shifting the left image small area by one pixel with respect to one right image small area. Therefore, each time the value of the city block distance H is output, it is compared with the maximum value HMAX and the minimum value HMIN of the previous values and updated, so that the small area is almost simultaneously output with the output of the last city block distance H. , The maximum value HMAX and the minimum value HMIN of the city block distance H are determined.

  The shift amount determination unit 60 is configured as a relatively small-scale RISC processor, and has a computing unit 61 as a center, two 16-bit data buses 62a and 62b, a latch 63a for holding a shift amount x, a first Latch 63b holding threshold value HA as a specified value, latch 63c holding threshold value HB as a second specified value, latch 63d holding threshold value HC as a third specified value, right image Two shift registers 64a and 64b for holding the luminance data of the above, a switch circuit 65 for receiving the output of the arithmetic unit 61 and outputting the shift amount x or "0", and an output buffer memory 66a for temporarily storing the output result , 66b, and a 16-bit width ROM 67 in which a control program for the operation timing of the circuit and the function of the arithmetic unit 61 is written.

  The arithmetic unit 61 includes an A register 71, a B register 72, an F register 73, and a selector 74 with an ALU 70 as a center. The A register 71 is connected to the data bus 62a (hereinafter, referred to as an A bus 62a). In addition, a B register 72 is connected to the data bus 62b (hereinafter, referred to as a B bus 62b), and the switch circuit 65 is operated based on the operation result of the ALU 70, and the shift amount x or “0” is stored in the output buffer memory 66a. , 66b.

  The A bus 62a is connected to latches 63b, 63c, 63d holding the threshold values HA, HB, HC, and the maximum value detection circuit 52, and the B bus 62b is connected to the minimum value detection circuit 51. It is connected. Further, the shift registers 64a and 64b are connected to the A bus 62a and the B bus 62b.

  The arithmetic unit 61 is connected to the switch circuit 65, and the minimum value detection circuit 51 is connected to the switch circuit 65 via the latch 63a. The arithmetic unit 61 determines three check conditions described later. The output to the output buffer memories 66a and 66b is switched according to the determination result.

  The shift amount determining unit 60 checks whether or not the obtained minimum value HMIN of the city block distance H really indicates the coincidence of the left and right small areas, and only the output buffer memories 66a and 66b which satisfy the condition are satisfied. Is output to the position of the pixel corresponding to.

That is, the displacement amount that minimizes the city block distance H is the displacement amount x to be obtained. If the following three check conditions are satisfied, the displacement amount x is output. "0" is output without adoption.
(1) HMIN ≦ HA (When HMIN> HA, the distance cannot be detected.)
(2) HMAX−HMIN ≧ HB (This is a condition for checking that the obtained minimum value HMIN is clearly lower than the fluctuation due to noise, and is not a difference from a value near the minimum value HMIN but a maximum value. By using the difference from the value HMAX as a check target, distance detection can be performed even on an object such as a curved surface whose luminance gradually changes.)
(3) Luminance difference between adjacent pixels in the horizontal direction in the small region of the right image> HC (If the threshold value HC is increased, edge detection is performed, but it is possible to cope with a case where the luminance changes slowly. In addition, the threshold value HC is much lower than the normal edge detection level, which is based on the basic principle that distance detection cannot be performed in a portion where there is no change in luminance. Since the process is performed for each pixel, only the pixels for which the distance is actually detected are used in the small area, and a natural result is obtained.)
The distance distribution information as the final result output from the shift amount determination unit 60 is written via a common bus 80 to a dual port memory 90 as the distance image memory 20b.

  The distance distribution information output from the image processor 20 has a form like an image (distance image), and an image captured by the CCD camera 11, for example, as shown in FIG. When an image obtained by capturing a three-dimensional object such as the guardrail 400 (FIG. 9 shows an image captured by one camera) is processed by the image processor 20, an image shown in FIG. 10 is obtained.

  In the example of the distance image shown in FIG. 10, the image size is 400 pixels horizontally × 200 pixels vertically, and the portion having the distance data is a black dot portion, which is one of the pixels in the image shown in FIG. This is a portion where the brightness change is large between adjacent pixels. As shown in FIG. 10, the coordinate system on the image has an origin at the upper left corner, an i coordinate axis in the horizontal direction, a j coordinate axis in the vertical direction, and a unit of pixel.

  This distance image is read by the distance image processing computer 120, a plurality of objects such as other vehicles and obstacles present ahead are detected, their positions and sizes are calculated, and a contour image of the detected object is obtained. Is extracted. Further, it is also possible to calculate the relative speed with respect to the own vehicle due to the time change of the position.

  In this case, the distance image processing computer 120 uses the three-dimensional position information of the object, distinguishes the road from the object by the height from the road surface, and distinguishes the object from the background by the value of the distance. . Therefore, in the distance image processing computer 120, first, the coordinate system of the distance image from the image processor 20 is converted into the coordinate system of the real space surrounding the own vehicle (vehicle 1). Calculate position and size.

That is, as shown in FIGS. 11 and 12, the coordinate system in the real space is a fixed coordinate system of the vehicle 1, the X axis is on the right side of the vehicle 1, the Y axis is above the vehicle 1, and the Z axis is the vehicle 1 Assuming that the front and the origin are the road surface just below the center of the two CCD cameras 11a (12b) and 11b (12b), the XZ plane (Y = 0) matches the road surface when the road is flat. In order to calculate the three-dimensional position (X, Y, Z) of the subject from the distance information (i, j, Z) in the image, a kind of coordinate transformation is performed by the following equations (3) and (4). .
Y = CH−Z × PW × (j−JV) (3)
X = r / 2 + Z × PW × (i-IV) (4)
Here, CH: CCD camera 11 (CCD camera 12)
Mounting height,
PW: viewing angle per pixel,
JV, IV: on the image at infinity point in front of vehicle 1
Coordinates.

An expression for calculating the position (i, j) on the image from the three-dimensional coordinates (X, Y, Z) in the real space is also obtained by modifying the expressions (3) and (4).
j = (CH−Y) / (Z × PW) + JV (5)
i = (X−r / 2) / (Z × PW) + IV (6)
When the mounting position of the CCD camera 11 is indicated by the XYZ coordinate system in the real space, for example, the right CCD camera 11b has X = 0.20m, Y = 1.24m, Z = 0.0m, In the left CCD camera 11a, X = −0.20 m, Y = 1.24 m, and Z = 0.0 m.

  FIG. 13 shows a functional configuration of the distance image processing computer 120, which mainly includes an object recognition unit 130 using a microprocessor 120a, a side wall detection unit 140 using a microprocessor 120b, and a gap distance calculation unit 160 using a microprocessor 120c. The processing results of the object recognizing unit 130 and the side wall detecting unit 140 are roughly stored in the three-dimensional object parameter storing unit 150 including the output memory 125, and are used for the microprocessor 120c (gap distance calculating unit 160) and the ADA. Is read by another device (not shown).

  Further, the function of the object recognition unit 130 is subdivided into an object detection unit 131, a three-dimensional window generation unit 132, and an object contour image extraction unit 133. The object detection unit 131 converts the distance image from the image processor 20 into a grid. The data is divided at predetermined intervals (for example, at intervals of 8 to 20 pixels), and only data of a three-dimensional object that may be an obstacle to traveling is selected for each region, and the detection distance is calculated.

  FIG. 14 is an explanatory diagram in which a grid-like region is set on a bright and dark image obtained by capturing a scene in front of the vehicle 1. In practice, the distance image in FIG. 10 is divided as described above. In this embodiment, 33 small areas AR1 to AR33 are divided at intervals of 12 pixels. That is, a plurality of objects can be detected simultaneously by dividing the image into a number of regions and searching for the objects.

For the subject in each area, the three-dimensional position (X, Y, Z) in the real space is calculated from the coordinates (i, j) on the image and the distance data Z using the above-described equations (3) and (4). Further, the height H of the subject from the road surface at the distance Z can be calculated by the following equation (7), where Yr is the height of the road surface, as shown in FIGS. Is set, Yr = 0.0 m can be set as long as the vehicle 1 does not have a special inclination or vertical movement.
H = Y-Yr (7)

  A subject having a height H of about 0.1 m or less is considered to be a white line, dirt, shadow, or the like on a road, and is not considered to be an obstacle to traveling. Therefore, data of the subject is rejected. In addition, subjects located above the height of the vehicle are rejected because they are considered to be pedestrian bridges or signs, and only data on three-dimensional objects that become obstacles on the road are selected. Thus, even if an object overlaps a road or the like on a two-dimensional image, data can be distinguished based on the height from the road surface, and only the object can be detected.

  Next, with respect to the data of the three-dimensional object extracted in this manner, the number of data included in the section of the preset distance Z is counted, and a histogram having the distance Z as a horizontal axis is created. FIG. 15 is a histogram in which, for example, the parking vehicle 300 on the front right side in the image example of FIG. 9 is a detection object, and the length and the number of sections of the distance Z to be set are the detection limit and accuracy of the distance Z and the detection target. Is determined in consideration of the shape of the object.

  In the histogram, since the distance data in the input distance image includes a value which is erroneously detected, some data appears at a position where no object is actually present. However, when there is an object of a certain size, the frequency at the position shows a large value, while when there is no object, the frequency generated only by erroneous distance data becomes a small value.

  Therefore, if there is a section in which the frequency of the created histogram is equal to or greater than a predetermined determination value and has a maximum value, it is determined that an object exists in that section, and if the maximum value of the frequency is equal to or less than the determination value, the object is determined. By determining that the object does not exist, the object can be detected while minimizing the influence of the noise even when the image data includes some noise.

  When it is determined that an object exists, the average value of the distance Z between the detected section and the data of the three-dimensional object included in the preceding and succeeding sections is calculated, and this value is regarded as the distance to the object. After performing such a process of detecting the distance to the object for the entire area, the detection distance of the object in each area is checked, and if the difference in the detection distance to the object in the adjacent area is equal to or less than the set value, the same object is detected. On the other hand, if it is equal to or greater than the set value, it is regarded as a separate object.

  Specifically, first, the left end area AR1 is examined, and if an object is detected, it is set as the object S1 and the distance is set as Z1. Next, the area AR2 on the right side is examined. If no object is detected, the object S1 exists inside and near the area AR1, the distance is determined to be Z1, the object is detected, and the detection distance is determined. Is Z2, the difference between the distances Z1 and Z2 is checked.

  If the difference between the distances Z1 and Z2 is equal to or greater than the set value, it is determined that the object detected in the area AR2 is different from the previously detected object S1, and is newly set as the object S2 and the distance Z2. Check the area.

  On the other hand, when the difference between the distances Z1 and Z2 is equal to or less than the set value, the object detected in the area AR2 is determined to be the previously detected object S1, and the distance is set to the average value of Z1 and Z2. Further, the area on the right side is sequentially examined, and if it is determined that the object S1 is continuously present, the distance and the existing area are updated.

  Conventionally, it has been difficult to extract only the data of an object when a distant background appears around the object on a two-dimensional image. However, the above processing is performed from the left end area AR1 to the right end area AR1. By performing the processing up to the area AR33 and distinguishing the data by the value of the distance, it is possible to detect a plurality of objects, their distances, and the existence area separately from the background, and furthermore, the plurality of objects overlap on the two-dimensional image. Even in the case where they are shown, they can be distinguished and detected depending on the difference in the distance between the objects.

  It has been experimentally obtained that the set value should be about 4 m to 6 m when detecting an automobile, and about 1 m to 2 m when detecting a pedestrian.

  FIG. 16 shows the existence area of the object detected by the above processing by a frame line. In this example, two objects of the left vehicle 200 and the right vehicle 300 are detected. Numerical values shown on the lower side of the figure are detection distances of each object.

  In the three-dimensional window generating unit 132, for each of the objects detected by the object detecting unit 131, in a three-dimensional space as shown in FIG. That is, a three-dimensional window WD3 is set, how the set three-dimensional window WD3 looks on a two-dimensional image is calculated, and the inside of the window contour is defined as a two-dimensional window WD2, and only the data in the window is defined. Make it a detection target.

  The horizontal width of the three-dimensional window WD3 is a range extended by one region to the left and right from the region where the object exists. This is because, if only a part of the left and right ends of an object hangs over a certain area, that object does not have a large value on the histogram and another object may be detected. In consideration of the above, the range of the window is expanded.

  Further, the length of the three-dimensional window WD3 in the distance Z direction is a range obtained by adding the section length of the histogram at the detection distance of the object before and after the detection distance. The lower end of the three-dimensional window WD3 is a position obtained by adding about 0.1 m to the height of the road surface, and the upper end is the upper end of each area divided for object detection.

  In order to obtain the two-dimensional window WD2 from the three-dimensional window WD3, from the coordinates (Xn, Yn, Zn) of the eight vertices of the three-dimensional window WD3, using the above-described equations (5) and (6). The coordinates (in, jn) on the image are calculated, and a polygon enveloping these points is calculated. FIG. 18 shows an example in which a two-dimensional window WD2 is set for one of the detected objects (parked vehicle 300) shown in FIG.

  The object contour image extraction unit 133 performs a process of sequentially surveying each data in the two-dimensional window WD2, selecting only data included in the three-dimensional window WD3, and extracting a contour image of the detected object. Of the vehicle 1 with respect to the own vehicle 1 is detected.

  That is, first, each data in the two-dimensional window WD2 is sequentially surveyed for each object, and the three-dimensional positions (X, Y, and X) of the pixels having the distance data are calculated using the above-described equations (3) and (4). After calculating Z), only data whose distance and height values are within the range of the three-dimensional window WD3 are extracted, and the others are rejected.

  FIG. 19 shows the data extracted in this manner projected and displayed on a two-dimensional image. Further, when the outlines of these data are connected by a line segment, an outline image of the object as shown in FIG. 20 is obtained. The coordinates (i, j) of the left and right ends and the upper end of this contour image on the image are detected, and subsequently, using the detection distance Z of the object and the equations (3) and (4), a three-dimensional space is obtained. When the positions of the left and right ends and the upper end of the object are calculated, the horizontal width of the object is obtained from the positions of the left and right ends, and the height of the object is obtained from the upper end position. In FIG. 20, it can be determined that the object has a width of 1.7 m and a height of 1.3 m.

  On the other hand, the function of the side wall detection unit 140 by the microprocessor 120b is subdivided into a three-dimensional object data extraction unit 141, a side wall straight line detection unit 142, and a side wall range detection unit 143, and distinguishes the side wall from the road from the road surface height. By distinguishing the side wall from the distant background based on the distance in the front-rear direction and the horizontal direction, only the peripheral data where the side wall is presumed to be present is extracted, and then the data of the side wall is horizontally shifted. Attention is paid to features that are arranged in a straight line, and this is detected by Hough transform to determine the position.

  That is, the three-dimensional object data extracting unit 141 extracts only data located above a predetermined road surface from the information of the distance distribution, and the side wall straight line detecting unit 142 extracts the three-dimensional object data from the extracted three-dimensional object data. Then, only the data within the preset search area for the side wall is extracted and processed by the Hough transform to detect the presence or absence of the side wall and the linear expression indicating the position of the side wall. Further, the side wall range detection unit 143 sets a side wall candidate region in which a side wall is presumed to exist based on a linear expression indicating the position of the side wall, and determines a side wall candidate region from the distribution state of the three-dimensional object data in the side wall candidate region. Detect front and rear end positions.

  More specifically, since the side wall is a kind of three-dimensional object, first, the three-dimensional object data extraction unit 141 extracts data of the three-dimensional object above the road surface from the distance image. At this time, since a three-dimensional object having a height H of about 0.1 m or less is considered to be a white line, dirt, shadow, or the like on the road as described above, the data of the three-dimensional object is rejected. Also, the three-dimensional objects above the height of the vehicle are discarded because they are considered to be pedestrian bridges and signs, and only the data of the three-dimensional objects on the road are selected.

  The three-dimensional object data extraction unit 141 extracts a wide range of three-dimensional object data shown on the screen, but it is not reasonable to process all of them. Limits.

  In this case, when the range in which the distance image is measured is viewed from above, it is limited to the field of view of the CCD camera 11 as shown in FIG. 21. When the vehicle is normally traveling on a road, the side wall of the vehicle 1 is On the left side and the right side, they are substantially parallel to the vehicle 1. On the other hand, distant side walls are difficult to detect from the viewpoint of accuracy of distance data, and the necessity of detection is small. Therefore, in consideration of these, two search areas SL and SR on the left and right sides as shown are set, and the left and right side walls are separately detected.

  That is, in order to detect the left and right side walls, first, the side wall straight line detection unit 142 sets the left search area SL, performs the side wall straight line detection processing and the side wall range detection processing, detects the left side wall, and then performs again. The right side search area SR is set by the side wall straight line detection unit 142, and the same processing is repeated to detect the right side wall.

  In order to extract the three-dimensional object data included in each of the search areas SL and SR, the three-dimensional position (X, Z coordinates) of the subject of each data extracted by the three-dimensional object data extraction unit 141 is calculated, and the three-dimensional position is calculated. The position (X, Z) and each of the search areas SL and SR are compared and determined.

  For example, in the situation shown in FIG. 22, the above-described search areas SL and SR are shown as broken lines on the image, and various three-dimensional objects other than the target side wall also exist in these search areas. I do. Furthermore, the range image also includes false data in the form of noise, and only data is dispersed in a space where no object actually exists. FIG. 23 schematically shows these data, in which the side wall has a feature that the data is arranged in a straight line. Therefore, the side wall is detected by detecting the linear equation of the data column using the Hough transform.

Describing the detection of a straight line equation by the Hough transform, first, a straight line Fi passing through a point of the solid object data Pi (coordinates Xi, Zi) in FIG. 24 is assumed. The equation of this straight line is represented by the following equation (8).
X = afi × Z + bfi (8)

  Next, as shown in FIG. 25, the vertical axis sets the parameter space of the slope af of the equation (8), and the horizontal axis sets the parameter space of the intercept bf, and performs voting at positions corresponding to the parameters afi and bfi of the equation (8).

  Here, as described above, the value of the slope afi is considered to be practically used on a general road if it is changed within a range of about ± 20 ° (afi: ± 0.36), for example, since the side wall is considered to be substantially parallel to the vehicle 1. Above is enough. Further, the value of the intercept bfi is, for example, X = -1 m to about −10 m on the left side of the vehicle 1 when detecting the left side wall, and X = + 1 m when detecting the right side wall. Limit to a range of about +10 m. Thus, the reason why the limit range is set to, for example, about ± 10 m is that the detection of the side wall that is too far away requires little practical use.

  Due to such a limitation, the range in which voting is performed in the parameter space is a rectangular area as shown in FIG. 25, and this rectangular area is further divided into a grid shape and voted for each grid. The slope afi of the equation (8) is within a predetermined change range (for example, ± 10 ° to ± 20 °), and is set by sequentially changing the grid interval Δaf. The intercept bfi is calculated by substituting the set inclination afi and the coordinates (Xi, Zi) of the three-dimensional object data Pi into the equation (8). You.

  The position of the detected side wall, that is, the linear inclination and intercept detection accuracy is determined by the lattice intervals Δaf and Δbf, and the setting of the lattice intervals Δaf and Δbf is based on a request from the external device using the information on the side wall. It is done. For example, when used for detecting danger such as a collision in the case of normal traveling on a road, the grid interval Δaf is preferably about 1 to 2 °, and the grid interval Δbf is preferably about 0.3 to 0.6 m.

  When voting in the parameter space for all three-dimensional object data in the search area as described above, if there is linearly arranged data as shown in FIG. In the parameter space grid corresponding to the straight line parameters afi and bfi set in the above, many votes are obtained, and local maxima appear in the left and right voting regions SL and SR.

  If there is a side wall and there is a clear column of three-dimensional object data, the local maximum value in the parameter space shows a large value. On the other hand, if there are no side walls and multiple objects are dispersed, the local maximum value is a small value. Is shown. Therefore, the local maximum value is detected for each of the left and right voting regions SL and SR in the parameter space, and if the detected local maximum value is equal to or greater than the determination value, it can be determined that the side wall exists. The judgment value is set in consideration of the size of the search area to be set, the interval between the lattices, and the like.

Next, when the side wall straight line detecting section 142 determines that there is a side wall, the side wall range detecting section 143 detects the positions of the front and rear ends of the side wall. When the parameters af and bf corresponding to the local maximum value grid are read, it is estimated that the side wall exists along the following linear equation (9), and FIG. 26 shows the linear equation detected in the example of FIG. A straight line Ff shown in FIG.
X = af × Z + bf (9)

  First, assuming that a region having a width of about 0.3 m to 1.0 m around the straight line Ff is a side wall candidate region Tf, this region is further divided in the Z direction as shown in FIG. The width of the side wall candidate region Tf is set in consideration of a data error or the like in the lattice space Δbf of the parameter space.

  Next, the three-dimensional object data in the search area is surveyed sequentially, and only the data in the side wall candidate area Tf is extracted. Then, the number of three-dimensional object data is counted for each section, and a histogram is created. This is schematically shown in FIG. 27, where the portion where the side wall exists has a large frequency. Therefore, by detecting a section whose frequency is equal to or greater than the determination value, it can be determined that a side wall exists in this range, and the three-dimensional positions at both ends are calculated to be the front and rear end positions of the side wall. In the example of FIG. 24, the upper end of the search region SR is regarded as the rear end position of the side wall (guardrail 400). FIG. 28 shows the side wall detected by the above processing by a frame line. In this example, the right guardrail 400 is detected.

  As described above, when the parameters such as the position and shape of each object on the road are obtained from the distance image and written into the three-dimensional object parameter storage unit 150, the gap distance calculation unit 160 including the microprocessor 120c determines the own vehicle. The left and right gap distances between 1 and the detected object are calculated.

For example, in the image example of FIG. 9 described above, if the X coordinates of the front edges of the parked vehicles 200 and 300 on the own vehicle 1 side are XL and XR, and the X coordinates of the front end of the guard rail 400 are XWR, FIG. As described above, the distance DL between the left extension line of the own vehicle 1 and the edge of the left parking vehicle 200 is calculated by the following equation (10), where W is the vehicle width of the own vehicle 1. The distance DR between the right extension of the vehicle 1 and the right parking vehicle 300 is calculated by the equation (11), and the distance DWR between the right extension of the vehicle 1 and the right guardrail 400 is ( It is calculated by equation (12).
DL = | XL | -W / 2 (10)
DR = | XR | -W / 2 (11)
DWR = | XWR | -W / 2 (12)

  Then, calculations based on the equations (10), (11), and (12) are performed for each edge. When the minimum value of each of the left and right sides is obtained, the minimum value becomes the closest distance, that is, the clearance distance. The clearance distance is added to a margin, for example, a rearview mirror of the opponent vehicle, and output to the display 3. Are displayed on the display 3 as shown in FIG.

  In the image example of FIG. 9, since the distance DR to the parking vehicle 300 is smaller than the distance DWR to the guardrail 400, the distance DR to the parking vehicle 300 is adopted as the right gap distance and output to the display 3.

  Further, a threshold value such as 1 m is provided for the numerical value of the gap distance displayed on the display 3, and a numerical value exceeding this threshold value is not displayed. This is because when the gap is sufficiently large, there is little meaning in displaying a numerical value.

  Next, the operations of the image processor 20 and the range image processing computer 120 will be described.

  FIG. 30 is a flowchart showing the flow of the operation of the image processor 20. First, when images captured by the left and right CCD cameras 11a and 11b are input in step S101, the input analog image is converted into an A / D converter in step S102. After performing A / D conversion into digital images having a predetermined luminance gradation in 31a and 31b, the LUTs 32a and 32b enhance the contrast of low-luminance portions, compensate the characteristics of the left and right CCD cameras 11a and 11b, and perform other operations. , 33b.

  The images stored in these image memories 33a and 33b are only the lines necessary for the subsequent processing among all the lines of the CCD elements of the CCD cameras 11a and 11b, for example, once every 0.1 second ( (One out of three TV images).

  Next, in step S103, the left and right image data are read from the image memories 33a and 33b for the left and right images to the input buffer memories 41a, 41b, 42a and 42b via the common bus 80, for example, four lines at a time. Matching of the left and right images, that is, evaluation of the degree of coincidence is performed.

  At this time, a read operation from the image memories 33a, 33b to the input buffer memories 41a, 41b, 42a, 42b and a write operation to the shift registers 43a, 43b, 44a, 44b are alternately performed for each of the left and right images. It is. For example, in the left image, while the image data is being read from the image memory 33a to one input buffer memory 41a, the image data read from the other input buffer memory 41b to the shift register 43b is written out, and the right image is read. In the process, while the image data is being read from the image memory 33b to the one input buffer memory 42a, the image data read from the other input buffer memory 42b to the shift register 44b is written out.

  As shown in FIG. 31, the shift registers 43a, 43b, 44a, and 44b store image data (1, 1)... (4, 4) of a small area of 4 × 4 pixels on the left and right sides. The register 43a (44a) receives data of one or two lines, and the other shift register 43b (44b) receives data of three or four lines in the order of odd lines and even lines for each pixel. Each of the shift registers 43a, 43b, 44a, and 44b has an independent transfer line, and data of 4 × 4 pixels is transferred at, for example, eight clocks.

  Then, the contents of the even-numbered stages of the eight stages are simultaneously output from the shift registers 43a, 43b, 44a, and 44b to the city block distance calculation circuit 45, and when the calculation of the city block distance H starts, the data of the right image is changed to The data of the odd-numbered lines and the data of the even-numbered lines are alternately output at each clock while being held in the shift registers 44a and 44b. On the other hand, the data of the left image continues to be transferred to the shift registers 43a and 43b. While the data is output alternately, it is replaced by data shifted to the right by one pixel every two clocks. This operation is repeated until, for example, a shift of 100 pixels (200 clocks).

  Thereafter, when the transfer for one small area is completed, the contents of the right image address counter (the head address of the next 4 × 4 pixel small area) are set in the left image address counter in the # 2 address controller 87, and Processing of the next small area starts.

  In the city block distance calculation circuit 45, as shown in the timing chart of FIG. 32, first, data of eight pixels is input to the absolute value calculator of the first stage of the pyramid structure, and the absolute value of the luminance difference between the left and right images is calculated. . In other words, the luminance of the corresponding left pixel is subtracted from the luminance of the right pixel, and if the result is negative, the arithmetic instruction is changed so that the subtraction and the subtraction are performed again and the subtraction is performed again. Calculate the value. Therefore, the subtraction may be performed twice in the first stage.

  Next, after passing through the first stage, the first to third adders of the second to fourth stages add and output two simultaneous input data. Then, the total sum adder at the last stage adds two consecutive data to calculate the sum, and outputs the required city block distance H for 16 pixels to the minimum / maximum value detection unit 50 every two clocks.

  Next, the process proceeds to step S104, where the maximum value HMAX and the minimum value HMIN of the city block distance H calculated in step S103 are detected. As described above, the detection of the maximum value HMAX and the detection of the minimum value HMIN are exactly the same except that the logic is reversed and the deviation amount is not preserved. The detection of the value HMIN will be described.

  First, the initially output city block distance H (shift amount x = 0) is input to the B register 46b of the arithmetic unit 46 via the C latch 53 of the minimum value detection circuit 51 shown in FIG. The city block distance H (displacement amount δ = 1) output at the next clock is input to the C latch 53 and the A register 46a of the arithmetic unit 46, and the arithmetic unit 46 simultaneously compares it with the B register 46b. The operation starts.

  As a result of the comparison operation in the arithmetic unit 46, if the contents of the A register 46a are smaller than the contents of the B register 46b, the contents of the C latch 53 (that is, the contents of the A register 46a) are changed at the next clock. The shift amount δ at this time is stored in the D latch 55. At the same time as this clock, the next city block distance H (shift amount δ = 2) is input to the A register 46a and the C latch 53, and the comparison operation starts again.

  In this way, the calculation is continued until the shift amount δ reaches 100 while the minimum value during the calculation is always stored in the B register 46b and the shift amount δ at that time is stored in the D latch 55. When the calculation is completed (one clock after the output of the last city block distance H), the contents of the B register 46b and the D latch 55 are read into the shift amount determination unit 60.

  During this time, the above-mentioned city block distance calculation circuit 45 reads the initial value of the next small area so that time is not wasted. For example, four clocks are used to calculate one city block distance H. However, because of the pipeline structure, a new calculation result is obtained every two clocks.

  When the minimum value HMIN and the maximum value HMAX of the city block distance H are determined in step 104, in step S105, the displacement amount determination unit 60 checks the above three conditions and determines the displacement amount x. .

  That is, as shown in the timing chart of FIG. 33, the minimum value HMIN is latched by the B register 72 of the arithmetic unit 61 via the B bus 62b, and the threshold value HA to be compared with the value of the B register 72 is The data is latched by the A register 71 via the A bus 62a. The two are compared by the ALU 70, and if the minimum value HMIN is larger than the threshold value HA, the switch circuit 65 is reset, and 0 is always output regardless of the subsequent checks.

  Next, the maximum value HMAX is latched in the A register 71, and the difference between the maximum value HMAX latched in the A register 71 and the minimum value HMIN stored in the B register 72 is calculated. 73 is output. At the next clock, the threshold value HB is latched in the A register 71 and compared with the value of the F register 73. If the content of the F register 73 is smaller than the threshold value HB latched by the A register 71, the switch circuit 65 is similarly reset.

  From the next clock, calculation of the luminance difference between adjacent pixels starts. The two sets of shift registers 64a and 64b in which the luminance data is stored have a ten-stage configuration. The shift registers 44a for the first and second lines of the city block distance calculator 40 and the shift registers for the third and fourth lines are respectively provided. 44b is connected to the subsequent stage. The outputs of the shift registers 64a and 64b are taken from the last stage and the stage immediately before the last stage, and output to the A bus 62a and the B bus 62b, respectively.

  When the calculation of the luminance difference is started, the luminance data of each location in the small area is held in each stage of the shift registers 64a and 64b. And the luminance data of the first row and first column of the current small area are latched by the A register 71 and the B register 72 of the arithmetic unit 61.

  Then, the absolute value of the difference between the contents of the A register 71 and the contents of the B register 72 is calculated, and the result is stored in the F register 73. At the next clock, the threshold value HC is latched in the A register 71 and is compared with the value of the F register 73.

  As a result of the comparison by the arithmetic unit 61, if the content of the F register 73 (the absolute value of the luminance difference) is larger than the content of the A register (the threshold value HC), the shift amount x or “0” from the switch circuit 65 is obtained. Is output, and if the content of the F register 73 is smaller than the content of the A register, "0" is output and written to a position corresponding to the first row and first column of the corresponding small area of the output buffer memories 66a and 66b. It is.

  While the arithmetic unit 61 compares the luminance difference between adjacent pixels with the threshold value HC, the shift registers 64a and 64b shift one stage. This time, the calculation is started for the luminance data of the fourth row and the second column of the previous small area and the luminance data of the first row and the second column of the current small area. After the calculation is alternately performed on the first and second columns of the small area in this manner, the calculation is similarly performed on the third and fourth columns.

  During the calculation, the last stage and the first stage of the shift registers 64a and 64b are connected to form a ring register. If the shift clock is added twice after calculating the entire small area, the contents of the registers are in a state before the calculation. When the transfer of the luminance data of the next small area is completed, the data of the fourth row of the current small area is retained in the last stage and the previous stage.

  In this way, during the calculation for determining the shift amount, the next data is prepared on the A bus 62a and the B bus 62b, and the result is written, so that one data is processed only by two clocks necessary for the calculation. You. As a result, even if the check of the minimum value HMIN and the maximum value HMAX performed at the beginning is included, all calculations are completed at, for example, 43 clocks, and the minimum value HMIN and the maximum value of the city block distance H are calculated for one small area. There is plenty of time to find HMAX, and more features can be added.

  When the shift amount x is determined, the shift amount x is output as distance distribution information from the output buffer memories 66a and 66b to the dual port memory 90 in step S106, and the processing in the image processor 20 ends.

  The output buffer memories 66a and 66b have a capacity of, for example, four lines, similar to the input buffer memories 41a, 41b, 42a and 42b described above. The distance distribution information is sent to 90.

  From the distance distribution information written in the dual port memory 90, the three-dimensional position of the object corresponding to each pixel in the XYZ space is calculated from the mounting positions of the CCD cameras 11 and 12 and lens parameters such as the focal length. Thus, the distance to the object outside the vehicle can be accurately detected without a decrease in the amount of information.

  Here, the overall timing of the image processor 20 will be described with reference to the timing chart shown in FIG.

  First, the field signals from the synchronized left and right CCD cameras 11a and 11b are written into the image memories 33a and 33b every 0.1 seconds (the ratio of one screen to three screens).

  Next, upon receiving the capture end signal, block transfer for every four lines starts. In this transfer, three blocks are transferred in the order of the right image, the left image, and the resulting distance distribution image.

  During this time, the calculation of the shift amount δ is performed for one input / output buffer memory. Then, in consideration of the calculation time of the deviation amount δ, the transfer to the other input / output buffer memory is started after waiting for a predetermined time.

  The calculation of the city block distance H for a small area of 4 × 4 pixels of one right image is performed 100 times because the calculation is performed while shifting the left image by 100 pixels. While the city block distance H of one area is being calculated, the shift amount δ of the previous area is output as a distance distribution through each check.

  Assuming that the number of lines to be processed is 200, the processing for four lines is repeated 50 times, the processing time for four lines for transferring the first data at the start of the calculation, and the final result after the calculation is completed. Further, a processing time for four lines for transferring to the unit and a processing time for a total of eight lines are further required.

  The time from the start of the transfer of the first input image line to the end of the transfer of the last distance distribution is 0.076 seconds as a result of the actual circuit operation.

  On the other hand, the flowcharts of FIGS. 35 and 36 are object detection processing executed by the microprocessor 120a. When the position of the road surface is set in step S201, in step S202, the distance image from the image processor 20 is grid-shaped. The area is divided, and in step S203, the data of the first area is read.

  Next, proceeding to step S204, when the first data in the area is set, the three-dimensional position (X, Y, Z) of the subject, that is, the distance and height are calculated in step S205, and the distance is calculated in step S206. The height of the road surface at Z is calculated, and in step S207, data located above the road surface is selected.

  Then, the process proceeds to step S208 to check whether the data is the final data. If the data is not the final data, the next data in the area is set in step S209, and the process returns to the above-described step S205 to repeat the processing. The process proceeds from step S208 to step S210.

  In step S210, a histogram is created. In step S211, a section in which the frequency of this histogram is equal to or greater than the determination value and has a maximum value is detected. If a section where the frequency of the histogram is equal to or greater than the determination value and has the maximum value is detected, in step S212, it is determined that an object exists in the section, and the distance to the object is detected.

  Then, in step S213, it is checked whether or not it is the last area. If it is not the last area, the data of the next area is read in step S214, the process returns to step S204, and the same processing is continued. If there is, the process proceeds to step S215, the detection of the distance and the existence area of each object ends, and the process proceeds to step S216 and subsequent steps.

  In step S216, the parameters of the first object are set. Next, in step S217, the height and distance range of the lower end of the three-dimensional window WD3 are set. In step S218, the two-dimensional window WD3 is set based on the three-dimensional window WD3. The shape of WD2 is calculated, and the process proceeds to step S219.

  In step S219, the data in the two-dimensional window WD2 is read out, and after calculating the three-dimensional position of the subject in step S220, the process proceeds to step S221 to select and extract the data included in the three-dimensional window WD3.

  Then, the process proceeds to step S222, where the data extracted in step S221 is projected on a two-dimensional image, and in step S223, each data is connected by a line segment to create a contour image. Subsequently, in step S224, the shape, size, position, and speed of the object are calculated, and in step S225, it is determined whether the object is the final object.

  If it is not the final object, the parameters of the next object are set in step S226, and the process returns to step S217.If it is the final object, the process proceeds to step S227, where the position, shape, velocity, acceleration of each object Then, parameters such as the possibility of collision are written in the output memory 125, and the process is terminated.

  The microprocessor 120b performs the side wall detection processing shown in FIGS. 37 and 38 in parallel with the object detection processing by the microprocessor 120a. In this side wall detection processing, first, when the position of the road surface is set in step S301, the first distance data is read from the distance image in step S302.

  Next, the process proceeds to step S303, where the position (X, Z coordinates) and height (Y coordinate) of the subject are calculated, and in step S304, the height H (Y coordinate) of the road surface at the distance Z is calculated. Then, in step S305, data above the road surface and below the height of the vehicle 1 is extracted as three-dimensional object data.

  Then, the process proceeds to step S306 to check whether the data is the final data. If the data is not the final data, the next distance data is read in step S307, and the process returns to the above-described step S303 to repeat the processing. Proceed to step S308.

  In step S308, the first three-dimensional object data is read. In step S309, the position (X, Z coordinate) of the subject is calculated. In step S310, whether the calculated position (X, Z coordinate) is within the search area is determined. Find out.

  When the calculated position (X, Z coordinates) is outside the search area, the process jumps from step S310 to step S312. When the calculated position (X, Z coordinates) is within the search area, the process proceeds from step S310 to step S311 to vote in the parameter space. Proceed to.

  In step S312, it is checked whether or not the processed three-dimensional object data is final data.If not, the next three-dimensional object data is read in step S313, and the processing from step S309 described above is repeated. Proceeding to step S314, the local maximum value in the parameter space is detected.

  Next, when the process proceeds to step S315, it is determined whether the detected local maximum value is equal to or greater than the determination value.If the detected local maximum value is smaller than the determination value, it is determined that there is no side wall in step S316. , It is determined that the side wall exists, and the process proceeds to step S318.

  In step S318, a parameter corresponding to the lattice of the local maximum value detected in step S314, that is, a linear expression parameter (af, bf) indicated by the local maximum point is read, and then, in step S319, a side wall candidate region is set.

  Then, the process proceeds to step S320, where the first three-dimensional object data in the search area is read. In step S321, the position (X, Z coordinates) of the subject is calculated. In step S322, data in the side wall candidate area is extracted. After that, in step S323, it is checked whether or not the processed data is the last data in the search area.

  If it is not the last data in the search area, the process branches from step S323 to step S324 to read the next three-dimensional object data in the search area, and returns to step S321.If it is the last data in the search area, Proceeding from step S323 to step S325, a histogram is created using the data in the side wall candidate region.

  Next, the process proceeds to step S326, and when a section in which the frequency of the created histogram is equal to or greater than the determination value is detected, in step S327, the three-dimensional positions of both ends of the section in which the frequency of the histogram is equal to or greater than the determination value, that is, the front and rear end positions of the side wall are determined. Then, in step S328, parameters such as the presence / absence of the side wall, the position, the direction, and the positions of the front and rear ends are written in the output memory 125, and the program is terminated. This program is executed on the left side wall and then on the right side wall.

  Through the above processing, the position and size of the wall, the parked vehicle, and the like existing in front of the host vehicle 1 are known, and the microprocessor 120c executes the program of the gap distance calculation processing of FIG.

  In this gap distance calculation process, in step S401, when the position of the edge of the side of the own vehicle 1 and the position of the end of the side wall of each object on the road are input from the output memory 125, the edge of each object is The distance in the X direction between the end of the side wall and the extension of the side of the vehicle 1 is determined separately for the left and right sides.

  Next, the process proceeds to step S403, in which the minimum value of each distance obtained in step S402 is separately obtained for the left and right sides. In step S404, the minimum value for each of the left and right sides is set as a gap distance, and a margin is added. To display the numerical value.

Overall configuration diagram of travel guide device Circuit block diagram of travel guide device Illustration showing the screen of the display Front view of vehicle Explanatory diagram showing the relationship between the camera and the subject Detailed circuit diagram of image processor Illustration of the city block distance calculation circuit Block diagram of minimum value detection circuit Explanatory diagram showing an example of an image captured by a vehicle-mounted CCD camera Explanatory diagram showing an example of a distance image Top view of vehicle Side view of vehicle Functional block diagram of range image processing computer Explanatory diagram showing how to classify images Explanatory diagram showing the relationship between the detected object and the histogram Explanatory diagram showing an example of detection results and detection distances of an object existence region Explanatory drawing showing the shape of a three-dimensional window for object detection Explanatory drawing showing the shape of a two-dimensional window for object detection Explanatory diagram showing an example of data constituting the contour of an object Explanatory diagram showing an example of a contour image of an object and detected external dimensions Explanatory drawing showing the shape of the search area in side wall detection Explanatory drawing showing the side wall search area on the image Explanatory diagram showing the distribution status of three-dimensional object data Explanatory diagram showing the assumption of a straight line in the Hough transform Explanatory diagram showing the voting area in the parameter space Explanatory drawing showing a side wall candidate area Explanatory diagram showing the relationship between the histogram and the existence range of the side wall Explanatory drawing showing the detection result of the side wall Illustration of gap distance calculation Flow chart showing the operation of the image processor Explanatory drawing showing the storage order in the shift register Timing chart showing the operation of the city block distance calculation circuit Timing chart showing the operation of the shift amount determining unit Timing chart showing the overall operation of the image processor Flowchart of object detection processing Flowchart of object detection processing (continued) Flow chart of side wall detection processing Flowchart of sidewall detection processing (continued) Flow chart of gap distance calculation processing

Explanation of reference numerals

DESCRIPTION OF SYMBOLS 1 Vehicle 2 Travel guidance device 3 Display 10 Stereo optical system 15 Stereo image processing means 100 Solid object detection means 110 Gap distance calculation means
Attorney Attorney Susumu Ito

Claims (4)

Three-dimensional object detection means for detecting a three-dimensional object based on a captured image outside the vehicle,
Gap distance calculation means for calculating the gap distance between the three-dimensional object detected by the three-dimensional object detection means,
A display device for displaying information relating to the gap distance calculated by the gap distance calculation means.   The travel guide device for a vehicle according to claim 1, wherein the display device displays the gap distance.   The vehicle travel guide device according to claim 1, wherein the gap distance calculation unit calculates a distance between an edge of the three-dimensional object on the own vehicle side and a side portion of the own vehicle as a gap distance. A stereo image processing means for processing a stereo image pair of an object outside the vehicle imaged by an imaging system mounted on the vehicle, and obtaining a distance distribution over the entire image by a principle of triangulation from a shift amount of a corresponding position of the stereo image pair; ,
The three-dimensional object detection means calculates a three-dimensional position of each part of the subject corresponding to the distance distribution information from the stereo image processing means, and detects a plurality of three-dimensional objects using the calculated three-dimensional position information. ,
The gap distance calculation means may calculate the closest distance between the edge of the plurality of three-dimensional objects detected by the three-dimensional object detection means on the side of the own vehicle and an extension of the side of the own vehicle as a gap distance for each of the right and left sides. The travel guide device for a vehicle according to any one of claims 1 to 3, wherein:

Images