JP7106332B2 - External recognition device - Google Patents

Description

The present invention relates to an external world recognition device.

A vehicle image processing device capable of improving the reliability of information analysis based on camera images is known (see, for example, Patent Literature 1).

In Patent Document 1, ``In an image captured by an imaging means for imaging the front of the own vehicle, an attention area including a forward vehicle, a road white line, or a sign and a non-attention area are set, and the imaging means Detecting luminance information in an attention area and a non-attention area in a captured image, and performing image processing based on the luminance information of the attention area and the non-attention area in a driving environment where it is difficult to analyze the situation ahead of the vehicle. determine whether or not.”

JP-A-2005-135308

2. Description of the Related Art In recent years, with the increase in the number of pixels of external recognition devices such as stereo cameras, the capacity of images has increased. Therefore, a transfer delay may occur when image data is output to the outside. Patent Document 1 does not consider such a transfer delay.

SUMMARY OF THE INVENTION An object of the present invention is to provide an external world recognition device capable of suppressing a transfer delay when outputting image data to the outside.

In order to achieve the above object, an example of the present invention includes: an imaging unit that captures an image; a data generation unit that generates a plurality of types of data from the image captured by the imaging unit; An external world recognition device comprising: a data extraction unit for extracting data; a data output unit for outputting the data extracted by the data extraction unit to the outside ; and a request reception unit for receiving a request from a control device for controlling a vehicle. the request includes position information indicating a position on the image corresponding to the position of the object detected by a sensor that detects the position of the object using electromagnetic waves or sound waves; and the data output section outputs the data extracted by the data extraction section to the control device .

According to the present invention, transfer delay can be suppressed when image data is output to the outside. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.

1 is a configuration diagram of a system including stereo cameras according to an embodiment of the present invention; FIG. 1 is a configuration diagram of a stereo camera according to an embodiment of the present invention; FIG. 3 is a block diagram illustrating the functionality of a stereo camera according to an embodiment of the present invention; FIG. FIG. 3 is a diagram showing an example of a communication interface; FIG. 4 is a flow chart showing the operation of the stereo camera according to the embodiment of the present invention; FIG. 11 is a block diagram showing functions of a stereo camera of a modified example; 10 is a flow chart showing the operation of the stereo camera of the modified example;

The configuration and operation of a stereo camera (external world recognition device) according to an embodiment of the present invention will be described below with reference to the drawings. In addition, in each figure, the same code|symbol shows the same part.

(embodiment)
First, the configuration of the system will be described with reference to FIG. FIG. 1 is a configuration diagram of a system 1 including a stereo camera 100 according to an embodiment of the invention.

The system 1 includes a stereo camera 100, an automatic driving ECU 200 (Electronic Control Unit), a millimeter wave radar 300, a LIDAR 400 (Light Detecting And Ranging), and the like.

The stereo camera 100 captures images of the same target object from a plurality of different viewpoints, and calculates the distance to the target object from the difference in appearance (parallax) in the obtained images. As an example, the stereo camera 100 is connected to the automatic driving ECU 200 via a LAN (Local Area Network) cable conforming to Ethernet (registered trademark). Here, since Ethernet has a higher transfer speed than CAN (Controller Area Network), which will be described later, it is suitable for transmitting a large amount of data.

The automatic driving ECU 200 controls automatic driving of the vehicle based on the distance, angle, relative speed, etc. of the target object detected by sensors such as the millimeter wave radar 300 and the LIDAR 400 . As an example, the automatic driving ECU 200 is connected to sensors such as a millimeter wave radar 300 and a LIDAR 400 via a CAN-compliant two-wire bus.

The millimeter wave radar 300 uses millimeter waves (electromagnetic waves) to detect the distance, angle, relative velocity, etc. of a target object. The LIDAR 400 uses light waves (electromagnetic waves) such as ultraviolet rays, visible rays, and near-infrared rays to detect the distance, angle, relative velocity, and the like of a target object.

Next, the configuration of stereo camera 100 will be described with reference to FIG. FIG. 2 is a configuration diagram of the stereo camera 100 according to the embodiment of the present invention.

The stereo camera 100 includes, for example, an imaging unit 101 , a CPU 102 (Central Processing Unit), a memory 103 , an image processing unit 104 and a communication circuit 105 .

The imaging section 101 is composed of a left imaging section 101L and a right imaging section 101R, and captures images. Specifically, each of the left imaging unit 101L and the right imaging unit 101R includes an optical element such as a lens and an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The optical element refracts light to form an image on the imaging element. The imaging element receives an image of light refracted by the optical element and generates an image according to the intensity of the light.

The CPU 102 implements various functions, which will be described later, by executing a predetermined program. In this embodiment, the CPU 102 generates a vehicle control command based on, for example, the type of target object (pedestrian, vehicle, etc.).

The memory 103 is composed of, for example, a non-volatile memory such as FLASH and a volatile memory such as RAM (Random Access Memory), and stores various information.

The image processing unit 104 is configured by a circuit such as an FPGA (Field-Programmable Gate Array), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or the like. The image processing unit 104 performs image correction, parallax calculation, and the like, for example. Details of the functions of the image processing unit 104 will be described later with reference to FIG.

The communication circuit 105 is composed of, for example, a transceiver conforming to Ethernet. Communication circuit 105 transmits or receives various data.

Next, functions of the stereo camera 100 will be described with reference to FIG. FIG. 3 is a block diagram illustrating the functionality of stereo camera 100 according to an embodiment of the present invention.

The left imaging unit 101L captures a first image of the target object. The right imaging unit 101R captures a second image of the target object. The captured first image and second image are temporarily stored in, for example, the memory 103 as "raw images" (RAW data).

The image processing unit 104 includes an image correction unit 1041 , a parallax calculation unit 1042 , a three-dimensional object extraction unit 1043 , a three-dimensional grouping unit 1044 and an object identification unit 1045 . Note that the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 may be configured separately.

The image correction unit 1041 corrects the first image captured by the left imaging unit 101L and the second image captured by the right imaging unit 101R. Specifically, the image correction unit 1041 corrects, for example, distortion and optical axis misalignment between the first image and the second image. The corrected first image and second image are temporarily stored, for example, in the memory 103 as "corrected images".

The parallax calculation unit 1042 performs stereo matching to calculate parallax from the first image and the second image corrected by the image correction unit 1041 . The parallax calculator 1042 calculates the distance from the parallax to the target object by triangulation. The parallax calculator 1042 generates a parallax image that associates the position on the image with the distance corresponding to the parallax. Parallax images (first data) are stored in the memory 103, for example.

The three-dimensional object extraction unit 1043 extracts objects positioned at the same distance from the parallax images generated by the parallax calculation unit 1042 as one three-dimensional object. Three-dimensional object life information (second data), which is data of the extracted three-dimensional object, is stored in the memory 103, for example.

The three-dimensional grouping unit 1044 groups the three-dimensional objects extracted by the three-dimensional object extraction unit 1043 based on brightness, edges, and the like. Post-grouping three-dimensional object information (third data), which is data of the grouped three-dimensional objects, is stored in the memory 103, for example.

The object identification unit 1045 identifies what the target object (pedestrian, vehicle, etc.) is from the three-dimensional objects grouped by the three-dimensional grouping unit 1044 .

CPU 102 functions as vehicle control unit 1021 and data extraction unit 1022 by executing a predetermined program. Vehicle control unit 1021 generates a command for controlling the vehicle according to the type of object identified by object identification unit 1045 . For example, the vehicle control unit 1021 generates a command to decelerate the vehicle when determining that a pedestrian is on the route of the vehicle.

The data extraction unit 1022 extracts (selects) data stored in the memory 103 in response to a request from the automatic driving ECU 200 accepted by the request acceptance unit 1051, which will be described later. That is, the data extraction unit 1022 extracts predetermined data from multiple types of data. In other words, the data extractor 1022 extracts data upon request.

Communication circuit 105 functions as request reception unit 1051 and data output unit 1052 . The request receiving unit 1051 receives requests from the external automatic driving ECU 200 . Data output unit 1052 outputs the data extracted by data extraction unit 1022 to external automatic driving ECU 200 . In other words, the data output unit 1052 outputs data upon request.

The image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, and the three-dimensional grouping unit 1044 constitute a data generation unit 104a that generates multiple types of data from the image captured by the imaging unit 101.

Next, the communication interface of communication circuit 105 will be described with reference to FIG. FIG. 4 is a diagram showing an example of a communication interface.

The second row of the table in FIG. 4 shows the “request” input from the automatic driving ECU 200 to the communication circuit 105 . The third line of the table in FIG. 4 shows “data according to the request” output from the communication circuit 105 to the automatic driving ECU 200 .

The request from the automatic driving ECU 200 includes the coordinate values on the image as information indicating the attention area (information indicating the position on the image), the type ID as information indicating the output content (data type), and the output resolution (resolution). and the number of times of output per unit time (second) (times/s) as information indicating the output frequency.

The region of interest is, for example, a region estimated by the automatic driving ECU 200 to include a three-dimensional object based on the distance, angle, relative speed, etc. of the target object detected by sensors such as the millimeter wave radar 300 and the LIDAR 400. The type ID indicates the type of data to be output. Data types include, for example, raw images captured by the imaging unit 101 (RAW data), parallax images (first data), three-dimensional object biological information (second data), and information after three-dimensional object grouping (third data). be.

Next, the operation of stereo camera 100 will be described with reference to FIG. FIG. 5 is a flow chart showing the operation of stereo camera 100 according to an embodiment of the present invention.

The stereo camera 100 receives the request shown in FIG. 4 from the automatic driving ECU 200 (S10). Stereo camera 100 extracts (selects) data stored in memory 103 in response to the request received in S10 (S15).

Stereo camera 100 outputs the data extracted in S15 (S20). For example, in response to the request shown in FIG. 4, the stereo camera 100 reads the data of the type indicated by the "type ID" from the memory 103 for the attention area indicated by the "coordinate values on the image". The data is output (transmitted) to the automatic driving ECU 200 at the output resolution indicated by "the degree of image thinning/the number of outputs" and the output frequency indicated by "times/s". Note that the stereo camera 100 may output synchronization information.

Here, the data extraction unit 1022 extracts the data of the “coordinate values on the image” (the position on the image indicated by the position information) included in the request. Thereby, the automatic driving ECU 200 can acquire data only for the region of interest.

The data extraction unit 1022 extracts data of the type indicated by the "type ID" included in the request. Thereby, the automatic driving ECU 200 can acquire only desired types of data.

The data output unit 1052 outputs the data extracted by the data extraction unit 1022 to the external automatic driving ECU 200 at the output resolution (resolution) indicated by the "image thinning degree/output number" included in the request. Thereby, the automatic driving ECU 200 can acquire data of a desired output resolution.

Data output unit 1052 outputs the data extracted by data extraction unit 1022 to the outside at the output frequency indicated by “times/s” included in the request. Thereby, the automatic driving ECU 200 can acquire data at a desired output frequency.

The data output unit 1052 outputs the data extracted by the data extraction unit 1022 to the external autonomous driving ECU 200 via Ethernet. As a result, the transfer speed is improved compared to CAN or the like.

The data generation unit 104a generates a parallax image (first data) indicating a parallax or a distance corresponding to the parallax, and converts data of a three-dimensional object from the parallax image (first data) into three-dimensional object biological information (second data). After extraction, three-dimensional object grouping information (third data) is generated by grouping from three-dimensional object raw information (second data).

The data extraction unit 1022 extracts the data of the type indicated by the type ID into a raw image (RAW data), a parallax image (first data), three-dimensional object biological information (second data), three-dimensional object grouping information (third data). Extract (select) from

The object identification unit 1045 identifies objects from the image captured by the imaging unit 101 . The data extracting unit 1022 extracts data of objects associated with a degree of importance equal to or greater than a threshold.

The system 1 includes a stereo camera 100 (external recognition device), sensors such as a millimeter wave radar 300 and LIDAR 400 that detect the position of an object using electromagnetic waves, and an automatic driving ECU 200 (control device).

The automatic driving ECU 200 (control device) converts the position of the object detected by the sensor into a position on the image, transmits a request including the position information of the position to the stereo camera 100 (external world recognition device), and the data extraction unit The data extracted by 1022 is received from the stereo camera 100 (external recognition device), and the vehicle is controlled based on the received data.

In this way, the automatic driving ECU 200 can extract data of various layers (from the imaging unit 101, the image correction unit 1041 to the stereoscopic grouping unit 1044, etc.).

As described above, according to the present embodiment, transfer delay can be suppressed when image data is output to the outside. In addition, it is possible to reduce the volume of data output from a stereo camera with a high pixel count.

(Modification)
Next, a modification will be described with reference to FIGS. 6 and 7. FIG. FIG. 6 is a block diagram showing the functions of the stereo camera 100A of the modified example. Note that the hardware configuration of the stereo camera 100A of this modified example is the same as that of the stereo camera 100 shown in FIG.

In this modification shown in FIG. 6, the communication circuit 105 does not have the request reception unit 1051 . The data extracting unit 1022A of this modified example extracts the data of the position (coordinate values) on the image of the object identified by the object identifying unit 1045 and associated with the degree of importance equal to or greater than the threshold. That is, the data extraction unit 1022A extracts data of objects of high importance (pedestrians, dangerous parts). Note that the degree of importance and the type of object are stored in association with each other in the memory 103, for example.

FIG. 7 is a flow chart showing the operation of the stereo camera 100A of the modified example.

Stereo camera 100A identifies the object (S13). The stereo camera 100A extracts data of objects of high importance (pedestrians, dangerous parts) among the objects identified in S13 (S18). Stereo camera 100A outputs the data extracted in S18 (S20).

In addition, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations.

In the above embodiment, the stereo camera 100 is used as the external world recognition device, but the number of imaging elements is arbitrary, and a monocular camera may be used.

In the above embodiments, millimeter wave radar and LIDAR are used as sensors, but the sensor may be a sonar that uses sound waves to detect the distance, angle, relative velocity, and the like of a target object.

In the above embodiment, images such as raw images (RAW data), parallax images (first data), three-dimensional object biological information (second data), and information after three-dimensional object grouping (third data) are used for easy understanding. Data relating to are stored in the memory 103 as an example, but may be stored in the built-in memory of the image processing unit 104 .

In the above-described embodiment, the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 are realized as functions of the image processing unit 104 as an example. may be implemented. Note that each of the image correction unit 1041, the parallax calculation unit 1042, the three-dimensional object extraction unit 1043, the three-dimensional grouping unit 1044, and the object identification unit 1045 may be configured as a circuit such as an ASIC.

In the above embodiment, Ethernet is used as the standard between the stereo camera 100 and the automatic driving ECU 200, but other standards may be used as long as a transfer rate equal to or higher than that of Ethernet can be realized.

In the above-described embodiment, the "request" from the automatic driving ECU 200 includes the coordinate values on the image, but may include the coordinate values of another coordinate system such as the world coordinate system instead of the coordinate values on the image. The CPU 102 of the stereo camera 100 transforms the coordinate values of the world coordinate system into coordinate values on the image. Also, the vehicle speed may be included instead of the coordinate values on the image. When the vehicle speed is equal to or higher than the threshold (high speed), the data extraction unit 1022 extracts (selects) the high-resolution image and the parallax image, and extracts (selects) the low-resolution image and the parallax image, for example.

The request from the automatic driving ECU 200 includes information specifying either or both of data of the first image captured by the left imaging unit 101L and data of the second image captured by the right imaging unit 101R. good too. The data extraction units 1022 and 1022A extract (select) data of designated images. The vehicle speed is detected by a vehicle speed sensor.

The autonomous driving ECU 200 refers to the parallax image (first data), the three-dimensional object biological information (second data), the three-dimensional object grouping information (third data), etc. In this case, the coordinate values of the image coordinate system or the world coordinate system corresponding to the position may be included in the "request" as information indicating the attention area, and the request may be input to the stereo camera 100 again.

Further, each of the above configurations, functions, and the like may be realized by hardware, for example, by designing a part or all of them using an integrated circuit. Moreover, each of the above configurations, functions, and the like may be realized by software by a processor (CPU) interpreting and executing a program for realizing each function. Information such as programs, tables, and files that implement each function can be stored in recording devices such as memory, hard disks, SSDs (Solid State Drives), or recording media such as IC cards, SD cards, and DVDs.

DESCRIPTION OF SYMBOLS 1... System 100, 100A... Stereo camera 101... Imaging part 101L... Left imaging part 101R... Right imaging part 103... Memory 104... Image processing part 104a... Data generation part 105... Communication circuit 300... Millimeter wave radar 400... LIDAR
1021 Vehicle control unit 1022 Data extraction unit 1022A Data extraction unit 1041 Image correction unit 1042 Parallax calculation unit 1043 Three-dimensional object extraction unit 1044 Three-dimensional grouping unit 1045 Object identification unit 1051 Request reception unit 1052 Data output Department

Claims (9)

an imaging unit that captures an image, a data generation unit that generates a plurality of types of data from the image captured by the imaging unit, a data extraction unit that extracts data from the plurality of types of data, and the data extraction unit An external world recognition device comprising: a data output unit that outputs extracted data to the outside; and a request reception unit that receives a request from a control device that controls a vehicle ,
The request includes position information indicating a position on the image corresponding to the position of the object detected by a sensor that detects the position of the object using electromagnetic waves or sound waves;
The data extraction unit extracts data of a position on the image indicated by the position information,
The data output unit outputs the data extracted by the data extraction unit to the control device.
An external world recognition device characterized by: The external world recognition device according to claim 1 ,
The data output unit
An external world recognition device that outputs data in response to the request. The external world recognition device according to claim 2,
The data extractor is
An external world recognition device that extracts data according to the request. The external world recognition device according to claim 1 ,
Said request is
including a type ID indicating the type of data,
The data extractor is
An external world recognition device, characterized by extracting data of a type indicated by the type ID. The external world recognition device according to claim 1 ,
Said request is
including resolution,
The data output unit
An external world recognition device, wherein the data extracted by the data extraction unit is output to the outside at the resolution. The external world recognition device according to claim 1 ,
Said request is
including output frequency,
The data output unit
An external world recognition device, wherein the data extracted by the data extraction unit is output to the outside at the output frequency. The external world recognition device according to claim 4 ,
The data generation unit
generating first data indicating a parallax or a distance corresponding to the parallax;
extracting data of a three-dimensional object from the first data as second data;
generating third data grouped from the second data;
The data extractor is
An external world recognition device, wherein the data of the type indicated by the type ID is extracted from the RAW data, the first data, the second data, and the third data of the image. The external world recognition device according to claim 1,
An object identification unit that identifies an object from the image captured by the imaging unit,
The data extractor is
An external world recognition device, characterized by extracting data of the object associated with a degree of importance equal to or greater than a threshold. A system including the external world recognition device according to claim 1 ,
a sensor that detects the position of an object using electromagnetic waves or sound waves;
converts the position of the object detected by the sensor into a position on the image, transmits the request including the position information of the position to the external world recognition device, and converts the data extracted by the data extraction unit to the external world recognition device. a controller that receives data from the device and controls the vehicle based on the received data .

Images