JP2017102838A - Database construction system for article recognition algorism machine-learning - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a database construction system for automatically collecting machine-learning tutor-presence learning-data for executing an article recognition from an output of another sensor using, as tutor data, a detection result of a certain sensor in a technique for recognizing an article in surroundings of a vehicle using plural kinds of censors.SOLUTION: A database construction system includes: first article recognition means for recognizing an article in a surrounding zone of a vehicle on the basis of output data of a first sensor that detects a state of a the surrounding zone of the vehicle; data correlation means for sequentially correlating tutor data and input data using, as input data, output data sequentially obtained from a second sensor that detects a state of the surrounding zone of the vehicle, and using, as tutor data, result data of highly reliable recognition of an article by the first article recognition means; and learning-data storage means for sequentially storing, as tutor-presence learning-data, a set of the correlated input data and tutor data.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for recognizing or detecting an event such as the presence or state of an object through an algorithm using machine learning from an output of a certain sensor, and more particularly, during traveling of a vehicle such as an automobile. , A system for constructing a database for machine learning in a recognition algorithm using machine learning that can be used when detecting nearby vehicles, pedestrians or other obstacles using a vehicle-mounted sensor Concerning.

  For example, in a moving vehicle or other moving body, detection of an object such as a preceding vehicle or moving body or other obstacles using an in-vehicle sensor such as a camera, LIDAR (Laser Imaging Detection and Ranging), radar, etc. Various techniques have been proposed. In the detection of objects by these in-vehicle sensors, there are weaknesses in detection performance depending on the surrounding state of the vehicle or moving body, the state of the object to be detected, etc., for example, object detection by imaging with a camera In this case, the detection of an object can be achieved with high accuracy in a bright environment such as during the day, but the detection accuracy of the object decreases at night, and the presence of the object depends on the distance between the vehicle-mounted sensor and the object. Presence / absence and accuracy of distance to the object may change. On the other hand, in the case of object detection by LIDAR, the detection accuracy is not significantly affected by the brightness of the surroundings, and the distance and direction to the object can be detected with high accuracy. In general, the recognition accuracy of the type of object is slightly lower than that of object detection by imaging with a camera. Therefore, from the past, when performing the recognition or detection of the presence of the surrounding object and / or its state by the sensor mounted on the vehicle or the moving body as described above, using a plurality of different types of sensors, Various configurations for recognizing or detecting an object from outputs obtained from these sensors, or configurations for recognizing or detecting an object by integrating outputs obtained from these sensors, have been proposed. .

  For example, in Patent Document 1, the LIDAR point sequence representing the position information of the other vehicle in front of the host vehicle obtained by LIDAR is used as the position of the other vehicle in front of the vehicle specified by the image obtained by the camera. It is disclosed that the position of the end portion of another vehicle is specified by being superimposed on the end portion. Further, in Patent Document 2, an object in front of a running vehicle is recognized by a recognition device using stereo image processing of images taken by two electronic cameras 1 and 2 and a laser range finder, respectively. In the plane area, the position where the object exists is detected, and the frequency of detection of the presence of the object in the detection result is weighted and added together. The structure which determines with exists is proposed. In this case, it is described that the weighting ratio is adjusted by learning. In Patent Document 3, in recognition of an object in front of a traveling vehicle, a group of reflection points that are present at approximately equidistant positions within a range of approximately vehicle widths based on the position of each reflection point specified by a laser radar. Is a vehicle candidate point cloud, and this vehicle candidate point cloud is coordinate-converted to the camera coordinate system and collated with a rectangular area extracted by the camera. If the vehicle candidate point cloud after the coordinate conversion substantially matches the rectangular area, A configuration is described in which the vehicle candidate point group is determined to be a forward vehicle. In Patent Document 4, in a device that detects the degree of risk by referring to risk information from a state quantity obtained by converting a feature vector extracted from image data ahead of the vehicle into a one-dimensional state, The risk information is learned by using the correlation between the state quantity converted to one dimension of the feature quantity vector and the teacher information, with the risk information for determining the degree extracted from the operation information of the driver as the teacher information. It has been proposed. In Patent Document 5, in an apparatus for determining a road surface state from the reflection intensity (reflected light data) of laser light irradiated on a road surface while a vehicle is traveling, reflected light data used for such determination is known. A configuration is disclosed that is generated by learning using reflected light data obtained while a vehicle is traveling on a road surface. And in patent document 6, in recognition of the object ahead of the vehicle in driving | running | working, about the gaze area | region in the captured image at the time of actual driving | running | working, the edge histogram in a camera image, and the light reception intensity in a laser radar From the histogram, X and Y direction vectors and laser vectors for the vehicle candidate region are created and fused to generate a fusion vector. In the vector space, the distance between the fusion vector and the prepared dictionary fusion vector is A configuration is disclosed that predicts that the vehicle candidate region is a vehicle when the vehicle is small.

JP 2009-98025 A JP 2000-329852 A JP 2003-84064 A JP2008-238831 JP2014-228300 JP 2003-99762 A

  By the way, as in the case where an object such as a preceding vehicle or an obstacle in front of the traveling vehicle is detected using a camera, a LIDAR, a radar, or the like, a certain detection or recognition target is detected by a plurality of types of sensors. In detecting or recognizing objects in the form of detecting or recognizing using a sensor, as already mentioned, there are differences in the object or environment that can be detected accurately depending on the type of sensor. Therefore, with respect to a certain detection target or in a certain detection environment, one sensor can accurately recognize the presence and / or type of an object, but the other sensor alone However, there are cases where accurate recognition is difficult, or proper signal processing, for example, processing for recognition of an object type, has not been established in the first place. In addition, depending on the type of sensor, processing conditions for performing accurate recognition may change during use. In such a case, that is, an appropriate signal processing method and conditions for the presence of an object, recognition of the type and / or estimation of a motion state (hereinafter referred to as “object recognition”) from the output of a certain sensor. Alternatively, if the procedure is unknown, uncertain, or variable, the appropriate signal processing method, condition, or procedure for the output of such a sensor can be adjusted by machine learning techniques to separate it from the sensor output. The signal processing method is performed so that the object can be recognized as accurately as possible from the output of the former sensor by collating with the detection result using the output of the sensor in a state where the object can be recognized with high accuracy. It is conceivable to configure or adjust conditions or procedures. That is, according to the configuration in which the detection result of the first sensor is used as teacher data and the signal processing method, condition, or procedure for obtaining the detection result from the output of the second sensor is configured or adjusted by machine learning, the second It is expected that the accuracy of detection results by these sensors will be improved. At that time, in general, the higher the accuracy of the teacher data and the larger and more diverse the database, that is, the more data such as vehicles of various distances, orientations and types, the more machine learning is performed. It is known that the accuracy of the detection result obtained by the second sensor obtained as a result of this is improved.

  In this regard, a large amount of supervised learning in the signal processing method of a sensor using so-called “machine learning”, the construction or adjustment of conditions or procedures, ie the construction of a recognition algorithm using machine learning. Data, that is, data obtained by collating the detection result of the first sensor serving as teacher data with the output and / or detection result of the second sensor serving as input data in machine learning is required. With respect to such collation data, conventionally, teacher data or learning data is created by a process in which a person manually tags the detection result of the first sensor and the output and / or detection result of the second sensor. It was. However, it is difficult to manually create a large amount of supervised learning data at a low cost, and it may be difficult to manually create teacher data depending on the type of data. For example, in an image taken in front of the vehicle, if a bounding box (frame set in the image) is given to the image of the preceding vehicle, it is easy to manually attach the bounding box if the image is good. However, it is difficult to accurately give a bounding box to the image of the preceding vehicle in backlit or night-time images. Also, such as point cloud acquired by LIDAR and millimeter wave radar intensity map It is not easy to manually create teacher data for data having a complicated shape or ambiguous boundaries. Furthermore, it is difficult to manually provide teacher data for information such as vehicle speed that cannot be observed directly from images and moving images. Therefore, in order to prepare a large amount of supervised learning data for constructing a recognition algorithm using machine learning, it is possible to automatically collect such learning data regardless of a human manual operation. preferable.

  Thus, one object of the present invention is a technique for recognizing objects such as other vehicles, pedestrians, and obstacles around a vehicle using a plurality of types of sensors, and the detection result of a certain sensor is obtained. Provide a system for automatically collecting supervised learning data and constructing a supervised learning data database for machine learning to perform object recognition from the output of another sensor used as supervised data It is to be.

The above problem is a system for constructing a database for storing supervised learning data used for machine learning for configuration or adjustment of an algorithm for performing recognition of an object in a surrounding area of a vehicle based on an output of a sensor,
A first sensor for sequentially detecting the state of the surrounding area of the vehicle;
First object recognition means for sequentially recognizing objects in the surrounding area of the vehicle based on output data obtained sequentially by the first sensor;
A second sensor for sequentially detecting the state of the surrounding area of the vehicle;
When the reliability of the recognition result data of the object by the first object recognition means is equal to or higher than a predetermined degree, the output data obtained sequentially from the second sensor is used as input data in machine learning. Data association means for associating teacher data with input data corresponding to the teacher data, using the recognition result data of the objects sequentially recognized by the object recognition means as teacher data in machine learning;
This is achieved by a system including learning data storage means for storing a set of associated input data and teacher data as supervised learning data for machine learning.

  In the above configuration, the “sensor” may be an arbitrary sensor. Typically, the surrounding area of the vehicle such as a camera, a LIDAR, or a millimeter wave radar that detects the state of the surrounding area of the vehicle as an image. It may be one normally used for detecting the state of The “configuration or adjustment of an algorithm for performing object recognition” as used herein means an appropriate signal processing method for performing object recognition based on the output of a certain sensor, as already described. Configure conditions or procedures, or adjust various conditions in an already configured algorithm. As is well known in the field of machine learning, when an object recognition algorithm is constructed or developed using machine learning, a recognition result (corrected) for an output (input data) of a certain sensor ( Configuration and / or use of arithmetic processing and judgment processing so that a correct recognition result is given when an arbitrary output of the sensor is given with reference to a data group corresponding to (teacher data) Various parameters are determined. The “algorithm configuration or adjustment” means such a configuration of arithmetic processing and determination processing and / or determination of various parameters. In the present invention, in the case of “recognition of an object”, as described above, detection of the presence of an object and / or identification of the type of the object (whether the object is stationary or not, It shall mean the identification of whether it is a vehicle, person, animal or other obstacle) and / or estimation of the motion state (position, posture (orientation), speed, etc.) of the object. In the above, the first sensor and the second sensor are typically sensors of different types or different specifications (measurement position, measurement angle range, sensitivity is different, etc.), For example, a combination of a camera and LIDAR may be used, but in some embodiments, the same type of sensor may be used. The “first object recognition means” gives the result of object recognition based on the output of the first sensor by any method known in this field depending on the type of the “first sensor”. It may be a means.

  In the above configuration, in short, at least two sensors, that is, a first sensor and a second sensor, are prepared as described above, and each of them is sequentially configured according to the respective mode. Detection of the state of the surrounding area is executed. In this case, as the first sensor, as described above, a sensor for which an algorithm for performing object recognition is determined based on the output is selected, and the second sensor is selected by machine learning. The algorithm that performs object recognition based on the output is selected to be configured or adjusted. Then, object recognition result data (which may or may not be present) obtained sequentially based on the output of the first sensor may be included in machine learning. It is selected as teacher data, the output of the second sensor is selected as input data in machine learning, these data are associated, and stored in the database as supervised learning data.

  The correspondence between the recognition result data of the object of the first sensor and the output of the second sensor and the storage of the supervised learning data are automatically executed by a computer process. In one embodiment, for example, the output data of the first sensor, the recognition result data of the object of the first sensor, and the output data of the second sensor are stored in an arbitrary data storage device. In addition, later (by offline processing), referring to the traveling log data of the vehicle, the correspondence between the recognition result data of the object of the first sensor and the output of the second sensor, and storage of supervised learning data, May be executed. The data storage device may be mounted on the host vehicle, or the data may be transmitted to and stored in a data storage device provided in an external facility via network communication. Good. Moreover, as another aspect, you may perform sequentially, during driving | running | working of a vehicle. As described above, the correspondence between the recognition result data of the object by the first sensor and the output of the second sensor indicates that the reliability of the recognition result data of the object by the first sensor is a predetermined degree. Only when the above is true, that is, when the recognition result data of the object by the first sensor is available as teacher data.

  The above-described configuration of the present invention may further include a configuration in which an object recognition result obtained based on the output data of the second sensor is used as teacher data. That is, before the object recognition algorithm by the second sensor is configured, the object recognition result based on the output data of the first sensor is used as teacher data, and after the object recognition algorithm by the second sensor is configured, , Which is obtained by integrating the object recognition data based on the output of the first sensor and the object recognition data based on the output of the second sensor as teacher data, and the output of the second sensor actually obtained The association may be performed. In this case, a learning data loop is formed for the object recognition algorithm based on the output of the second sensor, and further improvement in accuracy is expected.

  Thus, the above-described apparatus of the present invention further includes second object recognition means for sequentially recognizing objects in the surrounding area of the vehicle based on the output data sequentially obtained by the second sensor, The data association means may further be configured to use the recognition result data of the objects sequentially recognized by the second object recognition means as teacher data in machine learning. According to such a configuration, the object recognition accuracy can be improved by the loop repetitive processing, and the effect of compensating for the shortage of the data amount can be obtained. In the configuration in which the recognition result data of the object obtained based on the output data of the second sensor is used as teacher data to form a learning data loop, the recognition result data of the object It is preferable to have a higher degree of reliability than when used for the control. Accordingly, the recognition result data of the object obtained based on the output data of the second sensor is the teacher data when the reliability is equal to or higher than a predetermined degree higher than the degree to be satisfied when used for various controls. May be used.

  Furthermore, in another aspect, when at least two sensors are used, the recognition result data of the object of at least two sensors is used each time depending on the reliability of the recognition result data of each object. May be selected as the first sensor (provided that the reliability is equal to or higher than a predetermined level), and the other is selected as the second sensor. Further, a plurality of sensors may be used as the first sensor, and the first object recognition means may be configured to integrate the outputs of the plurality of sensors (sensor fusion) to give an object recognition result. It should be understood that such cases are also within the scope of the present invention.

  Thus, according to the present invention described above, in a technique for recognizing an object such as a vehicle, a pedestrian, or an obstacle around the host vehicle using a plurality of types of sensors, the detection result of a certain sensor is obtained. A system for constructing a database of supervised learning data by automatically preparing and collecting supervised learning data for machine learning to execute object recognition from the output of another sensor used as supervised data It becomes possible to provide. According to such a configuration, it is possible to greatly reduce the user's labor and the cost required for system construction, or to construct a database of supervised learning data even when it is very difficult to construct manually. Is obtained. In addition, by using the object recognition algorithm by the second sensor for the preparation of learning data, if there is a configuration that forms a loop of object recognition algorithm configuration by learning data accumulation and machine learning, the object recognition algorithm As a result, the accuracy, diversity, and quantity of the resulting database are improved, and further improvement in the accuracy of the object recognition algorithm is expected. In other words, if the loop processing of object recognition algorithm configuration by learning data accumulation and machine learning is repeated, it becomes possible to construct a larger amount or more efficiently a database of supervised learning data with higher accuracy and variety. Become.

  Other objects and advantages of the present invention will become apparent from the following description of preferred embodiments of the present invention.

FIG. 1A shows the configuration of one embodiment of a database construction system for collecting and storing machine learning learning data in a sensor output recognition algorithm according to the present invention in the form of a block diagram. FIG. FIG. 1B is a diagram showing processing in database construction in the form of a flowchart. 2A to 2C are diagrams showing the configuration of another embodiment of the database construction system according to the present invention in the form of a block diagram. 3A to 3C are diagrams showing the configuration of still another embodiment of the database construction system according to the present invention in the form of a block diagram. FIGS. 4A to 4B are still other embodiments of the database construction system according to the present invention. In particular, FIG. 4A is a diagram showing the configuration in the case of two types of teacher data in the form of a block diagram. . FIGS. 5A to 5B show still another embodiment of the database construction system according to the present invention, in particular, an object obtained by an object recognition algorithm established using learning data stored in the database. It is a figure showing the structure in the case of using a recognition result as teacher data in the format of a block diagram.

SC ... Camera image output SL ... LIDAR point cloud output SR ... Millimeter wave radar output Rr, Rr1, Rr2 ... Recognition results

  The present invention will now be described in detail with reference to a few preferred embodiments with reference to the accompanying drawings. In the figure, the same reference numerals indicate the same parts.

Basic Configuration and Operation of the System In the database construction system that executes learning data collection for machine learning of the object recognition algorithm according to the present invention, as described in the “Summary of Invention” section, The recognition result of the object obtained sequentially based on the output of the first sensor is used as teacher data, and the sequential output of the second sensor is used as input data. Preparation, collection and storage of “supervised learning data” used for machine learning to construct or adjust an algorithm for recognizing an object based on the output are sequentially performed. Here, “supervised learning data” used for machine learning is a set of the output of the second sensor and the recognition result of the first sensor associated therewith. Therefore, the database construction system has, as a basic configuration, a first sensor, a recognition unit for sequentially recognizing an object based on the output of the first sensor, a second sensor, and a sequential by the recognition unit. An association means for associating the obtained recognition results with the output of the second sensor sequentially obtained, the output of the associated second sensor and the recognition result of the first sensor Data storage means for storing the set as “supervised learning data”. The recognition means for sequentially recognizing the object based on the first sensor, the association means, and the data storage means are connected to each other by a computer system, that is, a normal type bidirectional bidirectional bus. It is realized by a system including a microcomputer having a CPU, a ROM, a RAM, and an input / output port device and a driving circuit, and the operation of each means described above is automatically achieved by execution of a computer program in the system.

  As described above, one of the most basic embodiments of the database construction system according to the present invention is for a system for recognizing an object in a surrounding area of a vehicle as illustrated in FIG. Used for database construction. Referring to the figure, in this database construction system, first, an in-vehicle camera is adopted as the “first sensor”, and an in-vehicle LIDAR is adopted as the “second sensor”. Each of the camera and the LIDAR may be of any or well-known type used in this field for recognizing an object in the surrounding area (particularly the front area) of the vehicle. In the case of the illustrated embodiment, the image output SC of the in-vehicle camera that is the first sensor is sequentially given to the object recognition unit, in which the object in the image taken by the camera The presence and / or type of images of, eg, (preceding) vehicles, pedestrians, animals, roadside belt fixtures, roadside obstacles, etc., are any or known The recognition result Rr is sequentially output. Then, the recognition result Rr obtained based on the camera image and the point cloud data SL, which is the output data of LIDAR, about the same region as the camera image or the same region are sequentially given to the association unit, In the associating unit, a process of associating the recognition result Rr and the corresponding point cloud data SL with the former as teacher data and the latter as input data is executed, and the learning data is sequentially obtained. The prepared learning data is sequentially stored in a database, that is, data storage means. In the configuration illustrated in FIG. 1A, the camera and the LIDAR are mounted on the vehicle, but the object recognition unit, the association unit, and the database may be mounted on the vehicle, or It may be installed in any facility. When the object recognizing unit, the associating unit, and the database are installed in an external facility, the output of the camera and LIDAR and / or the recognition result of the object recognizing unit is transmitted to the external facility through any form of wireless communication means, network, etc. May be sent to.

Example of association processing Preparation of learning data in the above-described association unit, that is, association processing between the output of the second sensor and the recognition result of the first sensor, Depending on the expression form or aspect of each of the outputs of the two sensors, it can be achieved by various arbitrary methods, for example, the processing procedures described in the above series of patent documents or any other arbitrary procedures. . For example, as shown in FIG. 1A, the case where the recognition result Rr based on the camera image is adopted as the recognition result of the first sensor and the LIDAR point cloud data SL is adopted as the output of the second sensor. The attaching process may be achieved by the process illustrated in FIG. Specifically, referring to the figure, in the associating unit, first, recognition result Rr based on camera image (step 10) and LIDAR point cloud data acquisition (step 12) are executed. Is done. Here, the expression format of the recognition result Rr based on the camera image is defined, for example, in a state in which the range of the image of the vehicle or the other object in the image specifies the type of the object of the image. The range of the image may be represented by coordinates in the image or coordinates in the space captured by the image. Further, the LIDAR point cloud data may be represented by position coordinates in each space of detection points (light reflection points). Thus, when each data is acquired, the LIDAR point cloud data is divided into partial point clouds by grouping according to the position in the space in a general manner as processing of the LIDAR point cloud data. (Step 14). Here, the division pattern may be arbitrarily determined in advance, for example, a division method by equal solid angle division, or for each point, the distance between the point and the nearest point is less than a threshold value. A division method or the like based on a process of setting the same group and a group having a threshold value or more as another group may be employed.

  Thereafter, the partial point group obtained above is projected onto the camera image (step 16), and the number of points included in the range of the image to be recognized in the camera image, for example, the image of the vehicle, for each partial point group. Is calculated (step 18). In this case, the projection of the partial point cloud onto the camera image is performed according to the representation format of the range of the recognition target image and the representation format of the point cloud data. It may be performed using a geometric transformation between the two so that the coordinates of the space in which the group data is represented are aligned with each other. Note that, in the recognition result Rr based on the camera image, typically, when an image of a certain object is recognized, the degree of reliability or probability of existence is expressed as a percentage or the like. (For example, the existence probability is 75%). In that case, in calculating the ratio of the number of points included in the range of the image to be recognized in the camera image, the reliability of the existence of the object in the recognition result in order to reduce the matching error. The ratio of the number of points included in the image range is calculated only for the range of the image that is greater than or equal to the predetermined degree, or only for the range of the image of the object closest to the camera. Good. The predetermined degree may be appropriately set experimentally or theoretically. Thus, a partial point group in which the ratio of the number of points included in the range of the image is not less than a predetermined threshold that is arbitrarily set is associated as a recognition target, for example, a point group that belongs to the vehicle, and is a part that is less than the predetermined threshold The point cloud is associated as a point cloud belonging to an object that is not a recognition target, and is stored in the database as supervised learning data (steps 20 to 24).

  The process illustrated in FIG. 1B is automatically executed by a computer process. Typically, output data of the camera (first sensor), recognition result Rr (recognition result data of the object of the first sensor), and LIDAR point cloud data (output data of the second sensor) are arbitrary data. The series of data association and supervised learning data storage illustrated in FIG. 1B stored in the storage device is offline processing, for example, vehicle log data (sensor data and accompanying data). May be executed with reference to FIG. As another aspect, data association and supervised learning data storage may be performed sequentially with sensor data acquisition, in which case supervised learning data is prepared from moment to moment. And will be accumulated.

  The accumulated “supervised learning data” was used in machine learning to construct or adjust an object recognition algorithm based on point cloud data detected by LIDAR in any manner, and thus obtained Based on the point cloud data detected by LIDAR using an algorithm, it may be used for recognition of an object in the surrounding area of the vehicle, and the recognition result may be used for various controls in the vehicle. It should be understood that the recognition result based on the camera image may also be used for various controls in the vehicle. The illustrated example is one example of the association processing, and the association is performed according to the expression format or form of the recognition result (teacher data) based on the first sensor and the output (input data) of the second sensor. It should be understood that processing may be performed. What is important is that the process of acquiring the recognition result based on the first sensor and the output of the second sensor in sequence, preparing the learning data by associating them, and storing them automatically by the computer Is to achieve it.

Examples of Other Embodiments of Database Construction System According to the Present Invention The database construction system according to the present invention may be realized by the forms shown in FIGS. 2 to 5 in addition to the configuration illustrated in FIG. In either case, the association processing between the teacher data and the input data according to the respective expression formats is performed, and the learning data may be prepared and stored in the same manner as described above.

(1) When the recognition result based on the output of LIDAR is used as the teacher data (FIG. 2A)
In situations where LIDAR can obtain a more accurate recognition result than the camera (backlight, night, rainy weather, etc.), the recognition result based on the output of LIDAR is used as teacher data, and learning data is prepared and stored using the camera image as input data May be. In this case, using learning data, an algorithm for recognizing an object based on a camera image is configured or adjusted by machine learning. Also, either the configuration of FIG. 1A or the configuration of FIG. 2A can be selected according to the situation in which a correct recognition result can be obtained between the LIDAR and the camera. May be.

(2) In the case of configuration (sensor fusion) in which object recognition is performed using both the camera image and the LIDAR point cloud data (FIGS. 2B and 2C)
In this case, the recognition result of the object based on the camera image and the LIDAR point cloud data is used as teacher data, and the LIDAR point cloud data (FIG. 2B) or the camera image (FIG. 2C) is used as input data. The learning data may be prepared and stored. The teacher data may be information such as the distance to the object and the speed of the object extracted from the recognition result of the object. The camera image used as input data may be a moving image.

(3) In the case of a configuration in which an object is recognized by LIDAR and millimeter wave radar (RADAR) (FIGS. 3A and 3B)
In the configurations of FIGS. 1A and 2A to 2C, a millimeter wave radar may be used instead of the camera. Since the output SR of the millimeter wave radar is a radar reflection intensity map, when the millimeter wave radar is used as the first sensor, the object recognition unit uses any method based on the radar reflection intensity map SR. The teacher data becomes an object recognition result Rr based on the radar reflection intensity map SR (FIG. 3A). In the case of a configuration (sensor fusion) in which object recognition is performed using both the radar reflection intensity map and the LIDAR point cloud data, the teacher data is the radar reflection intensity map SR and the LIDAR point cloud data SL. The object recognition result Rr based on the above, and the input data is the radar reflection intensity map SR (FIG. 3B) or LIDAR point cloud data (not shown). In particular, since it is difficult to perform processing such as tagging a radar reflection intensity map of a millimeter wave radar by a human hand, it is very advantageous that the processing can be automatically performed by a computer as described above.

(4) When other information is added to the learning data (FIG. 3C)
In addition to camera images, LIDAR point cloud data or radar reflection intensity maps, vehicle motion information such as vehicle speed acquired by an arbitrary sensor or detection device, and environmental information Dt such as weather are added to learning data. May be. In this case, it is expected that machine learning adapted to vehicle motion information and environmental information is possible.

(5) When using a plurality of teacher data (FIGS. 4A and 4B)
Two or more types of data (Rr1, Rr2) may be used as the teacher data (in the example described above, one type). When there are two or more teacher data, information appropriately extracted from each data may be used in the association process. For example, in the example of FIG. 4A, information on the position and type of the image of the object in the image is adopted as the information referred to as the teacher data from the recognition result Rr1 based on the camera image, and the radar reflection intensity of the RADAR From the recognition result Rr2 based on the map SR, information on the distance to the object and speed information may be adopted. In addition, as shown in FIG. 4B, when the recognition algorithm in the LIDAR point cloud data that is the object of machine learning is established with a certain degree of accuracy, the recognition based on the LIDAR point cloud data SL is performed. The result Rr2 may be adopted as the second teacher data.

(6) When using the recognition result by the recognition algorithm obtained by machine learning as teacher data (FIGS. 5A and 5B)
After the object recognition algorithm based on the output of the second sensor is constructed or adjusted by machine learning using learning data stored in the database, the recognition result using the recognition algorithm is further adopted as teacher data. It's okay. For example, in the case of the configuration illustrated in FIG. 5A, first, similarly to the configuration described in FIG. 1A, first, the recognition result Rr1 based on the camera image and the point cloud data SL of LIDAR After the learning data is prepared and stored for a certain period of time through the association process, an object recognition algorithm is configured based on the LIDAR point cloud data SL by machine learning using the learning data. Or adjusted. Thereafter, object recognition based on the LIDAR point cloud data SL is executed by the object recognition algorithm obtained by the machine learning, and the recognition result Rr2 is also associated with the LIDAR point cloud data SL as teacher data. Thus, the learning data is prepared and stored. Here, there are two teacher data, Rr1 and Rr2. For example, according to the measurement situation, one of the more accurate teacher data is preferentially selected and associated with the input data. It may be like this. More specifically, for example, by adjusting the ratio of the contribution of the teacher data Rr1 and Rr2 in a mode in which the weight of the recognition result with high reliability that can be determined by an arbitrary method is increased, one teacher data is obtained. It may be prepared and associated with input data. In this regard, when the recognition result Rr2 of the object recognition algorithm based on the output of the second sensor configured or adjusted by machine learning is used as teacher data, the reliability of the result is sufficiently high. It is preferable. Therefore, the recognition result Rr2 may be used as teacher data only when the reliability is equal to or higher than a predetermined degree higher than the degree to be satisfied when used for various controls. In the above processing, machine learning using the learning data stored in the database may be executed in an arbitrary manner.

  As described above, when the recognition result obtained by machine learning using learning data stored in the database is further used as teacher data, a so-called learning data loop in machine learning is formed. As the loop is repeated, the recognition accuracy of the recognition algorithm that is the object of machine learning is expected to improve.

  Further, as illustrated in FIG. 5B, the recognition algorithm for each object is configured by machine learning using the learning data accumulated in the database for both outputs of two different types of sensors. It may be adjustable. In the case of the example of FIG. 5B, when object recognition based on the camera image and object recognition based on the LIDAR point cloud data are executed, each object recognition algorithm is stored in the database. It is configured or adjusted by machine learning using the learned data. In the association processing, teacher data based on the output of different types of sensors may be appropriately associated with each of the outputs of different types of sensors (that is, learning data is prepared for each sensor). .) According to such a configuration, a loop of learning data in machine learning is formed in each object recognition algorithm of different types of sensor outputs, and the recognition accuracy of the object recognition algorithm based on both sensors is improved. Improvement is expected.

  Thus, according to the above series of database construction systems, supervised learning data for machine learning for executing recognition of an object from the output of another sensor using the detection result of one sensor as teacher data. It is possible to provide a system that automatically adjusts and collects and constructs a database of supervised learning data, which significantly reduces the labor of the user and the cost required to construct the system, or builds it manually. Even when it is very difficult, it is possible to construct a database of supervised learning data.

  Although the above description has been made in relation to the embodiment of the present invention, many modifications and changes can be easily made by those skilled in the art, and the present invention is limited to the embodiment exemplified above. It will be apparent that the invention is not limited and applies to various devices without departing from the inventive concept.

  For example, the sensor used in the present invention may be a sensor fixed outside the vehicle, such as a fixed camera installed at an intersection.

Claims (1)

A system for constructing a database for accumulating supervised learning data used for machine learning for configuration or adjustment of an algorithm for executing recognition of an object in a surrounding area of a vehicle based on an output of a sensor,
A first sensor for sequentially detecting the state of the surrounding area of the vehicle;
First object recognition means for sequentially recognizing objects in the surrounding area of the vehicle based on output data obtained sequentially by the first sensor;
A second sensor for sequentially detecting the state of the surrounding area of the vehicle;
When the reliability of the recognition result data of the object by the first object recognition means is equal to or higher than a predetermined level, output data obtained sequentially from the second sensor is used as input data in the machine learning. Using the recognition result data of the objects sequentially recognized by the first object recognition means as teacher data in the machine learning, and associating the teacher data with the input data corresponding to the teacher data. Data association means to perform;
A learning data storage means for storing a set of the associated input data and teacher data as supervised learning data for the machine learning.

Images