JP6965325B2 - Auto-discovery system and auto-detection program - Google Patents

Description

The present invention relates to a technique for automatically detecting a traffic safety facility (traffic signal lamp, road sign, etc.).

In order to drive an automobile automatically, it is necessary to automatically recognize a traffic signal by using an in-vehicle camera. However, it is difficult to secure sufficient recognition performance by using only the image obtained by the in-vehicle camera.

Patent Document 1 discloses a method of calculating the position of a vehicle with high accuracy by using the road marking information DB.

JP-A-2007-108043

An object of the present invention is to improve the performance of a function of automatically detecting a traffic signal lamp or the like.

The automatic detection system of the present invention automatically detects at least one of the traffic safety facilities of a traffic signal lamp and a road sign.
The automatic detection system
Data acquisition to acquire a road image obtained by shooting with a camera mounted on a vehicle, vehicle position data indicating the position of the vehicle at the time of shooting, and vehicle posture data indicating the posture of the vehicle at the time of shooting. Department and
Based on the vehicle position data and the vehicle attitude data from an information database in which information data including position data, size data, and feature data is registered for each traffic safety facility provided at each point, the vehicle is taken at the time of shooting. A database search unit that acquires information data of the target facility, which is a traffic safety facility located in the direction of travel,
The target facility is projected from the road image based on the vehicle position data, the vehicle attitude data, the position data included in the information data of the target facility, and the size data included in the information data of the target facility. An image range limiting part that extracts a range as a limited range, and
A detection unit that detects the target facility by collating the limited range with the feature data included in the information data of the target facility is provided.

The detection unit performs projection conversion and size conversion on the limited range, and collates the converted limited range with the feature data.

The feature data includes a reference image showing the target facility.
The detection unit compares the limited range after the conversion with the reference image, and determines whether or not the limited range matches the feature data based on the comparison result.

The target facility is a traffic signal lamp.
The feature data includes display area data.
The display area data indicates a display area which is each area of one or more display units provided in the traffic signal lamp.
The detection unit extracts a display range corresponding to each display area from the limited range after conversion, and determines and discriminates the signal display reflected in the limited range after conversion by image processing for each display range. The signal display is compared with the signal display shown in the display information data at the time of shooting, and it is determined based on the comparison result whether or not the limited range matches the feature data.

The target facility is a road sign.
The feature data includes display area data.
The display area data indicates a display area which is an area of a display unit provided on the road sign.
The detection unit extracts a display range corresponding to the display area from the limited range after conversion, and determines whether or not the display unit of the road sign is reflected in the display range by image processing on the display range. , Whether or not the limited range matches the feature data is determined based on the determination result.

The feature data includes adjacency data indicating an adjacency that is an object adjacent to the target facility.
The detection unit extracts a reference range, which is a range including the limited range, from the road image, performs projection conversion and size conversion on the reference range, and performs image processing on the converted reference range to obtain the adjacent reference range. It is determined whether or not the object is reflected in the reference range, and whether or not the limited range matches the feature data is determined based on the determination result.

The camera has a directivity function for adjusting the line-of-sight direction.
The automatic detection system
Based on the vehicle position data and the vehicle attitude data, the position data of the facility of interest, which is a traffic safety facility located in the traveling direction of the vehicle, is acquired from the information database, and the position data of the facility of interest is used as the basis for the position data. A directional control unit that controls the directional function of the camera so that the camera captures the direction in which the facility of interest is located is provided.
The road image is obtained by taking a picture of the direction in which the facility of interest is located by the camera.

The camera has an angle of view adjusting function for adjusting the angle of view.
The directional control unit narrows the angle of view of the camera by controlling the angle of view adjusting function.
The road image is obtained by taking a picture after the angle of view of the camera is narrowed.

The automatic detection program of the present invention automatically detects at least one of the traffic safety facilities of a traffic signal lamp and a road sign.
The automatic detection program
Data acquisition to acquire a road image obtained by shooting with a camera mounted on a vehicle, vehicle position data indicating the position of the vehicle at the time of shooting, and vehicle posture data indicating the posture of the vehicle at the time of shooting. Department and
Based on the vehicle position data and the vehicle attitude data from an information database in which information data including position data, size data, and feature data is registered for each traffic safety facility provided at each point, the vehicle is taken at the time of shooting. A database search unit that acquires information data of the target facility, which is a traffic safety facility located in the direction of travel,
The target facility is projected from the road image based on the vehicle position data, the vehicle attitude data, the position data included in the information data of the target facility, and the size data included in the information data of the target facility. An image range limiting part that extracts a range as a limited range, and
As a detection unit that detects the target facility by collating the limited range with the feature data included in the information data of the target facility.
Make your computer work.

The detection unit performs projection conversion and size conversion on the limited range, and collates the converted limited range with the feature data.

The feature data includes a reference image showing the target facility.
The detection unit compares the limited range after the conversion with the reference image, and determines whether or not the limited range matches the feature data based on the comparison result.

The target facility is a traffic signal lamp.
The feature data includes display area data.
The display area data indicates a display area which is each area of one or more display units provided in the traffic signal lamp.
The detection unit extracts a display range corresponding to each display area from the limited range after conversion, and determines and discriminates the signal display reflected in the limited range after conversion by image processing for each display range. The signal display is compared with the signal display shown in the display information data at the time of shooting, and it is determined based on the comparison result whether or not the limited range matches the feature data.

The target facility is a road sign.
The feature data includes display area data.
The display area data indicates a display area which is an area of a display unit provided on the road sign.
The detection unit extracts a display range corresponding to the display area from the limited range after conversion, and determines whether or not the display unit of the road sign is reflected in the display range by image processing on the display range. , Whether or not the limited range matches the feature data is determined based on the determination result.

The feature data includes adjacency data indicating an adjacency that is an object adjacent to the target facility.
The detection unit extracts a reference range, which is a range including the limited range, from the road image, performs projection conversion and size conversion on the reference range, and performs image processing on the converted reference range to obtain the adjacent reference range. It is determined whether or not the object is reflected in the reference range, and whether or not the limited range matches the feature data is determined based on the determination result.

The camera has a directivity function for adjusting the line-of-sight direction.
The automatic detection program further
Based on the vehicle position data and the vehicle attitude data, the position data of the facility of interest, which is a traffic safety facility located in the traveling direction of the vehicle, is acquired from the information database, and the position data of the facility of interest is used as the basis for the position data. As a directional control unit that controls the directional function of the camera so that the camera captures the direction in which the facility of interest is located.
Make your computer work.
The road image is obtained by taking a picture of the direction in which the facility of interest is located by the camera.

The camera has an angle of view adjusting function for adjusting the angle of view.
The directional control unit narrows the angle of view of the camera by controlling the angle of view adjusting function.
The road image is obtained by taking a picture after the angle of view of the camera is narrowed.

According to the present invention, it is possible to improve the performance of the function of automatically detecting a traffic signal lamp or the like.

The block diagram of the automatic detection system 100 in Embodiment 1. FIG. The block diagram of the vehicle 110 in Embodiment 1. The explanatory view for the directivity of the camera 119 in Embodiment 1. FIG. The figure which shows the specific example of the road image in Embodiment 1. FIG. FIG. 6 is a simplified diagram of a specific example of a road image according to the first embodiment. The block diagram of the sensor management apparatus 300 in Embodiment 1. FIG. The block diagram of the automatic detection apparatus 200 in Embodiment 1. FIG. The flowchart of the sensor management method in Embodiment 1. The flowchart of the automatic detection method in Embodiment 1. The figure which shows the specific example of the adjacent object of the target signal lamp in Embodiment 1. FIG. The figure which shows the specific example of the road image in Embodiment 2. FIG. FIG. 6 is a simplified diagram of a specific example of a road image according to the second embodiment. The flowchart of the sensor management method in Embodiment 2. The flowchart of the automatic detection method in Embodiment 2. The figure which shows the specific example of the adjacent thing of the target sign in Embodiment 2. FIG.

In embodiments and drawings, the same or corresponding elements are designated by the same reference numerals. The description of the elements with the same reference numerals as the described elements will be omitted or abbreviated as appropriate. The arrows in the figure mainly indicate the flow of data or the flow of processing.

Embodiment 1.
A mode for automatically detecting the traffic signal lamps provided at each point will be described with reference to FIGS. 1 to 10.

Traffic lights are part of traffic lights.
The traffic signal includes a traffic signal lamp and a traffic signal controller.
The traffic signal lamp displays a traffic signal. Traffic signals include red, green, yellow and arrow signals.
The traffic signal controller controls the display of the traffic signal lamp.
The traffic signal may be any of a traffic light for automobiles, a traffic light for pedestrians, or a traffic light for other purposes.
A traffic signal is a type of traffic safety facility.

*** Explanation of configuration ***
The configuration of the automatic detection system 100 will be described with reference to FIG.
The automatic detection system 100 includes an automatic detection device 200 and a sensor management device 300.
The automatic detection device 200 and the sensor management device 300 are mounted on the vehicle 110.
Further, the vehicle 110 is equipped with an automatic driving system 130.

While traveling on the road, the vehicle 110 measures the position and posture at each time and takes a picture. For example, vehicle 110 is an automobile used in a mobile mapping system (MMS).
The data indicating the position of the vehicle 110 is referred to as "vehicle position data". The position of the vehicle 110 is identified by three-dimensional coordinate values.
The data indicating the posture of the vehicle 110 is referred to as "vehicle posture data". The posture of the vehicle 110 is identified by a roll angle, a pitch angle and a yaw angle.
The image obtained by taking a picture from the vehicle 110 is referred to as a "road image". The road image shows the area around the road (including the road).

The automatic detection device 200 uses the vehicle position data at each time, the vehicle attitude data at each time, the road image at each time, and the information database 120 to obtain a traffic signal lamp located in the traveling direction of the vehicle 110. To detect.

The information database 120 is a database in which information data of traffic signal lamps provided at each point is registered.
The information data of the traffic signal lamp indicates various information of the traffic signal lamp.

The information data of the traffic signal lamp includes an identification number, basic data, and feature data.
The basic data includes position data, posture data, size data, and display content data.
The feature data includes display area data, adjacency data, and a reference image.
The identification number of the traffic signal lamp identifies the traffic signal lamp.
The position data of the traffic signal lamp indicates the position of the traffic signal lamp. The position of the traffic signal lamp is identified by the three-dimensional coordinate value of the reference point of the traffic signal lamp.
The attitude data of the traffic signal lamp indicates the attitude of the traffic signal lamp. The posture of the traffic signal lamp is identified by the inclination angle of the front portion of the traffic signal lamp.
The size data of the traffic signal lamp indicates the size of the traffic signal lamp. The size of a traffic light is identified by its dimensions and shape.
The display content data of the traffic signal lamp indicates each type of one or more display units provided in the traffic signal lamp. For example, the display unit is classified into a type such as a red traffic light, a green traffic light, a yellow traffic light, or an arrow signal.
The display area data of the traffic signal lamp indicates each area (display area) of one or more display units provided in the traffic signal lamp. For example, the display area is identified by a two-dimensional area in front of the traffic signal lamp.
Adjacent object data of a traffic signal lamp indicates an object (adjacent object) adjacent to the traffic signal lamp. For example, adjacency data includes identification numbers, type identifiers, relative position data, relative size data, and the like. The identification number identifies the adjacent object. The type identifier identifies the type of adjacency. The relative position data indicates the relative position of the adjacent object with respect to the position of the traffic signal lamp. Relative size data indicates the relative size of an adjacent object to the size of a traffic signal light.
The reference image of the traffic signal lamp is an image showing the traffic signal lamp. For example, the reference image shows a traffic signal lamp viewed from the front in a reference size.

The information database 120 may be managed by any of the sensor management device 300, the automatic detection device 200, the automatic operation system 130, or other device (for example, a database server).

The automatic driving system 130 is a system for automatic driving. The automatic driving realized by the automatic driving system 130 may be any level of automatic driving such as fully automatic driving, partially automatic driving, or driving assist.

The sensor management device 300 generates vehicle position data and vehicle attitude data.

The configuration of the vehicle 110 will be described with reference to FIG.
The vehicle 110 is provided with various sensors on the roof of the vehicle body.
Specifically, the vehicle 110 includes antennas (111A to 111C), an IMU 112, cameras (113A, 113B), and a laser scanner 114.
These sensors are the basic elements of a vehicle for MMS.

The antennas (111A to 111C) are antennas for a satellite positioning system. A specific example of a satellite positioning system is the Global Positioning System (GPS).
The IMU 112 is an inertial measurement unit.
The cameras (113A, 113B) are cameras for MMS.

The vehicle 110 includes a camera 119 at the front of the vehicle body.
The camera 119 is a camera for the automatic detection system 100. The camera 119 has an image sensor.

The sensor management device 300 (see FIG. 1) controls the directivity of the camera 119.

The directivity of the camera 119 will be described with reference to FIG.
The camera 119 has a function of adjusting the line-of-sight direction (direction function). The line-of-sight direction is a shooting direction with the position of the camera 119 as a base point. The shooting direction is the direction in which the camera 119 shoots. Specifically, the directional function adjusts the orientation of the image sensor (the visual axis of the image sensor). However, the directional function may adjust the posture of the camera 119 instead of adjusting the orientation of the image sensor (the visual axis of the image sensor). For example, the directional function is realized by a device that drives an image sensor or a camera 119. The camera 119 can shoot a specific direction by the directional function.
The camera 119 has a function of adjusting the angle of view (angle of view adjusting function). For example, the camera 119 can take a picture by narrowing the angle of view by the angle of view adjusting function. That is, the camera 119 can shoot in a narrow field of view. Shooting in a narrow field of view narrows the shooting range and increases the resolution of the shooting range. That is, the number of elements of the image sensor is increased.

A road image obtained by controlling the directivity of the camera 119 will be described with reference to FIGS. 4 and 5.
FIG. 4 shows a specific example of a road image. FIG. 5 briefly shows a specific example of a road image.
If the directivity of the camera 119 is not controlled, a wide field of view is taken and a road image with a small traffic light towards the edge is obtained (see solid rectangle). In this case, it becomes difficult to recognize the traffic signal lamp shown in the road image.
When the directivity of the camera 119 is controlled, a narrow field of view is taken and a road image with a large traffic signal lamp in the center is obtained (see the dashed rectangle). In this case, it becomes easy to recognize the traffic signal lamp shown in the road image.

The configuration of the sensor management device 300 will be described with reference to FIG.
The sensor management device 300 is a computer including hardware such as a processor 301, a memory 302, an auxiliary storage device 303, a communication device 304, and an input / output interface 305. These hardware are connected to each other via signal lines.

The processor 301 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 301 is a CPU.
IC is an abbreviation for Integrated Circuit.
CPU is an abbreviation for Central Processing Unit.

The memory 302 is a volatile or non-volatile storage device. The memory 302 is also referred to as a main storage device or a main memory. For example, the memory 302 is a RAM. The data stored in the memory 302 is stored in the auxiliary storage device 303 as needed.
RAM is an abbreviation for Random Access Memory.

The auxiliary storage device 303 is a non-volatile storage device. For example, the auxiliary storage device 303 is a ROM, HDD, or flash memory. The data stored in the auxiliary storage device 303 is loaded into the memory 302 as needed.
ROM is an abbreviation for Read Only Memory.
HDD is an abbreviation for Hard Disk Drive.

The communication device 304 is a receiver and a transmitter. For example, the communication device 304 is a communication chip or NIC.
NIC is an abbreviation for Network Interface Card.

The input / output interface 305 is an input / output port. For example, one end of a cable is connected to each of the various sensors and the camera 119, and the other end of each cable is connected to the input / output interface 305. As a result, various sensors and the camera 119 are connected to the sensor management device 300.

The sensor management device 300 includes elements such as a data collection unit 311, a data processing unit 312, and a directional control unit 313. These elements are realized in software.

The auxiliary storage device 303 stores a sensor management program for operating the computer as a data collection unit 311, a data processing unit 312, and a directional control unit 313. The sensor management program is loaded into memory 302 and executed by processor 301.
The OS is further stored in the auxiliary storage device 303. At least part of the OS is loaded into memory 302 and executed by processor 301.
The processor 301 executes the sensor management program while executing the OS.
OS is an abbreviation for Operating System.

The input / output data of the sensor management program is stored in the storage unit 390.
The memory 302 functions as a storage unit 390. However, a storage device such as an auxiliary storage device 303, a register in the processor 301, and a cache memory in the processor 301 may function as a storage unit 390 instead of the memory 302 or together with the memory 302.

The sensor management program can record (store) in a non-volatile recording medium such as an optical disk or a flash memory so that it can be read by a computer.

The configuration of the automatic detection device 200 will be described with reference to FIG. 7.
The automatic detection device 200 is a computer including hardware such as a processor 201, a memory 202, an auxiliary storage device 203, a communication device 204, and an input / output interface 205. These hardware are connected to each other via signal lines.

The processor 201 is an IC that performs arithmetic processing and controls other hardware. For example, the processor 201 is a CPU.

The memory 202 is a volatile or non-volatile storage device. The memory 202 is also referred to as a main storage device or a main memory. For example, the memory 202 is a RAM. The data stored in the memory 202 is stored in the auxiliary storage device 203 as needed.

The auxiliary storage device 203 is a non-volatile storage device. For example, the auxiliary storage device 203 is a ROM, HDD, or flash memory. The data stored in the auxiliary storage device 203 is loaded into the memory 202 as needed.

Communication device 204 is a receiver and a transmitter. For example, the communication device 204 is a communication chip or NIC.

The input / output interface 205 is an input / output port.

The automatic detection device 200 includes elements such as a data acquisition unit 211, a database search unit 212, an image range limiting unit 213, a detection unit 214, a result output unit 215, and a display information acquisition unit 216. These elements are realized in software.

The auxiliary storage device 203 stores a database generation program for operating the computer as a data acquisition unit 211, a database search unit 212, an image range limiting unit 213, a detection unit 214, a result output unit 215, and a display information acquisition unit 216. Has been done. The database generator is loaded into memory 202 and executed by processor 201.
The OS is further stored in the auxiliary storage device 203. At least part of the OS is loaded into memory 202 and executed by processor 201.
The processor 201 executes the database generation program while executing the OS.

The input / output data of the database generation program is stored in the storage unit 290.
The memory 202 functions as a storage unit 290. However, a storage device such as an auxiliary storage device 203, a register in the processor 201, and a cache memory in the processor 201 may function as a storage unit 290 instead of the memory 202 or together with the memory 202.

The database generation program can record (store) in a non-volatile recording medium such as an optical disk or a flash memory so that it can be read by a computer.

*** Explanation of operation ***
The operation procedure of the automatic detection system 100 corresponds to the automatic detection method.
The operation procedure of the automatic detection system 100 (particularly, the automatic detection device 200) corresponds to the processing procedure by the automatic detection program. A program for the sensor management device 300 (sensor management program) may be included in the automatic detection program.

The sensor management method will be described with reference to FIG.
The sensor management method corresponds to the procedure for operating the sensor management device 300.

When shooting with the camera 119, the direction control unit 313 narrows the angle of view of the camera 119 by controlling the angle of view adjusting function of the camera 119. Therefore, the camera 119 shoots in a narrow field of view.

Steps S101 to S104 are executed at each time.

In step S101, the data collecting unit 311 acquires sensor data from each sensor and stores each sensor data in the storage unit 390.
The sensor data is the data obtained by the sensor.

The sensor data obtained by the antennas (111A to 111C) is referred to as "positioning signal data".
The sensor data obtained by the IMU 112 is referred to as "inertial measurement data".
The sensor data obtained by photographing with the camera 119 is referred to as a "road image".

In step S102, the data processing unit 312 generates vehicle position data and vehicle posture data using the sensor data acquired in step S101, and stores the vehicle position data and vehicle posture data in the storage unit 390.

The data processing unit 312 generates vehicle position data and vehicle attitude data as follows.
The data processing unit 312 calculates the position of the vehicle 110 by performing satellite navigation using the positioning signal data. Then, the data processing unit 312 generates vehicle position data indicating the position of the vehicle 110.
The data processing unit 312 calculates the attitude of the vehicle 110 by performing inertial navigation using the inertial measurement data. Then, the data processing unit 312 generates vehicle posture data indicating the posture of the vehicle 110.

In step S103, the directional control unit 313 acquires the position data of the attention signal lamp (an example of the attention facility) from the information database 120 based on the vehicle position data and the vehicle attitude data.

The attention signal lamp is a traffic signal lamp located in the traveling direction of the vehicle 110. In particular, a traffic signal lamp located in the conical region on the traveling direction side of the vehicle 110 is referred to as a attention signal lamp. The cone-shaped region is a cone-shaped (for example, a quadrangular pyramid) region having the position of the vehicle 110 as an apex. The size of the cone region is predefined.

The traveling direction of the vehicle 110 is a direction determined by the posture of the vehicle 110 with the position of the vehicle 110 as a base point.

For example, the directional control unit 313 calculates the traveling direction of the vehicle 110 using the vehicle position data and the vehicle attitude data, and calculates the cone-shaped region on the traveling direction side of the vehicle 110. Then, the directional control unit 313 finds the position data of the traffic signal lamp (attention signal lamp) located in the calculated cone-shaped region by searching the information database 120.

In step S104, the directional control unit 313 controls the directional function of the camera 119 so that the camera 119 captures the direction in which the attention signal lamp is located based on the position data of the attention signal lamp.

For example, the directional control unit 313 calculates the shooting direction of the camera 119 based on the position of the vehicle 110, the posture of the vehicle 110, and the line-of-sight direction of the camera 119. The directional control unit 313 calculates the amount of deviation between the direction from the camera 119 to the position of the signal lamp of interest and the shooting direction of the camera 119. The calculated deviation amount is called an adjustment amount. Then, the directional control unit 313 controls the directional function of the camera 119 to adjust the line-of-sight direction of the camera 119 as well as the adjustment amount.

A database generation method will be described with reference to FIG.

The display information acquisition unit 216 acquires the display information data of the traffic signal lamps located around the vehicle 110 at each time.
The display information data indicates an identification number and a signal display. The identification number identifies the traffic signal lamp. The signal display is a display of a traffic signal lamp at that time.
For example, the display information acquisition unit 216 communicates with a device that manages the display of a traffic signal lamp, such as a traffic signal controller or a computer of a traffic control center, and receives display information data from the device.

Steps S111 through S115 are performed on the dataset at each time.
The data set includes a road image obtained by shooting with a camera 119 mounted on the vehicle 110, vehicle position data indicating the position of the vehicle 110 at the time of shooting, and vehicle posture data indicating the posture of the vehicle 110 at the time of shooting. ,including.

In step S111, the data acquisition unit 211 acquires the data set.
For example, the data acquisition unit 211 receives the data set at each time from the sensor management device 300 by communicating with the sensor management device 300. Then, the data acquisition unit 211 stores the data set at each time in the memory 302.

In step S112, the database search unit 212 acquires information data of the target signal lamp (an example of the target equipment) from the information database 120 based on the vehicle position data and the vehicle attitude data.

The target signal lamp is a traffic signal lamp that was located in the traveling direction of the vehicle 110 at the time of shooting. In particular, a traffic signal lamp located in the conical region on the traveling direction side of the vehicle 110 at the time of photographing is referred to as a target signal lamp. The size of the cone region is predefined.

For example, the database search unit 212 calculates the traveling direction of the vehicle 110 using the vehicle position data and the vehicle posture data, and calculates the cone-shaped region on the traveling direction side of the vehicle 110. Then, the database search unit 212 finds the position data of the traffic signal lamp (target signal lamp) located in the calculated cone-shaped region by searching the information database 120. The information data including the found position data is the information data of the target signal lamp.

In step S113, the image range limiting unit 213 extracts a limited range from the road image based on the vehicle position data, the vehicle attitude data, the position data of the target signal lamp, and the size data of the target signal lamp.
The limited range is the range in which the target signal lamp is reflected in the road image.

For example, the image range limiting unit 213 calculates the shooting direction of the camera 119 based on the position of the vehicle 110, the posture of the vehicle 110, and the line-of-sight direction of the camera 119. The image range limiting unit 213 calculates the shooting range of the camera 119 based on the shooting direction of the camera 119 and the angle of view of the camera 119. The image range limiting unit 213 identifies pixels (target pixels) corresponding to the positions of the target signal lamps in the road image based on the shooting range of the camera 119. Further, the image range limiting unit 213 calculates the size (target size) of the target signal lamp reflected in the road image based on the distance from the camera 119 to the target signal lamp and the size of the target signal lamp. Then, the image range limiting unit 213 extracts a range having a target size from the road image with the target pixel as a base point. The extracted range is a limited range.

In step S114, the detection unit 214 detects the target signal lamp by collating the limited range with the feature data included in the information data of the target signal lamp.
The limited range is collated with the feature data as follows.

The detection unit 214 performs projection conversion and size conversion on the limited range.
The projective transformation for a limited range is a process for converting a subject (target signal lamp) reflected in the limited range into an image obtained by shooting from the front.
The size conversion for the limited range is a process for enlarging or reducing the limited range and converting the limited range into a reference size image. For example, the detection unit 214 expands or contracts the limited range at a magnification according to the distance between the vehicle 110 and the target signal lamp.
Then, the detection unit 214 collates the limited range after conversion with the feature data.

The feature data includes a reference image.
The reference image is an image showing a traffic signal lamp.
The detection unit 214 compares the limited range after conversion with the reference image.
If the converted limited range is similar to the reference image, the limited range matches the feature data.
If the limited range after the conversion process is not similar to the reference image, the limited range does not match the feature data.

The feature data includes display area data.
The display area data indicates each area (display area) of one or more display units provided in the traffic signal lamp.
First, the detection unit 214 extracts a range corresponding to each display area from the limited range after conversion. Each extracted range is called a display range.
Next, the detection unit 214 determines the signal display reflected in the limited range after conversion by image processing for each display range.
Then, the detection unit 214 compares the determined signal display with the signal display shown in the display information data at the time of shooting.
The display information data used is the display information data including the same identification number as the identification number included in the information data of the target signal lamp.
When the signal display of the limited range matches the signal display shown in the display information data at the time of shooting, the limited range matches the feature data.
If the signal display of the limited range does not match the signal display shown in the display information data at the time of shooting, the limited range does not match the feature data.

The feature data includes adjacency data.
Adjacent object data indicates an object (adjacent object) adjacent to a traffic signal lamp. Specific examples of adjacent objects will be described later.
First, the detection unit 214 extracts a range including the limited range from the road image. The extracted range is called the reference range. For example, the size of the reference range is determined by the size of the limited range. The larger the limited range, the larger the reference range, and the smaller the limited range, the smaller the reference range.
Next, the detection unit 214 performs a projective transformation and a size transformation on the reference range.
The projective transformation with respect to the reference range is a process for converting a subject (target signal lamp and an adjacent object) reflected in the reference range into an image obtained by photographing from the front.
The size conversion for the reference range is a process for enlarging or reducing the reference range and converting the reference range into a reference size image. For example, the detection unit 214 expands or contracts the reference range at a magnification according to the distance between the vehicle 110 and the target signal lamp.
Then, the detection unit 214 determines whether or not the adjacent object shown in the adjacent object data is reflected in the reference range by image processing on the reference range after conversion. For example, the adjacent object is determined by the type, the relative position with respect to the target signal lamp, the relative size with respect to the target signal lamp, and the like. The type of adjacency is determined, for example, by pattern matching of appearance such as shape and color.
If the reference range shows an adjacent object, the limited range matches the feature data.
If the reference range does not show adjacent objects, the limited range does not match the feature data.

The detection unit 214 determines the detection result of the target signal lamp based on the result of each collation.
For example, if the limited range matches the feature data in at least one of the collations, the detection unit 214 determines that the target signal lamp has been detected.

A specific example of an adjacent object of the target signal lamp will be described with reference to FIG.
The column supporting the target signal lamp is an example of an adjacent object of the target signal lamp. Further, the traffic signal controller attached to the support of the target signal lamp is an example of an adjacent object of the target signal lamp.

Returning to FIG. 9, step S115 will be described.
In step S115, the result output unit 215 outputs the detection result data to the automatic operation system 130.
The detection result data is data showing the detection result of the target signal lamp. For example, the detection result data includes the detection result and the target data. The detection result indicates whether or not the target signal lamp is detected. The target data indicates various information (identification number, position, etc.) of the target signal lamp.
The detection result data is input to the automatic driving system 130 and used by the automatic driving system 130.

*** Effect of Embodiment 1 ***
According to the first embodiment, the performance of the function of automatically detecting the traffic signal lamp can be improved.
The feature data of each traffic signal lamp is registered in the information database 120. By collating the feature data of each traffic signal lamp with the road image, it is possible to accurately detect the traffic signal lamp without omission.

Embodiment 2.
The mode for automatically detecting the road sign provided at each point will be described mainly different from the first embodiment with reference to FIGS. 11 to 15.

Road signs are a type of traffic safety facility.

*** Explanation of configuration ***
The configuration of the automatic detection system 100 is the same as the configuration in the first embodiment (see FIG. 1).
However, the information database 120 is a database in which information data of road signs provided at each point is registered. That is, in the second embodiment, the content of the information data described in the first embodiment replaces the information of the "traffic signal lamp" with the information of the "road sign".

The configuration of the vehicle 110 is the same as the configuration in the first embodiment (see FIG. 2).

The directivity of the camera 119 is the same as the directivity in the first embodiment (see FIG. 3).

A road image obtained by controlling the directivity of the camera 119 will be described with reference to FIGS. 11 and 12.
FIG. 11 shows a specific example of a road image. FIG. 12 briefly shows a specific example of a road image.
If the directivity of camera 119 is not controlled, wide field of view is taken and a road image with road signs smaller towards the edges is obtained (see solid rectangle). In this case, it becomes difficult to recognize the road sign shown in the road image.
When the directivity of the camera 119 is controlled, a narrow field of view is taken and a road image with a large road sign in the center is obtained (see the dashed rectangle). In this case, it becomes easy to recognize the road sign reflected in the road image.

The configuration of the sensor management device 300 is the same as the configuration in the first embodiment (see FIG. 6).

The configuration of the automatic detection device 200 is the same as the configuration in the first embodiment (see FIG. 7).

*** Explanation of operation ***
The sensor management method will be described with reference to FIG.
The sensor management method corresponds to the procedure for operating the sensor management device 300.

When shooting with the camera 119, the direction control unit 313 narrows the angle of view of the camera 119 by controlling the angle of view adjusting function of the camera 119. Therefore, the camera 119 shoots in a narrow field of view.

Steps S201 to S204 are executed at each time.

In step S201, the data collecting unit 311 acquires sensor data from each sensor and stores each sensor data in the storage unit 390.
Step S201 is the same as step S101 (see FIG. 8) of the first embodiment.

In step S202, the data processing unit 312 generates vehicle position data and vehicle posture data using the sensor data acquired in step S201, and stores the vehicle position data and vehicle posture data in the storage unit 390.
Step S201 is the same as step S101 (see FIG. 8) of the first embodiment.

In step S203, the directional control unit 313 acquires the position data of the attention sign (an example of the facility of interest) from the information database 120 based on the vehicle position data and the vehicle attitude data.
The attention sign is a road sign located in the traveling direction of the vehicle 110.
The process of step S203 is the same as the process of replacing the “attention signal lamp” with the “attention sign” in step S103 (see FIG. 8) of the first embodiment.

In step S204, the directional control unit 313 controls the directional function of the camera 119 so that the camera 119 captures the direction in which the attention sign is located based on the position data of the attention sign.
The process of step S204 is the same as the process of replacing the “attention signal lamp” with the “attention sign” in step S104 (see FIG. 8) of the first embodiment.

A database generation method will be described with reference to FIG.
Steps S211 to S215 are performed on the dataset at each time.

In step S211 the data acquisition unit 211 acquires the data set.
Step S211 is the same as step S111 (see FIG. 9) of the first embodiment.

In step S212, the database search unit 212 acquires information data of the target indicator (an example of the target equipment) from the information database 120 based on the vehicle position data and the vehicle attitude data.
The target sign is a road sign that was located in the traveling direction of the vehicle 110 at the time of shooting.
The process of step S212 is the same as the process of replacing the “target signal lamp” with the “target sign” in step S112 (see FIG. 9) of the first embodiment.

In step S213, the image range limiting unit 213 extracts the limited range from the road image based on the information data of the target sign, the vehicle position data, and the vehicle posture data.
The process of step S213 is the same as the process of replacing the “target signal lamp” with the “target sign” in step S113 (see FIG. 9) of the first embodiment.

In step S214, the detection unit 214 detects the target sign by collating the limited range with the feature data included in the information data of the target sign.
The process of step S214 is the same as the process of replacing the “target signal lamp” with the “target sign” in step S114 (see FIG. 9) of the first embodiment.
The limited range is collated with the feature data as follows.

The detection unit 214 performs projection conversion and size conversion on the limited range.
The projective transformation for a limited range is a process for converting a subject (target sign) reflected in the limited range into an image obtained by shooting from the front.
The size conversion for the limited range is a process for enlarging or reducing the limited range and converting the limited range into a reference size image. For example, the detection unit 214 expands or contracts the limited range at a magnification according to the distance between the vehicle 110 and the target sign.
Then, the detection unit 214 collates the limited range after conversion with the feature data.

The feature data includes a reference image.
The reference image is an image showing a road sign.
The detection unit 214 compares the limited range after conversion with the reference image.
If the converted limited range is similar to the reference image, the limited range matches the feature data.
If the limited range after the conversion process is not similar to the reference image, the limited range does not match the feature data.

The feature data includes display area data.
The display area data indicates an area (display area) of the display unit provided on the road sign.
First, the detection unit 214 extracts a range corresponding to the display area shown in the display area data from the limited range after conversion. The extracted range is called the display range.
Next, the detection unit 214 determines whether or not the display unit of the road sign is reflected in the display range by image processing on the display range.
When the display part of the road sign is reflected in the display range, the limited range matches the feature data.
If the display part of the road sign is not reflected in the display range, the limited range does not match the feature data.

The feature data includes adjacency data.
Adjacent object data indicates an object (adjacent object) adjacent to a road sign. Specific examples of adjacent objects will be described later.
First, the detection unit 214 extracts a range including the limited range from the road image. The extracted range is called the reference range. For example, the size of the reference range is determined by the size of the limited range. The larger the limited range, the larger the reference range, and the smaller the limited range, the smaller the reference range.
Next, the detection unit 214 performs a projective transformation and a size transformation on the reference range.
The projective transformation with respect to the reference range is a process for converting a subject (target sign and an adjacent object) reflected in the reference range into an image obtained by photographing from the front.
The size conversion for the reference range is a process for enlarging or reducing the reference range and converting the reference range into a reference size image. For example, the detection unit 214 expands or contracts the reference range at a magnification according to the distance between the vehicle 110 and the target sign.
Then, the detection unit 214 determines whether or not the adjacent object shown in the adjacent object data is reflected in the reference range by image processing on the reference range after conversion. For example, adjacencies are determined by type, relative position relative to the target marker, relative size relative to the target marker, and the like. The type of adjacency is determined, for example, by pattern matching of appearance such as shape and color.
If the reference range shows an adjacent object, the limited range matches the feature data.
If the reference range does not show adjacent objects, the limited range does not match the feature data.

The detection unit 214 determines the detection result of the target marker based on the result of each collation.
For example, if the limited range matches the feature data in at least one of the collations, the detection unit 214 determines that the target marker has been detected.

A specific example of an object adjacent to the target sign will be described with reference to FIG.
The strut supporting the target sign is an example of an adjacent object of the target sign. Further, the arbitrary sign attached to the support of the target sign is an example of an adjacent object of the target sign. The optional sign is a road sign that is not managed by the information database 120.

Returning to FIG. 14, step S215 will be described.
In step S215, the result output unit 215 outputs the detection result data to the automatic operation system 130.
The process of step S215 is the same as the process of replacing the “target signal lamp” with the “target sign” in step S115 (see FIG. 9) of the first embodiment.

*** Effect of Embodiment 2 ***
According to the second embodiment, the performance of the function of automatically detecting a road sign can be improved.
The feature data of each road sign is registered in the information database 120. By collating the feature data of each road sign with the road image, it is possible to accurately detect the road sign without omission.

*** Supplement to the embodiment ***
The sensor management device 300, the automatic detection device 200, and the automatic operation system 130 may be integrated into one or two devices.

Instead of the camera 119, cameras for MMS (113A, 113B) may be used.
A camera 119 may be used instead of the cameras for MMS (113A, 113B).

The vehicle 110 may be a railroad vehicle. The traffic signal may be a railway signal (for example, a railroad crossing alarm).

The automatic detection system 100 may detect a traffic safety facility other than the traffic signal lamp and the road sign.
That is, the first embodiment may be carried out by replacing the "traffic signal lamp" with the "other traffic safety facility", or the second embodiment may be carried out by replacing the "road sign" with the "other traffic safety facility". May be done.

The hardware configuration of the automatic detection device 200 and the sensor management device 300 will be supplemented.
The automatic detection device 200 includes a processing circuit that realizes a data acquisition unit 211, a database search unit 212, an image range limiting unit 213, a detection unit 214, a result output unit 215, and a display information acquisition unit 216.
The sensor management device 300 includes a processing circuit that realizes a data collection unit 311, a data processing unit 312, and a directional control unit 313.
The processing circuit may be dedicated hardware or a processor that executes a program stored in memory.
In each of the automatic detection device 200 and the sensor management device 300, some functions may be realized by dedicated hardware, and the remaining functions may be realized by software or firmware.

Each embodiment is an example of a preferred embodiment and is not intended to limit the technical scope of the invention. Each embodiment may be partially implemented or may be implemented in combination with other embodiments. The procedure described using the flowchart or the like may be appropriately changed.
The "unit" which is an element of each of the automatic detection device 200 and the sensor management device 300 may be read as "processing" or "process".

100 automatic detection system, 110 vehicle, 111 antenna, 112 IMU, 113 camera, 114 laser scanner, 119 camera, 120 information database, 130 automatic driving system, 200 automatic detection device, 201 processor, 202 memory, 203 auxiliary storage device, 204 Communication device, 205 I / O interface, 211 data acquisition unit, 212 database search unit, 213 image range limitation unit, 214 detection unit, 215 result output unit, 216 display information acquisition unit, 290 storage unit, 300 sensor management unit, 301 Processor, 302 memory, 303 auxiliary storage device, 304 communication device, 305 input / output interface, 311 data collection unit, 312 data processing unit, 313 directional control unit, 390 storage unit.

Claims (14)

It is an automatic detection system that automatically detects at least one of the traffic safety facilities of traffic signal lights and road signs.
Data acquisition to acquire a road image obtained by shooting with a camera mounted on a vehicle, vehicle position data indicating the position of the vehicle at the time of shooting, and vehicle posture data indicating the posture of the vehicle at the time of shooting. Department and
Based on the vehicle position data and the vehicle attitude data from an information database in which information data including position data, size data, and feature data is registered for each traffic safety facility provided at each point, the vehicle is taken at the time of shooting. A database search unit that acquires information data of the target facility, which is a traffic safety facility located in the direction of travel,
The target facility is projected from the road image based on the vehicle position data, the vehicle attitude data, the position data included in the information data of the target facility, and the size data included in the information data of the target facility. An image range limiting part that extracts a range as a limited range, and
A detection unit that detects the target facility by collating the limited range with the feature data included in the information data of the target facility.
Equipped with a,
The feature data is adjacent object data indicating the relative position of the adjacent object with respect to the target facility and the relative size of the adjacent object with respect to the target facility as information for discriminating an adjacent object which is an object adjacent to the target facility. Including
The detection unit extracts a reference range, which is a range including the limited range, from the road image, performs projection conversion and size conversion on the reference range, and performs image processing on the converted reference range to obtain the adjacent reference range. Whether or not an object is reflected in the reference range is determined based on the relative position and the relative size, and whether or not the limited range matches the feature data is determined based on the determination result. <br /> Automatic detection system. The automatic detection system according to claim 1, wherein the detection unit performs projection conversion and size conversion on the limited range, and collates the converted limited range with the feature data. The feature data includes a reference image showing the target facility.
The automatic detection system according to claim 2, wherein the detection unit compares the converted limited range with the reference image, and determines whether or not the limited range matches the feature data based on the comparison result. .. The target facility is a traffic signal lamp,
The feature data includes display area data.
The display area data indicates a display area which is each area of one or more display units provided in the traffic signal lamp.
The detection unit extracts a display range corresponding to each display area from the limited range after conversion, and determines and discriminates the signal display reflected in the limited range after conversion by image processing for each display range. 2. The automatic detection system according to 3. The target facility is a road sign,
The feature data includes display area data.
The display area data indicates a display area which is an area of a display unit provided on the road sign.
The detection unit extracts a display range corresponding to the display area from the limited range after conversion, and determines whether or not the display unit of the road sign is reflected in the display range by image processing on the display range. The automatic detection system according to claim 2 or 3, wherein it is determined based on the determination result whether or not the limited range matches the feature data. The camera has a directivity function for adjusting the line-of-sight direction.
The automatic detection system
Based on the vehicle position data and the vehicle attitude data, the position data of the facility of interest, which is a traffic safety facility located in the traveling direction of the vehicle, is acquired from the information database, and the position data of the facility of interest is used as the basis for the position data. A directional control unit that controls the directional function of the camera so that the camera captures the direction in which the facility of interest is located is provided.
The automatic detection system according to any one of claims 1 to 5 , wherein the road image is obtained by taking a picture of the direction in which the facility of interest is located by the camera. The camera has an angle of view adjustment function for adjusting the angle of view.
The directional control unit narrows the angle of view of the camera by controlling the angle of view adjusting function.
The automatic detection system according to claim 6 , wherein the road image is obtained by taking a picture after the angle of view of the camera is narrowed. It is an automatic detection program for automatically detecting at least one of the traffic safety facilities of traffic signal lights and road signs.
Data acquisition to acquire a road image obtained by shooting with a camera mounted on a vehicle, vehicle position data indicating the position of the vehicle at the time of shooting, and vehicle posture data indicating the posture of the vehicle at the time of shooting. Department and
Based on the vehicle position data and the vehicle attitude data from an information database in which information data including position data, size data, and feature data is registered for each traffic safety facility provided at each point, the vehicle is taken at the time of shooting. A database search unit that acquires information data of the target facility, which is a traffic safety facility located in the direction of travel,
The target facility is projected from the road image based on the vehicle position data, the vehicle attitude data, the position data included in the information data of the target facility, and the size data included in the information data of the target facility. An image range limiting part that extracts a range as a limited range, and
As a detection unit that detects the target facility by collating the limited range with the feature data included in the information data of the target facility.
An automatic detection program that makes your computer work
The feature data is adjacent object data indicating the relative position of the adjacent object with respect to the target facility and the relative size of the adjacent object with respect to the target facility as information for discriminating an adjacent object which is an object adjacent to the target facility. Including
The detection unit extracts a reference range, which is a range including the limited range, from the road image, performs projection conversion and size conversion on the reference range, and performs image processing on the converted reference range to obtain the adjacent reference range. Whether or not an object is reflected in the reference range is determined based on the relative position and the relative size, and whether or not the limited range matches the feature data is determined based on the determination result. <br /> Automatic detection program. The automatic detection program according to claim 8 , wherein the detection unit performs projection conversion and size conversion on the limited range, and collates the converted limited range with the feature data. The feature data includes a reference image showing the target facility.
The automatic detection program according to claim 9 , wherein the detection unit compares the converted limited range with the reference image, and determines whether or not the limited range matches the feature data based on the comparison result. .. The target facility is a traffic signal lamp,
The feature data includes display area data.
The display area data indicates a display area which is each area of one or more display units provided in the traffic signal lamp.
The detection unit extracts a display range corresponding to each display area from the limited range after conversion, and determines and discriminates the signal display reflected in the limited range after conversion by image processing for each display range. 9. 10. The automatic detection program according to 10. The target facility is a road sign,
The feature data includes display area data.
The display area data indicates a display area which is an area of a display unit provided on the road sign.
The detection unit extracts a display range corresponding to the display area from the limited range after conversion, and determines whether or not the display unit of the road sign is reflected in the display range by image processing on the display range. The automatic detection program according to claim 9 or 10 , wherein it is determined based on the determination result whether or not the limited range matches the feature data. The camera has a directivity function for adjusting the line-of-sight direction.
The automatic detection program further
Based on the vehicle position data and the vehicle attitude data, the position data of the facility of interest, which is a traffic safety facility located in the traveling direction of the vehicle, is acquired from the information database, and the position data of the facility of interest is used as the basis for the position data. As a directional control unit that controls the directional function of the camera so that the camera captures the direction in which the facility of interest is located.
Make your computer work
The automatic detection program according to any one of claims 8 to 12 , wherein the road image is obtained by taking a picture of the direction in which the facility of interest is located by the camera. The camera has an angle of view adjustment function for adjusting the angle of view.
The directional control unit narrows the angle of view of the camera by controlling the angle of view adjusting function.
The automatic detection program according to claim 13 , wherein the road image is obtained by taking a picture after the angle of view of the camera is narrowed.

Images