KR20230017365A - controlling host vehicle based on detected spacing between stationary vehicles - Google Patents

Abstract

A system and method for controlling driving of an autonomous vehicle are described. One embodiment relates to a system and method for driving a host vehicle based on detection of a door opening event. Another embodiment relates to a system and method for controlling driving of a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is driving. A third embodiment relates to a system and method for detecting whether a road on which a host vehicle is driving is a one-way road. A fourth embodiment relates to a system and method for determining a predicted state of a parked vehicle within the host vehicle's surroundings. A fifth embodiment relates to a system and method for driving a host vehicle based on a distance between two stationary vehicles.

Description

Controlling host vehicle based on detected spacing between stationary vehicles}

The present invention relates to autonomous driving of vehicles. In addition, the present invention relates to driving of the host vehicle based on detection of door opening, driving of the host vehicle based on detection of host vehicle lane entry of a target vehicle, and detection of whether the road on which the host vehicle is traveling is one-way. It relates to a system and method for driving a host vehicle and predicting situation of a parked vehicle based on the host vehicle.

With the continued development of technology, the realization of fully autonomous vehicles capable of driving on the road is just around the corner. Autonomous vehicles may need to consider a variety of factors and make appropriate decisions based on those factors in order to arrive safely and accurately at their intended destination. For example, an autonomous vehicle may need to process and interpret visual information (e.g., information taken by a camera), information from radar or lidar (LIdaR), as well as other sources (e.g., GPS devices, Information acquired from speed sensors, accelerometers, suspension sensors, etc.) can also be used. At the same time, the self-driving car can identify its location on a particular road (e.g. lane on a multi-lane road), follow other vehicles, avoid obstacles and pedestrians, and follow traffic lights and road signs in order to drive to its destination. It may be necessary to observe, move from one road to another at appropriate intersections or forks, and react to other situations that arise during operation of the vehicle.

Autonomous vehicles must react to changing conditions in sufficient time to adjust the vehicle's driving path or apply the brakes. The response times of many traditional algorithms, such as those used in existing brake systems, are not comparable to human response times. As a result, these algorithms are often less suitable for use in fully autonomous vehicles and are more suitable for use as backups for human drivers.

Also, the characteristics of parked vehicles are often good indicators of road characteristics. For example, the direction of a parked vehicle may indicate whether it is a one-way street or not, and the spacing between vehicles may indicate whether a pedestrian may emerge from between vehicles. However, existing self-driving car algorithms do not take advantage of this feature.

Autonomous vehicle systems can leverage measurements that are inaccessible to human drivers. For example, autonomous vehicle systems can use infrared cameras to access and make predictions about their surroundings. However, many existing systems do not utilize measurement combinations such as visual and infrared cameras. Embodiments of the present invention may address one or more of the disadvantages of existing systems described above.

An object of the present invention is to realize a fully autonomous vehicle capable of driving on the road.

It also provides a driving response based on analysis of images captured by one, two, or more cameras that monitor the surrounding environment of the vehicle, and provides systems and methods for autonomous driving.

Embodiments according to the present invention provide a system and method for autonomous driving. The described embodiments may use cameras to provide autonomous driving features. For example, according to a described embodiment, a described system may include one, two, or more cameras that monitor the vehicle's surroundings. The described system may provide navigational responses based, for example, on analysis of images captured by one or more cameras. Some embodiments may further include one, two, or more infrared cameras to monitor the environment. Thus, some embodiments may provide navigational responses based on analysis of, for example, visual images, infrared images, or combinations thereof.

The navigation response may also take into account other data including, for example, gPS data, sensor data (eg, data from accelerometers, speed sensors, suspension sensors, etc.), and/or other map data.

In one embodiment, a navigation system for driving a host vehicle based on detection of a door opening event in the surrounding environment of the host vehicle may include at least one processing unit. The at least one processing processing unit may be programmed to receive from the image capture device and analyze at least one image associated with the environment of the host vehicle to identify the side of the parked vehicle. The at least one processing unit identifies in the at least one image a first structural feature of a parked vehicle in a front region of the side of the parked vehicle and a second structural feature of the parked vehicle in a rear region of the side of the parked vehicle; and identify in the at least one image a door edge of a parked vehicle in the vicinity of the second structural feature. Further, the at least one processing processing unit determines a change in the image feature of the door edge of the parked vehicle based on analysis of one or more subsequent images received from the image capture device, and the change in the image feature of the door edge of the parked vehicle It may be programmed to change the driving path of the host vehicle based at least in part on the

In another embodiment, a navigation method for controlling driving of a host vehicle based on detection of a door opening event in the surrounding environment of the host vehicle includes receiving from an image capture device at least one image associated with the surrounding environment of the host vehicle. Analyzing the at least one image to identify a side of the parked vehicle. The method comprises identifying in at least one image a first structural feature of a parked vehicle in an area forward of the side of the parked vehicle and a second structural feature of the parked vehicle in an area aft of the side of the parked vehicle; The method may further include identifying in the at least one image a door edge of a parked vehicle in the vicinity of the feature. The navigating method may also include determining a change in an image feature of the door edge of the parked vehicle based on analysis of one or more subsequent images received from the image capture device and at least partially determining a change in the image feature of the door edge of the parked vehicle. The method may further include changing the driving route of the host vehicle based on the .

In another embodiment, a navigation system for driving a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is driving may include at least one processing unit. The at least one processing unit receives a plurality of images associated with the surrounding environment of the host vehicle from an image capture device and analyzes at least one of the plurality of images to obtain a target vehicle and at least one wheel component on the side of the target vehicle. can be programmed to identify The at least one processing unit analyzes a region including at least one wheel component of the target vehicle in at least two images of the plurality of images to determine at least one wheel component of the target vehicle. and to cause at least one navigational change in the host vehicle based on the identified motion associated with at least one wheel component of the target vehicle.

In yet another embodiment, a navigation method for controlling driving of a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is driving includes receiving a plurality of images associated with surrounding environments of the host vehicle from an image capture device. and analyzing at least one of the plurality of images to identify a target vehicle and at least one wheel component on the side of the target vehicle. The navigating method analyzes an area including at least one wheel component of the target vehicle in at least two images of the plurality of images, and performs an operation related to the at least one wheel component of the target vehicle. and inducing at least one navigational change in the host vehicle based on the identified motion associated with at least one wheel component of the target vehicle. .

In another embodiment, a system for detecting whether the road the host vehicle is driving on is a one-way road may include at least one processing unit. The at least one processing unit receives at least one image associated with the surrounding environment of the host vehicle from an image capture device, and captures a plurality of first vehicles on a first side of the road on which the host vehicle is driving, as the at least one image. and identify a plurality of second vehicles on the second side of the road on which the host vehicle travels based on the analysis of the at least one image. The at least one processing unit determines a first forward direction associated with the plurality of first vehicles, determines a second forward direction associated with the plurality of second vehicles, and determines the first forward direction and a second forward direction. All of which may be further programmed to cause at least one navigational change of the host vehicle when the direction of travel of the host vehicle is reversed.

In another embodiment, a method for detecting whether a road on which the host vehicle is driving is a one-way road includes receiving at least one image associated with the environment of the host vehicle from an image capture device, identifying a plurality of first vehicles on a first side based on the analysis of the at least one image; and identifying a plurality of second vehicles on a second side of the road on which the host vehicle is traveling based on the analysis of the at least one image. It may include the step of identifying based on the analysis of. The method includes determining a first forward direction associated with the plurality of first vehicles, determining a second forward direction associated with the plurality of second vehicles, and wherein the first forward direction and the second forward direction are both The method may further include inducing at least one navigational change of the host vehicle when the direction of travel of the host vehicle is opposite.

In another embodiment, a system for driving a host vehicle may include at least one processing unit. The at least one processing unit receives a driving command for driving the host vehicle from a first road on which the host vehicle is running to a second road, and receives at least one image associated with the environment of the second road from an image capture device. can be programmed to do The at least one processing processing unit identifies a first plurality of vehicles on the first side of the second road based on the analysis of the at least one image and identifies a first plurality of vehicles on the second side of the second road. 2 can be further programmed to identify the vehicle based on the analysis of said at least one image. The at least one processing unit may further determine a first forward direction associated with the plurality of first vehicles, determine a second forward direction associated with the plurality of second vehicles, and determine the first forward direction and a second forward direction. It may be further programmed to determine that all of the directions are opposite to the traveling direction in which the host vehicle will travel when the host vehicle enters the second road. The at least one processing unit travels in response to a determination that both the first forward direction and the second forward direction are opposite to the traveling direction in which the host vehicle will travel when the host vehicle travels on the second road. It can be further programmed to abort commands.

In another embodiment, a system for determining a predicted state of a parked vehicle within the environment of the host vehicle may include an image capture device, an infrared image capture device, and at least one processing processing device. The at least one processing processing unit receives a plurality of images associated with the environment of the host vehicle from the image capture device, analyzes at least one of the plurality of images to identify the parked vehicle, and determines at least one of the plurality of images. It may be programmed to analyze the two images to identify a change in the lighting state of at least one light associated with the parked vehicle. The at least one processing processing unit receives at least one thermal image of the parked vehicle from the infrared image capture device, and sets the predicted state of the parked vehicle to a change in the lighting condition and the at least one thermal image. and based on the analysis of the thermal image, and based on the predicted state of the parked vehicle, may be further programmed to trigger at least one driving response in the host vehicle.

In yet another embodiment, a method for determining a predicted state of a parked vehicle within the host vehicle's surroundings comprises receiving a plurality of images associated with the host vehicle's surroundings from an image capture device, analyzing at least one of the plurality of images. and identifying a change in lighting state of at least one light associated with the parked vehicle by analyzing at least two images of the plurality of images. The method comprises the steps of receiving at least one thermal image of the parked vehicle from an infrared image capture device, the predicted state of the parked vehicle as a combination of the change in the lighting condition and the at least one thermal image. The method may further include determining based on the analysis, inducing at least one driving response in the host vehicle based on the predicted state of the parked vehicle.

In yet another embodiment, a system for determining a predicted state of a parked vehicle within a host vehicle's surroundings may include an image capture device and at least one processing processing device. The at least one processing processing unit may be programmed to receive from the image capture device a plurality of images associated with the environment of the host vehicle. The at least one processing unit analyzes at least one of the plurality of images to identify the parked vehicle and analyzes at least two images of the plurality of images to determine a change in lighting state of at least one light associated with the parked vehicle. can be further programmed to identify. The at least one processing unit is further configured to determine a predicted state of the parked vehicle based on a change in lighting conditions, and to trigger at least one driving response in the host vehicle based on the predicted state of the parked vehicle. can be programmed

In another embodiment, a system for driving a host vehicle may include at least one processing unit. The at least one processing processing unit may be programmed to receive a plurality of images of the host vehicle's surroundings from the camera and analyze at least one of the plurality of images to identify at least two stationary vehicles. The at least one processing processing unit determines a distance between the two stationary vehicles based on at least one analysis of the plurality of images and based on the magnitude of the determined distance between the two stationary vehicles. may be further programmed to cause at least one navigational change in the host vehicle by placing a.

In another embodiment, a driving control method of a host vehicle includes the steps of receiving a plurality of images showing a surrounding environment of the host vehicle from a camera and analyzing at least one of the plurality of images to identify at least two vehicles in a stationary state. steps may be included. The method further comprises determining a distance between the two stationary vehicles based on at least one analysis of the plurality of images and determining the distance between the two stationary vehicles based on the magnitude of the determined distance. It may further include causing at least one navigational change to the host vehicle.

According to another described embodiment, a non-transitory computer readable medium may store program instructions that are executed by at least one processing unit and perform any one or more of the above methods.

The above general description and the following detailed description are for purposes of illustration and description only and do not limit the scope of the present invention.

One embodiment provides a system and method for driving control of a host vehicle based on detection of a door opening event.

Another embodiment provides a system and method for driving a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is driving.

A third embodiment provides a system and method for detecting whether a road on which a host vehicle is driving is a one-way road.

A fourth embodiment provides a system and method for determining a predicted state of a parked vehicle within the host vehicle's surroundings.

A fifth embodiment provides a system and method for driving a host vehicle based on a distance between two stationary vehicles.

In some embodiments, a travel change may be executed if it is determined that the gap between the two vehicles is sufficient for the target vehicle to cross. The travel change may be executed when it is determined that the gap between the two stopped vehicles is sufficient for a pedestrian to cross.

In some embodiments, processing unit 110 may then detect a pedestrian within the gap between the two stopped vehicles based on analysis of the plurality of images. This detection method can improve existing motion detection.

1 is a schematic diagram of an example system in accordance with a described embodiment.
2A is a side view of an exemplary automobile incorporating a system according to a described embodiment.
FIG. 2B is a plan view of the vehicle and system of FIG. 1A according to a disclosed embodiment.
2C is a top plan view of another embodiment of a vehicle incorporating a system according to a disclosed embodiment.
2D is a plan view of another embodiment of a vehicle incorporating a system according to a disclosed embodiment.
2E is a top plan view of another embodiment of a vehicle incorporating a system in accordance with a disclosed embodiment.
2F is a schematic diagram of an exemplary vehicle control system in accordance with the disclosed embodiments.
3A is a schematic illustration of an interior of a vehicle including a rear view mirror and a user interface for a vehicle imaging system according to a disclosed embodiment.
3B is an example of a camera mount configured to be positioned on a vehicle windshield behind a rearview mirror in accordance with the disclosed embodiments.
FIG. 3C illustrates the camera mount of FIG. 3B from another perspective, in accordance with a disclosed embodiment.
3D is an example of a camera mount configured to be positioned on a vehicle windshield behind a rearview mirror in accordance with the disclosed embodiments.
4 is an exemplary schematic diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments.
5A is a flow chart illustrating an example of a process for inducing one or more navigational responses based on monocular image analysis in accordance with the disclosed embodiments.
5B is a flow chart illustrating an exemplary process for detecting one or more vehicles and/or pedestrians in a series of images in accordance with a disclosed embodiment.
5C is a flowchart illustrating an exemplary process of detecting road sign and/or lane shape information from a series of images according to the disclosed embodiment.
5D is a flow chart illustrating an exemplary process for detecting traffic lights in a series of images according to a disclosed embodiment.
5E is a flowchart illustrating an exemplary process for inducing one or more navigational responses based on a vehicle path according to a disclosed embodiment.
5F is a flowchart illustrating an exemplary process for determining whether a leading vehicle is changing lanes according to the disclosed embodiments.
6 is a flow chart illustrating an exemplary process for inducing one or more navigational responses based on stereoscopic image analysis in accordance with a disclosed embodiment.
7 is a flowchart illustrating an exemplary process for inducing one or more navigational responses based on three sets of images, according to a disclosed embodiment.
8 is another exemplary configuration diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments.
9 is a schematic diagram of a road from the perspective of the system according to the disclosed embodiment.
10 is another schematic view of a road from the perspective of a system according to a disclosed embodiment.
11 is a schematic diagram of a parked vehicle from the point of view of the system according to the disclosed embodiment.
12A is a schematic diagram of a door opening event viewed from the perspective of the system according to the disclosed embodiment.
12B is another schematic diagram of a door opening event from the perspective of the system according to the disclosed embodiment.
13 is another schematic view of a road from the point of view of the system according to the disclosed embodiment.
14 is a flow chart illustrating an exemplary process for inducing one or more navigational responses based on detection of a door opening event in accordance with a disclosed embodiment.
15 is another exemplary configuration diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments.
16A is a schematic diagram of a parked vehicle viewed from the point of view of the system according to a disclosed embodiment.
16B is another schematic diagram of a parked vehicle from the point of view of the system according to the disclosed embodiment.
17 is a flow chart illustrating an exemplary process for invoking one or more navigational responses based on detection of a target vehicle entering the host vehicle's lane in accordance with the disclosed embodiments.
18 is a flow chart illustrating an exemplary process for twisting the homography of a road according to a disclosed embodiment.
19 is another exemplary configuration diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments.
20A is a schematic diagram of a one-way road viewed from the perspective of the system according to the disclosed embodiments.
10B is another schematic diagram of a one-way road viewed from the point of view of the system according to the disclosed embodiment.
21 is a flow chart illustrating an exemplary process for invoking one or more navigational responses based on detecting whether the road the host vehicle is driving on is a one-way road according to the disclosed embodiments.
22 is another exemplary configuration diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments.
23A is a schematic diagram of a parked vehicle viewed from the point of view of the system according to a disclosed embodiment.
23B is a schematic diagram of a parked vehicle with varying lighting from the perspective of the system according to the disclosed embodiments.
23C is a schematic diagram of a heat map of a parked vehicle from the perspective of the system according to the disclosed embodiment.
24 is a flow chart illustrating an exemplary process for determining a predicted state of a parked vehicle in accordance with a disclosed embodiment.
25 is a flow chart illustrating an exemplary process for aligning visual and infrared images from a system in accordance with a disclosed embodiment.
26 is another exemplary configuration diagram of a memory configured to store instructions for performing one or more operations in accordance with the disclosed embodiments.
27A is another schematic diagram of a roadway from the perspective of a system according to a disclosed embodiment.
27B is another schematic diagram of a road with detection hotspots according to the described embodiment.
28 is a flowchart illustrating an exemplary process for driving control of a host vehicle according to the disclosed embodiment.

The following detailed description relates to the accompanying drawings. Wherever possible, like reference numerals are used for like or like elements in the drawings and description. While several exemplary embodiments have been described, various modifications, adaptations, implementations, and the like are possible. For example, components illustrated in the drawings may be substituted, added, or modified, and methods included in the description may be modified by substituting, changing the order of, or adding steps. Accordingly, the following detailed description is not limited to the described embodiments and examples, and the scope of claims of the present invention is defined by the appended claims.

[Overview of autonomous vehicles]

The term "autonomous vehicle" used in the description of the present invention refers to a vehicle capable of implementing at least one navigational change without a driver's input. "Navigational change" means a controlled change in one or more of the steering, braking, acceleration/deceleration of a vehicle. For a vehicle to be autonomous, it must be fully autonomous (i.e. fully operational without driver or driver input). On the other hand, autonomous vehicles include vehicles that can operate under the control of a driver during certain times and without the control of the driver during other times. An autonomous vehicle controls only some driving elements, such as steering (e.g., keeping the vehicle between lanes in its path) or some steering actions under certain circumstances (but not under all conditions), while remaining elements (e.g., braking or braking under certain conditions) are controlled by the driver. It may also include a vehicle entrusted to you. In some cases, an autonomous vehicle may handle some or all elements of steering, braking and/or speed control of the vehicle.

Drivers often rely on visual cues and observation to control their vehicles, so transportation infrastructure has been built to include lane markings, traffic signs and traffic lights designed to provide visual information to drivers. Considering these design features of the transportation infrastructure, autonomous vehicles may include cameras and processing units that analyze visual information obtained from the vehicle's surroundings. Visual information can include, for example, images representing components of transportation infrastructure (eg, lane markings, traffic signs, traffic lights, etc.) and other obstacles (eg, other vehicles, pedestrians, debris, etc.) that can be seen by drivers. can include Autonomous vehicles may also include infrared cameras. In such an embodiment, the processing unit may analyze the thermal information captured from the environment individually or together with the visual information.

In addition, the self-driving vehicle can also use stored information such as information that provides a model around the vehicle while driving. For example, a vehicle may utilize GPS data and/or sensor data (e.g., data from accelerometers, speed sensors, suspension sensors, etc.), and other map data to provide information related to the vehicle's environment as it moves, and the vehicle ( Other vehicles (along with other vehicles) can use this information to determine the location of the vehicle on the model. Some vehicles may also communicate with each other, exchange information and divert other vehicles to hazards or changes around the vehicle.

[System Overview]

1 is a block diagram of a system 100 according to an illustratively described embodiment. System 100 may include various components depending on the requirements of the implementation. In some embodiments, system 100 includes processing unit 110, image acquisition unit 120, position sensor 130, one or more memory units 140 and 150, map database 160, user interface 170, A radio transceiver 172 may be included. Processing unit 110 may include one or more processing devices. In some embodiments, processing unit 110 may include applications processor 180, image processor 190, or other suitable processing device. Similarly, the image acquisition unit 120 may include several image acquisition devices and elements according to requirements of specific applications. In some embodiments, image acquisition unit 120 may include one or more image capture devices (eg, cameras, CCDs, or other types of image sensors) such as image capture devices 122 , 124 , and 126 . In some embodiments, the image acquisition unit 120 may further include one or more infrared capture devices (eg, an infrared camera, a far infrared detector, or other types of infrared sensors). For example, one or more of the image capture devices 122, 124, and 126 may include an infrared image capture device.

System 100 may also include a data interface 128 that couples processing unit 110 and image acquisition unit 120 in communication with each other. For example, the data interface 128 may include one or more wired and/or wireless links for transmitting image data obtained by the image acquisition unit 120 to the processing unit 110 .

The wireless transceiver 172 may include one or more devices configured to transmit and receive radio waves to and from one or more networks (eg, mobile communication, Internet, etc.) through a wireless interface using radio frequency, infrared frequency, magnetic field, or electric field. The radio transceiver 172 may transmit and/or receive data using known standards (eg, Wi-Fi, Bluetooth®, Bluetooth Smart, 802.15.4, ZigBee, etc.). These radio waves may include communications from the host vehicle to one or more remotely located servers. These radio waves may also be used for communication (either one way or the other) between the host vehicle and one or more target vehicles in the host vehicle's environment (e.g., driving of the host vehicle to or with the target vehicle in the host vehicle's environment). communication for adjustment) or even broadcast radio waves sent to unspecified recipients around the vehicle that transmits the radio waves.

Both the application processor 180 and the image processor 190 may include various types of hardware-based processing devices. For example, one or both of application processor 180 and image processor 190 may include a microprocessor, a pre-processing unit (e.g., an image pre-processing unit), a graphics processor, a central processing unit (CPU), auxiliary circuitry, a digital signal processor, It may include an integrated circuit, memory, graphics processing unit (GPU), or other type of device suitable for running applications and processing and analyzing images. In some embodiments, applications processor 180 and/or image processor 190 may include one type of single or multi-core processor, mobile device microcontroller, or central processing unit. Various processing units may be used, which may include, for example, processors provided by vendors such as Intel®, AMD®, etc. or GPUs provided by vendors such as NVIDIA®, ATI®, etc., and various architectures (eg , x86 processors, ARM®, etc.).

In some embodiments, application processor 180 and/or image processor 190 may include an EyeQ series of processor chips provided by Mobileye®. These processor designs include multiple processing units with local memory and instruction sets. Such a processor may include a video input for receiving image data from a plurality of image sensors and may also include a video output function. For example, the EyeQ2® uses 90nm-micron technology running at 332Mhz. The EyeQ2® architecture consists of two floating point hyper-threaded 32-bit RISC CPUs (MIPS32® 34K® cores), five Vision Computing Engines (VCE), three Vector Microcode Processors (VMP® ), It consists of a Denali 64-bit Mobile DDR Controller, 128-bit internal Sonics Interconnect, dual 16-bit video input and 18-bit video output control units, 16-channel DMA and several peripherals. A MIPS34K CPU manages five VCEs, three VMP??, DMA, a second MIPS34K CPU, multi-channel DMA, and other peripherals. Five VCEs, three VMP® and MIPS34K CPUs can perform the intensive video computations required by multi-functional bundled applications. In another example, EyeQ3®, a 3rd generation processor that is 6 times more powerful than EyeQ2®, may be used in an embodiment. In another example, EyeQ4® and/or EyeQ5® may be used in the described embodiments. Of course, new or future EyeQ processing devices may also be used with the described embodiments.

Each of the processing processing devices described herein may be configured to perform a specific function. Configuring a processing device, such as an EyeQ processor or other control unit or microprocessor, to perform a particular function may include programming computer-executed instructions and providing these instructions to the processing device for execution while the processing device is operating. can In some embodiments, configuring the processing device may include directly programming the processing device with structured instructions. For example, processing devices such as field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), and the like may be constructed using one or more hardware description languages (HDLs).

In other embodiments, configuring the processing device may include storing executable instructions in a memory accessible to the processing device during operation. For example, a processing processing unit may access memory during operation to obtain and execute stored instructions. In either case, the processing devices configured to perform the sensing, image analysis, and/or driving control functions described herein represent hardware-based systems specialized for controlling multiple hardware-based components of a host vehicle.

Although FIG. 1 illustrates that two processing units are included in the processing unit 110, more or fewer processing units may be used. For example, in some embodiments, a single processing unit may be used to perform the tasks of applications processor 180 and image processor 190 . In other embodiments, these tasks may be performed using two or more processing processing units. Also, in some embodiments, the system 100 may include one or more processing units 110 but may not include other components such as the image acquisition unit 120 or the like.

Processing unit 110 may include various types of devices. For example, the processing unit 110 may be a control unit, an image pre-processing unit, a central processing unit (CPU), a graphics processing unit (GPU), an auxiliary circuit, a digital signal processing unit, an integrated circuit, a memory, or image processing and analysis. It may include various devices, such as other types of devices for The image pre-processing device may include an image processor for photographing, digitizing, and processing an image from an image sensor. A CPU may include one or more microcontrollers or microprocessors. A GPU may also contain one or more microcontrollers or microprocessors. Auxiliary circuits may include one or more circuits generally known in the art, such as cache, power supply, clock, input/output circuits, and the like. The memory may store software that, when executed by the processing unit, controls the operation of the system. Memory may include one or more random access memory (RAM), read only memory (ROM), flash memory, disk drives, optical storage, tape storage, removable storage, and other types of storage. . In one example, the memory may be separate from processing unit 110 . In another example, the memory may be integrated with the processing unit 110 .

Each of memories 140 and 150 may contain software instructions that, when executed by a processing device (eg, applications processor 180 and/or image processor 190), may control the operation of various aspects of system 100. can Such a memory device may contain learning systems such as, for example, neural networks or deep neural networks, as well as various database and image processing software. Such memory devices may include RAM, ROM, flash memory, disk drives, optical storage devices, tape storage devices, removable storage devices, and/or other types of storage devices. In some embodiments, memory devices 140 and 150 may be separate from application processor 180 and/or image processor 190 . In another embodiment, such a memory device may be integrated with the application processor 180 and/or image processor 190 .

Position sensor 130 may include any type of device suitable for determining a position associated with at least one component of system 100 . In some embodiments, the position sensor 130 may include a GPS receiver. These receivers can determine the user's location and speed by processing signals transmitted from GPS satellites. Location information from the location sensor 130 may be provided to the application processor 180 and/or the image processor 190 .

In some embodiments, system 100 may include a component such as a speed sensor (eg, speedometer) for measuring the speed of vehicle 200 . System 100 may also include one or more accelerometers (uniaxial or multi-axis) for measuring acceleration of vehicle 200 along one or more axes.

The memory devices 140 and 150 may contain data organized into a database or other type of data containing one or more indicators and/or locations of known landmarks. In order to determine the vehicle's current position relative to known landmarks and to make the vehicle's position more accurate, sensory information (e.g., images, radar signals, stereo processing of two or more images, or lidar depth information) of the surrounding environment is used to determine GPS coordinates, It can be processed together with location information such as the movement of the vehicle itself. Certain aspects of this technology are included in a positioning technology known as REM ^™ which is being commercialized by the assignee of the present application.

User interface 170 may include any device suitable for providing information to and receiving input from one or more users of system 100 . In some embodiments, user interface 170 may include user input devices such as, for example, a touchscreen, microphone, keyboard, pointer device, trackwheel, camera, knobs, and buttons. Such input devices may allow a user to type commands or information, provide voice commands, select on-screen menus using buttons, pointers, or eye-tracking features, or other suitable methods for exchanging information with system 100. Technology allows information input or commands to be provided to the system 100 .

User interface 170 may include one or more processing devices configured to provide information to a user, receive information from a user, and process such information for use by, for example, application processor 180 . In some embodiments, these processing devices may recognize and track eye movements, receive and interpret voice commands, recognize and interpret touches and/or gestures on a touchscreen, respond to keyboard input or menu selections, and the like. can carry out instructions. In some embodiments, user interface 170 may include a display, speaker, tactile device, and/or other device that provides output information to the user.

Map database 160 may include any type of database for storing map data usable in system 100 . In some embodiments, map database 160 may include data relating to the locations on reference coordinates of various items such as roads, river features, geographic features, businesses, points of interest, restaurants, gas stations, and the like. The map database 160 may store not only the locations of these items, but also descriptions including, for example, names associated with the stored points. In some embodiments, map database 160 may be physically co-located with other components of system 100. Alternatively, or additionally, map database 160 or portions of map database 160 may be located remotely relative to other components of system 100 (eg, processing unit 110). In this case, information from the map database 160 may be downloaded through a wired or wireless data connection with a network (eg, a mobile communication network and/or the Internet). In some cases, map database 160 may store simplified data models that include polynomial representations of specific road features (eg, lane markings) or target trajectories of the host vehicle. Map database 160 may also include stored representations of various recognized landmarks that may be utilized to determine or update the host vehicle's known position relative to a target trajectory. The landmark expression may include data fields such as landmark type and landmark location among potential identifiers.

Image capture devices 122, 124, and 126 may each include any type of device suitable for capturing at least one image from the surrounding environment. In addition, multiple image capture devices may be used to capture images required for input to the image processor. Some embodiments may include a single image capture device, while other embodiments may include two, three, or four or more image capture devices.

Further, as described above, image capture devices 122, 124, and 126 may each include a type of device suitable for capturing at least one infrared image from the surrounding environment. Any number of infrared image capture devices may be used. Some embodiments may include a single infrared image capture device, while other embodiments may include two, three, or four or more infrared image capture devices. In addition, some embodiments may include different numbers of infrared image capture devices and combinations of different numbers of image capture devices. Image capture devices 122, 124 and 126 are further described below with reference to FIGS. 1B-1E.

One or more cameras (eg, image capture devices 122, 124, 126) may be part of a sensing block included in the vehicle. The sensing block may further include one or more infrared cameras separately or in combination with the one or more cameras.

A variety of other sensors may be included in the sensing block, and may rely on some or all of the sensors to build an understanding of the vehicle's driving state. In addition to cameras (front, side, rear, etc.), other sensors such as radar, lidar, and acoustic sensors may be included in the sensing block. Additionally, the sensing block may include one or more components configured to communicate and send and receive information about the vehicle's surroundings. For example, these components may include a radio transceiver (RF, etc.) capable of receiving sensor-based or other types of information about the host vehicle from a source located remotely about the host vehicle. This information may include sensor output information or related information received from a vehicle other than the host vehicle. In some embodiments, this information may include information received from remote computing devices, central servers, and the like. Furthermore, the camera may have various configurations, such as a single camera device, a plurality of cameras, a camera cluster, a telescopic field of view, a short field of view, a wide angle view, a fisheye view, and the like.

System 100, or various components of system 100, may be implemented on various platforms. In some embodiments, system 100 may be included in vehicle 200, as illustrated in FIG. 1A. For example, vehicle 200 may be equipped with processing unit 110 and other components of system 100 described in FIG. 1 . Some embodiments vehicle 200 may be equipped with only a single image capture device (eg, camera) and/or a single infrared image capture device, while in other embodiments vehicles 200 may have multiple, as illustrated in FIGS. 1B-2E . An image capture device and/or multiple infrared image capture devices may be used. For example, image capture devices 122 and 124 of vehicle 200 illustrated in FIG. 1A may be part of an Advanced Driver Assistance Systems (ADAS) image set.

An image capture device included in vehicle 200 as part of image acquisition unit 120 may be located in a suitable location. In some embodiments, as shown in FIGS. 1A-2E and 3A-3C , image capture device 122 may be placed around the rearview mirror. This location may provide a line of sight similar to that of the driver of the vehicle 200 and thus may help determine what is visible and invisible to the driver. The image capture device 122 can be placed anywhere around the rearview mirror, but placing the image capture device 122 on the driver's side of the rearview mirror will be more helpful in obtaining an image representing the driver's field of view and/or line of sight. can

The image capture device of the image acquisition unit 120 may be disposed in another location. For example, image capture device 124 may be placed in or on the bumper of vehicle 200 . Such a location may be particularly suitable for image capture devices having a wide field of view. Since the gaze of the image capture device located in the bumper may be different from that of the driver, the image capture device located in the bumper and the driver may not always see the same object. Image capture devices (eg, 122, 124, 126) may also be located in other locations. For example, the image capture device may be placed on or inside a side mirror of one or both sides of the vehicle 200, on the roof of the vehicle 200, on the bonnet of the vehicle 200, on the side of the vehicle 200, It may be placed on the windows, rear or front of the vehicle 200, on or around the front and/or rear lights of the vehicle 200, and the like.

Vehicle 200 may include various other components of system 100 in addition to the image capture device. For example, the processing unit 110 may be included in the vehicle 200 either integrally with or separate from an engine control unit (ECU) of the vehicle 200 . The vehicle 200 may also include a location sensor 130 such as a GPS receiver, a map database 160, and memory units 140 and 150.

As described above, the wireless transceiver 172 may transmit and/or receive data through one or more networks (eg, a mobile communication network, the Internet, etc.). For example, wireless transceiver 172 can upload data collected by system 100 to one or more servers and download data from one or more servers. System 100 may receive periodic or temporary updates of data stored, for example, in map database 160 and/or memories 140 and 150 via radio transceiver 172 .

Similarly, the wireless transceiver 172 transmits data of the system 100 (eg, images captured by the image acquisition unit 120, data received by the position sensor 130, other sensors, or vehicle control systems, etc.) and/or Data processed by the processing unit 110 may be uploaded to one or more servers.

System 100 may upload data to a server (eg, cloud) based on privacy settings. For example, system 100 may implement privacy settings to restrict or limit the types of data (including metadata) that individually identify vehicles and their drivers/owners that are sent to the server. These settings may be set by a user through the wireless transceiver 172, initialized to factory settings, or set by data received by the wireless transceiver 172, for example.

In some embodiments, system 100 may upload data in accordance with a "high" privacy level, which allows system 100 to upload data (e.g., routes) without specific vehicle and/or driver/owner information. related location information, captured images, etc.) can be transmitted. For example, when uploading data under a "high" privacy level, system 100 may not include the vehicle identification number or the name of the driver or owner of the vehicle, but instead capture images and/or route and Data such as related limited location information can be transmitted.

Other privacy levels are possible. For example, system 100 may transmit data to a server under a "medium" privacy level, in which case the make and/or model and/or make and/or model of a vehicle not included under a "high" privacy level. Or additional information such as vehicle type (eg, car, SUV, truck, etc.) may be included. In some embodiments, system 100 may upload data based on a “low” privacy level. Under the "low" privacy level, the system 100 may upload data including information that may specify the individual vehicle, its owner/driver, and/or all or part of the route the vehicle has traveled. Such “low” privacy level data may include, for example, vehicle identification number, driver/owner name, vehicle origin, vehicle destination, vehicle make and/or model, vehicle type, and the like.

2A is a side view of an exemplary vehicle imaging system in accordance with a described embodiment. Figure 2b is a plan view of the embodiment shown in Figure 2a. As shown in FIG. 2B , the vehicle 200 according to the present embodiment includes a first image capture device 122 disposed around the rearview mirror and/or near the driver, a bumper area (eg, a portion of the bumper area 210) The system 100 having the second image capture device 124 disposed on or inside the body and the processing unit 110 is included inside the main body. 2A and 2B, one or more of the first image capture device 122 and the second image capture device 124 may include an infrared image capture device.

As shown in FIG. 2C , image capture devices 122 and 124 may both be positioned near the rearview mirror of vehicle 200 and/or near the driver. In FIG. 2C, similar to FIGS. 2A and 2B, one or more of the first image capture device 122 and the second image capture device 124 may include an infrared image capture device.

In addition, although two image capture devices 122 and 124 are illustrated in FIGS. 2B and 2C , it will be natural that three or more image capture devices may be included in other embodiments. For example, in the embodiment shown in FIGS. 2D and 2E , first, second, and third image capture devices 122 , 124 , and 126 are included in system 100 of vehicle 200 . Similar to FIGS. 2A, 2B, and 2C, one or more of the first, second, and third image capture devices 122, 124, and 126 of FIGS. 2D and 2E may include an infrared image capture device.

As shown in FIG. 2D , image capture device 122 may be positioned near the rearview mirror of vehicle 200 and/or near the driver, and image capture devices 124 and 126 may be located in the bumper area of vehicle 200 (e.g., , one area of the bumper area 210) or inside it. Additionally, as shown in FIG. 2E , image capture devices 122 , 124 , and 126 may be positioned near the rearview mirror of vehicle 200 and/or near the driver. The present embodiment is not limited to a specific number and configuration of image capture devices, and image capture devices may be placed at any appropriate location in and/or on a vehicle.

The described embodiments are not limited to vehicles and can of course be applied to other situations as well. In addition, the described embodiments are not limited to a specific type of vehicle 200 and can be applied to all types of vehicles, such as cars, trucks, trailers, and other types of vehicles.

The first image capture device 122 may include an image capture device of any suitable type or an infrared image capture device. The image capture device 122 may include an optical axis. In one example, image capture device 122 may include a global shutter Aptina M9V024 WVGA sensor. In another embodiment, image capture device 122 may provide a resolution of 1280x960 pixels and include a rolling shutter. The image capture device 122 may include a variety of optical elements. In some embodiments, one or more lenses may be included to provide a focal length and field of view required by the image capture device. In some embodiments, image capture device 122 may be combined with a 6 mm lens or a 12 mm lens. In some embodiments, image capture device 122 may have a required field of view 202, as shown in FIG. 2D. It can be configured to secure. For example, image capture device 122 may be configured to have a normal field of view ranging from 40 to 56 degrees, such as a 46 degree field of view, a 50 degree field of view, a 52 degree field of view, or more. Alternatively, the image capture device 122 may be configured to have a narrow field of view ranging from 23 to 40 degrees, such as a 28 degree field of view or a 36 degree field of view.

In addition, the image capture device 122 may be configured to secure a wide field of view ranging from 100 to 180 degrees. In some embodiments, the image capture device 122 may include a wide-angle bumper camera or a camera capable of securing a 180-degree field of view. In some embodiments, image capture device 122 may be a 7.2 megapixel image capture device with an aspect ratio of about 2:1 (eg, HxV = 3800x1900 pixels) and a horizontal field of view of about 100 degrees. This image capture device can replace the three image capture device configuration. The vertical field of view of these image capture devices using radially symmetric lenses may be less than 50 degrees due to lens distortion. For example, a radially asymmetric lens can be used to achieve a vertical field of view of 50 degrees or more when a horizontal field of view is 100 degrees.

The first image capture device 122 may acquire a plurality of first images of a scene related to the vehicle 200 . Each of the plurality of first images may be obtained as a continuous scan line captured using a rolling shutter. Each scan line may include a plurality of pixels. In embodiments where the first image capture device 122 includes an infrared image capture device, each of the plurality of first images may be acquired as a continuous scan line that may be captured utilizing an electronic scanning system.

The first image capture device 122 may have a scan rate related to acquisition of the first continuous scan line. The scan rate may refer to a speed at which an image sensor acquires image data related to each pixel included in a specific scan line. In an embodiment in which the first image capture device 122 includes an infrared image capture device, the scan rate may mean a speed at which an infrared image sensor acquires thermal data related to each pixel included in a specific scan line.

Image capture devices 122, 124, and 126 may include any suitable type and number of image sensors, such as, for example, CCD sensors or CMOS sensors. In one embodiment, a rolling shutter CMOS image sensor may be introduced to read each pixel in a row one by one, and scanning may be performed row by row until the entire image frame is captured. In some embodiments, each column may be captured sequentially from top to bottom for a frame. In embodiments where one or more of the image capture devices 122, 124, 126 includes an infrared image capture device, an uncooled focal plane array (UFPA) may be utilized in conjunction with an electronic scanning system to capture a full heat map. Scanning may proceed row-by-row until

In some embodiments, one or more of the image capture devices 122, 124, 126 described herein constitute a high resolution imager and may have a resolution of 5 Mega, 7 Mega, 10 Mega or more pixels.

With a rolling shutter, different rows of pixels can be exposed and photographed at different times, which can introduce skew and other image artifacts in the captured image frame. On the other hand, if image capture device 122 is configured to operate in a global or synchronous shutter mode, all pixels may be exposed for the same amount of time and for a typical exposure time. As a result, image data of a frame collected in a system using the global shutter method represents a snapshot of the entire field of view (eg, 202 ) at a specific time. On the other hand, in the rolling shutter method, each column of one frame is exposed at different times and data is collected. Therefore, a moving object in an image of a rolling shutter type image capture device may appear distorted. This phenomenon (similar to the case of using electronic scanning of an infrared image capture device) is described in more detail below.

The second image capture device 124 and the third image capture device 126 may each be a suitable type of image capture device or infrared image capture device. Like the first image capture device 122, the second image capture device 124 and the third image capture device 126 may each include an optical axis. In one embodiment, the second image capture device 124 and the third image capture device 126 may each include a global shutter type Aptina M9V024 WVGA sensor. Alternatively, the second image capture device 124 and the third image capture device 126 may each include a rolling shutter method. Like the first image capture device 122, the second image capture device 124 and the third image capture device 126 may be configured to include various lenses and optical elements, respectively. In some embodiments, the lenses associated with the second image capture device 124 and the third image capture device 126 have a field of view 204 that is equal to or narrower than the field of view (e.g., 122) associated with the first image capture device 122. 206) can be provided. For example, the fields of view of the second image capture device 124 and the third image capture device 126 may be 40 degrees, 30 degrees, 26 degrees, 20 degrees, or less, respectively.

The second image capture device 124 and the third image capture device 126 may acquire a plurality of second and third images of a scene related to the vehicle 200 . Each of the plurality of second and third images may be obtained with the second and third consecutive scan lines captured using the rolling shutter. Each scan line or column may include a plurality of pixels. The second image capture device 124 and the third image capture device 126 may have second and third scan rates associated with acquisition of each scan line included in the second and third continuous scan lines. In embodiments where at least one of second image capture device 124 and third image capture device 126 includes an infrared image capture device, each of the plurality of second and third images is captured using an electronic scanning system. It can be obtained with second and third consecutive heat scan lines. In such an embodiment, each scan line or row may include a plurality of pixels, and the second image capture device 124 and/or the third image capture device 126 may be used for the second and third consecutive scan lines. It may have second and third scan rates associated with the acquisition of each included column scan line.

Image capture devices 122 , 124 , and 126 may be positioned in suitable locations and orientations of vehicle 200 , respectively. The relative placement of image capture devices 122, 124 and 126 may be selected to facilitate fusion of information obtained from the image capture devices. For example, in some embodiments, the field of view (e.g., 204) of second image capture device 124 is the same as that of first image capture device 122 (e.g., 202) and third image capture device 126. It may partially or completely overlap the field of view (eg, 206 ).

Image capture devices 122 , 124 , and 126 may be positioned at suitable relative heights of vehicle 200 . In one example, a difference in height between image capture devices 122, 124, and 126 may provide parallax information enabling stereoscopic analysis. For example, as shown in FIG. 2A, image capture device 122 and image capture device 124 have different heights. There is also transverse displacement between the image capture devices 122, 124, and 126 to provide additional parallax information for stereoscopic analysis, such as by processing unit 110. As shown in Figs. 2c and 2d, the difference in transverse displacement can be _denoted as dx. In some embodiments, there may be forward or backward displacement (eg, range displacement) between image capture devices 122, 124, and 126. For example, image capture device 122 may be positioned 0.5 to 2 meters behind image capture device 124 and/or image capture device 126 . This type of displacement allows one of the image capture devices to compensate for the blind spot of the other image capture device.

Similarly, there may be no height difference between image capture devices 122, 124, and 126, which may align the heat map produced by one or more of the image capture devices with the visual image produced by one or more of the image capture devices. can help you do that.

Image capture device 122 may have any suitable resolution (e.g., number of pixels associated with an image sensor), the resolution of the image sensor associated with image capture device 122 being the resolution of the image sensor associated with image capture devices 124, 126. It can be higher, lower, or equal to . In some embodiments, the resolution of image capture device 122 and/or image sensors associated with image capture devices 124 and 126 may be 640 x 480, 1024 x 768, 1280 x 960, or other suitable resolutions.

The frame rate (i.e., the rate at which the image capture device acquires a set of pixel data or column data from one image frame before moving on to acquire pixel data or column data from the next image frame) is controllable. The frame rate of image capture device 122 may be higher, lower, or equal to the frame rate of image capture devices 124 and 126 . The frame rate of image capture devices 122, 124, and 126 may depend on a variety of factors that affect the timing of the frame rate. For example, one or more of the image capture devices 122, 124, 126 may include a selectable pixel delay period imposed before or after acquiring image data related to one or more pixels of an onboard image sensor. can In general, image data corresponding to each pixel is obtained based on the clock rate of the device (eg, 1 pixel per clock cycle). Additionally, in an embodiment that includes a rolling shutter method, one or more of the image capture devices 122, 124, 126 may acquire an optional horizontal retrace period ( selectable horizontal blanking period) may be included. Further, one or more of the image capture devices 122, 124, 126 may include a selectable vertical blanking period imposed before or after acquiring the image data associated with the image frame. Similarly, in embodiments involving electronic scanning, one or more of image capture devices 122, 124, 126 may include a dynamically variable scan rate.

Due to such timing control, frame rates of the image capture devices 122, 124, and 126 may be synchronized even when line scan rates of the image capture devices 122, 124, and 126 are different from each other. Also, as will be discussed in more detail below, among other factors (eg, image sensor resolution, maximum pre-scan speed, etc.), this timing control causes the field of view of image capture device 122 to be different from the field of view of image capture devices 124, 126. In other cases, synchronization of images captured from an area where the field of view of image capture device 122 and one or more fields of view of image capture devices 124 and 126 overlap may be possible.

Frame rate timing of image capture devices 122, 124, 126 may be based on the resolution of the corresponding image sensor. For example, assuming that the pre-scan speeds of the two devices are similar, if the resolution of the image sensor of one device is 640 X 480 and the resolution of the image sensor of the other device is 1280 X 960, an image from an image sensor with a higher resolution Acquiring a frame of data will take more time.

Another factor affecting the image data acquisition timing of the image capture devices 122, 124, and 126 is the maximum line scan rate. For example, acquiring one row of image data from image sensors included in the image capture devices 122, 124, and 126 may take more than a minimum amount of time. Assuming no pixel delay period is added, the minimum time to acquire a column of image data will be related to the maximum pre-scan speed of a particular device. A device with a higher peak preload rate is likely to deliver higher frame rates than a device with a lower peak preload rate. In some embodiments, the maximum pre-scan speed of one or more of image capture devices 124 and 126 may be higher than the maximum pre-scan speed of image capture device 122 . In some embodiments, the maximum pre-scan speed of image capture devices 124 and/or 126 may be 1.25 times, 1.5 times, 1.75 times, 2 times, or more higher than the maximum pre-scan speed of image capture device 122 .

In another embodiment, the maximum pre-scan speeds of image capture devices 122, 124, and 126 are all the same, but image capture device 122 may be operated at scan speeds less than or equal to the maximum scan speed.

The system may be configured such that one or more of image capture devices 124 and 126 operate at the same pre-scan rate as the pre-scan rate of image capture device 122 . In another example, the system may be configured such that the pre-scan speed of image capture devices 124 and/or 126 is 1.25 times, 1.5 times, 1.75 times, 2 times, or more higher than the pre-scan speed of image capture device 122. there is.

In some embodiments, image capture devices 122, 124, and 126 may be asymmetrical. That is, the image capture devices 122, 124, and 126 may include cameras having different fields of view and different focal lengths. The field of view of the image capture devices 122 , 124 , and 126 may include a necessary area related to the surroundings of the vehicle 200 . In some embodiments, one or more of image capture devices 122 , 124 , 126 may be configured to acquire image data from an environment in front of, behind, to the side of vehicle 200 , or a combination thereof.

Additionally, the focal length of each image capture device 122 , 124 , 126 may be selected (eg, with a suitable lens applied) such that each device acquires an image of an object at a specific distance range relative to vehicle 200 . For example, in some embodiments, image capture devices 122, 124, and 126 may acquire close-up images of objects within a few meters of the vehicle. Image capture devices 122, 124, and 126 may be configured to acquire images of objects at ranges away from the vehicle (eg, 25 m, 50 m, 100 m, 150 m, or more). In addition, one image capture device (eg, 122) acquires an image of an object relatively close to the vehicle (eg, within 10 m or 20 m), and the other image capture devices (eg, 124, 126) are further away (eg, 20 m or less) from the vehicle. , 50 m, 100 m, 150 m or more) The focal length of the image capture device 122, 124, 126 can be selected to acquire an image of the object.

According to some embodiments, the field of view of one or more image capture devices 122, 124, 126 may be wide. For example, it may be advantageous if the field of view of the image capture devices 122, 124, and 126 used to acquire images of the area around the vehicle 200 is 140 degrees. For example, image capture device 122 may be used to capture an image of an area to the right or left of vehicle 200, in which case image capture device 122 may capture a wide field of view (eg, 140 degrees or greater). It may be desirable to have

The fields of view of image capture devices 122, 124, and 126 may be based on their respective focal lengths. For example, as the focal length increases, the corresponding field of view decreases.

Image capture devices 122, 124, and 126 may be configured to have a suitable field of view. In one example, the horizontal field of view of image capture device 122 is 46 degrees, the horizontal field of view of image capture device 124 is 23 degrees, and the horizontal field of view of image capture device 126 may be between 23 and 46 degrees. In another example, the horizontal field of view of image capture device 122 is 52 degrees, the horizontal field of view of image capture device 124 is 26 degrees, and the horizontal field of view of image capture device 126 may be between 26 and 52 degrees. In some embodiments, the ratio of the field of view of image capture device 122 to the field of view of image capture device 124 and/or image capture device 126 may be between 1.5 and 2.0. In other embodiments, this ratio may be between 1.25 and 2.25.

System 100 may be configured such that part or all of the field of view of image capture device 126 overlaps the field of view of image capture device 124 and/or image capture device 126 . In some embodiments, system 100 may be configured such that the fields of view of image capture devices 124 and 126 fall in the center of the field of view of image capture device 126 (eg, if the field of view is narrow) and are centered together. can In other embodiments, image capture devices 122, 124, and 126 may capture adjacent fields of view or may partially overlap adjacent fields of view. In some embodiments, the field of view of image capture devices 122, 124, 126 is such that the center of image capture devices 124 and/or 126 with a narrow field of view is positioned below the field of view of image capture device 122 with a wide field of view. can be sorted

2F is a schematic diagram of an exemplary vehicle control system in accordance with the disclosed embodiments. As shown in FIG. 2F , vehicle 200 may include a driving system 220 , a braking system 230 , and a steering system 240 . The system 100 inputs (eg, controls) one or more of the driving system 220, the braking system 230, and the steering system 240 through one or more data links (eg, wired and/or wireless links for data transmission). signal) can be provided. For example, based on analysis of images acquired by image capture devices 122, 124, and 126, system 100 may provide control signals to drive vehicle 200 (eg, accelerate, turn, change lanes, etc.) may be provided as one or more of the driving system 220, the braking system 230, and the steering system 240. In addition, the system 100 transmits an input indicating an operating condition (eg, speed, braking, and/or rotation) of the vehicle 200 to one of the driving system 220, the braking system 230, and the steering system 240. can be received from above. This will be described in detail below with reference to FIGS. 4 to 7 .

As shown in FIG. 3A , vehicle 200 also includes a user interface 170 for interacting with a driver or occupant of vehicle 200 . For example, the user interface 170 applied to a vehicle may include a touch screen 320, a dial 330, a button 340, and a microphone 350. A driver or occupant of the vehicle 200 may also use a knob (eg, a turn lamp knob disposed on or around a steering shaft of the vehicle), a button (eg, a button disposed on a steering wheel of the vehicle, etc.) 100). In some embodiments, microphone 350 may be positioned adjacent rearview mirror 310 . Similarly, in some embodiments, image capture device 122 may be placed near rearview mirror 310 . In some embodiments, user interface 170 may also include one or more speakers 360 (eg, the speakers of a vehicle audio system). For example, the system 100 may provide various information (eg, alerts) through the speaker 360 .

3B-3D illustrate an exemplary camera mount 370 configured to be positioned on a vehicle windshield behind a rearview mirror (eg, 310) in accordance with the disclosed embodiments. As shown in FIG. 3B , camera mount 370 may include image capture devices 122 , 124 , and 126 . Image capture devices 124 and 126 may be disposed behind a glare shield 380 that faces the vehicle windshield and includes a structure of film and/or anti-reflective material. For example, the glare shield 380 may be arranged to be aligned with the vehicle windshield with an inclination corresponding to the vehicle windshield. In some embodiments, each image capture device 122, 124, 126 may be placed behind the glare shield 380, as shown in FIG. 3D. In embodiments where one or more of the image capture devices 122, 124, 126 includes an infrared image capture device, such image capture device may be placed in front of the glare shield 380 so that the entry of the infrared light is unobstructed (or The glare shield 380 may not extend to the front of the infrared image capture device). The embodiments described herein are not limited to any particular configuration of image capture devices 122 , 124 , 126 , camera mount 370 , or glare shield 380 . 3C is an example of the camera mount 370 shown in FIG. 3B viewed from the front.

Those skilled in the art will understand that the above embodiments are subject to various variations and/or modifications. For example, not all components are necessarily required for operation of system 100 . Further, while providing the functionality of the described embodiments, any component may be placed in a suitable portion of system 100, and components may be rearranged in various configurations. Therefore, the above configurations are only examples, and the system 100 can analyze the surrounding environment of the vehicle 200 and control the driving of the vehicle 200 in response to the analysis by providing a wide range of functions regardless of the above configurations. there is.

In accordance with the descriptions and various embodiments described below, system 100 may provide various features related to autonomous driving and/or driver assistance technology. For example, system 100 may analyze image data, infrared image data, location data (eg, GPS location information), map data, speed data, and/or data from sensors included in vehicle 200. can The system 100 may collect data to be analyzed from, for example, the image acquisition unit 120, the position sensor 130, and other sensors. In addition, the system 100 may analyze the collected data to determine whether the vehicle 200 should perform a specific action, and then determine whether to automatically perform the determined action without human intervention. For example, when vehicle 200 is driven without human intervention, system 100 automatically controls braking, acceleration, and/or steering of vehicle 200 (e.g., drive system 220, braking system (230), a control signal may be transmitted to one or more of the steering system (240). Additionally, system 100 may analyze the collected data and provide warnings and/or alerts to vehicle occupants according to the results of the analysis. Various embodiments provided by the system 100 are further described below.

[Front multi-view system]

As described above, system 100 may provide driving assistance features using a multi-camera system. A multi-camera system may use one or more cameras (and/or infrared cameras) facing the front of the vehicle. In other embodiments, a multi-camera system may include one or more cameras (and/or infrared cameras) that face to the side or rear of the vehicle. For example, in one embodiment, system 100 uses a dual camera imaging system, where a first camera and a second camera (eg, image capture devices 122, 124) are positioned in front of a vehicle (eg, 200). and/or lateral. There are other camera configurations according to the described embodiments, and the configurations described herein are exemplary only. For example, system 100 may have different numbers of cameras (eg, 1, 2, 3, 4, 5, 6, 7, 8, etc.) and combinations of different types of cameras (eg, two visual cameras and one infrared camera, one visual camera and two infrared cameras, two visual cameras and two infrared cameras, etc.). Furthermore, system 100 may include “clusters” of cameras. For example, a camera cluster (including a variable number of cameras, such as one, four, eight, etc., and cameras of the appropriate type, such as visual, infrared, etc.) may be directed towards the front of the vehicle or in another direction (e.g., rearward, sideward). , at a certain angle, etc.). Accordingly, system 100 includes a plurality of camera clusters, each cluster capable of capturing images from a specific area around the vehicle directed in a specific direction.

The field of view of the first camera may be wider, narrower, or partially overlap with the field of view of the second camera. In addition, the first camera is connected to the first image processor to perform monocular image analysis of the image provided by the first camera, and the second camera is connected to the second image processor to perform monocular image analysis of the image provided by the second camera. It can perform monocular image analysis of images. In embodiments where at least one of the first and second cameras includes an infrared camera, the first image processor and/or the second image processor may perform heat map analysis of the heat map provided by the infrared cameras.

Outputs (eg, processed information) of the first and second image processors may be combined.

In some embodiments, the second image processor may receive images from both the first and second cameras to perform stereoscopic analysis or to perform analysis on aligned visual and infrared images. In another embodiment, system 100 may use a three-camera imaging system where each camera has a different field of view. Thus, these systems can make decisions based on information obtained from objects located at various distances in front of and to the side of the vehicle. Monocular image analysis may refer to a case in which image analysis is performed based on images taken from a single field of view (eg, images captured by a single camera). Stereoscopic image analysis may refer to a case in which image analysis is performed based on two or more images captured with one or more image capture parameters.

For example, captured images suitable for stereoscopic image analysis may include images captured from two or more locations, images captured from different views, images captured using different focal lengths, images captured according to parallax information, and the like. can Hybrid image analysis may refer to cases where one or more visual images are aligned with one or more infrared images and image analysis is performed based on the aligned images. For example, in one embodiment, system 100 may implement a triple camera configuration using image capture devices 122, 124, and 126. In this configuration, image capture device 122 may provide a narrow field of view (eg, 34 degrees, or a field of view selected from about 20 to 45 degrees), while image capture device 124 may provide a wide field of view (eg, 150 degrees). or a field of view selected from about 100 to 180 degrees), and image capture device 126 can provide an intermediate field of view (eg, 46 degrees or a field of view selected from about 35 to 60 degrees). In some embodiments, image capture device 126 may act as a primary camera. The image capture devices 122, 124, and 126 may be disposed behind the rearview mirror 310 and may be disposed substantially side by side with each other (eg, 6 cm apart). Also, as described above, in some embodiments, one or more image capture devices 122 , 124 , 126 may be mounted behind glare shield 380 against the windshield of vehicle 200 . Such shielding can minimize the effect of reflections from inside the vehicle on the image capture devices 122, 124, and 126.

In another embodiment, as described above with reference to FIGS. 3B-3C , a wide-field camera (e.g., image capture device 124 in the above example) may include a narrow-field camera and a main-field camera (e.g., image capture device 124 in the above example). It can be mounted at a lower position than the image capture devices 122 and 126. This configuration can provide an unobstructed line of sight from the wide-field cameras To reduce reflections, the cameras can be mounted close to the windshield of vehicle 200. A polarizing plate may be included to mitigate reflected light.

A three-camera system may provide certain capabilities. For example, some embodiments may include the ability to verify objects detected by one camera based on detections of other cameras. In the triple camera configuration described above, processing unit 110 includes, for example, three processing units (e.g., three EyeQ series processor chips), each processing unit comprising one or more image capture devices 122, 124, 126 ) can exclusively process the captured image.

In a triple camera system, the first processing unit receives images from the main camera and the narrow field of view camera, and performs vision processing of the narrow field of view camera, such as other vehicles, pedestrians, lane markings, traffic signs, traffic lights, Other objects on the road can be detected. Furthermore, the first processing unit generates a 3D reconstruction of the surrounding environment of the vehicle 200 by calculating a pixel difference between the main camera and the narrow field of view camera, and then the 3D reconstruction is converted into 3D map data or information received from other cameras. It can be combined with 3D information calculated based on

The second processing unit may receive images from the main camera and perform vision processing to detect other vehicles, pedestrians, lane markings, traffic signs, traffic lights, and other objects on the road. In addition, the second processing unit may generate a 3D reconstruction of the scene (eg, motion structure) by calculating the camera displacement and calculating pixel differences between successive images based on the calculated displacement. The second processing device may transmit the motion structure based on the 3D reconstruction to the first processing device and combine it with the 3D stereoscopic image.

A third processing unit may receive and process images from the wide-field camera to detect vehicles, pedestrians, lane markings, traffic signs, and other objects on the road. The third processing unit may also perform additional processing instructions for image analysis to identify moving objects in the image, such as vehicles changing lanes, pedestrians, and the like.

In some embodiments, streams of image-based information may be obtained and processed independently, providing redundancy on the system. Here, redundancy includes, for example, using a first image capture device and images processed from the device to capture and process image information from at least a second image capture device to verify and/or supplement information obtained. can do.

In some embodiments, system 100 uses two image capture devices (e.g., 122 and 124) to provide driving control support for vehicle 200 and uses a third image capture device (e.g., 126) to perform the Redundancy and verification of analysis results of data received from two image capture devices can be provided. For example, in this configuration, image capture devices 122 and 124 provide images for stereo analysis by system 100 for driving of vehicle 200, and image capture device 126 provides images for stereoscopic analysis by system 100. ) to provide an image for monocular analysis, thereby providing redundancy and verification of information obtained based on the image captured by the image capture device 122 and/or 126. That is, image capture device 126 (and corresponding processing device) is redundant to provide validation of the analysis obtained from image capture devices 122 and 124 (e.g., to provide automatic emergency braking (AEB)). It can be considered as providing a subsystem. Further, in some embodiments, redundancy and verification of received data may be supplemented based on information received from one or more sensors (e.g., radar, lidar, acoustic sensors, information received from one or more transceivers external to the vehicle, etc.). .

Those skilled in the art will be able to recognize that the configuration, arrangement, number, position, etc. of the cameras are only examples. These and other components described in connection with the overall system may be combined and used in various configurations without departing from the scope of the described embodiments. Details related to the use of multi-camera systems to provide driver assistance and/or autonomous driving functions are discussed below.

4 is an exemplary functional configuration diagram of memories 140 and/or 150 in which instructions for performing one or more operations according to the disclosed embodiments may be stored/programmed. Although memory 140 is described below, it will be apparent to those skilled in the art that instructions may be stored in memory 140 and/or memory 150 .

As shown in FIG. 4 , the memory 140 may store a monocular image analysis module 402, a stereoscopic image analysis module 404, a speed and acceleration module 406, and a driving response module 408. there is. The embodiments described herein are not limited to any particular configuration of memory 14. Also, the application processor 180 and/or the image processor 190 may execute instructions stored in all modules 420 , 404 , 406 , and 408 included in the memory 140 . Those skilled in the art will understand that the processing unit 110 may refer to the application processor 180 and/or the image processor 190 individually or collectively in the following description. Accordingly, the steps of the process described below may be performed by one or more processing devices.

In one embodiment, the monocular image analysis module 402, when executed by the processing unit 110, is a monocular image ( may store instructions (eg, computer vision software) that perform analysis of monocular images. In some embodiments, processing unit 110 may perform monocular image analysis by merging information from a set of images with additional sensing information (eg, information obtained from radar, lidar, etc.). As will be described below with reference to FIGS. 5A-5D , the monocular image analysis module 402 may be used to analyze lane markings, vehicles, pedestrians, road signs, highway interchanges, traffic lights, dangerous objects, and other things associated with the vehicle's surroundings. It may include instructions for detecting features within a set of images, such as features. Based on this analysis, system 100 may, for example, via processing unit 110, cause one or more navigational responses, such as turning vehicle 200, changing lanes, changing acceleration, etc., as described below. The reaction module 408 is described.

In one embodiment, stereo image analysis module 404, when executed by processing unit 110, sets first and second images obtained by a combination of image capture devices selected from among image capture devices 122, 124, and 126. (eg, computer vision software) to perform stereo image analysis of the set. In some embodiments, processing unit 110 may perform stereoscopic image analysis by merging information of the first and second image sets with additional sensing information (eg, information obtained from radar). For example, the stereoscopic image analysis module 404 may perform a stereoscopic image based on a first image set obtained by the image capture device 124 and a second image set obtained by the image capture device 126. It may contain instructions for performing the analysis. As will be described below with reference to FIG. 6 , the stereo image analysis module 404 determines features such as lane markings, vehicles, pedestrians, road signs, highway interchanges, traffic lights, hazardous objects, and other characteristics associated with the vehicle's surroundings. It may include instructions for detecting within the first and second image sets. Based on this analysis, processing unit 110 may cause one or more navigational responses, such as turning vehicle 200, changing lanes, changing acceleration, etc., as described below in navigational response module 408. Further, in some embodiments, stereoscopic image analysis module 404 is a learning system, such as a system that may be configured to use computer vision algorithms to detect and/or mark objects within the environment in which sensory information is captured and processed. You can implement approaches that involve (such as neural networks or deep neural networks) or non-learning systems. In one embodiment, stereoscopic image analysis module 404 and/or other image processing modules may be configured to utilize a combination of learning and non-learning systems.

In one embodiment, speed and acceleration module 406 is software configured to analyze data received from one or more computing and mechanical devices included in vehicle 200 that are configured to cause changes in speed and/or acceleration of vehicle 200. can be saved. For example, processing unit 110 may perform commands associated with velocity and acceleration module 406 to obtain data obtained by executing monocular image analysis module 402 and/or stereoscopic image analysis module 404. Based on this, the target speed of the vehicle 200 may be calculated. The data may include, for example, target position, speed, and/or acceleration, position and/or speed of vehicle 200 relative to nearby vehicles, pedestrians or objects on the road, position of vehicle 200 relative to lane markings on the road, for example. information may be included. Additionally, processing unit 110 may respond to sensing inputs (eg, information obtained from radar) and inputs of other systems, such as drive system 220, braking system 230, and/or steering system 240 of vehicle 200. Based on this, the target speed of the vehicle 200 may be calculated. Based on the calculated target speed, processing unit 110 sends electronic signals to drive system 220, braking system 230, and/or steering system 240 of vehicle 200 to, for example, physically As a result, a change in speed and/or acceleration may be caused by pressing the brake pedal of the vehicle 200 or depressing the accelerator pedal.

In one embodiment, the navigation response module 408 is a processing unit (monocular image analysis module 402) and/or stereo image analysis module 404 to determine a necessary navigation response based on data obtained by executing the processing unit (408). 110) may store executable software. The data may include location and speed of nearby vehicles, pedestrians, and objects on the road, target location information of the vehicle 200, and the like. In addition, in some embodiments, the navigation response may trigger map data, a preset position of the vehicle 200, and/or a monocular image analysis module 402 and/or stereo image analysis module 404. may be based in part or entirely on the relative velocity or relative acceleration between the vehicle 200 and one or more objects obtained by doing so. Driving response module 408 may also include sensing inputs (eg, information obtained from radar) and inputs of other systems such as drive system 220 , braking system 230 , and/or steering system 240 of vehicle 200 . Based on this, it is possible to determine the required driving response. Based on the desired driving response, processing unit 110 sends electronic signals to drive system 220, braking system 230, and/or steering system 240 of vehicle 200 to, for example, the vehicle ( 200) to induce a rotation of a preset angle by rotating the steering wheel, thereby generating a necessary driving response. In some embodiments, processing unit 110 uses the output of navigation response module 408 (eg, desired navigation response) as input for execution of speed and acceleration module 406 to calculate a change in speed of vehicle 200. can be used

Furthermore, any of the modules described herein (eg, 402, 404, 406) may implement a learning system (eg, a neural network or a deep neural network) or a method associated with a non-learning system.

5A is a flow chart illustrating an exemplary process 500a for inducing one or more navigational responses based on monocular image analysis in accordance with a disclosed embodiment. In step 510 , the processing unit 110 may receive a plurality of images through the data interface 128 between the processing unit 110 and the image acquisition unit 120 . For example, the camera included in the image acquisition unit 120 (eg, the image capture device 122 having the field of view 202) captures a plurality of images of the front area (or side or rear area) of the vehicle 200. capture and transmit this image to processing unit 110 via a data connection (eg, digital, wired, USB, wireless, Bluetooth, etc.). The processing unit 110 may execute the monocular image analysis module 402 at step 520 to analyze the plurality of images. This will be described in detail with reference to FIGS. 5B to 5D. By performing the analysis, processing unit 110 may detect features such as lane markings, vehicles, pedestrians, road signs, highway interchanges, traffic lights, and the like in a set of images.

Processing unit 110 also executes monocular image analysis module 402 at step 520 to detect various road hazards such as, for example, pieces of truck tires, fallen signs on the road, unloaded vehicles, animals, and the like. can do. Hazards on the road vary in structure, shape, size, color, etc., making it more difficult to detect such hazards. In some embodiments, processing unit 110 may execute monocular image analysis module 402 to perform multi-frame analysis of multiple images to detect road hazards. For example, processing unit 110 may construct a three-dimensional map of a road by predicting camera movement between successive frames and calculating pixel differences between frames. Processing unit 110 may then use the three-dimensional map to detect hazards present on the road surface as well as the road surface.

At step 530, processing unit 110 may execute drive response module 408 to cause one or more drive responses to vehicle 200 based on the analysis performed at step 520 and the method described above with reference to FIG. 4. . Driving responses may include, for example, turning, changing lanes, changing acceleration, and the like. In some embodiments, processing unit 110 may utilize data obtained from executing speed and acceleration module 406 to cause one or more navigational responses. Additionally, multiple navigational responses may occur simultaneously, sequentially, or in combination. For example, processing unit 110 may sequentially send control signals to steering system 240 and drive system 220 of vehicle 200 to cause vehicle 200 to change lanes and then accelerate. Alternatively, processing unit 110 may simultaneously transmit control signals to braking system 230 and steering system 240 of vehicle 200 so that vehicle 200 brakes and simultaneously changes lanes.

5B is a flow chart illustrating an exemplary process 500b of detecting one or more vehicles and/or pedestrians in a set of images in accordance with the disclosed embodiments. Processing unit 110 may implement this process 500b by executing monocular image analysis module 402 . At step 540, processing unit 110 may determine sets of candidate objects that are likely to be vehicles and/or pedestrians. For example, processing unit 110 may scan one or more images and compare the images to one or more preset patterns to determine likely locations of objects of interest (eg, vehicles, pedestrians, etc.) within each image. there is. Pre-established patterns can be designed to achieve a high rate of false hits and a low rate of misses. For example, processing unit 110 may use a pre-established pattern with a lower similarity threshold to identify candidate objects that are likely vehicles or pedestrians. As a result, the probability that processing unit 110 misses (ie, fails to identify) a candidate object representing a vehicle or pedestrian may be reduced.

At step 542, processing unit 110 may filter the set of candidate objects based on the classification criteria to exclude particular candidates (eg, irrelevant or minor objects). Here, the criterion can be obtained from various properties of object types stored in a database (eg, a database stored in the memory 140). Here, the properties of the object type may include the shape, size, texture, location (eg, location with respect to the vehicle 200) of the object. Accordingly, processing unit 110 may use one or more criteria to exclude false candidates from a set of candidate objects.

At step 544, processing unit 110 may analyze multiple frames of the image to determine whether objects in the set of candidate objects represent vehicles and/or pedestrians. For example, processing unit 110 may track candidate sensed objects over successive frames to accumulate frame-by-frame data associated with the sensed object (eg, size, position, etc. related to vehicle 200 ). Also, the processing unit 110 may estimate a parameter of the sensed object and compare position data of each frame of the object with the predicted position.

At step 546, processing unit 110 may construct a set of measurements of the sensed object. Here, the measurement may include, for example, a position associated with the detected object, a velocity, and an acceleration value (eg, a relative acceleration value with respect to the vehicle 200). In some embodiments, processing unit 110 may use estimation methods using time-based observations, such as Kalman filters or linear quadratic estimation (LQE), and/or other object types (e.g., cars, trucks). , pedestrians, bicycles, road signs, etc.) can construct measurements based on existing modeling data. The Kalman filter is based on a measure of the size of an object, and the size measure can be proportional to the time to impact (eg, the time vehicle 200 takes to reach the object). Accordingly, by performing steps 540, 542, 544, and 546, processing unit 110 identifies vehicles and pedestrians appearing within the set of captured images and provides information associated with these vehicles and pedestrians (e.g., location, speed, size). ) can be derived. Based on the identified content and the derived information, processing unit 110 may cause vehicle 200 to make one or more navigational responses, as described above with reference to FIG. 5A.

At step 548, processing unit 110 may perform optical flow analysis on one or more images to reduce the probability of false positives and misses for candidate objects representing vehicles or pedestrians. Here, optical flow analysis may mean analyzing motion patterns of the vehicle 200 and other than road surface motion in one or more images related to other vehicles and pedestrians, for example. The processing unit 110 may use the position and time values as inputs to a mathematical model to calculate the motion of the candidate object. Thus, optical flow analysis may provide another method of detecting vehicles and pedestrians proximate to vehicle 200 . Processing unit 110 may perform optical flow analysis in conjunction with steps 540, 542, 544, and 546 to provide redundancy in detecting vehicles and pedestrians and improve reliability of system 100.

5C is a flowchart illustrating an example of a process 500C of detecting road sign and/or lane shape information from an image set according to the described embodiment. The processing unit 110 may implement the process 500C by executing the monocular image analysis module 402 . At step 550, processing unit 110 may scan one or more images to detect a set of objects. To detect portions of lane markings, lane shape information, and other relevant road signs, processing unit 110 filters a set of objects to exclude objects that are determined to be irrelevant (e.g., insignificant potholes, gravel, etc.) can make it At step 552, processing unit 110 may group together pieces detected at step 550 that correspond to the same road sign or lane markings. Based on this set, processing unit 110 may create a model, eg, a mathematical model, to represent the detected pieces.

At step 554, processing unit 110 may construct a set of measurements corresponding to the sensed segments. In some embodiments, processing unit 110 may project the sensed fragments from an image plane to a real-world plane. Here, the projection may be characterized by using a third-order polynomial having coefficients corresponding to physical properties such as the position, slope, curvature, and curvature derivative of the detected road. When generating the projection, processing unit 110 may take into account the pitch and roll ratio of vehicle 200 as well as changes in the road surface. In addition, the processing unit 110 may estimate the height of the road by analyzing the position and motion signals present on the road surface. Further, processing unit 110 may estimate a pitch and roll rate of vehicle 200 by tracking a set of feature points of one or more images.

At step 556, processing unit 110 may perform multi-frame analysis, for example, by tracking the detected segments across successive image frames and accumulating frame-by-frame data associated with the detected segments. As processing unit 110 performs multi-frame analysis, the set of measurements constructed in step 554 becomes more reliable and has a higher confidence level. Accordingly, by performing steps 550 to 556, the processing unit 110 can identify road signs appearing in the captured image set and derive lane shape information. Based on this identification and derived information, processing unit 110 may cause vehicle 200 to take a navigational response, as described above with reference to FIG. 5A.

At step 558, processing unit 110 may take the additional information into account to create a safety model of the surroundings of vehicle 200. Processing unit 110 may use the safety model to define an environment in which system 100 may safely control autonomous control of vehicle 200 . In some embodiments, to create a safety model, processing unit 100 may use general information extracted from the positions and movements of other vehicles, detected road edges and barriers, and/or map data (eg, data from map database 160). Road shape descriptions can be considered. By considering the additional information, processing unit 110 may provide redundancy for road markings and lane shapes and improve reliability of system 100 .

5D is a flowchart illustrating an exemplary process 500D for detecting traffic lights in a set of images according to the disclosed embodiments. The processing unit 110 may implement the process 500D by executing the monocular image analysis module 402 . At step 560, processing unit 110 scans the set of images and identifies objects appearing at locations in the images that likely contain traffic lights. For example, processing unit 110 may filter the identified objects to construct a set of candidate objects excluding objects unlikely to correspond to traffic lights. Filtering may be performed based on various properties such as the shape, size, texture, and position (eg, relative position to the vehicle 200) of the traffic light. These properties may be based on multiple examples of traffic lights and traffic control signals stored in the database. In some embodiments, processing unit 110 may perform multi-frame analysis on a set of possible traffic light candidate objects. For example, processing unit 110 may track candidate objects across successive image frames, estimate real-world locations of candidate objects, and filter out moving objects (ie, objects less likely to be traffic lights). In some embodiments, processing unit 110 may perform color analysis on the candidate objects and identify the relative location of the sensed color within the object as a possible traffic light.

At step 562, processing unit 110 may analyze the geometry of the intersection. This analysis is based on (i) the number of lanes detected on both sides of the vehicle 200, (ii) marks (arrows, etc.) detected on the road, (iii) map data (e.g., map data stored in the database 160). It can be performed based on a combination of extracted intersection descriptions. The processing unit 110 may perform analysis using information derived by executing the monocular analysis module 402 . Additionally, processing unit 110 may determine in step 560 whether the detected traffic light corresponds to a visible lane in the vicinity of vehicle 200 .

As vehicle 200 approaches an intersection, at step 564, processing unit 110 may update the confidence level regarding the analyzed intersection geometry and detected traffic lights. For example, the confidence level can be determined by comparing the number of traffic lights estimated to appear at an intersection with the number of traffic lights that actually appear at an intersection. Accordingly, based on this confidence level, processing unit 110 may delegate control of vehicle 200 to the driver to improve safety conditions. By performing steps 560, 562, and 564, processing unit 110 may identify traffic lights appearing within the set of captured images and analyze intersection geometry information. Based on this identification and analysis, processing unit 110 may cause vehicle 200 to perform one or more navigational responses described above with reference to FIG. 5A.

FIG. 5E is a flowchart illustrating an exemplary process 500E for causing one or more navigational responses in vehicle 200 based on the vehicle path in accordance with the disclosed embodiments. At step 570 , processing unit 110 may construct an initial vehicle path for vehicle 200 . The vehicle path may be represented by a set of points represented by coordinates ( x , z ), and an interval d _i between two points of the set of points may be in the range of 1 to 5 meters. In one embodiment, processing unit 110 may construct the initial vehicle path using two polynomials, such as a left road polynomial and a right road polynomial. Processing unit 110 calculates the midpoint between the two polynomials, and if there is an offset (zero offset corresponds to center lane driving), each point included in the vehicle path result is assigned a preset offset (e.g., Smart Lane Offset). The offset may be in a direction perpendicular to the interval between two points in the vehicle's path. In another embodiment, processing unit 110 may use a polynomial and the estimated lane width to offset each point in the vehicle's path by the estimated lane width plus a preset offset (eg, smart lane offset). there is.

At step 572 , processing unit 110 may update the vehicle path constructed at step 570 . Processing unit 110 may reconstruct the vehicle path constructed in step 570 using a higher resolution such that the distance d _k between two points in the set of points representing the vehicle path is less than the distance d _i described above. there is. For example, d _k may be in the range of 0.1 to 0.3 meters. Processing unit 110 uses a parabolic spline algorithm to calculate a cumulative distance vector S corresponding to the full length of the vehicle path (i.e., based on a set of points representing the vehicle path). can be reconstructed.

At step 574, processing unit 110 may determine a look-ahead point (expressed as coordinates ( x _l , z _l )) based on the updated vehicle path constructed at step 572. Processing unit 110 may extract a look-ahead point from the cumulative distance vector S , and the look-ahead point may be associated with a look-ahead distance and a look-ahead time. The look-ahead distance, which may have a lower limit of 10 to 20 meters, may be calculated by multiplying the look-ahead time by the speed of the vehicle 200 . For example, if the speed of the vehicle 200 decreases, the look-ahead distance may also decrease (eg, until a lower limit is reached). The look-ahead time, which may range from 0.5 to 1.5 seconds, may be inversely proportional to the gain of one or more control loops involved in inducing a driving response in the vehicle 200 (eg, an azimuth error tracking control loop). there is. For example, the gain of the azimuth error tracking control loop may vary depending on the bandwidth of the yaw rate loop, steering actuator loop, vehicle lateral dynamics, and the like. Therefore, as the gain of the azimuth error tracking control loop increases, the prediction time may decrease.

At step 576, processing unit 110 may determine an azimuth error and a yaw rate command based on the look-ahead determined at step 574. Processing unit 110 may determine by calculating the inverse tangent of the lookahead point, for example, arctan ( xl / _{zl )} . The processing unit 110 may determine a yaw rate command by calculating the product of the azimuth error and the high level control gain. The high-level control gain can be equal to (2 / look-ahead time) if the look-ahead distance is not at the lower limit. Alternatively, the high-level control gain can be equal to (2 * vehicle speed / look-ahead distance).

5F is a flowchart illustrating an exemplary process 500F for determining whether a lead vehicle is changing lanes in accordance with the disclosed embodiments. At step 580, processing unit 110 may determine driving information for a lead vehicle (eg, a vehicle traveling ahead of vehicle 200). For example, processing unit 110 may utilize the methods described with reference to FIGS. 5A and 5B above to determine the position, speed (eg, direction and speed), and/or acceleration of the lead vehicle. Processing unit 110 may also utilize one or more road polynomials, look-ahead points (associated with vehicle 200), and/or snail trails utilizing the methods described with reference to FIG. 5E above. A set of points describing the can be determined.

In step 582, the processing unit 110 may analyze the driving information determined in step 580. In one embodiment, processing unit 110 may calculate the distance between the snail trail and the road polynomial (eg, the polynomial along the snail trail). If the variance of the distance along the snail trail exceeds a preset threshold (e.g., 0.1 to 0.2 on a straight road, 0.3 to 0.4 on a gentle curve, 0.5 to 0.6 on a sharp curve), processing unit 110 It may be determined that the leading vehicle has a possibility of lane change. If multiple vehicles are detected in front of vehicle 200, processing unit 110 may compare the snail trails of each vehicle. Based on the comparison, processing unit 110 may determine that a vehicle whose snail trail does not match the snail trail of other vehicles is likely to change lanes. Processing unit 110 may additionally compare the curvature of the snail trail of the lead vehicle with the expected curvature of the section of road the lead vehicle is traveling on. The expected curvature may be extracted from map data (eg, data from database 16), road polynomials, snail trails of other vehicles, prior knowledge of roads, and the like. If the difference between the curvature of the snail trail and the expected curvature of the road segment exceeds a preset threshold, processing unit 110 may determine that the leading vehicle is likely changing lanes.

을 통하여 판단할 수 있다. 여기서,

는 횡방향 주행 거리이고,

는 종방향 주행 거리이다. 횡방향 주행 거리와 예상 곡률 사이의 차이가 미리 설정한 임계치(예, 500 내지 700미터)를 초과하는 경우, 처리부(110)는 선두 차량이 차선 변경을 하고 있을 가능성이 있는 것으로 판단할 수 있다. 다른 실시예에서, 처리부(110)는 선두 차량의 위치를 분석할 수 있다. 선두 차량의 위치가 도로 다항식을 안 보이게 하는 경우(예, 선두 차량이 도로 다항식의 상부에 덮어씌워 있는 경우), 처리부(110)는 선두 차량이 차선 변경을 하고 있을 가능성이 있는 것으로 판단할 수 있다. 다른 차량이 선두 차량의 앞에 검출되고 선두 차량과 앞 차량의 스네일 트레일이 서로 평행하지 않은 경우, 처리부(110)는 선두 차량(즉, 차량(200)과 가까운 차량)이 차선 변경을 하고 있을 가능성이 있는 것으로 판단할 수 있다. In another embodiment, processing unit 110 may compare the instantaneous position of the lead vehicle with the look-ahead point (the look-ahead point of vehicle 200) for a specific time period (eg, 0.5 to 1.5 seconds). During a specific time, the distance between the instantaneous position of the leading vehicle and the look-ahead changes, and the cumulative sum of the changes is a preset threshold (e.g., 0.3 to 0.4 meters on a straight road, 0.7 to 0.8 meters on a gentle curve, 0.7 to 0.8 meters on a sharp curve) 1.3 to 1.8 meters), processing unit 110 may determine that the lead vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the geometry of the snail trail by comparing the lateral distance traveled along the snail trail with the expected curvature of the snail trail. The expected radius of curvature is

can be judged through here,

is the transverse travel distance,

is the longitudinal travel distance. If the difference between the lateral travel distance and the expected curvature exceeds a predetermined threshold (eg, 500 to 700 meters), processing unit 110 may determine that the leading vehicle is likely changing lanes. In another embodiment, processing unit 110 may analyze the position of the lead vehicle. If the position of the lead vehicle obscures the road polynomial (e.g., the lead vehicle is overlaid on top of the road polynomial), processing unit 110 may determine that the lead vehicle is likely changing lanes. . If another vehicle is detected in front of the lead vehicle and the snail trails of the lead vehicle and the preceding vehicle are not parallel to each other, processing unit 110 determines the likelihood that the leading vehicle (i.e., a vehicle close to vehicle 200) is changing lanes. It can be judged that there is

At step 584, processing unit 110 may determine whether the lead vehicle is changing lanes based on the analysis performed at step 582. For example, processing unit 110 may make the determination based on a weighted average of the individual analyzes performed in step 582. In this method, for example, if processing unit 110 determines, based on some type of analysis, that the lead vehicle is likely to be changing lanes, it may assign a value of “1” (the lead vehicle may be changing lanes). In the case of determining the possibility of not changing lanes, a value of “0” is given). Different analyzes performed in step 582 may be given different weights, and the present embodiment is not limited to specific combinations of analyzes and weights. Further, some embodiments may utilize a learning system (eg, a machine learning system or a deep learning system) that may, for example, estimate a future path ahead of the vehicle's current location based on images captured at the current location. there is.

6 is a flow chart illustrating an exemplary process 600 for inducing one or more navigational responses based on stereoscopic image analysis in accordance with a described embodiment. At step 610 , processing unit 110 may receive a plurality of first and second images via data interface 128 . For example, a camera included in the image acquisition unit 120, the image capture devices 122 and 124 having fields of view 202 and 204, etc., captures a plurality of first and second images of the front area of the vehicle 200. and may be transmitted to the processing unit 110 through a digital connection (eg, USB, wireless communication, Bluetooth, etc.). In some embodiments, processing unit 110 may receive a plurality of first and second images through more than one data interface. The embodiments described herein are not limited to a specific data interface configuration or protocol.

At step 620, processing unit 110 executes stereoscopic image analysis module 404 to perform stereoscopic image analysis on the plurality of first and second images to generate a three-dimensional map of the road in front of the vehicle, and drive within the image. Features such as markings, vehicles, pedestrians, road signs, highway interchanges, traffic lights, and hazards on the road can be detected. Stereoscopic image analysis may be performed in a manner similar to the steps described above with reference to FIGS. 5A to 5D . For example, processing unit 110 may execute stereo image analysis module 404 to detect candidate objects (eg, vehicles, pedestrians, road signs, traffic lights, road hazards, etc.) within the plurality of first and second images. and filter a set of candidate objects based on various criteria, perform multi-frame analysis, construct measurements, and determine confidence levels for the remaining candidate objects. In performing the above steps, processing unit 110 may consider information of both the plurality of first and second images rather than a group of images. For example, processing unit 110 may analyze differences in pixel level data (or other data subsets among the captured images of the two streams) for candidate objects appearing in both the first and second plurality of images. As another example, processing unit 110 may observe that a candidate object appears in one of the plurality of images but not another, or through other differences that exist with respect to objects appearing in the two image streams, the candidate object ( For example, a position and/or speed of a candidate object relative to vehicle 200 may be estimated. For example, the position, speed, and/or acceleration associated with vehicle 200 may be determined based on trajectories, positions, motion characteristics, and the like, of features associated with objects appearing in one or both of the image streams.

At step 630, processing unit 110 executes navigation response module 408 to generate one or more navigation responses to vehicle 200 based on the analysis performed in step 620 and the methods described above with reference to FIG. can Driving responses may include, for example, turning, changing lanes, changing acceleration, changing speed, braking, and the like. In some embodiments, processing unit 110 may use data derived from executing speed and acceleration module 406 to generate one or more navigational responses. In addition, multiple navigation responses can occur simultaneously, sequentially, or combinations thereof.

7 is a flow chart illustrating an exemplary process 700 for generating one or more navigational responses based on three sets of images in accordance with the described embodiments. At step 710 , processing unit 110 may receive a plurality of first, second, and third images through data interface 128 . For example, a camera included in the image acquisition unit 120 (image capture device 122, 124, 126, etc. having a field of view 202, 204, 206) may capture a plurality of images of the front and/or side areas of vehicle 200. The first, second, and third images may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless communication, Bluetooth, etc.). In some embodiments, processing unit 110 may receive a plurality of first, second, and third images through three or more data interfaces. For example, image capture devices 122, 124, and 126 may have a data interface associated with each device to communicate data with processing unit 110. The embodiments described herein are not limited to a specific data interface configuration or protocol.

At step 720, processing unit 110 analyzes the plurality of first, second, and third images to detect features in the images, such as lane markings, vehicles, pedestrians, road signs, highway interchanges, traffic lights, road hazards, and the like. can This analysis may be performed in a manner similar to the steps described above with reference to FIGS. 5A to 5D and FIG. 6 . For example, the processing unit 110 analyzes a monocular image (eg, executes a monocular image analysis module 402 and FIGS. 5A to 5D) for each of the plurality of first, second, and third images. The step) described with reference to may be performed. Alternatively, the processing unit 110 may perform stereoscopic image analysis on the plurality of first and second images, the plurality of second and third images, and/or the plurality of first and third images (e.g., the stereoscopic image analysis module 404). ) and the above steps described with reference to FIG. 6) may be performed. Processing information corresponding to a plurality of first, second and/or third images may be merged. In some embodiments, processing unit 110 may perform a combination of monocular image analysis and stereoscopic image analysis. For example, the processing unit 110 performs monocular image analysis (eg, execution of the monocular image analysis module 402 ) on a plurality of first images, and a plurality of second and third images. Stereoscopic image analysis (eg, execution of the stereoscopic image analysis module 404 ) may be performed on the image. The configuration of image capture devices 122, 124, 126, each with its position and field of view 202, 204, 206, may affect the type of analysis performed on the plurality of first, second, and third images. . The embodiments described herein are not limited to the particular configuration of image capture devices 122, 124, 126 or the type of analysis performed on the plurality of first, second, and third images.

In some embodiments, processing unit 110 may perform inspections of system 100 based on the images acquired and analyzed at steps 710 and 720 . Such inspection may provide an indicator of the overall performance of system 100 for a particular configuration of image capture devices 122, 124, and 126. For example, processing unit 110 may determine a percentage of "false hits" (eg, when system 100 falsely determines that a vehicle or pedestrian is present) and "misses."

At step 730, processing unit 110 may generate one or more navigational responses to vehicle 200 based on information derived from two of the plurality of first, second, and third images. Two of the plurality of first, second, and third images may be selected based on various factors such as, for example, the number, type, and size of objects detected in each of the plurality of images. The processing unit 110 also determines the quality and resolution of the image, the effective field of view reflected in the image, the number of captured frames, and the degree to which the object of interest actually appears in a frame (e.g., the percentage of frames in which the object appears, the number of frames in which the object appears in each frame). You can select an image according to the ratio occupied by the object, etc.).

In some embodiments, processing unit 110 may determine the extent to which information derived from one image source is consistent with information derived from another image source to determine the extent to which information derived from two of the plurality of first, second, and third images can choose For example, the processing unit 110 merges processing information derived from each of the image capture devices 122, 124, and 126 (monocular analysis, stereoscopic analysis, or a combination thereof) and captures the image capture devices 122, 124, and 126. Consistent visual indicators (eg, lane markings, detected vehicles and their locations and/or routes, detected traffic lights, etc.) may be determined from each captured image. Processing unit 110 may also exclude information that is inconsistent across captured images (eg, vehicles changing lanes, lane models indicating vehicles too close to vehicle 200 , etc.). Accordingly, processing unit 110 may select information derived from two of the plurality of first, second, and third images based on the determination of consistent information and inconsistent information.

The driving response may include, for example, turning, changing lanes, changing acceleration, and the like. Processing unit 110 may generate one or more navigational responses based on the analysis performed at step 720 and the methods described with reference to FIG. 4 . Processing unit 110 may also use data derived from executing speed and acceleration module 406 to generate one or more navigational responses. In some embodiments, processing unit 110 may determine based on a relative position, relative speed, and/or relative acceleration between vehicle 200 and an object detected within any one of the plurality of first, second, and third images. One or more driving responses may be triggered. Multiple travel responses can occur simultaneously, sequentially, or a combination thereof.

By analyzing the captured images and/or heatmaps, certain characteristics of both parked and moving vehicles can be detected. Based on the detected characteristics, a driving change can be calculated. Embodiments for detecting specific characteristics based on one or more specific analyzes of captured images and/or heatmaps are described below with reference to FIGS. 8-28 .

[Vehicle door open event detection]

For example, targeted monitoring of a door opening event is possible by identifying a wheel component of the identified vehicle after identification of the vehicle. By target monitoring, the system can identify and respond to door opening events with shorter reaction times than conventional motion detection. Embodiments of the present disclosure described below relate to systems and methods for detecting door opening events utilizing targeted monitoring.

8 is an exemplary configuration diagram of memories 140 and/or 150 in which instructions for performing one or more operations according to the disclosed embodiments may be stored/programmed. Although memory 140 is discussed below, those skilled in the art will recognize that instructions may be stored in memory 140 and/or 150 .

As shown in FIG. 8 , the memory 140 may store a vehicle side identification module 802, a wheel identification module 804, a door edge identification module 806, and a driving response module 808. there is. The described embodiments are not limited to the particular configuration of memory 140. Additionally, application processor 180 and/or image processor 190 may execute instructions stored in one or more of modules 802 - 808 included in memory 140 . It will be understood by those skilled in the art that the following information described through the processing unit 110 may also apply to the application processor 180 and the image processor 190 individually or collectively. Accordingly, the steps of the process below may be performed by one or more processing units.

In one embodiment, vehicle side identification module 802, when executed by processing unit 110, performs instructions (computer computer) to perform analysis of one or more images obtained by one of image capture devices 122, 124, 126. vision software, etc.) As described below with reference to FIGS. 9-14 , the vehicle side identification module 802 may include instructions for determining a bounding box representing one or more sides of the vehicle.

In one embodiment, wheel identification module 804, when executed by processing unit 110, performs an analysis of one or more images obtained by one of image capture devices 122, 124, 126 (computer vision). software, etc.) As described below with reference to FIGS. 9-14 , the wheel identification module 804 may include instructions for determining an ellipse representing one or more vehicle wheels.

In one embodiment, door edge identification module 806, when executed by processing unit 110, performs instructions (computer vision software, etc.) As described below with reference to FIGS. 9-14 , the door edge identification module 806 can include instructions for identifying the appearance of a door edge and monitoring movement of the identified door edge.

In one embodiment, the navigation response module 808 determines the required navigation response based on data derived from executing the vehicle side identification module 802, wheel identification module 804, and/or door edge identification module 806. It may store software executable by the processing unit 110 to determine. For example, the navigation response module 808 can cause a navigation change according to the method 1400 of FIG. 14 described below.

Furthermore, one or more of the modules (eg, 802, 804, 806) described herein may implement a learning system (eg, a neural network or a deep neural network) or a manner in conjunction with a non-learning system.

9 is a schematic diagram of a road 902 viewed from the perspective of a system included in a host vehicle (eg, system 100 described above) according to the described embodiment. As shown in FIG. 9 , there may be one or more parked vehicles (eg, parked vehicle 904 or parked vehicle 906 ) on roadway 902 .

For example, system 100 utilizes an attention mechanism to return a suspect patch and feed the suspect patch to a more complex bulk classifier to determine if the patch is actually a vehicle to park a vehicle. can be detected. Attention mechanisms and classifiers can be trained on true and false patches as described below.

For example, system 100 may utilize an attention mechanism to detect the rear of a parked vehicle (eg, vehicle rear 908, 910). Utilizing the detected vehicle rear, system 100 may detect the side of the parked vehicle (eg, vehicle sides 912 and 914 ). Once detected, the rear and/or side of the vehicle can be tracked.

To detect the bounding box shown in FIG. 9, system 100 inputs an image (e.g., an image received from one of image capture devices 122, 124, 126) into one or more learned algorithms. can do. The input image may be, for example, a grayscale image whose original size is 1280x9560. The learned algorithm may output a scaled (eg, 256x192) attention image. From this attention image, a suspect patch can be identified and the (x, y) points of the suspect patch can be scaled (eg, by 5) and mapped onto the original image coordinates. The learned algorithm can use the (x, y) coordinates to define the center of the suspect patch. Suspect patches can be scanned within a region of (x, y) coordinates, eg +/- 5 pixels in each direction. In this example, each suspect patch then generates a total of 11×11 candidate patches.

Further, in this example, each candidate patch is a square of size 2R+1, where R is the radius of the bounding ellipse. For example, as shown in FIG. 10 , a vehicle (eg, vehicle 1002) may have an associated bounding ellipse (eg, ellipse 1004) with a center (eg, center 1006).

Candidate patches can be scaled to a basic size (eg, 40x40) and used as input to one or more learning networks, eg, one or more convolutional neural networks (CNNs) described below. For example, one or more learning networks may each score an input patch. Using this score, each candidate patch can be assigned a label based on the highest score. (In this example, the candidate patch can be mapped back to the original image using the radius R and the circular coordinates (x, y).)

Thereafter, each candidate patch having a highest score higher than a threshold (which may be preset or variable and may be learned through learning) may be input to a final classifier. The final classifier can output the (x, y) coordinates of the three points at the bottom of the bounding box. These coordinates can be scaled back to the original image coordinates by multiplying by the appropriate factor.

In the example described above, a suitable scale factor might be (2R + 1) / 40. In addition to using scale factors, the actual position (x ₀ , y ₀ ) can be added to the scale coordinates (x, y). Using unique labels, the system can determine which two of the three (x, y) coordinates belong to the side (and by extension which side) and which two of the three (x, y) coordinates belong to the posterior or anterior. can

Those skilled in the art will recognize that variations on this exemplary algorithm are possible. For example, the size of the scaled attention image may vary, the movement of the suspect patch may vary, and the size of the candidate patch may vary. As an additional example, the upper coordinates of the bounding box may also be calculated. (In this example, the final classifier may output the (x, y) coordinates of the additional three points.) Also, other algorithms are possible in place of or in combination with the example algorithms described above. For example, different algorithms may include different and/or additional classifiers.

As described above, the attention mechanism and subsequent classifiers can be learned. For example, the learning mechanism may utilize more than one million example images, which may be gray scale images, for example 1280x960. In this exemplary learning setup, the visible side of the bounding box may be indicated as left, right, back, or front. For example, they may be shown as yellow, blue, red, and green, respectively. If one side is partially visible, only the unobscured portion is displayed in the image and the partially obscured can be recorded in the database.

In this learning example, for each bounding box, the system calculates the two farthest edges of the bounding box, the center is between these two edges, and the distance to the farthest edge is the radius. Boundary ellipses can be constructed.

In this learning example, the system can then extract an attention image that is a 256×192 image (ie reduced by a factor of 5) from the whole. Each vehicle displayed in the training image may be replaced with a point in the attention image located at the coordinates of the center of the ellipse divided by 5, and the value of this point may be the radius of the boundary ellipse.

In this training example, example images can be used to train a CNN. Those skilled in the art will appreciate that other machine learning methods may be used in place of or incorporating CNNs. Thus, the neural network can map the original image to a sparse, low-resolution attention image. This method can merge scene understanding (eg, location of the road, viewpoint of the image, etc.) with detection of objects that appear to be vehicles. Other design choices are also possible. For example, the neural network may first apply a filter bank designed to detect vehicles in locations where vehicles are predicted (eg, non-sky locations).

The neural network may send the suspect patch to a first classifier that can score each possible scene. For example, the first classifier may designate one of four major labels: LeftRear, LeftFront, RightRear, and RightFront. If only one side is visible, one of the two possible labels can be arbitrarily assigned. Each major label can be further divided. For example, each major label could be further divided into whether a patch contains more "sides" than "ends" or vice versa. This sub-segmentation may be performed, for example, by comparing the image widths of the displayed side and cross-section. When the widths are the same, subdivisions can be arbitrarily designated.

For another example, each subdivision may be further divided into three subdivisions. For example, a "side" subdivision of LeftRear may include three subdivisions. An "end" LeftRear10 may represent a patch where the rear surface is less than 10% of the width of the left surface, and an "end" LeftRear50 may represent a patch where the rear surface is greater than 10% and less than 50%. , and the "end" LearRear may indicate a patch with a rear surface greater than 50%.

For another example, each sub-subdivision may further indicate whether at least one side is obscured. In this example, the total combination of lower subdivisions and labels is 48. One skilled in the art will appreciate that other means of partitioning and labeling of the same or different numbers of total combinations are possible. For example, "side" LeftRear20 may represent a patch in which the left surface is less than 20% of the total, "side" LeftRear50 may represent a patch in which the left surface is 20% to 50% of the total, , "side" LeftRear80 may represent a patch in which the left surface is 50% to 80% of the total, and "side" LeftRear100 may represent a patch in which the left surface is 80% to 100% of the total. .

For another example, the four main labels could be replaced with the two main labels, ie 'side/short' or 'short/side' depending on whether the side is viewed on the left or right side of the section, respectively. The choice of splits and labels depends on the amount of available data, since the number of examples required for training increases as the number of subdivisions and labels increases.

Accordingly, the neural network may input a patch centered on a point in the attention map scaled back to the original image coordinates and shifted by +/-5 pixels in the x and y directions to the first classifier. This movement produces 121 moved instances. Those skilled in the art will appreciate that other means of creating a moved instance are possible. For example, a patch may be moved +/- 4 pixels (or +/- 3 pixels, etc.) thoroughly in the x and y directions.

In this example, each patch may be formed by utilizing the radius R of the boundary ellipse to cut a square of size (2R + 1) x (2R + 1) scaled to a basic size (eg, 40x40 pixels). One skilled in the art will recognize that other means of creating a patch are possible. For example, you can cut a square of size (2R - 1) × (2R - 1). For another example, one or more bounding box lengths may be used instead of 2R.

The neural network can input each labeled patch into a final classifier. The final input could output the position (x, y), within the labeled patch, of each of the three points defining the bottom of the bounding box. The output (x, y) coordinates may be related to the patch. In some embodiments, a neural network may be trained for each combination of subdivisions and labels. In other embodiments, fewer neural networks may be trained.

A similar learning method can be used to train the classifier to extract more specific features from the identified vehicles. For example, the classifier can be trained to identify wheels, tires, 'A' pillars, side mirrors, and the like. In the example of FIG. 11 , wheels 1102 and 1104 of vehicle 1106 are represented by ovals. The system can draw an ellipse, for example, by scaling a bounding box to a default size (eg, 40 pixels long and 20 pixels high) and inputting it into an appropriate classifier.

Based on the identification of the wheel, the system may determine one or more “hot spots” on the identified vehicle where a door opening event is predicted to occur. For example, one or more hotspots may be located between the identified tires and/or over the identified rear tires. As shown in FIG. 12A , one or more hotspots may be observed for the appearance of vertical stripes 1202 on vehicle 1204 . The appearance of the stripe 1202 may indicate the start of a door opening event.

In some embodiments, the system may track the movement of the edge of the stripes 1202 relative to the side of the vehicle 1204 using one or more features on the vehicle 1204 as reference points. For example, the one or more features may include an identified tire or other identified feature, such as a front edge of vehicle 1204, one or more tail lights of vehicle 1204.

As shown in FIG. 12B , stripes 1202 may expand as the doors of vehicle 1204 are opened. Edge 1202a of stripes 1202 can be seen stationary along the body of vehicle 1204, while edge 1202b of stripes 1202 is moving towards the front of vehicle 1204. Thus, the system can determine that there is a door open event based on observation of stripes 1202 .

Based on the presence of a door open event, the host vehicle may perform a driving change. For example, as shown in FIG. 13 , the host vehicle's yaw 1301 has changed, meaning that the host vehicle is moving away from the door open event.

14 is a flowchart illustrating an exemplary process 1400 for inducing one or more navigational responses based on detection of a door opening event in accordance with the described embodiments. At step 1402 , processing unit 110 may receive at least one image of the environment of the host vehicle via data interface 128 . For example, a camera included in image acquisition unit 120 (eg, image capture device 122 , 124 , 126 having a field of view 202 , 204 , 206 ) may capture at least a portion of the front area and/or side of the host vehicle. One image may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless, Bluetooth, etc.).

At step 1404, processing unit 110 may analyze the at least one image to identify a side of the parked vehicle. Step 1404 may further include associating at least one bounding box with the shape of the side of the parked vehicle. For example, the analysis may be performed using the learning algorithm described above with reference to FIGS. 9 and 10 .

At step 1406, processing unit 110 may identify structural features of the parked vehicle. In some embodiments, processing unit 110 identifies a first structural feature of the parked vehicle within a forward region of the side of the parked vehicle and identifies a second structural feature of the parked vehicle within a rear region of the side of the parked vehicle. can be identified. For example, structural features may include wheel components (eg, tires, hubcaps, or wheel structures), mirrors, 'A' pillars, 'B' pillars, 'C' pillars, and the like. The first and/or second structural feature may be identified within an area proximate to the identified side surface. For example, the analysis may be performed using the learning algorithm described with reference to FIG. 11 .

At step 1408, processing unit 110 may identify a door edge of a parked vehicle. A door edge can be identified within the area around the structural feature. For example, in an embodiment where processing unit 110 identifies a wheel component of a front wheel and a wheel component of a rear wheel, the vicinity of the first wheel and the second wheel is the wheel component of the front wheel. ) and the wheel component of the rear wheel. For another example, in an embodiment in which processing unit 110 identifies a wheel component of a front wheel and a wheel component of a rear wheel, the vicinity of the first wheel and the second wheel is the wheel component of the rear wheel. component) may include the upper region. The analysis may be performed using the learning algorithm described with reference to FIG. 12A.

At step 1410, processing unit 110 may determine a change in image characteristics of the door edge. For example, the processing unit 110 may observe whether vertical stripes are visible in the vicinity of the first and second wheel components as described above with reference to FIG. 12B in at least two images received from the image capture device. can In this example, the first edge of the vertical stripe is stationary in the viewed image along the body of the parked vehicle, and the second edge of the vertical stripe moves towards the area in front of the parked vehicle in the observed image. After the door edge is viewed, the width of the door edge over time (ie, the width of the vertical stripes) can be tracked. In some embodiments, changing the image feature of the door edge may include increasing the width of the door edge. In such an embodiment, determining a change in the image feature of the door edge may include observing an extension of the vertical stripes.

In some embodiments, processing unit 110 may estimate the degree of door opening. For example, the processing unit 110 may extend the vertical line of the fixed edge to intersect the bounding box and estimate the distance to the door using the y coordinate of the intersection point. In this example, the degree of door opening can be estimated using the distance to the door and the width of the stripe. Accordingly, processing unit 110 may determine the distance the door edge extends away from the parked vehicle based on the determined width. Accordingly, it is possible to track the separation of the door edge from the body of the vehicle over time (ie, the degree of door opening).

At step 1412, processing unit 110 may alter the driving route of the host vehicle. For example, the driving response may include a turn (shown in FIG. 13 ), a lane change, a change in acceleration, and the like. Processing unit 110 may trigger one or more navigational responses based on the determination made in step 1410 . For example, processing unit 110 may move the host vehicle away from the door open event and/or decelerate the host vehicle in response to the door open event. In this example, processing unit 110 determines a lateral safety distance to the host vehicle based on the determined distance the door edge extends away from the parked vehicle, and a change in the driving path of the host vehicle is at least It may be based in part on a judged lateral safety distance. In another example, processing unit 110 may determine a lateral safety distance relative to the host vehicle based on a predefined value, such as a value corresponding to a general overhang associated with opening a vehicle door. Furthermore, other predefined values may be used for different types of vehicles, eg, for vehicles of different sizes, such as trucks, a larger predefined safety distance value may be used than the safety distance value used for small vehicles. can

Processing unit 110 may also utilize data derived from execution of speed and acceleration module 406 to generate one or more navigational responses. Multiple travel responses can occur simultaneously, sequentially, or a combination thereof. For example, driving response may be determined by a learning system. Furthermore, the learning system can be configured to not compromise certain safety constraints while optimizing performance, and the learning system can be configured to react to door open detection by causing a change in travel. In another example, when a door opening event (of a parked vehicle) is detected, a set of rules may be utilized to determine the required reaction.

[Detection of Vehicle Entering Lane of Host Vehicle]

Systems and methods for identifying road homography and identifying wheel components of a vehicle may enable targeted monitoring of vehicle motion. By target monitoring, the system can detect and react to motion entering the host vehicle's lane from another lane or from a parked location, at least under certain circumstances, with a shorter response time than conventional motion detection. Embodiments of the present disclosure described below relate to systems and methods for detecting vehicles entering a lane of a host vehicle utilizing target monitoring.

15 is an exemplary functional configuration diagram of memories 140 and/or 150 in which instructions for performing one or more operations according to the disclosed embodiments may be stored/programmed. Although memory 140 is discussed below, those skilled in the art will recognize that instructions may be stored in memory 140 and/or 150 .

As shown in FIG. 15 , the memory 140 may store a road homography module 1502 , a wheel identification module 1504 , a motion analysis module 1506 , and a driving response module 1508 . The described embodiments are not limited to the particular configuration of memory 140. Additionally, applications processor 180 and/or image processor 190 may execute instructions stored in one or more of modules 1502-1508 included in memory 140. It will be understood by those skilled in the art that the following information described through the processing unit 110 may also apply to the application processor 180 and the image processor 190 individually or collectively. Accordingly, the steps of the process below may be performed by one or more processing units.

In one embodiment, road homography module 1502, when executed by processing unit 110, distorts the homography of roads in one or more images obtained by one of image capture devices 122, 124, 126. Can store instructions (such as computer vision software). For example, road homography module 1502 can include instructions for executing method 1800 of FIG. 18 described below.

In one embodiment, wheel identification module 1504, when executed by processing unit 110, provides instructions (computer vision) to perform analysis of one or more images obtained by one of image capture devices 122, 124, 126. software, etc.) As described above with reference to FIGS. 9-14 , the wheel identification module 1504 may include instructions for determining an ellipse representing one or more vehicle wheels.

In one embodiment, motion analysis module 1506, when executed by processing unit 110, may include one or more images obtained by one of image capture devices 122, 124, 126 (and/or road homography module ( 1502) to perform analysis of the one or more images processed by 1502 to track the motion of one or more identified vehicle elements (such as computer vision software). For example, in conjunction with wheel identification module 1504, motion analysis module 1506 can track motion over time of identified wheel components of the vehicle.

In one embodiment, the navigation response module 1508 determines the required navigation response based on data derived from executing the road homography module 1502, wheel identification module 1504, and/or motion analysis module 1506. To do so, software executable by the processing unit 110 may be stored. For example, the navigational response module 1508 can cause a navigational change according to the method 1700 of FIG. 17 described below.

Furthermore, one or more of the modules (eg, 1502, 1504, and 1506) described herein may implement a learning system (eg, a neural network or a deep neural network) or a method in conjunction with a non-learning system.

16A shows a parked vehicle 1602 from the perspective of the system according to the disclosed embodiments. In the example of FIG. 16A , wheels 1604 and 1606 of vehicle 1602 are represented by ovals. As described above, the system can draw an ellipse, for example, by scaling a bounding box to a default size (eg, 40 pixels long and 20 pixels high) and inputting it into an appropriate classifier.

16B shows vehicle 1602 starting to move from a parked state (ie, a stationary state). In the example of FIG. 16B , wheels 1604 and 1606 of vehicle 1602 are rotating as vehicle 1602 is moving. The system can determine movement of vehicle 1602 by tracking rotation of wheels 1604 and 1606 as described below.

17 is a flowchart illustrating an exemplary process 1700 for invoking one or more navigational responses based on detection of a target vehicle entering the host vehicle's lane in accordance with the disclosed embodiments. At step 1702 , processing unit 110 may receive a plurality of images of the surrounding environment of the host vehicle via data interface 128 . For example, a camera included in image acquisition unit 120 (e.g., image capture devices 122, 124, 126 having fields of view 202, 204, 206) may be used to view multiple views of the front area and/or side of the host vehicle. An image of may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless, Bluetooth, etc.).

A plurality of images may be collected over a period of time. For example, the plurality of images may include a first image captured at time t = 0, a second image captured at t = 0.5 second, and a third image captured at t = 1.0 second. The time between images may vary depending on the scan rate of at least one image capture device.

At step 1704, processing unit 110 may analyze the at least one image to identify a target vehicle. For example, identifying the target vehicle may further include associating at least one bounding box with the profile of the target vehicle. For example, the analysis may be performed using a learning algorithm as described above with reference to FIGS. 9 and 10 . In some embodiments, step 1704 may further include identifying wheels on the side of the identified target vehicle. For example, the analysis may be performed using the learning algorithm described above with reference to FIG. 11 .

Step 1704 is not limited to wheels and may also include wheel components. For example, processing unit 110 may identify a wheel component that includes at least one of a tire, hubcap, and wheel structure.

At step 1706, processing unit 110 may identify an action associated with the identified wheel. For example, the processing unit may identify motion within an area containing at least one wheel component of the target vehicle (eg, an area adjacent to a road surface). By monitoring at least two images of the plurality of images, the processing unit may identify motion by ascertaining rotation of at least one wheel component.

In another example, processing unit 110 may identify at least one feature associated with at least one wheel component (e.g., a logo of a wheel, a tire size of a wheel, a hubcap size of a wheel, a specific patch of pixels). can Utilizing this at least one feature, processing unit 110 may identify an indicator of a change in location of the at least one feature (eg, blurring of a logo, change in coordinates of a patch of pixels).

In another example, processing unit 110 may distort the homography of the road, as described with reference to method 1800 of FIG. 18 below, and the junction between the identified wheel and the twisted road (which may be stationary). , and the movement can be identified by tracking the point on the top of the contact point.

Processing unit 110 may utilize an indicator of rotation, a change in position of at least one feature, and/or a tracked point to determine the rate at which the target vehicle is moving. Processing unit 110 may utilize the determined speed to cause a navigational change in step 1708 below. Additionally, processing unit 110 may utilize ground plane constraints to estimate the distance to the tire and estimate the lateral motion of the target vehicle based on the estimated distance.

At step 1708, processing unit 110 may cause a navigational change of the host vehicle. For example, the driving response may include a change in driving direction of the host vehicle (shown in FIG. 13 ), a lane change, a change in acceleration (eg, braking of the host vehicle), and the like. Processing unit 110 may trigger one or more navigational responses based on the determination made in step 1706 . For example, processing unit 110 may move the host vehicle away from the target vehicle and/or decelerate the host vehicle in response to movement of the target vehicle. In this example, processing unit 110 determines a lateral safety distance to the host vehicle based on the determined speed (and/or estimated lateral movement) of the target vehicle, and changes in the driving path of the host vehicle. may be based, at least in part, on the determined lateral safety distance.

Processing unit 110 may also utilize data derived from execution of speed and acceleration module 406 to generate one or more navigational responses. Multiple travel responses can occur simultaneously, sequentially, or a combination thereof.

18 is a flow chart illustrating an exemplary process 1800 for twisting the homography of a road in accordance with a disclosed embodiment. At step 1802 , processing unit 110 may receive a plurality of images of the environment of the host vehicle via data interface 128 . For example, a camera included in image acquisition unit 120 (e.g., image capture devices 122, 124, 126 having fields of view 202, 204, 206) may be used to view multiple views of the front area and/or side of the host vehicle. An image of may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless, Bluetooth, etc.).

A plurality of images may be collected over a period of time. For example, the plurality of images may include a first image captured at time t = 0 and a second image captured at time t = 0.5 seconds. The time between images may vary depending on the scan rate of at least one image capture device.

At step 1804, processing unit 110 may first warp a first image of the plurality of images toward a second image of the plurality of images. For example, one of the first image and the second image may be rotated according to the yaw, pitch, and roll of the host vehicle.

At step 1806, processing unit 110 may select a grid of points in either the first image or the second image as a grid of reference points. For example, the grid can be formed from any shape such as an ellipse, rectangle, trapezoid, and the like. Alternatively, a random distribution of points may be chosen.

At step 1808, processing unit 110 may find patches around the selected grid. For example, the patches may be of uniform size and shape. Alternatively, the patches may vary and/or may vary randomly.

At step 1810, processing unit 110 may track the selected grid of points, for example utilizing normalized correlation computations based on patches. From the tracked points, processing unit 110 may select a subset of points with the highest score based on tracking.

At step 1812, processing unit 110 may fit the tracked points to the homography. In some embodiments, processing unit 110 may utilize a random subset of tracked points to compute multiple homographies. In such an embodiment, processing unit 110 may maintain a random subset with the highest scoring homography.

At step 1814, processing unit 110 may utilize the homography of step 1812 to correct the initial distortion. For example, processing unit 110 may utilize a random subset of points with the highest scoring homography to warp a first image of the plurality of images back toward a second image of the plurality of images. Processing unit 110 may further compute the least squares homography directly from the re-distorted image. Those skilled in the art will appreciate that other algorithms for calculating road homography may be used.

[One-way road detection based on direction of parked vehicle]

Systems and methods for identifying a vehicle and identifying the front and/or rear of the identified vehicle may enable detection of one-way roads. By utilizing the identified front and/or rear of the vehicle, one-way roads can be detected without interpreting signs or even when there are no vehicles running on the road. Embodiments of the present disclosure described below relate to systems and methods for detecting a one-way road based on detection of a parked vehicle.

19 is an exemplary functional configuration diagram of memories 140 and/or 150 in which instructions for performing one or more operations according to the disclosed embodiments may be stored/programmed. Although memory 140 is discussed below, those skilled in the art will recognize that instructions may be stored in memory 140 and/or 150 .

As shown in FIG. 19 , the memory 140 may store a vehicle identification module 1902 , a direction determination module 1904 , a vehicle side identification module 1906 , and a driving response module 1908 . The described embodiments are not limited to the particular configuration of memory 140. Additionally, applications processor 180 and/or image processor 190 may execute instructions stored in one or more of modules 1902-1908 included in memory 140. It will be understood by those skilled in the art that the following information described through the processing unit 110 may also apply to the application processor 180 and the image processor 190 individually or collectively. Accordingly, the steps of the process below may be performed by one or more processing units.

In one embodiment, the vehicle identification module 1902 includes instructions (computer vision) that, when executed by processing unit 110, perform analysis of one or more images obtained by one of image capture devices 122, 124, 126. software, etc.) As described in connection with FIGS. 9-14 above, the vehicle identification module 1902 may include instructions for determining the bounding boxes of one or more vehicles.

In one embodiment, orientation determination module 1904, when executed by processing unit 110, provides instructions (computer vision) to perform an analysis of one or more images obtained by one of image capture devices 122, 124, 126. software, etc.) As described in connection with FIGS. 20A and 20B below, the direction determination module 1904 may include instructions for determining the forward direction of the identified vehicle.

For example, the forward direction may indicate whether the identified vehicle that is parallel parked is facing the host vehicle with the front of the host vehicle. For another example, a forward direction or "oblique direction" may indicate whether an identified vehicle parked in an angled parking spot is angled towards the host vehicle. In this example, the forward direction may further indicate whether the identified vehicle was front parked or rear parked in an oblique parking spot.

In one embodiment, vehicle side identification module 1906, when executed by processing unit 110, provides instructions (computer vision software, etc.) As described in connection with FIGS. 9-14 above, the vehicle side identification module 1906 may include instructions for classifying the identified bounding boxes of one or more vehicles.

In one embodiment, the navigation response module 1908 determines the required navigation response based on data derived from executing vehicle identification module 1902, direction determination module 1904, and/or vehicle side identification module 1906. To do so, software executable by the processing unit 110 may be stored. For example, the navigation response module 1908 can cause a navigation change according to the method 2100 of FIG. 21 described below.

Further, one or more of the modules (eg, 1902, 1904, 1906) described herein may implement a learning system (eg, a neural network or a deep neural network) or a manner in conjunction with a non-learning system.

20A illustrates a one-way road 2002 from the perspective of the system according to the disclosed embodiments. Road 2002 can include a first plurality of stationary vehicles on one side (eg, first vehicle 2004) and a second plurality of stationary vehicles (eg, second vehicle 2006) on the other side. can include As described below in method 2100 of FIG. 21 , the system may determine forward directions of vehicles in a first plurality of stationary vehicles and determine forward directions of vehicles in a second plurality of stationary vehicles.

As shown in FIG. 20A , the system may determine that road 2002 is a one-way road if the forward directions of the first plurality of stopped vehicles and the second plurality of stopped vehicles are both the same. In another embodiment where one side of road 2002 is skewed parking rather than parallel parking, the system determines that road 2002 is a one-way street if the forward direction of the parallel parking side is the same as the skewed direction of the oblique parking side. can do. In another embodiment where both sides of the road 2002 are skewed parking, the system determines that the road 2002 is a one-way road if the first plurality of stationary vehicles and the second plurality of stationary vehicles both have the same slope direction. can be judged. In certain aspects, this determination may be different depending on whether the first and/or second plurality of stationary vehicles are front parked or rear parked in an oblique parking spot.

20B also shows a one-way road 2008 from the perspective of the system according to the described embodiment. Similar to road 2002, road 2002 can include a first plurality of stationary vehicles (eg, first vehicle 2010) on one side and a second plurality of stationary vehicles (eg, first vehicle 2010) on the other side. A second vehicle 2012). As described below with respect to method 2100 of FIG. 21 , the system may determine forward directions of vehicles in a first plurality of stationary vehicles and determine forward directions of vehicles in a second plurality of stationary vehicles. there is.

As shown in FIG. 20B , the system may determine that a vehicle (eg, 2010) is incorrectly parked if the forward direction of the vehicle (eg, 2010) is different from that of the plurality of associated vehicles. For example, a vehicle may be determined to be incorrectly parked if the number of other vehicles heading in the other direction is greater than a threshold value. As another example, the system may determine that a vehicle is incorrectly parked if the ratio of vehicles heading in the same direction to other vehicles heading in a different direction is above a threshold (e.g., greater than 50%, greater than 60%, greater than 70%, etc.). It can be judged that there is In some embodiments, this determination may be utilized to issue a traffic ticket to the owner or driver of vehicle 2010 (or to command the driver of the host vehicle to issue a traffic ticket).

21 is a flow chart illustrating an exemplary process 2100 for invoking one or more navigational responses based on detecting whether the road the host vehicle is driving on is a one-way road in accordance with the described embodiments. At step 2102 , processing unit 110 may receive at least one image of the environment of the host vehicle via data interface 128 . For example, a camera included in image acquisition unit 120 (eg, image capture device 122 , 124 , 126 having a field of view 202 , 204 , 206 ) may capture at least a portion of the front area and/or side of the host vehicle. One image may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless, Bluetooth, etc.).

At step 2104, processing unit 110 may analyze the at least one image to identify a first plurality of vehicles on a side of the road. For example, identifying the first plurality of vehicles may further include associating a bounding box with a shape of the first plurality of vehicles. For example, the analysis may be performed using the learning algorithm described above with reference to FIGS. 9 and 10 .

Step 2104 may further include identifying a side of at least one vehicle of the first plurality or at least one vehicle of the second plurality based on the analysis of the at least one image. For example, the analysis may be performed using the learning algorithm described above with reference to FIGS. 9 and 10 . In some embodiments, identification of the aspect may be based on at least two characteristics associated with at least one of the first plurality of vehicles or at least one of the second plurality of vehicles. For example, features associated with a vehicle may include mirrors, windows, door handles, door shape, number of doors, slope of front and/or rear glass, and the like. In some embodiments, the identified side may be the right side. In another embodiment, the side identified may be the left side.

At step 2106, processing unit 110 may analyze the at least one image to identify a second plurality of vehicles on the other side of the road. Step 2106 may be performed similarly to and/or concurrently with step 2104. At step 2108, processing unit 110 may determine a first forward direction for a first plurality of vehicles. In some embodiments, the first plurality of vehicles may all be in the same forward direction. In other embodiments, the forward direction may be different.

At step 2110, processing unit 110 may determine a second forward direction for a second plurality of vehicles. In some embodiments, the second plurality of vehicles may all be in the same forward direction. In other embodiments, the forward direction may be different.

At step 2112, processing unit 110 may cause a change in driving of the host vehicle. For example, the driving response may include a turn (shown in FIG. 13 ), a lane change, a change in acceleration (eg, application of the host vehicle's brakes), and the like. Processing unit 110 may trigger one or more navigational responses based on the determination made in step 2112 . For example, processing unit 110 may determine that the road is a one-way road based on the first and second forward directions. Based on this determination, the processing unit may effect a deceleration or stop of the host vehicle and/or a U-turn.

In some embodiments, method 2100 may include additional steps. For example, method 2100 may include receiving a drive command to cause the host vehicle to drive from a first road on which the host vehicle is traveling to a second road. The driving command may include a command to turn the host vehicle onto the second road, change direction to the second road, merge into a ramp entering the second road, and the like.

In this embodiment, the method 2100 may further include determining that both the first forward direction and the second forward direction are opposite to the traveling direction the host vehicle will travel when the host vehicle turns onto the second road. can For example, processing unit 110 may analyze the image of the vehicle on the second road to determine the first and second traveling directions, and then determine whether the direction is opposite to the expected traveling direction of the host vehicle. When it is determined that both the first forward direction and the second forward direction are opposite to the driving direction in which the host vehicle will travel when the host vehicle turns onto the second road, the driving command may be stopped. For example, since processing unit 110 determines that the second road is a one-way road in the opposite direction to the expected direction of travel of the host vehicle, processing unit 110 may cancel the command to turn, switch, merge, etc. to the second road.

In another embodiment, method 2100 may include receiving an override instruction to resume an aborted drive instruction. For example, an override instruction may be initiated based on manual confirmation received from a user inside the host vehicle, access to map data, crowdsourced data regarding the direction of travel on a second road, etc. .

Processing unit 110 may also utilize data derived from execution of speed and acceleration module 406 to generate one or more navigational responses. Multiple travel responses can occur simultaneously, sequentially, or a combination thereof.

[Status prediction of parked vehicle based on thermal profile]

A system and method for predicting the state of a parked vehicle based on its heat profile enables prediction of vehicle motion before the parked vehicle begins to move. This allows the system to identify predicted motion and proactively respond to it, rather than waiting for motion to actually be detected as in conventional motion detection. Embodiments of the present disclosure described below relate to systems and methods for predicting the condition of a parked vehicle based on a thermal profile.

22 is an exemplary functional configuration diagram of memories 140 and/or 150 in which instructions for performing one or more operations according to the disclosed embodiments may be stored/programmed. Although memory 140 is discussed below, those skilled in the art will recognize that instructions may be stored in memory 140 and/or 150 .

As shown in FIG. 22 , the memory 140 may store a visual-infrared alignment module 2202, a vehicle identification module 2204, a state prediction module 2206, and a driving response module 2208. The described embodiments are not limited to the particular configuration of memory 140. Additionally, applications processor 180 and/or image processor 190 may execute instructions stored in one or more of modules 2202-2208 included in memory 140. It will be understood by those skilled in the art that the following information described through the processing unit 110 may also apply to the application processor 180 and the image processor 190 individually or collectively. Accordingly, the steps of the process below may be performed by one or more processing units.

In one embodiment, the visual to infrared alignment module 2202, when executed by processing unit 110, converts one or more visual images obtained by one of image capture devices 122, 124, 126 to image capture device 122. , 124, 126) may store instructions (such as computer vision software) for aligning with one or more infrared images acquired by one of them. For example, the visual to infrared alignment module 2202 can include instructions for executing the method 2500 of FIG. 25 described below.

In one embodiment, vehicle identification module 2204 provides instructions (computer vision) that, when executed by processing unit 110, perform analysis of one or more images obtained by one of image capture devices 122, 124, 126. software, etc.) As described in connection with FIGS. 9-14 above, the vehicle identification module 2204 may include instructions for determining the bounding boxes of one or more vehicles.

In one embodiment, state prediction module 2206, when executed by processing unit 110, performs analysis of one or more aligned images from visual-infrared alignment module 2202 to determine the state of one or more identified vehicles. can store instructions (such as computer vision software) to predict For example, condition prediction module 2206 can output a predicted condition based on the identified vehicle's visual and thermal indicators.

In one embodiment, the driving response module 2208 determines the required driving response based on data derived from executing the visual-infrared alignment module 2202, vehicle identification module 2204, and/or state prediction module 2206. It may store software executable by the processing unit 110 to determine. For example, the navigation response module 2208 can trigger a navigation change according to the method 2400 of FIG. 24 described below.

Furthermore, one or more of the modules (eg, 2202, 2204, 2206) described herein may implement a learning system (eg, a neural network or a deep neural network) or a manner in conjunction with a non-learning system.

23A shows a parked vehicle 2302 from the perspective of the system according to the disclosed embodiments. For example, the system may monitor vehicle 2302 for changes in lighting conditions and/or temperature characteristics. The system may determine a predicted state of the vehicle 2302 based on lighting condition changes and/or temperature characteristics utilizing method 2400 of FIG. 24 , for example.

23B shows a parked vehicle 2302 with a changed lighting condition. In the example of FIG. 23B , the taillights 2304a and 2304b of the vehicle 2302 have changed from off to on. Other embodiments in which the illumination state of the headlamp is changed are also possible. Other embodiments are also possible in which the headlights and/or taillights of the vehicle change from an on state to an off state.

23C shows a parked vehicle 2302 (viewed from another angle) with a warm engine 2306 and cold tires 2308a and 2308b. Here, the expressions 'warm' and 'cold' may be terms denoting a deviation from an expected temperature that may be set and/or learned in advance. For example, an engine can be described as 'warm' if it is above the ambient temperature, and 'cold' if it is less than or equal to the ambient temperature. In embodiments of this disclosure, reference to an engine or engine temperature may refer to a specific area of a vehicle whose temperature is often affected by that of the engine, such as, for example, an engine hood located at the front of the vehicle. . The 'warm' and 'cold' temperature thresholds can be selected to reflect the expected temperature of the engine hood under the specific conditions required, possibly allowing some tolerance to reduce false positive or false negative detection. In one example, under sunny conditions, the threshold temperature may be adjusted to account for heating from sunlight (eg, after determining that the vehicle's hood is exposed to sunlight). In another example, the effect of the sun can be taken into account given the color of the vehicle, which can be determined by spectral analysis of the image. In another example, the threshold may be determined by averaging the temperature of a previously detected parked vehicle and/or a specific area of one or more parked vehicles within the local area of the currently monitored vehicle.

Similarly, tire temperature can be used as an indicator of the condition of the vehicle. For example, a tire may be 'warm' if it is higher than the road temperature, or 'cold' if it is lower than or equal to the road temperature. In another example, the threshold temperature (eg, the temperature that distinguishes a 'warm' tire from a 'cold' tire) may be related to an expected or calculated operating temperature of the tire. In addition to the ambient conditions, the operating temperature calculation can take into account the driving model and the effect of the driving model on the tire temperature. In another example, the operating temperature calculation may also include a tire cooling model. Tire cooling models can also account for ambient conditions. In the example of FIG. 23C , the system can determine a predicted state of vehicle 2302 based on the temperatures of engine 2306 and tires 2308a, 2308b utilizing, for example, method 2400 of FIG. 24 .

24 is a flowchart illustrating an exemplary process 2400 for determining the predicted state of a parked vehicle in accordance with the disclosed embodiments. At step 2402 , processing unit 110 may receive a plurality of images of the surrounding environment of the host vehicle via data interface 128 . For example, a camera included in image acquisition unit 120 (e.g., image capture devices 122, 124, 126 having fields of view 202, 204, 206) may be used to view multiple views of the front area and/or side of the host vehicle. An image of may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless, Bluetooth, etc.).

A plurality of images may be collected over a period of time. For example, the plurality of images may include a first image captured at time t = 0, a second image captured at t = 0.5 second, and a third image captured at t = 1.0 second. The time between images may vary depending on the scan rate of at least one image capture device.

At step 2404, processing unit 110 may analyze the plurality of images to identify a parked vehicle. For example, identifying the parked vehicle may further include associating at least one bounding box with the profile of the target vehicle. For example, the analysis may be performed using a learning algorithm as described above with reference to FIGS. 9 and 10 . In some embodiments, step 2404 may further include identifying an engine in front of the identified target vehicle and/or wheels on the side of the identified target vehicle. For example, the analysis may be performed using the learning algorithm described above with reference to FIG. 11 .

At step 2406, processing unit 110 may analyze the plurality of images to identify changes in lighting conditions of the parked vehicle. For example, processing unit 110 may identify taillights and/or headlights of a parked vehicle and monitor whether the identified taillights and/or headlights change from off to on (example in FIG. 23B ) or from on to off. Other embodiments may include more detailed changes, such as changing from lighting only the parking lights to lighting of the brake lights, for example.

In some embodiments, method 2400 may include determining the predicted state of the parked vehicle based on the change in lighting conditions and skip to step 2412 (i.e., determine the host vehicle based on the predicted state of the parked vehicle). trigger at least one driving response). For example, based on a change from off to on in the lighting state of at least one light associated with the parked vehicle, processing unit 110 may indicate that the predicted state of the parked vehicle is an indication that the parked vehicle's engine has been started. It can be judged to include. Similarly, based on a change from an on to an unlit state of the lighting state of at least one light associated with the parked vehicle, processing unit 110 determines that the predicted state of the parked vehicle is an indication that the engine of the parked vehicle has been turned off. It can be judged to include. Accordingly, in such an embodiment, this determination may not utilize a thermal image.

At step 2408 , processing unit 110 may receive at least one thermal image (ie, an infrared image) of the environment of the host vehicle via data interface 128 . For example, a camera included in image acquisition unit 120 (eg, image capture device 122 , 124 , 126 having a field of view 202 , 204 , 206 ) may capture at least a portion of the front area and/or side of the host vehicle. A single thermal image may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless, Bluetooth, etc.).

Step 2408 may be performed separately or concurrently with step 2402. Thus, in some embodiments, visual images and infrared images may be received simultaneously. In another embodiment, there may be a delay between the visual image and the infrared image by varying the scan rate and/or transfer rate between the one or more image capture devices and the one or more infrared image capture devices.

In some embodiments, method 2400 may further include aligning at least one thermal image with at least one of the plurality of images. In accordance with this alignment, method 2400 may further include identifying at least one of an engine region and at least one wheel component region of the parked vehicle in the aligned at least one thermal image. there is.

At step 2410, processing unit 110 may determine a predicted state of the parked vehicle based on the analysis of step 2406 and/or the analysis of the at least one thermal image. For example, the predicted state may include an indication that the engine of the parked vehicle is turned on or an indication that the engine of the parked vehicle is turned off. For another example, the prediction state indicates that a parked vehicle is not expected to move within a predetermined time, an indication that a parked vehicle is expected to move within a predetermined time, or a door of a parked vehicle is expected to open within a predetermined time. may contain indications.

In some embodiments, analyzing the at least one thermal image may include determining a temperature of an engine area of the parked vehicle. For example, if the temperature is below a threshold, processing unit 110 may determine a prediction condition that includes an indication that the parked vehicle is not expected to move within a predetermined amount of time. The predetermined time may vary, for example, depending on known characteristics of the parked vehicle or the engine of the parked vehicle.

In some embodiments, analyzing the at least one thermal image may include determining a first temperature of an engine region of the parked vehicle and a second temperature of at least one wheel component of the parked vehicle. can In this embodiment, the predicted state of the parked vehicle may be determined based on a comparison of the first temperature with the first threshold value and the comparison of the second temperature with the second threshold value. For example, if the first temperature is higher than the first threshold and the second temperature is lower than the second threshold, processing unit 110 determines a prediction condition that includes an indication that the parked vehicle is expected to move within a predetermined amount of time. can do.

In another example, if the first temperature is higher than the first threshold and the second temperature is higher than the second threshold, processing unit 110 makes a prediction including an indication that the door of the parked vehicle is expected to open within a predetermined amount of time. status can be judged.

In some embodiments, processing unit 110 may perform additional monitoring of the received image based at least in part on a predicted condition based on a change in illumination, analysis of at least one thermal image, or a combination thereof. can For example, if the predicted condition indicates that the engine of the parked vehicle is off and/or that the door of the parked vehicle is expected to open within a predetermined time, processing unit 110 monitors one or more portions of the received image. Accordingly, it is possible to determine whether there is a change in the image characteristics of the door edge of the parked vehicle. An example of such monitoring is described above with respect to method 1400 of FIG. 14 . For another example, if the predicted condition indicates that the engine of the parked vehicle is being started and/or that the parked vehicle is expected to move within a predetermined time, processing unit 110 may determine one or more wheel elements of the received image. component) to check the movement of the wheel component. An example of such monitoring is described above with respect to method 1700 of FIG. 17 .

At step 2412, processing unit 110 may cause a change in driving of the host vehicle. For example, the driving response may include a change in the direction of travel of the host vehicle (shown in FIG. 13 ), a lane change, a change in acceleration, and the like. Processing unit 110 may perform step 2410 to trigger one or more driving responses based on the determined predicted condition. For example, processing unit 110 may move and/or move the host vehicle from a parked vehicle in response to a predicted condition indicating that the parked vehicle is expected to move within a predetermined time or that a door of the parked vehicle is expected to open within a predetermined time. Alternatively, the host vehicle may be decelerated.

25 is a flow chart illustrating an exemplary process 2500 for aligning a visual image with an infrared image. At steps 2502 and 2504, processing unit 110 may receive at least one visual image and at least one infrared image of the environment of the host vehicle via data interface 128. For example, a camera included in image acquisition unit 120 (eg, image capture device 122 , 124 , 126 having a field of view 202 , 204 , 206 ) may capture at least a portion of the front area and/or side of the host vehicle. One visual image and one or more infrared images may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless, Bluetooth, etc.).

In some embodiments, visual images and infrared images may be received simultaneously. In another embodiment, there may be a delay between the visual image and the infrared image by varying the scan rate and/or transfer rate between the one or more image capture devices and the one or more infrared image capture devices.

At step 2506, processing unit 110 may select a set of fiducials in at least one infrared image. For example, a set of fiducials may include identification of known objects (eg, pedestrians, trees, vehicles, etc.) that are randomly selected or based on known characteristics.

At step 2508, processing unit 110 may project a reference point from the at least one infrared image to the at least one visual image. For example, processing unit 110 may project a shape (eg, ellipse, rectangle, trapezoid, etc.) representing (eg, enclosing) a reference point to a location within the at least one visual image.

At step 2510, processing unit 110 may optimize gain and/or exposure for the portion of the visual image corresponding to the reference point. For example, improved contrast as a result of optimization can further improve alignment reliability. Step 2510 is optional and need not be performed in all embodiments.

At step 2512, processing unit 110 may align the at least one infrared image with the at least one visual image. For example, aligning (or matching) the images may include searching along the epipolar line for a distance that optimizes the degree of alignment. In this example, by optimizing the degree of alignment, the distance between the fiducial and the viewer and/or between the fiducial and another object can be the same in both the visual and infrared images.

[Travel control based on distance detection between vehicles]

Systems and methods for identifying gaps between vehicles may enable driving control based on the detected gaps. By performing driving control through this method, preliminary monitoring and reaction to motion within the detected interval can be performed in a shorter reaction time than conventional motion detection. Embodiments of the present disclosure described below relate to a system and method for driving control based on distance detection between vehicles.

26 is an exemplary functional configuration diagram of memories 140 and/or 150 in which instructions for performing one or more operations according to the disclosed embodiments may be stored/programmed. Although memory 140 is discussed below, those skilled in the art will recognize that instructions may be stored in memory 140 and/or 150 .

As shown in FIG. 26 , the memory 140 may store an image analysis module 2602 , a vehicle identification module 2604 , an interval calculation module 2606 , and a driving response module 2608 . The described embodiments are not limited to the particular configuration of memory 140. Also, the application processor 180 and/or the image processor 190 may execute instructions stored in one or more of the modules 2602 to 2608 included in the memory 140 . It will be understood by those skilled in the art that the following information described through the processing unit 110 may also apply to the application processor 180 and the image processor 190 individually or collectively. Accordingly, the steps of the process below may be performed by one or more processing units.

In one embodiment, image analysis module 2602 provides instructions (computer vision) that, when executed by processing unit 110, perform analysis of one or more images obtained by one of image capture devices 122, 124, 126. software, etc.) In some embodiments, processing unit 110 may perform Easy analysis by merging information from a set of images with additional sensory information (eg, information from radar, lidar, etc.). As described below with respect to vehicle identification module 2604, image analysis module 2602 may include instructions for detecting a vehicle utilizing one or more features (eg, front, rear, side, etc.) .

In one embodiment, vehicle identification module 2604 provides instructions (computer vision) that, when executed by processing unit 110, perform analysis of one or more images obtained by one of image capture devices 122, 124, 126. software, etc.) As described in connection with FIGS. 9-14 above, the vehicle identification module 2604 may include instructions for determining the bounding boxes of one or more vehicles.

In one embodiment, interval calculation module 2606, when executed by processing unit 110, performs an analysis of one or more images obtained by one of image capture devices 122, 124, 126 (computer vision). software, etc.) As described with respect to FIGS. 27A and 27B below, also in cooperation with vehicle identification module 2604, distance calculation module 2606 may include instructions for calculating one or more distances between identified vehicles. there is.

In one embodiment, the navigation response module 2608 determines a desired navigation response based on data derived from executing the image analysis module 2602, vehicle identification module 2604, and/or distance calculation module 2606. Software executable by the risk processing unit 110 may be stored. For example, the navigation response module 2608 can cause a navigation change according to the method 2800 of FIG. 28 described below.

Further, one or more of the modules (eg, 2602, 2604, and 2606) described herein may implement a learning system (eg, a neural network or a deep neural network) or a method in conjunction with a non-learning system.

27A shows a road 2702 from the perspective of the system according to the disclosed embodiments. Road 2702 can include a plurality of stationary vehicles (eg, 2704, 2706). As described below with reference to method 2800 of FIG. 28 , the system creates a bounding box for the side of a stationary vehicle, e.g., a side bounding box for vehicle 2704. 2708 and side bounding box 2710 for vehicle 2706 can be identified. As further described below with reference to method 2800 of FIG. 28 , the system uses the front end of one bounding box and the back end of an adjacent bounding box, for example, a bounding box. The front 2712 of the box 2708 and the back 2714 of the bounding box 2710 can be used to identify the gap between the identified bounding boxes.

27B also shows road 2702 including vehicle 2704 and vehicle 2706 as seen from the perspective of the system according to the described embodiment. Similar to FIG. 27A , the system has identified a side bounding box 2708 for vehicle 2704 and a side bounding box 2710 for vehicle 2706 . As shown in FIG. 27B, a hotspot 2716 is identified based on the gap identified between the front 2712 of the bounding box 2708 and the back 2714 of the bounding box 2710. identified As described below with reference to method 2800 of FIG. 28 , the system can determine a driving response of the host vehicle based on the identified hotspot.

28 is a flow chart illustrating an exemplary process 2800 for driving control based on detected spacing between vehicles according to the disclosed embodiments. At step 2802 , processing unit 110 may receive a plurality of images of the surrounding environment of the host vehicle via data interface 128 . For example, a camera included in image acquisition unit 120 (e.g., image capture devices 122, 124, 126 having fields of view 202, 204, 206) may be used to view multiple views of the front area and/or side of the host vehicle. An image of may be captured and transmitted to the processing unit 110 through a digital connection (eg, USB, wireless, Bluetooth, etc.). Additional information from other sensors such as radar, lidar, acoustic sensors, etc. may also be utilized in lieu of or in combination with multiple images.

At step 2804, processing unit 110 may analyze at least one of the plurality of images to identify at least two stopped vehicles. For example, identifying the stationary vehicle may further include associating at least one bounding box with the profile of the stationary vehicle. For example, the analysis may be performed using a learning algorithm as described above with reference to FIGS. 9 and 10 .

At step 2806, processing unit 110 may determine the spacing between the identified vehicles. For example, processing unit 110 may scan the at least one image in a left-to-right direction (or right-to-left direction) to identify a right edge and an adjacent left edge. The identified right edge and the identified left edge may include the frontmost edge of one bounding box and the rearmost edge of another bounding box. The right edge and the left edge form an interval pair, and the interval between this interval pair can be calculated. In such an embodiment, the distance may correspond to the distance between the front of one of the stationary vehicles and the rear of another stationary vehicle.

In another embodiment, the identified right edge and the identified left edge may include one bounding box on one side of the road and another bounding box on the other side of the road. In this embodiment, the spacing may correspond to the distance between adjacent sides of a stationary vehicle.

In some embodiments, step 2806 may further include calculating a distance between the host vehicle and the determined gap. For example, processing unit 110 may calculate the distance based on the height and focal length of the host vehicle's image capture device (eg, camera). Based on known characteristics (eg, height or height) of the pedestrian or other object, processing unit 110 may determine features within and/or near the calculated interval as 'hotspots' from which the pedestrian or other object may emerge. there is. For example, this shape may be a rectangle, an ellipse, or other shape.

At step 2808, processing unit 110 may cause a change in driving of the host vehicle. For example, the driving response may include turning (shown in FIG. 13 ), changing lanes (eg, moving the host vehicle within the driving lane or changing the operating lane of the host vehicle), changing acceleration (eg, decelerating the host vehicle), etc. can include In some embodiments, the at least one travel change may be performed by activating one of the host vehicle's steering mechanism, brakes, or accelerator.

Processing unit 110 may trigger the at least one travel response based on the interval calculated in step 2806. For example, processing unit 110 may determine that the calculated spacing is sufficient to accommodate a pedestrian. Based on this determination, the processing unit may decelerate the host vehicle and/or move the host vehicle away from the gap. In other words, the driving change may be executed when it is determined that the distance between the two stopped vehicles is sufficient for a pedestrian to cross.

For another example, processing unit 110 may determine that the calculated spacing is sufficient to accommodate the vehicle. Based on this determination, the processing unit may decelerate the host vehicle and/or move the host vehicle away from the gap. In other words, the driving change may be executed when it is determined that the gap between the two stopped vehicles is sufficient for the target vehicle to cross.

For another example, based on monitoring of the hotspot, processing unit 110 may slow down and/or host the host vehicle if a pedestrian or other object is identified in or near the hotspot or if motion is detected in or near the hotspot. The vehicle can be moved away from the gap.

In some embodiments, processing unit 110 may then detect a pedestrian within the gap between the two stopped vehicles based on analysis of the plurality of images. For example, processing unit 110 may utilize the calculated distance and the expected height of the pedestrian to determine where within the received image the pedestrian's head may be expected to appear (e.g., hotspot 2716 in FIG. 27B). can do.

Detection of the pedestrian may be performed utilizing a classifier learned for the pedestrian, similar to the classifier previously learned for the side of the vehicle. In this example, if the classifier score of a test point inside the hotspot is higher than the upper threshold, processing unit 110 may detect an identified pedestrian at that point. On the other hand, if the classifier score of a test point inside the hotspot is higher than the lower threshold and lower than the upper threshold, processing unit 110 may detect a suspected pedestrian at that point. The suspected walk may further be tracked as to whether or not the person is moving towards the roadway, and if the motion toward the road is tracked, the processing unit may reclassify the suspect as a confirmed pedestrian. This detection method can improve existing motion detection.

In some embodiments, at least a portion of the gap between the two stationary vehicles may be obscured from the camera's view. In such an embodiment, processing unit 110 may perform additional analysis to compensate for the obscured areas.

Processing unit 110 may also utilize data derived from execution of speed and acceleration module 406 to generate one or more navigational responses. Multiple travel responses can occur simultaneously, sequentially, or a combination thereof.

The foregoing description has been presented for purposes of illustration. This description is not exhaustive and is not limited to the forms or examples as described. Modifications and adaptations will occur to those skilled in the art from consideration of the present disclosure and practice of the described embodiments. Further, while aspects of the disclosed embodiments have been described as being stored in memory, those skilled in the art know that such aspects may be stored on, for example, a hard disk or CD ROM, or other type of RAM or ROM, USB media, DVD, Blu-ray, 4K UHD Blue It will be appreciated that it may also be stored on other types of computer readable media, such as secondary storage such as a storage device, memory card, or other optical drive media.

A computer program based on the described descriptions and methods is well within the skill of those skilled in the art. Various programs or program modules can be created using techniques known to those skilled in the art or designed in conjunction with existing software. For example, a program section or program module may be .Net Framework, .Net Compact Framework (and related languages such as Visual Basic, C, etc.), Java, C++, Objective-C, HTML, HTML/AJAX combinations, XML, or Java applets. (Java applet) can be designed in HTML.

Further, while illustrative embodiments have been described herein, it is intended that all embodiments are within the scope of equivalent elements, modifications, omissions, combinations (eg, combinations of aspects across various embodiments), adaptations, and/or variations. It is natural for those skilled in the art. The limitations of the claims are to be interpreted broadly based on the language used in the claims and are not limited to the examples described herein or during the prosecution of the present invention. It should be understood that the examples are not exclusive. Furthermore, the steps of a described method may be modified in various ways, such as rearranging the order of steps and/or inserting or deleting steps. Accordingly, this specification and examples are to be considered for purposes of illustration only, with the true scope and spirit being defined by the following claims and their equivalents.

100: system
110: processing unit
120: image acquisition unit
122, 124, 126: image capture device
130: position sensor
140, 150: memory device
160: map database
170: user interface
172: radio transceiver
180: application processor
190: image processor
200, 1106, 1204: vehicle
1102, 1104, 1604, 1606: wheel
1502 road homography module
1504: wheel identification module
1506: motion analysis module
1508: driving response module

Claims (20)

A system for navigating a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is running, comprising:
The system includes at least one processing device, the at least one processing device comprising:
receiving a plurality of images associated with the environment of the host vehicle from an image capture device;
analyzing at least one of the plurality of images to identify the target vehicle and at least one wheel component of the side of the target vehicle;
identifying an action associated with the at least one wheel element of the target vehicle by analyzing a region including the at least one wheel element of the target vehicle in at least two images of the plurality of images; and
cause at least one navigational change in the host vehicle based on the identified motion associated with the at least one wheel element of the target vehicle;
Characterized in that, the system is programmed to do. According to claim 1,
wherein the at least one travel change comprises a change in the direction of travel of the host vehicle. According to claim 1,
The system of claim 1 , wherein the at least one travel change comprises activating brakes of the host vehicle. According to claim 1,
The system of claim 1 , wherein identifying the target vehicle includes associating at least one bounding box with a shape of the target vehicle. According to claim 1,
The system of claim 1 , wherein the area containing the at least one wheel element of the target vehicle includes an area adjacent to a road surface. According to claim 1,
The system of claim 1, wherein identifying motion associated with the at least one wheel element comprises monitoring whether rotation of the at least one wheel element is indicated in at least two images of the plurality of images. According to claim 1,
Wherein the process of identifying motion associated with the at least one wheel element includes identifying at least one feature associated with the at least one wheel element and identifying an indicator of a change in position of the at least one feature. . According to claim 7,
The system of claim 1 , wherein the at least one processing unit is further configured to determine a speed at which the target vehicle is traveling based on the change in position of the at least one feature associated with the at least one wheel element. According to claim 8,
wherein said at least one processing unit is further configured to determine a lateral safety distance to said host vehicle based on said speed of said target vehicle. According to claim 9,
wherein the at least one driving change of the target vehicle is based on the determined lateral safety distance. According to claim 1,
The system of claim 1, wherein the at least one wheel element comprises at least one of a tire, a wheel cap, and a wheel structure. A method for navigating a host vehicle based on movement of a target vehicle toward a lane in which the host vehicle is driving, comprising:
receiving a plurality of images associated with the environment of the host vehicle from an image capture device;
analyzing at least one of the plurality of images to identify the target vehicle and at least one wheel component of the side of the target vehicle;
identifying an action associated with the at least one wheel element of the target vehicle by analyzing a region including the at least one wheel element of the target vehicle in at least two images of the plurality of images; and
inducing at least one navigational change in the host vehicle based on the identified motion associated with the at least one wheel element of the target vehicle. According to claim 12,
wherein identifying the target vehicle comprises associating at least one bounding box with a shape of the target vehicle. According to claim 12,
wherein identifying motion associated with the at least one wheel element comprises monitoring whether rotation of the at least one wheel element is indicated in at least two images of the plurality of images. According to claim 12,
Wherein identifying an action associated with the at least one wheel element comprises identifying at least one feature associated with the at least one wheel element and identifying an indicator of a change in position of the at least one feature. method. According to claim 15,
determining a speed at which the target vehicle is moving based on the change in position of the at least one feature associated with the at least one wheel element; and
determining a lateral safety distance to the host vehicle based on the speed of the target vehicle. A non-transitory computer readable medium storing instructions that, when executed by at least one processing unit, perform a method, the method comprising:
receiving a plurality of images associated with the environment of the host vehicle from an image capture device;
analyzing at least one of the plurality of images to identify a target vehicle and at least one wheel component of a side of the target vehicle;
identifying an action associated with the at least one wheel element of the target vehicle by analyzing a region including the at least one wheel element of the target vehicle in at least two images of the plurality of images; and
inducing at least one navigational change in the host vehicle based on the identified motion associated with the at least one wheel element of the target vehicle. According to claim 17,
wherein identifying the target vehicle comprises associating at least one bounding box with a shape of the target vehicle. According to claim 17,
Wherein identifying an action associated with the at least one wheel element comprises identifying at least one feature associated with the at least one wheel element and identifying an indicator of a change in position of the at least one feature. , a non-transitory computer readable medium. According to claim 19,
determining a speed at which the target vehicle is moving based on the change in position of the at least one feature associated with the at least one wheel element; and
and further storing instructions for determining a lateral safety distance to the host vehicle based on the speed of the target vehicle.

Images