KR102419526B1 - Field theory-based perception of autonomous vehicles - Google Patents

Abstract

The techniques described in this document may be embodied in a method comprising generating, using one or more processing devices of a vehicle operating in the environment, a discretized representation of the environment, comprising a plurality of cells. A cell is occupied by a particle representing at least one of an object or free space in the environment. Sensor data indicative of a state of at least one particle is received, and updates to a time-varying particle density function associated with the position of the particle in the dynamic occupancy grating are computed from the sensor data and the one or more models. The method comprises generating a prediction of the occupancy of at least one cell in the discretized representation based on the updated particle density function, and using a controller circuit of the vehicle to operate the vehicle based at least in part on the prediction. further comprising a step.

Description

Field theory-based perception of autonomous vehicles

delete

delete

field of invention

This description relates to autonomous vehicles, such as self-driving cars, aircraft, boats, and other vehicles.

The autonomous vehicle includes specialized processing circuitry (sometimes referred to as cognitive circuitry) that incorporates input from one or more sensors to determine the position of an object in the environment surrounding the autonomous vehicle. The information determined about the environment can be used to drive the vehicle.
Danescu, Radu & Oniga, Florin & Nedevschi, Sergiu. (2012), “Modeling and Tracking the Driving Environment With a Particle-Based Occupancy Grid,” Intelligent Transportation Systems, IEEE Transactions on. 12. 1331-1342. 10.1109/TITS.2011.2158097 relates to an occupancy grid tracking solution for tracking dynamic driving environments based on particles.

In an aspect, the document features a method comprising, using one or more processing devices of a vehicle operating in the environment, generating a discretized representation of the environment, the discretized representation comprising a plurality of cells do. Each cell of the plurality of cells is occupied by a particle representing at least one of an object or free space in the environment. The method includes receiving, from one or more sensors of a vehicle, sensor data indicative of a state of at least one of a plurality of particles in an environment, using the one or more processing devices, the sensor data and the one or more sensors; determining an update to the time-varying particle density function associated with the position of the at least one particle in the dynamic occupancy grating from the associated one or more models. The method comprises generating, using one or more processing devices, a prediction of the occupancy of at least one cell in the discretized representation based on the updated particle density function, and, using the vehicle's controller circuitry, the prediction. The method further includes operating the vehicle based, at least in part, on the vehicle.

In another aspect, this document features one or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause performing the method.

In another aspect, the document includes instantiating, using processing circuitry mounted on one or more sensors, a set of interacting software components representing the contents of a cell of a discrete representation of an environment. characterized by the method. The method also includes receiving, from the one or more sensors, a sensor observation vector associated with an interactive software component of a set of interactive software components. A sensor observation vector includes one or more parameters associated with an interactive software component at a given location. The method includes, using processing circuitry, determining, from a vector of sensor observations based on one or more models associated with the sensor, an update to a time-varying particle density function associated with a cell of the cells of the discretized representation corresponding to a given location, the processing circuitry comprising: generating a prediction of the occupancy of the cell based on the updated particle density function, using the processing circuit, and augmenting, using the processing circuit, the operation of the vehicle based on the prediction.

In another aspect, the present document includes one or more computer processors; and one or more non-transitory storage media for storing instructions. The instructions cause, when executed by one or more processors, to perform various operations. The operations include generating a discretized representation of the environment, the discretized representation comprising a plurality of cells. Each cell of the plurality of cells is occupied by a plurality of particles representing at least one of an object or free space in the environment. These operations include receiving, from one or more sensors of the vehicle, sensor data indicative of a state of at least one of a plurality of particles in the environment, and in a dynamic occupancy grid from the sensor data and one or more models associated with the one or more sensors. Also included is determining an update to the time-varying particle density function associated with the position of the at least one particle. The operations further include generating a prediction of the occupancy of the at least one cell in the discretized representation based on the updated particle density function, and operating the vehicle based at least in part on the prediction.

In another aspect, this document provides, using processing circuitry mounted on one or more sensors, instantiating a set of interactive software components representing the contents of a cell of a discretized representation of an environment, and interacting, from the one or more sensors. A method comprising receiving a sensor observation vector associated with an interactive software component of a set of software components. A sensor observation vector includes one or more parameters associated with an interactive software component at a given location. The method further includes determining, using the processing circuitry, from the sensor observation vector, an update to a time-varying particle density function associated with a cell of the cells of the discretized representation corresponding to the given location. The method also includes generating, using the processing circuit, a prediction of occupancy of the cell based on the updated particle density function, and, using the processing circuit, augmenting the operation of the vehicle based on the prediction. do.

In another aspect, this document features one or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause performing the method.

In another aspect, the present document includes one or more computer processors; and one or more non-transitory storage media for storing instructions. The instructions cause, when executed by one or more processors, to perform various operations. These operations include instantiating a set of interactive software components representing the contents of a cell of the discretized representation of the environment, and receiving, from one or more sensors, a sensor observation vector associated with the interactive software component of the set of interactive software components. includes A sensor observation vector includes one or more parameters associated with an interactive software component at a given location. These operations also include determining, from the sensor observation vector, an update to a time-varying particle density function associated with a cell of the cells of the discretized representation corresponding to a given position. These operations further include generating a prediction of the occupancy of the cell based on the updated particle density function, and augmenting the operation of the vehicle based on the prediction.

Implementations of this aspect may include one or more of the following features.

The discretized representation may include a grid defined using a Cartesian coordinate system or a polar coordinate system. For each cell of the plurality of cells, an initial value of occupancy may first be assigned and then updated according to the evolution of the particle density function. A label may also be assigned to a number of particles, each label indicating whether the corresponding particle represents an object or free space. The label may be updated according to the sensor data. The operation of determining an update to the time-varying particle density function comprises, using a Eulerian solver or a Lagrangian solver, a solution of one or more differential equations defined for one or more parameters associated with a state of at least one particle. may include the operation of determining The state of the at least one particle may include at least one velocity associated with the at least one particle. The state of the at least one particle may include (i) a plurality of velocities along a corresponding direction and (ii) a covariance associated with the plurality of velocities. The state of the at least one particle may include a force acting on the at least one particle. Each of the one or more models represents a sensor model that has been trained to provide information about the occupancy probabilities of various cells in a discretized representation, the probabilities conditioned on corresponding sensor data. The operation of generating a prediction of occupancy of the at least one cell comprises determining the occupancy probability of the at least one cell by (i) a ratio of a conditional probability generated based on the sensor data to a conditional probability generated based on the one or more sensor models and (ii) ) as a product of the updated time-varying particle density function for the at least one cell. The ratio of the conditional probabilities is the ratio of (i) a first probability of receiving sensor data conditioned on the at least one cell being occupied and (ii) a second probability of receiving sensor data conditioned on the at least one cell being unoccupied. ratio, wherein the first probability and the second probability are determined using one or more sensor models. When it is determined that sensor data corresponding to a particular sensor is outside a threshold range of expected values determined using a corresponding model associated with the particular sensor, a fault condition for that particular sensor may be identified in response thereto. have. If it is determined that the sensor data is missing at least one parameter indicative of the state of the at least one particle, the previous value of the at least one parameter may be used to determine an update to the time-varying particle density function. At least a portion of the one or more processing devices may be disposed in a cognitive circuit of the vehicle.

In some embodiments, the techniques described herein may provide one or more of the following advantages.

By representing objects and free space in the environment as aggregates of particles (e.g., analogous to applying field theory in fluid mechanics) and tracing a time-varying particle density function using the Eulerian approach, The technique described in allows for improved dynamic range and, for example, potentially higher resolution with more manageable computational complexity compared to tracking individual grid cells. For example, to achieve 28-bit accuracy without using the techniques described herein, it may be necessary to track 2 ²⁸ particles per grating, which results in an enormous computational burden. In contrast, tracking a time-varying particle density function with 28-bit accuracy in an equivalent discretized space using the technique described herein is significantly lower, at least because individual particles within a given lattice cell of the discretized space need not be considered. This can be done with a computational burden. Additionally, the proposed approach is independent of any particular grating and thus does not need to be substantially modified for different gratings. Furthermore, since particles can be defined and tracked with respect to free space, the proposed technique makes it possible to track free space, which in turn could potentially improve the driving capability of autonomous vehicles. By selecting appropriate parameters for modeling particle dynamics (velocity, force, etc.), occluded or partially visible objects can be accurately tracked by tracking the corresponding particle density function.

These and other aspects, features, and implementations may be represented as a method, apparatus, system, component, program product, means, or step, and otherwise, for performing a function.

These and other aspects, features, and implementations will become apparent from the following description, including claims.

1 shows an example of an autonomous vehicle with autonomous capability.
2 illustrates an example “cloud” computing environment.
3 illustrates a computer system.
4 shows an example architecture for an autonomous vehicle.
5 shows an example of an input and output that may be used by a cognitive module.
6A shows an example of a LiDAR system.
6B shows the lidar system in operation.
6C illustrates the operation of the lidar system in further detail.
7 shows a block diagram of the relationship between the input and output of the planning module.
8 shows a directed graph used in route planning.
9 shows a block diagram of the input and output of the control module.
10 shows a block diagram of the inputs, outputs, and components of a controller.
11 illustrates a discretized representation of the environment of an autonomous vehicle, in which particles represent objects or free space in specific cells.
12 is an example of a cognitive module that may be used to implement the techniques described herein.
13A-13C show examples of user interfaces generated based on the output of a cognitive module according to techniques described herein.
14 is a flowchart of an example process for generating a prediction of occupation of a location in an environment of an autonomous vehicle.
15 is a flowchart of another example process for generating a prediction of occupancy of a location in an environment of an autonomous vehicle.

In the following description for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

In the drawings, in order to facilitate description, a specific arrangement or order of schematic elements such as those representing devices, modules, instruction blocks, and data elements is shown. However, one of ordinary skill in the art will understand that a specific order or arrangement of schematic elements in the drawings does not imply that a specific order or sequence of processing or separation of processes is required. Moreover, the inclusion of schematic elements in the drawings does not imply that such elements are required in all embodiments, or that features represented by such elements may not be included in some embodiments or combined with other elements. It does not imply that it may not be possible.

Also, in the drawings, where a connecting element such as a solid or dashed line or arrow is used to show a connection, relationship or association between two or more other schematic elements, the member of any such connecting element is a connection, relationship, or association. This does not imply that it cannot exist. In other words, some connections, relationships, or associations between elements are not shown in the figures in order not to obscure the present disclosure. Additionally, for ease of illustration, a single connecting element is used to indicate multiple connections, relationships, or associations between the elements. For example, where a connecting element represents the communication of a signal, data or command, one of ordinary skill in the art would identify one or more signal paths (e.g., For example, a bus) will be understood.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the detailed description that follows, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. It will be apparent, however, to one skilled in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail in order not to unnecessarily obscure aspects of the embodiments.

Several features are described below that may each be used independently of one another or in combination with any combination of other features. However, any individual feature may not solve any of the problems discussed above or may only solve one of the problems discussed above. Some of the problems discussed above may not be completely solved by any of the features described herein. Although multiple headings are provided, information relating to a particular heading but not found in the section having that heading may be found elsewhere in this description. Examples are described herein according to the following outline:

1. General overview

2. Hardware Overview

3. Autonomous Vehicle Architecture

4. Autonomous vehicle input

5. Route planning

6. Autonomous vehicle control

7. Field Theory-Based Cognition

general overview

This document presents a technique for improving the perception of an object and free space in the environment of an autonomous vehicle. In particular, this technique makes it possible to model objects and free space as aggregates of particles, similar to how fluids are modeled in field theory-based fluid mechanics. Particles instantiated as representations of objects and free space can be tracked by updating a time-varying particle density function over a discretized representation of the environment, such that the updated particle density function computes the occupancy probabilities of the various cells of the discretized representation. It can be used to (eg, predict).

In one embodiment, updates to the particle density function may be computed by the autonomous vehicle's cognitive module using real-time sensor observations as well as pre-computed models corresponding to the sensors. In some implementations, the sensor model may be a forward sensor model. The pre-computed model takes into account a number of parameters of the corresponding particle dynamics, including, for example, the velocity of the particles along different directions, the velocity covariance, and possibly any forces applied to the particles. Euler solvers or Lagrangian solvers are used to compute solutions of differential equations for a number of parameters used, which in turn have a higher dynamic range and higher resolution compared to other processes in which the cognitive module tracks the occupancy of the lattice cells. to create an image. Moreover, since statistics of particle density functions (rather than individual particles) are tracked, this technique makes it possible to generate such high dynamic range and high resolution perceptual images without increasing the computational burden to unacceptable levels. . The techniques described herein also avoid performance limiting assumptions, such as rigid-body and constant velocity assumptions, and allow various sensors (e.g., radars (e.g., radars) to It enables early fusion of information from RADAR) and LiDAR).

Hardware Overview

1 shows an example of an autonomous vehicle 100 having autonomous driving capability.

As used herein, the term “autonomous driving capability” refers to a vehicle partially or fully operating without real-time human intervention, including without limitation fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles. Refers to a function, feature, or facility that makes it possible.

As used herein, an autonomous vehicle (AV) is a vehicle with autonomous driving capability. As used herein, “vehicle” includes means of transport of goods or persons. For example, cars, buses, trains, airplanes, drones, trucks, boats, ships, submarines, airships, etc. A driverless vehicle is an example of a vehicle.

As used herein, “trajectory” refers to a path or route that traverses an AV from a first spatiotemporal location to a second spatiotemporal location. In one embodiment, the first spatiotemporal location is referred to as an initial or starting location and the second spatiotemporal location is referred to as a destination, final location, target, target location, or target location. In some examples, a trajectory is made up of one or more segments (eg, sections of a road), and each segment is made up of one or more blocks (eg, portions of lanes or intersections). In one embodiment, the spatiotemporal location corresponds to a real-world location. For example, the spatiotemporal location is a pick up location or drop-off location for picking up or unloading people or loading or unloading goods.

As used herein, “sensor(s)” includes one or more hardware components that detect information about the environment surrounding the sensor. Some of the hardware components include sensing components (eg, image sensors, biometric sensors), transmitting and/or receiving components (eg, laser or radio frequency wave transmitters and receivers), electronic components such as analog-to-digital converters. , data storage devices (eg, RAM and/or non-volatile storage), software or firmware components, and data processing components such as application-specific integrated circuits (ASICs), microprocessors and/or microcontrollers.

As used herein, a “scene description” is a data structure (e.g., one or more classified or labeled objects detected by one or more sensors on an AV vehicle or provided by a source external to the AV). For example, a list) or data stream.

As used herein, a “road” is a physical area that may be traversed by a vehicle and may correspond to a named major road (eg, city street, interstate freeway, etc.), or may be named It can correspond to unpaved main roads (eg, private roads within a house or office building, sections of parking lots, sections of open spaces, unpaved paths in rural areas, etc.). Because some vehicles (eg, four-wheel drive pickup trucks, sport utility vehicles, etc.) may traverse various physical areas that are not particularly suitable for vehicle driving, a "road" may be defined by any municipality or other governmental or administrative body. It may be a physical area that is not officially defined as a major road.

As used herein, a “lane” is a portion of a roadway that may be traversed by a vehicle and may correspond to most or all of the space between lane markings, or only a portion of the space between lane markings ( For example, less than 50%). For example, a road with distantly spaced lane markings may accommodate more than one vehicle between the lane markings, such that one vehicle may overtake another without crossing the lane markings, so that the lane markings It can be interpreted as having a narrower lane than the space in between or having two lanes between lane markings. Lane lanes can be interpreted even in the absence of lane markings. For example, lanes may be defined based on physical characteristics of the environment, such as rocks and trees along major roads in rural areas.

"one or more" means a function performed by one element, a function performed by more than one element, eg, in a distributed fashion, several functions performed by one element, several functions performed by several elements. function, or any combination thereof.

It will also be understood that, although the terms first, second, etc. are used herein to describe various elements, in some instances, such elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the various embodiments described. Both the first contact and the second contact are contact, but not the same contact.

The terminology used in the description of the various embodiments described herein is for the purpose of describing particular embodiments only, and is not intended to be limiting. As used in the description of the various embodiments described and the appended claims, the singular is intended to include the plural unless the context clearly indicates otherwise. It will also be understood that the term “and/or,” as used herein, refers to and includes any and all possible combinations of one or more of the listed associated items. Moreover, the terms "comprises" and/or "comprising," when used in this description, specify the presence of a recited feature, integer, step, operation, element, and/or component, but one or more other features; It will be understood that this does not exclude the presence or addition of integers, steps, acts, elements, components, and/or groups thereof.

As used herein, the term "if" is to be interpreted to mean "when," or "when," or "in response to a determination," or "in response to detecting," optionally depending on the context. do. Likewise, the phrases “if it is determined that” or “if [the stated condition or event] is detected” are, optionally, depending on the context, “in determining” or “in response to a determination” or “[the stated condition or event] ] or "in response to the detection of [the stated condition or event]".

As used herein, an AV system refers to an AV with an array of hardware, software, stored data, and real-time generated data that support the operation of the AV. In one embodiment, the AV system is contained within AV. In one embodiment, the AV system is spread across multiple locations. For example, some of the software of the AV system is implemented on a cloud computing environment similar to cloud computing environment 300 described below with respect to FIG. 3 .

In general, the present disclosure relates to fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles, eg, any having one or more autonomous driving capabilities, including so-called level 5 vehicles, level 4 vehicles and level 3 vehicles, respectively. (Details on classification of levels of autonomy in vehicles are described in SAE International Standard J3016: Taxonomy and Definitions for Terms Related to On-128 -172020-02-28 Road Motor Vehicle Automated Driving Systems). The technology disclosed herein is also applicable to partially autonomous vehicles and driver-assisted vehicles, such as so-called level 2 and level 1 vehicles (see SAE International Standard J3016: Classification and Definition of Terms Relating to On-Road Vehicle Automated Driving Systems) . In one embodiment, one or more of the level 1, level 2, level 3, level 4 and level 5 vehicle systems are capable of performing a particular vehicle operation (eg, steering, braking, and using maps) can be automated. The technology described herein may benefit vehicles at any level, from fully autonomous vehicles to human-driven vehicles.

1 , the AV system 120 avoids objects (eg, natural obstacles 191 , vehicles 193 , pedestrians 192 , cyclists, and other obstacles) and avoids road laws (eg, For example, operating the AV 100 along a trajectory 198 through the environment 190 to a destination 199 (sometimes referred to as an end location) while adhering to operating rules or driving preferences).

In one embodiment, AV system 120 includes device 101 equipped to receive operational commands from computer processor 146 and act accordingly. In one embodiment, computing processor 146 is similar to processor 304 described below with reference to FIG. 3 . Examples of the device 101 include a steering control 102 , a brake 103 , a gear, an accelerator pedal or other acceleration control mechanism, a windshield wiper, a side door lock, a window control, and a turn indicator light.

In one embodiment, AV system 120 provides AV 100 such as position, linear and angular velocity and linear and angular acceleration, and heading (e.g., orientation of the tip of AV 100) of the AV. a sensor 121 for measuring or inferring a state or attribute of a condition of Examples of sensors 121 include GPS, an inertial measurement unit (IMU) for measuring both vehicle linear acceleration and angular rate, a wheel speed sensor for measuring or estimating wheel slip ratio, and wheel These are a brake pressure or braking torque sensor, an engine torque or wheel torque sensor, and a steering angle and rate of change sensor.

In one embodiment, the sensor 121 also includes a sensor for sensing or measuring an attribute of the environment of the AV. For example, monocular or stereo video camera 122, lidar 123, radar, ultrasonic sensor, time-of-flight (TOF) depth sensor, speed sensor, temperature in the visible, infrared, or thermal (or both) spectrum. sensors, humidity sensors, and rainfall sensors.

In one embodiment, AV system 120 includes a data storage unit 142 and memory 144 for storing machine instructions associated with computer processor 146 or data collected by sensors 121 . In one embodiment, data storage unit 142 is similar to ROM 308 or storage device 310 described below with respect to FIG. 3 . In one embodiment, memory 144 is similar to main memory 306 described below. In one embodiment, data storage unit 142 and memory 144 store historical information, real-time information, and/or predictive information about environment 190 . In one embodiment, the stored information includes maps, driving performance, traffic congestion updates or weather conditions. In one embodiment, data regarding environment 190 is transmitted to AV 100 via a communication channel from a remotely located database 134 .

In one embodiment, AV system 120 provides other vehicle conditions and conditions, such as position, linear velocity and angular velocity, linear acceleration and angular acceleration, and a linear heading and angular heading towards AV 100 . a communication device 140 for communicating the measured or inferred attribute of the heading). These devices enable wireless communication over Vehicle-to-Vehicle (V2V) and Vehicle-to-Infrastructure (V2I) communication devices and point-to-point or ad hoc networks or both. including devices for In one embodiment, communication device 140 communicates over the electromagnetic spectrum (including radio and optical communication) or other media (eg, air and acoustic media). A combination of Vehicle-to-Vehicle (V2V), Vehicle-to-Infrastructure (V2I) communication (and in some embodiments, one or more other types of communication) is sometimes referred to as Vehicle-to-Everything (V2X) communication. V2X communication typically complies with one or more communication standards for communication with and between autonomous vehicles.

In one embodiment, communication device 140 includes a communication interface. For example, wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near field, infrared, or radio interfaces. The communication interface transmits data from the remotely located database 134 to the AV system 120 . In one embodiment, the remotely located database 134 is embedded in the cloud computing environment 200 as described in FIG. 2 . The communication interface 140 transmits data collected from the sensor 121 or other data related to the operation of the AV 100 to a database 134 located remotely. In one embodiment, the communication interface 140 transmits information related to teleoperation to the AV 100 . In some embodiments, AV 100 communicates with another remote (eg, “cloud”) server 136 .

In one embodiment, the remotely located database 134 also stores and transmits digital data (eg, stores data such as road and street locations). Such data may be stored in memory 144 on AV 100 or transmitted to AV 100 via a communication channel from a remotely located database 134 .

In one embodiment, the remotely located database 134 provides information on driving attributes (eg, speed profile and acceleration profile) of vehicles that have previously driven along trajectory 198 at a similar time of day. Store and transmit historical information about In one implementation, such data may be stored in memory 144 on AV 100 or transmitted to AV 100 via a communication channel from a remotely located database 134 .

Computing device 146 located on AV 100 algorithmically generates control actions based on both real-time sensor data and prior information so that AV system 120 can execute autonomous driving capabilities. let there be

In one embodiment, AV system 120 provides computer peripherals coupled to computing device 146 to provide information and alerts to, and receive input from, a user of AV 100 (eg, an occupant or remote user). (132). In one embodiment, peripheral 132 is similar to display 312 , input device 314 , and cursor controller 316 discussed below with reference to FIG. 3 . The bonding may be wireless or wired. Any two or more of the interface devices may be integrated into a single device.

2 illustrates an example “cloud” computing environment. Cloud computing is a service for enabling convenient on-demand network access to a shared pool of configurable computing resources (eg, networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services). It is a model of service delivery. In a typical cloud computing system, one or more large cloud data centers house the machines used to deliver services provided by the cloud. Referring now to FIG. 2 , a cloud computing environment 200 includes cloud data centers 204a , 204b , and 204c interconnected through a cloud 202 . Data centers 204a , 204b , and 204c provide cloud computing services to computer systems 206a , 206b , 206c , 206d , 206e , and 206f coupled to cloud 202 .

Cloud computing environment 200 includes one or more cloud data centers. In general, a cloud data center, eg, cloud data center 204a shown in FIG. 2 , is a cloud, eg, cloud 202 shown in FIG. 2 , or a physical arrangement of servers constituting a specific part of the cloud. refers to the body. For example, servers are physically arranged in rooms, groups, rows, and racks within a cloud data center. A cloud data center has one or more zones including one or more server rooms. Each room has one or more server rows, and each row includes one or more racks. Each rack contains one or more individual server nodes. In some implementations, servers within zones, rooms, racks, and/or rows are based on physical infrastructure requirements of the data center facility, including power requirements, energy requirements, thermal requirements, heating requirements, and/or other requirements. are arranged into groups. In one embodiment, the server node is similar to the computer system described in FIG. 3 . Data center 204a has multiple computing systems distributed across multiple racks.

Cloud 202 interconnects cloud data centers 204a , 204b , and 204c and includes networks and networking resources (eg, networking) that help facilitate access of computing systems 206a - 206f to cloud computing services. equipment, nodes, routers, switches, and networking cables) together with cloud data centers 204a, 204b, and 204c. In one embodiment, network represents any combination of one or more local networks, wide area networks, or internetworks coupled using wired or wireless links distributed using terrestrial or satellite connections. Data exchanged over the network is transmitted using any number of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), and Frame Relay. Moreover, in embodiments where the network represents a combination of multiple sub-networks, a different network layer protocol is used in each of the underlying sub-networks. In some embodiments, the network represents one or more interconnected internetworks, such as the public Internet.

Computing systems 206a - 206f or cloud computing service consumers are connected to cloud 202 through network links and network adapters. In one embodiment, the computing systems 206a - 206f may include various computing devices, e.g., servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, autonomous vehicles (cars, drones, shuttles, trains, buses, etc.) and consumer electronics. In one embodiment, computing systems 206a - 206f are implemented within or as part of another system.

3 illustrates a computer system 300 . In one implementation, computer system 300 is a special purpose computing device. A special purpose computing device is a digital electronic device, such as one or more application-specific integrated circuits (ASICs) or field programmable gate arrays (FPGAs) that are either hard-wired to perform the techniques or are continuously programmed to perform the techniques. or one or more general purpose hardware processors programmed to perform techniques according to program instructions in firmware, memory, other storage, or combination. Such special purpose computing devices may also realize the technology by combining custom hardwired logic, ASICs, or FPGAs with custom programming. In various embodiments, the special purpose computing device is a desktop computer system, portable computer system, handheld device, network device, or any other device that includes hardwired and/or program logic to implement the technology.

In one embodiment, computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a hardware processor 304 coupled with bus 302 for processing information. Hardware processor 304 is, for example, a general purpose microprocessor. Computer system 300 also includes main memory 306 , such as random access memory (RAM) or other dynamic storage device, coupled to bus 302 for storing instructions and information to be executed by processor 304 . do. In one implementation, main memory 306 is used to store temporary variables or other intermediate information during execution of instructions to be executed by processor 304 . Such instructions, when stored on a non-transitory storage medium accessible by the processor 304 , render the computer system 300 a special purpose machine customized to perform the operations specified in the instructions.

In one embodiment, computer system 300 further includes a read only memory (ROM) 308 or other static storage device coupled to bus 302 to store static information and instructions for processor 304 . include A storage device 310 is provided and coupled to bus 302 for storing information and instructions, such as a magnetic disk, optical disk, solid state drive, or three-dimensional cross-point memory.

In one embodiment, computer system 300 includes, via bus 302 , a cathode ray tube (CRT), liquid crystal display (LCD), plasma display, light emitting diode (LED) display, active matrix for displaying information to a computer user. displays, such as organic light emitting diode (AMOLED) displays, quantum dot light emitting diode (QLED) displays, vacuum fluorescent displays, e-ink displays, cold cathode (nixie tube) displays, or organic light emitting diode (OLED) displays ( 312). An input device 314 , including alphanumeric keys and other keys, is coupled to bus 302 for communicating information and command selections to processor 304 . Another type of user input device is a cursor controller, such as a mouse, trackball, touch display, or cursor direction keys, for communicating direction information and command selections to the processor 304 and for controlling cursor movement on the display 312 . (316). Such input devices are typically in two axes, a first (eg, x-axis) and a second (eg, y-axis) axis, that allow the device to position in a plane. It has 2 degrees of freedom.

According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306 . Such instructions are read into main memory 306 from another storage medium, such as storage device 310 . Execution of the sequence of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In an alternative embodiment, hard-wired circuitry is used instead of or in combination with software instructions.

The term “storage medium,” as used herein, refers to any non-transitory medium that stores data and/or instructions that cause a machine to operate in a particular manner. Such storage media includes non-volatile media and/or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, solid state drives, or three-dimensional cross-point memory, such as storage device 310 . Volatile media includes dynamic memory, such as main memory 306 . Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid state drives, magnetic tape, or any other magnetic data storage medium, CD-ROM, any other optical data storage medium, hole pattern. includes any physical medium, RAM, PROM, and EPROM, FLASH-EPROM, NV-RAM, or any other memory chip, or cartridge having a

A storage medium is separate from, but may be used with, a transmission medium. A transmission medium participates in transferring information between storage media. For example, transmission media includes coaxial cables, copper wires, and optical fibers, including wires including bus 302 . Transmission media may also take the form of light waves or acoustic waves, such as those generated during radio wave and infrared data communications.

In one embodiment, various forms of media are involved in carrying one or more sequences of one or more instructions to the processor 304 for execution. For example, the instructions are initially held on a magnetic disk or solid state drive of a remote computer. The remote computer loads the commands into dynamic memory and uses a modem to send the commands over the phone line. A modem local to computer system 300 receives data over a telephone line and uses an infrared transmitter to convert the data into infrared signals. The infrared detector receives the data carried in the infrared signal and appropriate circuitry places the data on bus 302 . Bus 302 carries data to main memory 306, and processor 304 retrieves and executes instructions from main memory. Instructions received by main memory 306 may optionally be stored in storage device 310 before or after being executed by processor 304 .

Computer system 300 also includes a communication interface 318 coupled to bus 302 . Communication interface 318 provides a two-way data communication coupling for network link 320 coupled to local network 322 . For example, communication interface 318 is an integrated service digital network (ISDN) card, cable modem, satellite modem, or modem for providing a data communication connection to a corresponding type of telephone line. As another example, communication interface 318 is a LAN card for providing a data communication connection to a compatible local area network (LAN). In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 transmits and receives electrical signals, electromagnetic signals, or optical signals that carry digital data streams representing various types of information.

Network link 320 typically provides data communication over one or more networks to other data devices. For example, network link 320 provides a connection to a host computer 324 via a local network 322 or to a cloud data center or equipment operated by an Internet Service Provider (ISP) 326 . ISP 326, in turn, provides data communication services over a world-wide packet data communication network, now commonly referred to as "Internet 328". Both the local network 322 and the Internet 328 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals over various networks and signals over network link 320 over communication interface 318, which carry digital data to and from computer system 300, are exemplary forms of transmission media. In one embodiment, network 320 includes cloud 202 or part of cloud 202 described above.

Computer system 300 sends messages and receives data, including program code, via network(s), network link 320 , and communication interface 318 . In one embodiment, computer system 300 receives code for processing. The received code is executed by the processor 304 when received and/or stored in the storage device 310 or other non-volatile storage for later execution.

Autonomous Vehicle Architecture

4 shows an example architecture 400 for an autonomous vehicle (eg, AV 100 shown in FIG. 1 ). Architecture 400 includes a cognitive module 402 (sometimes called a cognitive circuit), a planning module 404 (sometimes called a planning circuit), a control module 406 (sometimes called a control circuit) , a localization module 408 (sometimes referred to as a localization circuit), and a database module 410 (sometimes referred to as a database circuit). Each module plays a predetermined role in the operation of the AV 100 . Together, modules 402 , 404 , 406 , 408 , and 410 may be part of the AV system 120 shown in FIG. 1 . In some embodiments, any of modules 402 , 404 , 406 , 408 , and 410 include computer software (eg, executable code stored on a computer readable medium) and computer hardware (eg, one a combination of the above microprocessors, microcontrollers, application-specific integrated circuits (ASICs), hardware memory devices, other types of integrated circuits, other types of computer hardware, or a combination of any or all of these).

In use, the planning module 404 receives data indicative of the destination 412 and a trajectory 414 that may be driven by the AV 100 to arrive at (eg, arrive at) the destination 412 . (sometimes referred to as a route) determines the data representing it. For the planning module 404 to determine data representing the trajectory 414 , the planning module 404 receives data from the recognition module 402 , the location measurement module 408 , and the database module 410 .

The recognition module 402 identifies a nearby physical object using one or more sensors 121 , for example, as also shown in FIG. 1 . The objects are classified (eg, grouped into types such as pedestrians, bicycles, cars, traffic signs, etc.), and a scene description including the classified objects 416 is provided to the planning module 404 .

The planning module 404 also receives data indicative of the AV location 418 from the location measurement module 408 . The location measurement module 408 determines the AV location using data from the sensor 121 and data from the database module 410 (eg, geographic data) to calculate the location. For example, the location measurement module 408 uses data from Global Navigation Satellite System (GNSS) sensors and geographic data to calculate the longitude and latitude of the AV. In one embodiment, the data used by the location measurement module 408 may include a high-precision map of road geometric properties, a map describing road network connectivity properties, and road physical properties (eg, traffic speed, traffic volume, vehicle traffic lanes and bicycles). Maps that describe the number of rider traffic lanes, lane widths, lane traffic directions, or lane marker types and locations, or combinations thereof, and road features such as crosswalks, traffic signs, or other driving signals of various types ( It contains a map that describes the spatial location of the travel signal.

Control module 406 receives data indicative of trajectory 414 and data indicative of AV position 418 , and causes AV 100 to travel trajectory 414 towards destination 412 of the AV Operates control functions 420a - 420c (eg, steering, throttling, braking, and ignition). For example, if trajectory 414 includes a left turn, control module 406 can determine that the steering angle of the steering function causes AV 100 to turn left and throttling and braking cause AV 100 to turn left. It will operate the control functions 420a to 420c in such a way that it pauses and waits for a passing pedestrian or vehicle before the turn is made.

Autonomous vehicle input

5 illustrates inputs 502a - 502d (eg, sensor 121 shown in FIG. 1 ) and outputs 504a - 504d (eg, sensors) used by cognitive module 402 ( FIG. 4 ). data) is shown. One input 502a is a Light Detection and Ranging system (eg, lidar 123 shown in FIG. 1 ). A rider is a technique that uses light (eg, a burst of light, such as infrared light) to obtain data about a physical object in his gaze. The lidar system generates lidar data as output 504a. For example, lidar data is a collection of 3D or 2D points (also known as point clouds) used to construct a representation of environment 190 .

Another input 502b is a radar system. Radar is a technology that uses radio waves to acquire data about nearby physical objects. The radar may acquire data regarding an object that is not within the line of sight of the lidar system. Radar system 502b generates radar data as output 504b. For example, radar data is one or more radio frequency electromagnetic signals used to construct a representation of environment 190 .

Another input 502c is a camera system. The camera system uses one or more cameras (eg, digital cameras that use light sensors such as charge-coupled devices (CCDs)) to obtain information about nearby physical objects. The camera system generates camera data as output 504c. Camera data often takes the form of image data (eg, data in an image data format such as RAW, JPEG, PNG, etc.). In some examples, the camera system has multiple independent cameras that enable the camera system to perceive depth, eg, for stereopsis (stereo vision). Although an object perceived by the camera system is described herein as “nearby”, it is relative to the AV. In use, the camera system may be configured to “see” objects that are distant, eg, up to one kilometer or more in front of the AV. Thus, a camera system may have features such as sensors and lenses that are optimized to recognize distant objects.

Another input 502d is a traffic light detection (TLD) system. TLD systems use one or more cameras to obtain information about traffic lights, street signs, and other physical objects that provide visual driving information. The TLD system generates TLD data as output 504d. TLD data often takes the form of image data (eg, data in an image data format such as RAW, JPEG, PNG, etc.). The TLD system uses a camera with a wide field of view (eg, using a wide angle lens or fisheye lens) to obtain information about as many physical objects as possible that provides visual navigation information, such that the AV 100 ) differs from systems including cameras in that it accesses all relevant navigation information provided by these objects. For example, the viewing angle of the TLD system may be greater than or equal to about 120 degrees.

In some embodiments, outputs 504a - 504d are combined using sensor fusion techniques. Thus, either one of the individual outputs 504a - 504d is provided to another system of the AV 100 (eg, provided to the planning module 404 as shown in FIG. 4 ), or the combined output is Forms of single combined outputs or multiple combined outputs of the same type (using the same combining technique or combining the same outputs, or both) or different types (eg, using different respective combining techniques or different may be provided to the other system either in the form of a single combined output or multiple combined outputs, either by combining their respective outputs or both. In some embodiments, early fusion techniques are used. Early fusion techniques are characterized by combining the outputs before one or more data processing steps are applied to the combined outputs. In some embodiments, a late fusion technique is used. Late fusion techniques are characterized by combining the outputs after one or more data processing steps have been applied to the individual outputs.

6A shows an example of a lidar system 602 (eg, input 502a shown in FIG. 5 ). The lidar system 602 emits light 604a - c from a light emitter 606 (eg, a laser transmitter). The light emitted by the lidar system is typically not in the visible spectrum; For example, infrared light is often used. A portion of the emitted light 604b encounters a physical object 608 (eg, a vehicle) and is reflected back to the lidar system 602 . (Light emitted from the lidar system does not typically penetrate a physical object, eg, a physical object in solid form). The lidar system 602 also has one or more light detectors 610 that detect reflected light. In one embodiment, one or more data processing systems associated with the lidar system generate an image 612 representing a field of view 614 of the lidar system. The image 612 includes information representing the boundary 616 of the physical object 608 . In this manner, image 612 is used to determine boundaries 616 of one or more physical objects in the vicinity of the AV.

6B shows the lidar system 602 in operation. In the scenario shown in this figure, the AV 100 receives both a camera system output 504c in the form of an image 632 and a lidar system output 504a in the form of lidar data points 634 . In use, the data processing system of AV 100 compares image 632 to data points 634 . In particular, physical object 636 identified in image 632 is identified among data points 634 . In this way, the AV 100 recognizes the boundaries of the physical object based on the contour and density of the data points 634 .

6C illustrates the operation of lidar system 602 in additional detail. As described above, AV 100 detects the boundary of a physical object based on characteristics of data points detected by lidar system 602 . As shown in FIG. 6C , a flat object, such as the ground 652 , will reflect light 654a - 654d emitted from the lidar system 602 in a coherent manner. In other words, since the lidar system 602 emits light using a coherent interval, the ground 652 will reflect the light back to the lidar system 602 at the same coherent interval. As the AV 100 travels over the ground 652 , the lidar system 602 will continue to detect light reflected by the next valid ground point 656 if nothing is blocking the roadway. However, if object 658 is obstructing the roadway, light 654e and 654f emitted by lidar system 602 will be reflected from points 660a and 660b in a manner that is not consistent with the expected consistent manner. From this information, the AV 100 may determine that the object 658 is present.

route planning

7 shows a block diagram 700 of a relationship between an input and an output of a planning module 404 (eg, as shown in FIG. 4 ). Generally, the output of the planning module 404 is a route 702 from a starting point 704 (eg, a source location or initial location) to an ending point 706 (eg, a destination or final location). . Route 702 is typically defined by one or more segments. For example, a segment is a distance traveled over at least a portion of a street, road, arterial road, private road, or other physical area suitable for vehicle driving. In some examples, for example, when AV 100 is an off-road capable vehicle such as a four-wheel drive (4WD) or full-time four-wheel drive (AWD) car, SUV, pickup truck, etc., route 702 is an unpaved path or "off-road" segments such as open fields.

In addition to route 702 , the planning module also outputs lane level route planning data 708 . Lane level route planning data 708 is used to traverse a segment of route 702 based on the condition of the segment at a particular time. For example, if route 702 includes a multi-lane arterial road, lane level route planning data 708 may include, for example, whether an exit is approaching, whether one or more of the lanes have other vehicles, or the number It includes trajectory planning data 710 that AV 100 can use to select one of multiple lanes, based on other factors that vary over minutes or less. Similarly, in some implementations, the lane level route planning data 708 includes a speed constraint 712 that is specific to a segment of the route 702 . For example, if the segment includes pedestrians or unexpected traffic, the speed constraint 712 may set the AV 100 to a slower driving speed than the expected speed, eg, speed limit data for the segment. can be limited to a speed based on

In one embodiment, input to planning module 404 includes database data 714 (eg, from database module 410 shown in FIG. 4 ), current location data 716 (eg, FIG. 4 ). AV location 418 shown in ), destination data 718 (eg, for destination 412 shown in FIG. 4 ), and object data 720 (eg, cognition shown in FIG. 4 ) classified object 416 ) recognized by module 402 . In one embodiment, database data 714 includes rules used for planning. Rules are specified using a formal language, for example using Boolean logic. In any given situation that AV 100 encounters, at least some of the rules will apply to that situation. When a rule has a condition to be satisfied based on information available to the AV 100, for example, information about the surrounding environment, the rule is applied to a given situation. Rules may have priorities. For example, a rule that says "If the road is a freeway, go to the leftmost lane" would have a lower priority than "If the exit is coming within a mile, go to the rightmost lane" can have

FIG. 8 shows a directed graph 800 used for route planning, for example, by the planning module 404 ( FIG. 4 ). In general, a directed graph 800 as shown in FIG. 8 is used to determine a path between any start point 802 and an end point 804 . In the real world, the distance separating the start point 802 and the end point 804 may be relatively large (eg, in two different metropolitan areas) or relatively small (eg, a city block). two intersections or two lanes of a multi-lane road).

In one embodiment, directed graph 800 has nodes 806a - 806d representing different locations between start point 802 and end point 804 that may be occupied by AV 100 . In some examples, nodes 806a - 806d represent segments of a road, eg, when start point 802 and end point 804 represent different metropolitan areas. In some examples, for example, when start point 802 and end point 804 represent different locations on the same road, nodes 806a - 806d represent different locations on that road. In this manner, directed graph 800 contains information at various levels of granularity. In one embodiment, a directed graph with high granularity is also a subgraph of another directed graph with a larger scale. For example, a directed graph in which the start point 802 and the end point 804 are distant (eg, many miles apart), although most of its information is of low granularity and is based on stored data, the AV Also includes some high granularity information for the portion of the graph representing the physical location within the field of view of (100).

Nodes 806a - 806d are separate from objects 808a and 808b that cannot overlap nodes. In one embodiment, when the granularity is low, objects 808a and 808b represent areas that cannot be traversed by motor vehicles, eg, areas without streets or roads. When the granularity is high, objects 808a and 808b represent physical objects within the field of view of the AV 100 , eg, other cars, pedestrians, or other entities that cannot share a physical space with the AV 100 . In one embodiment, some or all of the objects 808a - 808b are a static object (eg, an object that does not change position, such as a street light or telephone pole) or a dynamic object (eg, a position such as a pedestrian or other automobile) object that can be changed).

Nodes 806a through 806d are connected by edges 810a through 810c. When the two nodes 806a and 806b are connected by an edge 810a, the AV 100 can, for example, travel to one node 806b before arriving at the other node 806b. It is possible to travel between 806a) and another node 806b. (When we refer to AV 100 traveling between nodes, we mean that AV 100 travels between two physical locations represented by respective nodes.) Edges 810a to 810c are, AV (100) is often bidirectional in the sense that it travels from a first node to a second node, or from a second node to a first node. In one embodiment, edges 810a - 810c mean that AV 100 can travel from a first node to a second node, but AV 100 cannot travel from a second node to a first node. It is unidirectional. Edges 810a - 810c are unidirectional when, for example, they represent individual lanes of a one-way street, street, road, or arterial road, or other features that may only be traversed in one direction due to legal or physical constraints. .

In one embodiment, the planning module 404 uses the directed graph 800 to identify a path 812 consisting of nodes and edges between the starting point 802 and the ending point 804 .

Edges 810a - 810c have associated costs 814a and 814b. The costs 814a and 814b are values representing resources to be consumed when the AV 100 selects the corresponding edge. A typical resource is time. For example, if one edge 810a represents a physical distance that is twice the physical distance of the other edge 810b, then the associated cost 814a of the first edge 810a is the associated cost of the second edge 810b. (814b) may be doubled. Other factors that affect time include anticipated traffic conditions, number of intersections, and speed limits. Another typical resource is fuel economy. The two edges 810a and 810b may represent the same physical distance, but for example, due to road conditions, expected weather, etc., one edge 810a may require more fuel than the other edge 810b. can do.

When the planning module 404 identifies the path 812 between the starting point 802 and the ending point 804 , the planning module 404 typically configures the path 812 between the starting point 802 and the ending point 804 to be a cost-optimized path, eg, at an edge. When the costs are added together, the path with the lowest overall cost is chosen.

Autonomous vehicle control

9 shows a block diagram 900 of the inputs and outputs of the control module 406 (eg, as shown in FIG. 4 ). The control module may include, for example, one or more processors similar to processor 304 (eg, one or more computer processors, such as a microprocessor or microcontroller or both), short-term and/or long-term similar to main memory 306 . A controller 902 including data storage (eg, memory random access memory or flash memory or both), a ROM 308 , and a storage device 210 , and operating in accordance with instructions stored within the memory, the instructions performs the operations of the controller 902 when the instructions are executed (eg, by one or more processors).

In one embodiment, the controller 902 receives data representative of the desired output 904 . Desired output 904 typically includes speed, eg, speed and heading. The desired output 904 may be based, for example, on data received from the planning module 404 (eg, as shown in FIG. 4 ). Depending on the desired output 904 , the controller 902 generates data usable as a throttle input 906 and a steering input 908 . The throttle input 906 may engage the throttle (eg, acceleration control) of the AV 100 to achieve the desired output 904 , eg, by engaging the steering pedal or engaging other throttle control. indicates the degree. In some examples, throttle input 906 also includes data usable to engage in braking (eg, deceleration control) of AV 100 . Steering input 908 is a steering angle, eg, the angle at which a steering control of an AV (eg, a steering wheel, a steering angle actuator, or other functionality for controlling the steering angle) should be positioned to achieve the desired output 904 . indicates

In one embodiment, the controller 902 receives feedback used to adjust the inputs provided to throttle and steering. For example, if the AV 100 encounters an obstruction 910 such as a hill, the measured speed 912 of the AV 100 is lowered below the desired output speed. In one embodiment, any measured output 914 is provided to the controller 902 so that the necessary adjustments are made, for example, based on the difference 913 between the measured speed and the desired output. The measured output 914 includes the measured position 916 , the measured velocity 918 (including speed and heading), the measured acceleration 920 , and other outputs measurable by the sensor of the AV 100 . do.

In one embodiment, information about the obstruction 910 is pre-detected by a sensor, such as a camera or lidar sensor, for example, and provided to the predictive feedback module 922 . The predictive feedback module 922 then provides the information to the controller 902, which can use this information to adjust accordingly. For example, if a sensor in AV 100 has detected (“seen”) a hill, this information can be used by controller 902 to prepare to engage throttle at an appropriate time to prevent significant deceleration. .

10 shows a block diagram 1000 of the inputs, outputs, and components of a controller 902 . The controller 902 has a speed profiler 1002 that influences the operation of the throttle/brake controller 1004 . For example, speed profiler 1002 may engage in acceleration using throttle/brake 1006 , for example, according to feedback received by controller 902 and processed by speed profiler 1002 , for example. Instructs throttle/brake controller 1004 to engage in deceleration.

The controller 902 also has a lateral tracking controller 1008 that affects the operation of the steering controller 1010 . For example, the lateral tracking controller 1008 may steer to adjust the position of the steering angle actuator 1012 according to, for example, feedback received by the controller 902 and processed by the lateral tracking controller 1008 . command to the controller 1004 .

Controller 902 receives several inputs that are used to determine how to control throttle/brake 1006 and steering angle actuator 1012 . The planning module 404 may be configured by the controller 902, for example, to select a heading when the AV 100 begins operation and to determine which road segment to cross when the AV 100 reaches an intersection. information used by The position measurement module 408 can enable the controller 902 to determine whether the AV 100 is in an expected position based, for example, on how the throttle/brake 1006 and steering angle actuator 1012 are being controlled. Thus, information describing the current location of the AV 100 is provided to the controller 902 . In one embodiment, the controller 902 receives information from other inputs 1014, eg, information received from a database, computer network, or the like.

Field Theory Based Cognition

In one embodiment, the classified object 416 as perceived by the cognitive module 402 ( FIG. 4 ) is placed on a discretized representation of the environment of the AV. An example of such a discretized representation 1100 is shown in FIG. 11 . In this example, the discretized representation 1100 includes a grid map having a number of individual cells 1105 (also referred to as grid cells), each representing a unit area (or volume) of the environment. In some implementations, the recognition module 402 is configured to update the occupancy probability of such individual grid cells, wherein the occupancy probability indicates a likelihood that one or more of the classified objects are present in the individual cell. To recursively combine the new sensor measurements with the current estimate of the posterior probability for the corresponding grid cell, e.g. using a Bayesian filter of each grid cell in the discretized representation of the vehicle's environment. Occupancy status can be calculated. Such dynamically updated grid maps are often referred to as dynamic occupancy grids. This method assumes that the environment is changing dynamically, and the dynamics of the environment is described by a Newtonian motion model. Thus, the method estimates not only occupancy, but also parameters of the dynamic model, such as, for example, velocity, force, and the like.

로 표기되어 있다. 격자 셀의 상태는 센서 관측치에 기초하여 업데이트된다. 이것은, 예를 들어, 시간 t + 1에서의 측정치 z_t+1에 기초하여 각각의 격자 셀에 이산적인 이진 점유 확률

을 할당하는 역 센서 모델(inverse sensor model)을 사용하여 행해질 수 있다. 그렇지만, 그러한 역 센서 모델의 사용은 AV에 대한 현실적인 가정이 아닐 수 있는 정적 환경의 가정을 필요로 할 수 있다. 격자 셀의 동적 상태는, 예를 들어, 필드 이론 기반 유체 역학에서 유체가 모델링되는 방식과 유사하게, 차량 또는 보행자와 같은 대상체를 입자의 집합체로서 모델링하는 것에 의해 다루어질 수 있다. 입자라는 용어는, 본원에서 사용되는 바와 같이, 물질의 물리적 단위를 지칭하지 않는다. 오히려, 소프트웨어 컴포넌트들이 함께 AV의 환경에서 대상체(예를 들면, 차량, 보행자 등) 및 자유 공간의 가상 표현을 형성하도록, 입자는 상호작용 소프트웨어 컴포넌트 세트를 나타낸다. 일부 구현예에서, 각각의 소프트웨어 컴포넌트는 개념 객체(conceptual object)의 단위의 인스턴스화를 나타내는 데이터이다. 도 11을 또다시 참조하면, 격자 셀(1110)의 확대 삽입도는 격자 셀(1110)의 내용물을 나타내는 다수의 입자(1115)를 예시한다. 입자(1115) 각각은 대응하는 입자의 상태를 나타내는 하나 이상의 파라미터와 연관될 수 있다. 예를 들어, 입자의 상태는 속도(x 방향, y 방향, z 방향 중 하나 이상을 따른 속도), 다수의 속도와 연관된 공분산, 및 입자에 작용하는 힘 중 하나 이상으로 표현될 수 있다. 그러한 파라미터는 입자의 다양한 동적 특성을 고려할 수 있다. 예를 들어, 힘 파라미터는 곡선 도로를 따른 역학 또는 가속 차량 등의 역학을 고려하는 것을 가능하게 한다.The occupancy grid map divides the environment of the autonomous vehicle into an aggregate of individual grid cells, and the occupancy probability of each grid cell is calculated. In some implementations, a cell is a map (or a map) based on a Cartesian grid, a polar coordinate system, a structured mesh, a block structured mesh, or an unstructured mesh. driving environment). In some implementations, a cell is a map (or by an unstructured mesh, for example, where the cell can be a triangle, quadrilateral, pentagon, hexagon, or any other polygon or combination of various polygons to a two-dimensional mesh). driving environment) regularly or irregularly. Similarly, a cell may be an irregular tetrahedron, hexahedron, or any other polytope or combination of polytopes for a three-dimensional mesh. In an unstructured mesh, the relationships between cells are determined by common vertices that cells can share. For example, two triangles defined as two sets of vertex indices [a, b, c] and [b, c, e] have a common edge defined as the line segment between vertices b and c. edge) is shared. In some implementations, a cell may be described by a graph, wherein each cell corresponds to a node and two adjacent cells are characterized by an edge on the graph. An edge may be assigned a value representing the dynamic interaction (described below) between two linked nodes/cells. Each lattice cell can be considered to be in one of two states - occupied or free. 11 , the probability that a given cell is empty is

is marked with The state of the grid cells is updated based on sensor observations. This is, for example, a discrete binary occupancy probability for each grid cell based on the measurement z _t+ 1 at time t+1.

This can be done using an inverse sensor model that assigns However, the use of such an inverse sensor model may require assumptions of a static environment, which may not be realistic assumptions about AV. The dynamic state of a grid cell can be addressed, for example, by modeling an object, such as a vehicle or pedestrian, as an aggregate of particles, similar to how fluids are modeled in field theory-based fluid mechanics. The term particle, as used herein, does not refer to a physical unit of matter. Rather, a particle represents a set of interactive software components such that the software components together form a virtual representation of an object (eg, vehicle, pedestrian, etc.) and free space in the environment of the AV. In some implementations, each software component is data representing an instantiation of a unit of a conceptual object. Referring again to FIG. 11 , an enlarged inset of grid cell 1110 illustrates a number of particles 1115 representing the contents of grid cell 1110 . Each particle 1115 may be associated with one or more parameters indicative of the state of the corresponding particle. For example, the state of a particle may be expressed in terms of one or more of a velocity (velocity along one or more of the x-, y-, and z-direction), a covariance associated with multiple velocities, and a force acting on the particle. Such parameters may take into account various dynamic properties of the particle. For example, the force parameter makes it possible to take into account the dynamics along a curved road or the dynamics of an accelerating vehicle, etc.

In such field theory-based modeling, the number of particles in a particular grid cell or the sum of the particle weights in a particular grid cell may represent a measure for the probability of occupancy of the corresponding grid cell. However, such grating cell specific approaches require tracking each individual particle within a given grating cell, which in some cases can cause significant computational overhead. For example, to achieve 28-bit accuracy without using the techniques described herein, it may be necessary to track 2 ²⁸ particles per grating, resulting in a computational burden that may not be achievable in many AV applications. do. The techniques described herein can improve field theory based cognitive models by avoiding large computational burdens. In particular, instead of tracking individual particles to determine the occupancy of a cell, the techniques described herein calculate the occupancy probability of a cell by tracking statistics of a particle density function. In other words, the state of the lattice cell in this approach depends on one or more parameters of the particle's binding distribution as it traverses the lattice cells. An Euler solver or Lagrangian solver is used to determine a time-varying coupling distribution by computing a solution of a differential equation defined for one or more particle dynamics parameters obtained using one or more sensors. The resulting updated particle density function is used with the forward sensor model associated with the corresponding sensor to generate predictions for the occupancy probabilities of the various grid cells.

로 표기되어 있다. 추가적으로, 본원에서 설명된 기술은, 2개의 서로 소(disjoint)인 체적

및

에 대해, 그 각자의 점유 확률(또는 비어 있을 확률)이 서로 관련이 없다고 가정한다. 이것은 다음과 같이 표현될 수 있다:As described above, the probability that a given cell is empty is

is marked with Additionally, the technique described herein allows two volumes that are disjoint to each other.

and

For , assume that their respective occupancy probabilities (or empty probabilities) are not related to each other. This can be expressed as:

는 상태 공간에 대한 가법 측도(additive measure)로서 정의되며, 밀도 함수

는 다음과 같이 측도와 연관된 것으로 정의될 수 있다:From this assumption,

is defined as an additive measure for the state space, and is a density function

can be defined as associated with a measure as follows:

개의 독립 항등 분포(identically distributed and independent) 입자의 확률 밀도 함수로서 해석될 수 있다. 특히, 입자가 동일한 것으로 간주되기 때문에, 본원에서 설명된 기술의 다른 내재된 가정은 센서 측정치가 입자를 구분하는 데 사용될 수 없다는 것이다. 오히려, 센서 측정치는 입자가 x에 위치된 경우의 관측치 γ의 확률로서 정의된다. 이러한 측정치는 순방향 센서 모델이라고 지칭되며,

로 표기된다. 또한, 센서 데이터가 입자를 구분할 수 없고 측정치가 단지 하나의 입자로부터 취해질 수 있기 때문에, 이산화된 표현에서의 격자 셀의 전체 체적 V가 점유된 경우(시각적 보조 목적으로,

로 표기된 상황)의 관측치 γ의 확률은 다음과 같이 표기될 수 있다:This is within the volume V of the state space.

It can be interpreted as a function of probability density of randomly distributed and independent particles. In particular, since particles are considered identical, another inherent assumption of the techniques described herein is that sensor measurements cannot be used to differentiate particles. Rather, the sensor measurement is defined as the probability of an observation γ if the particle is located at x. These measurements are referred to as forward sensor models,

is marked with Also, since the sensor data cannot distinguish particles and measurements can be taken from only one particle, if the total volume V of the grid cell in the discretized representation is occupied (for visual aid purposes,

The probability of an observation γ in the situation denoted by

로 정의될 수 있다. 이 함수는 시간 t에서, 위치 x에서, 속도 v로 이동하는 입자를 발견할 확률 밀도를 나타낼 수 있다. 일부 구현예에서, 확률 밀도는 센서 데이터로부터 경험적으로 추론된다. 일부 구현예에서, 확률 밀도가 알려진 확률 분포(예를 들면, 지수족(exponential family)) 또는 2개 이상의 알려진 확률 분포의 혼합으로서 모델링된다. 일부 구현예에서, 확률 밀도가 알려진 분포로서 모델링되지 않을 수 있고, 순전히 센서 데이터에 의해 특성화된다. 일부 구현예에서, 입자에 작용하는 힘, 하나 이상의 추가 방향을 따른 속도, 다수의 속도의 공분산 등과 같은 다른 입자 동적 파라미터가 시변 입자 밀도 함수에서 사용될 수 있다. 입자가 움직이지 않는(stationary) 것은 아니기 때문에, 입자 밀도 함수는 시간 경과에 따라 진화하고, 입자 밀도 함수를 구성하는 파라미터에 대해 정의된 미분 방정식 세트의 해를 결정하는 것에 의해 입자 밀도 함수의 시간 변화(time-variation)가 계산될 수 있다. 일부 구현예에서, 시간 경과에 따른 입자 밀도 함수의 진화는 확률 밀도 함수에 대한 볼츠만 방정식과 같은 운동 방정식(kinetic equation)을 사용하여 모델링될 수 있다. 예를 들어, 입자 수 보존(particle number conservation)의 기본 원칙으로부터, 다음과 같은 미분 방정식이 정의될 수 있다:For autonomous vehicle applications, particles represent objects, free space, etc., and are considered dynamic across grid cells of discrete representations. The reason for this is that the environment of the vehicle is constantly changing and the position of the particles relative to the vehicle changes over time. To account for particle dynamics, a particle density function can be defined for a multidimensional phase space. For example, in some embodiments, the particle density function is a function in a time-space-velocity coordinate frame.

can be defined as This function can represent the probability density of finding a particle moving at time t, at position x, with velocity v. In some implementations, probability densities are inferred empirically from sensor data. In some embodiments, the probability density is modeled as a known probability distribution (eg, an exponential family) or a mixture of two or more known probability distributions. In some implementations, the probability density may not be modeled as a known distribution, and is characterized purely by sensor data. In some embodiments, other particle dynamic parameters may be used in the time-varying particle density function, such as a force acting on the particle, velocities along one or more additional directions, covariance of multiple velocities, and the like. Because particles are not stationary, the particle density function evolves over time, and the time change of the particle density function by determining a solution to a set of differential equations defined for the parameters constituting the particle density function. (time-variation) can be calculated. In some embodiments, the evolution of a particle density function over time can be modeled using a kinetic equation, such as the Boltzmann equation for a probability density function. For example, from the basic principle of particle number conservation, the following differential equation can be defined:

By finding the time derivatives of position and velocity, the Boltzmann partial differential equation can be derived as follows:

The mechanics described in the above equations are based on a Cartesian coordinate system, but can be generalized to any coordinate system. In some implementations, when describing cells and their interactions graphically, a gradient operator on the graph can be used to capture the Boltzmann equation.

Solving multivariate partial differential equations, such as those given in Equation 4, can be computationally intensive, making this process undesirable for real-time applications such as those used to find objects and free space in the environment of AV. . To make this process less computationally intensive and feasible for real-time AV applications, the technique described herein uses an Euler solver that computes solutions of differential equations using numerical approximations. Euler solvers work by approximating a differential equation to an ordinary differential equation (ODE) using known initial values of a set of parameters, and predicts the values of the parameters at a future point in time using the forward Euler method. .

로서, 또는 2차원 이산화된 표현의 경우

로서 표현된다. 일부 실시예에서, 극 좌표계가 사용될 수 있으며, 밀도 분포의 표기법은

가 되고, 여기서 r은 위치(x, y)의 반경이다.12 is an example of a cognitive module that may be used to implement the techniques described herein. In this example, the cognitive module 402 includes a prior distribution calculator 1205 that computes a particle density function at a particular time point t _n . This may be done, for example, using sensor data received from one or more sensors 121 . In some implementations, the sensor data may include radar and/or lidar information with information about one or more parameters relating to the particle. For example, the parameter may include one or more of a particle's velocity along a particular direction as defined by a coordinate system governing a discrete representation of the environment, a force acting on the particle, a position of the particle, and the like. In some implementations, the prior distribution calculator 1205 may be configured to calculate one or more additional parameters based on information received from the sensor 121 . For example, information about velocities from the sensor 121 along multiple directions (eg, the x- and y-directions, and possibly also the z-direction, as defined according to a Cartesian coordinate system) may be obtained from the sensor 121 . When received, prior distribution calculator 1205 may be configured to calculate the covariance of such rates. In some implementations, when the received sensor information includes velocity information along the x-direction and the y-direction, the prior distribution calculator 1205 provides the following parameters associated with particle dynamics: the density of particles in the cell ρ, respectively, x An observation vector γ can be generated that includes velocity components v _x and v _y along the direction and y direction, and the corresponding covariances σ _xx , σ _xy , and σ _yy . The covariance term is used to account for the uncertainty in the velocity term. When applying field theory to fluid dynamics, the anisotropic assumption requires that the particle velocity be equally uncertain in all directions. Such assumptions are unrealistic for objects in the environment of AV and are therefore specially considered. For notational purposes, the particle density function is

as, or in the case of a two-dimensional discretized representation

is expressed as In some embodiments, a polar coordinate system may be used, and the notation of the density distribution is

, where r is the radius of the position (x, y).

을 계산하기 위해 셀 내의 개별 입자를 추적하는 입자 역학 기반 프로세스는, 확률

외에도, 그러한 프로세스가 또한,

와 같이, 각각의 셀 i에 대한 개별

의 계산을 필요로 하기 때문에, 계산 집약적이며, 여기서 p(x, y)는 특정 셀에 대한 입자 밀도 함수를 나타낸다. 오일러 솔버 접근법을 사용하는 것은, 그러한 프로세스와 비교하여, 계산 복잡도를 크게 감소시킬 수 있고, 계산 리소스에 대한 큰 부담 없이 더 높은 품질(예를 들면, 분해능) 및 다이내믹 레인지를 갖는 이미지를 생성할 수 있다.The observations may then be provided to an Euler solver or Lagrangian solver 1210 to compute a solution of a differential equation defined for one or more parameters. The Euler solver may include one or more processing devices programmed to compute a numerical solution of a differential equation using a forward Euler method. This may include predicting the variation of different parameters for a future time point t _n+1 . As mentioned above, the probability of occupancy

A particle dynamics-based process that tracks individual particles within a cell to calculate the probability

Besides, such a process also

For each cell i, as

It is computationally intensive because it requires the calculation of Using the Euler solver approach, compared to such a process, can greatly reduce computational complexity, and can produce images with higher quality (e.g., resolution) and dynamic range without a great burden on computational resources. have.

의 업데이트된 버전을 생성한다. 그러나 표기법 편의를 위해, 입자 밀도 함수가 또한 본 문서에서

로 표현될 수 있다. 분포 계산기(1215)에 의해 계산된 입자 밀도 함수는 사전 분포를 업데이트하기 위해 피드백 루프를 통해 사전 분포 계산기(1205)에 제공될 수 있다.The Euler solver predicts the evolution of various parameters of the particle for time t _n+1 and provides those predicted values to the distribution calculator 1215 . Distribution calculator 1215 computes the predicted distribution of the particle density function in a manner substantially similar to that in prior distribution calculator 1205 , the particle density function

Create an updated version of However, for notational convenience, the particle density function is also

can be expressed as The particle density function calculated by distribution calculator 1215 may be provided to prior distribution calculator 1205 through a feedback loop to update the prior distribution.

The recognition module 402 also includes a posterior probability calculator 1220 that calculates the probability that the particle location will be occupied at a future time point t _n+1 given the current observation vector γ. It is calculated by the posterior probability calculator 1220 as follows:

항은 순방향 센서 모델을 나타내고 점 (x, y)가 속도 v를 갖는 대상체에 의해 점유되어 있는 경우의 관측치 γ의 확률을 나타낸다. 다양한 센서 모달리티(예를 들면, 라이더, 레이더, 비전, 및 다른 센서 모달리티)에 대한 순방향 센서 모델은 주석이 달린 실측 데이터로부터 계산될 수 있다. 실측 데이터는, 예를 들어, 관측치의 통계 및 점유 정보를, 둘 모두를 독립적인 랜덤 샘플로 간주하여, 수집하는 것에 의해, 수집될 수 있다. 파라미터의 결합 분포를 점유를 조건으로 하기(condition) 위해 사후 확률 계산기(1220)에 의해 그러한 순방향 센서 모델이 사용된다. 일부 구현예에서, 측정치 및 점유 정보는 실질적으로 연속적인 분포로부터 도출된다. 그러한 연속 분포는 관측치 샘플을 적절히 이격된 빈에 배치함으로써 히스토그램을 기록하는 것 및 분석 밀도 함수를 이산 히스토그램에 피팅하는 것에 의해 근사화될 수 있다.here

The term represents the forward sensor model and represents the probability of an observation γ if point (x, y) is occupied by an object with velocity v. Forward sensor models for various sensor modalities (eg, lidar, radar, vision, and other sensor modalities) can be computed from the annotated ground truth data. Ground truth data may be collected, for example, by collecting statistical and occupancy information of observations, both as independent random samples. Such a forward sensor model is used by the posterior probability calculator 1220 to condition the occupation of a joint distribution of parameters. In some embodiments, the measurement and occupancy information is derived from a substantially continuous distribution. Such a continuous distribution can be approximated by recording a histogram by placing the observation samples in appropriately spaced bins and fitting the analysis density function to a discrete histogram.

In some implementations, the forward sensor model may also be configured to detect a fault condition in the sensor. For example, ground truth data obtained from such a model may indicate whether the received sensor data is outside the range of expected values for that particular sensor by a threshold amount, and/or whether the received data is consistent with data received for another sensor. It can be used to decide whether or not When sensor data from a particular sensor is determined to be out of range by a threshold amount, a fault condition may be determined/flagned for that sensor, and the corresponding sensor input may be ignored until the fault condition is resolved. .

에 대한 베이지안 추정치이다. 이 함수에서,

은 시간을 나타내고, (x, y)는 2차원 공간 W에서의 위치를 나타내며,

는 (x, y)에서의 속도 벡터이다. 이 출력은 동적 점유 격자와 같은 이산화된 표현에 걸쳐 다양한 방식으로 질의될 수 있다. 예를 들어, 질의의 한 형태는 위상 공간에 시간이 추가된 영역

에서 예상된 입자 수를 계산하는 것이다. 일부 구현예에서, 이것은 다음과 같이 계산될 수 있다:Thus, the output generated by the posterior probability calculator 1220 is a particle density function

is a Bayesian estimate for In this function,

denotes time, (x, y) denotes a position in two-dimensional space W,

is the velocity vector at (x, y). This output can be queried in a variety of ways across a discrete representation, such as a dynamic occupancy grid. For example, one form of query is the time-added region in topological space.

to calculate the expected number of particles in In some embodiments, it can be calculated as:

Assuming that the particles are independently and uniformly distributed and the number of particles is very large, this yields the probability that the time-added region in the phase space is empty given

Thus, the techniques described herein can be used to track not only objects but also free space. This may be important to the AV's planning module, for example as information about where the AV can be steered. In some implementations, this information may improve control of the AV, for example by providing several possibilities, possibly along with information about the object from which the AV should be steered away.

에 대해, 닫힌 다각형은 이러한 포인트들 중 임의의 것에서 나오는 광선이 홀수 개의 세그먼트

와 교차하도록 월드(world) W에서의 포인트 세트로서 정의될 수 있다. 이러한 다각형은 P로서 표기될 수 있다. 일부 구현예에서, 다각형은 AV 환경의 이산화된 표현의 격자 셀을 나타낼 수 있다. 그렇지만, 특히, 다각형의 정의가 임의의 특정 격자에 의존하지 않기 때문에, 본원에서 설명된 기술은 격자 무관 방식(grid-agnostic manner)으로 구현될 수 있다. 게다가, 일부 경우에 특정 시점에서의 입자 밀도 함수를 나타내는 조건부 분포

, 속도에 관계없이 공간과 시간에서의 입자 밀도 함수를 나타내는 비조건부 분포(unconditional distribution)

, 및 둘 모두의 조합을 정의하는 것이 유용할 수 있다. 그러한 수량은 AV의 동작에 대한 다양한 관심 수량을 결정하는 데 사용될 수 있다. 예를 들어, 다각형 P가 특정 시간 t₀에서 점유될 확률은 다음과 같이 계산될 수 있다:In some embodiments, one or more additional quantities may be defined to obtain more information from the particle density function. For example, a set of points

For , a closed polygon is an odd number of segments where a ray from any of these points has an odd number of segments.

It can be defined as a set of points in world W to intersect with . This polygon may be denoted as P. In some implementations, the polygons may represent grid cells of a discretized representation of the AV environment. However, in particular, since the definition of a polygon does not depend on any particular grid, the techniques described herein may be implemented in a grid-agnostic manner. Moreover, in some cases a conditional distribution representing a function of particle density at a particular time point.

, an unconditional distribution representing a function of particle density in space and time independent of velocity.

It may be useful to define , and combinations of both. Such quantities may be used to determine various quantities of interest for the operation of the AV. For example, the probability that polygon P is occupied at a particular time t ₀ can be calculated as:

Given multiple velocities (eg velocities along different directions), this can be extended by defining another polygon P _v in the velocity vector space. Under this extension, the probability that an object occupies a polygon Px and travels at a speed from Pv is given as:

In another example, various other probabilities may be calculated using the particle density function described above, such as the probability that a space will be occupied for a period of time. Such probabilities, along with labeling for particles, will be used to identify various classified objects 416 , including inanimate objects, people, and free space, and how they move over time through a discretized representation of the AV environment. can

Before tracking the particle density function across the discretized representation, the cognitive module 402 can define and label the particles (pedestrians, cars, free space, etc.) and assign initial probabilities to individual grid cells. In some implementations, each cell is initially assumed to be occupied (eg, through assignment of a high probability of occupancy) and is later updated based on sensor data. In some implementations, particles may be assigned different colors depending on whether they represent objects (using additional color coding to differentiate between cars, pedestrians, etc.) or free space. In some embodiments, the particle is described in the document - Nuss et. Interaction software as described in al, “A Random Finite Set Approach for Dynamic Occupancy Grid Maps with Real-Time Application,” International Journal of Robotics Research, Volume: 37 issue: 8, page(s): 841-866 - It can be defined, labeled, and updated as components.

13A shows an example of a user interface 1300 generated using the output of a cognitive module operating in accordance with the techniques described herein. Interface 1300 shows multiple camera views 1305a - 1305d as well as the relative positions of AV 100 relative to various objects in the environment. Specifically, 1305 indicates a front camera view, 1305b and 1305c indicate a side camera view, and 1305d indicates a rear camera view. In this example, only roads are considered to be part of the discretized representation of the environment, and particles are initialized across the corresponding grid cells. In this example, the object and free space are color coded differently. For example, a car seen in the front camera view 1305a is represented as an aggregate 1310 of particles, and a pedestrian seen in the side camera view 1305b is represented as an aggregate 1315 of particles. The evolution of an aggregate of particles as it traverses lattice cells is tracked using field theory based perception as described above.

13B and 13C show examples of additional user interfaces 1320 and 1330, respectively, created in accordance with the techniques described herein. Specifically, interface 1330 illustrates a time point that is less than a second later than the time point represented in user interface 1320 , and illustrates a key benefit of the techniques described herein. As can be seen from the front camera view 1305a , the vehicle 1325 (corresponding to the collection of particles 1330 ) occludes the object/objects view corresponding to the collection of particles 1325 . However, because the cognitive module is able to predict the evolution of particles through the grid cells, an aggregate 1335 of particles is seen just before being separated from the connected particles representing other objects. Indeed, as shown in FIG. 13C , after less than a second, another car 1340 is seen emerging from the portion occluded by the car 1325 , and continues to be traced as the evolution of the aggregate 1335 of particles. Thus, the techniques described herein can be used to accurately track the evolution of an aggregate of particles, even when some (and, in some cases, all) particles are occluded and rendered invisible. Specifically, when sensor data is missing one or more parameters related to a particle, previously obtained values for a particle (or aggregate of particles) may be used to determine an update to the corresponding time-varying particle density function. This can provide a significant advantage to the planning module by preemptively considering the object even before it enters the camera's field of view.

14 is a flowchart of an example process 1400 for generating a prediction of occupation of a location in an environment of an autonomous vehicle. In some implementations, at least a portion of process 1400 may be executed in a cognitive module of an AV (eg, cognitive module 402 shown in FIG. 4 ) using, for example, one or more processing devices. . The operations of process 1400 include generating 1402 a discretized representation of the environment in which the AV operates. The discretized representation may include a plurality of cells (also referred to herein as grid cells). Each cell may be defined as occupied by a number of particles representing at least one of an object or free space in the environment. Particles are not physical units of matter, but rather a set of interactive software components that represent virtual units of physical objects according to the principles of field theory in fluid mechanics. The discretized representation may include a grid defined using a Cartesian or polar coordinate system. In some implementations, the discretized representation may represent a dynamic occupancy grid.

In some embodiments, each cell in the discretized representation is assigned an initial value of occupancy, which is then updated as the particle density function evolves. For example, an initial value of occupancy or probability of occupancy may be assigned 100% (or some other high value) to represent a safe assumption that all of the environment around the AV is occupied. The cell's occupancy probability is then updated according to the evolution of the particle density function and/or sensor data. In some embodiments, labels are assigned to multiple particles, each label indicating, for example, whether the corresponding particle represents an object or free space. These labels may be updated according to the received sensor data. For example, if a car in front of the AV is turning left or right, the sensor data (and/or evolution of a particle density function corresponding to the vehicle) may update one or more particles in the cells in front of the AV as free space.

The operations of process 1400 also include receiving 1404 sensor data indicative of a status of at least one of the plurality of particles in the environment, from one or more sensors of the AV. The one or more sensors may include any combination of the sensors 121 described above. A state of a particle may include, for example, one or more velocities associated with the particle (eg, velocities along different directions), a covariance associated with multiple velocities, and/or a force acting on the at least one particle. . Taking forces into account makes it possible to take into account various dynamics of the corresponding object or free space, for example, motion of a vehicle along a curved path, an accelerating vehicle, and the like.

The operations of process 1400 also include determining 1406 an update to a time-varying particle density function associated with a position of at least one particle in the dynamic occupancy grating from the sensor data and one or more models associated with the one or more sensors. . Determining an update to the time-varying particle density function may include determining, using an Euler solver or a Lagrangian solver, a solution of one or more differential equations defined for one or more parameters associated with a state of the at least one particle. have. The Euler solver may be substantially similar to the Euler solver 1210 described above with reference to FIG. 12 . The one or more models may represent sensor models that have been trained to provide information about the occupancy probabilities of various cells in a discretized representation. Probabilities may be conditional on corresponding sensor data, for example, as described above with reference to a forward sensor model. In some implementations, determining the update to the particle density function includes determining that the sensor data is missing at least one parameter indicative of the state of the at least one particle, and in response, determining the update to the time-varying particle density function and using a previous value of the at least one parameter to determine In some cases, this makes it possible to update the particle density function corresponding to the location being occluded from the sensor.

The operations of process 1400 also include generating 1408 a prediction of occupancy of at least one cell in the discretized representation based on the updated particle density function. This is, for example, by determining the occupancy probability of at least one cell by (i) the ratio of a conditional probability generated based on the sensor data to a conditional probability generated based on the one or more sensor models and (ii) for the at least one cell and determining as a product of the updated time-varying particle density function. The ratio of the conditional probabilities is the ratio of (i) a first probability of receiving sensor data conditioned on the at least one cell being occupied and (ii) a second probability of receiving sensor data conditioned on the at least one cell being unoccupied. it can be rain The first probability and the second probability may be determined using one or more sensor models. In some implementations, predictions can be generated, for example, according to equation (5). The operations of process 1400 also include operating 1410 using the controller circuitry, the AV based at least in part on the prediction.

15 is a flowchart of an example process 1500 for generating a prediction of occupation of a location in an environment of an autonomous vehicle. In some implementations, at least a portion of process 1500 may be executed in a cognitive module of an AV (eg, cognitive module 402 shown in FIG. 4 ) using, for example, one or more processing devices. . In some implementations, at least a portion of the operations of process 1500 may be performed by processing circuitry mounted on one or more sensors. The operations of process 1500 include instantiating 1502 a set of interactive software components that represent the contents of a cell of a discretized representation of an environment. The discretized representation may contain multiple cells, such as in a dynamic occupancy grid. The discretized representation may be substantially similar to that described above with reference to FIG. 14 .

The operations of process 1500 also include receiving 1504, from one or more sensors of the AV, a sensor observation vector associated with an interactive software component of a set of interactive software components. A sensor observation vector may include one or more parameters associated with an interactive software component at a given location. The one or more sensors may include any combination of the sensors 121 described above. A state of a particle may include, for example, one or more velocities associated with the particle (eg, velocities along different directions), a covariance associated with multiple velocities, and/or a force acting on the at least one particle. .

The operations of process 1500 also include determining 1506 an update to a time-varying particle density function associated with a cell of the cells of the discretized representation corresponding to a given location from the sensor observation vector. Determining an update to the time-varying particle density function may include determining, using an Euler solver or a Lagrangian solver, a solution of one or more differential equations defined for one or more parameters associated with a state of the at least one particle. have. The Euler solver may be substantially similar to the Euler solver 1210 described above with reference to FIG. 12 . The one or more models may represent sensor models that have been trained to provide information about the occupancy probabilities of various cells in a discretized representation. Probabilities may be conditional on corresponding sensor data, for example, as described above with reference to a forward sensor model. In some implementations, determining the update to the particle density function includes determining that the sensor data is missing at least one parameter indicative of the state of the at least one particle, and in response, determining the update to the time-varying particle density function and using a previous value of the at least one parameter to determine In some cases, this makes it possible to update the particle density function corresponding to the location being occluded from the sensor.

Operations of process 1500 also include generating 1508 a prediction of occupancy of the cell based on the updated particle density function. This is, for example, by determining the occupancy probability of at least one cell by (i) the ratio of a conditional probability generated based on the sensor data to a conditional probability generated based on the one or more sensor models and (ii) for the at least one cell and determining as a product of the updated time-varying particle density function. The ratio of the conditional probabilities is the ratio of (i) a first probability of receiving sensor data conditioned on the at least one cell being occupied and (ii) a second probability of receiving sensor data conditioned on the at least one cell being unoccupied. it can be rain The first probability and the second probability may be determined using one or more sensor models. In some implementations, predictions can be generated, for example, according to equation (5). The operations of process 1500 also include augmenting 1510 the operation of the vehicle based on the prediction. This may include, for example, steering a vehicle away from a cell predicted to be occupied by an object, steering the vehicle into an area predicted to have free space, or a vehicle predicted to be occupied by one or more objects. This may include accelerating/braking to avoid the area.

In some embodiments, the solver used to solve the equation is: a finite difference scheme (eg, upwind difference, Lax-Friedrichs, Lax-Wendroff, Warming-Beam, artificial viscous method ( artificial viscosity method, etc.) or a finite volume scheme (e.g. Godunov method, flux limiting, flux splitting, random choice, etc.) or finite element method element scheme) (eg, Galerkin, Petrov-Galerkin, discrete Galerkin, first-order systems least squares, etc.) or particle-in-cell method or lattice Boltzmann method (lattice Boltzmann method) or boundary element method (boundary element method).

A method is disclosed. The method comprises, using one or more processing devices of a vehicle operating in the environment, generating a discretized representation of the environment, the discretized representation comprising a plurality of cells, each cell of the plurality of cells being an object in the environment or occupied by particles representing at least one of free space; receiving, from one or more sensors of the vehicle, sensor data indicative of a state of at least one of the plurality of particles in the environment; determining, using the one or more processing devices, an update to a time-varying particle density function associated with a position of the at least one particle in the dynamic occupancy grating from the sensor data and one or more models associated with the one or more sensors; generating, using the one or more processing devices, a prediction of the occupancy of the at least one cell in the discretized representation based on the updated particle density function; and operating, using the vehicle's controller circuitry, the vehicle based at least in part on the prediction.
In one or more exemplary methods, the discretized representation includes a grid defined using a structured mesh, a block structured mesh, or an unstructured mesh.
In one or more exemplary methods, a grid is defined using a Cartesian or polar coordinate system, or a graphical model.
In one or more example methods, the discretized representation comprises a graph wherein each cell corresponds to a node and two adjacent cells are characterized by an edge in the graph.
In one or more exemplary methods, the method further comprises assigning, for each cell of the plurality of cells, an initial value of occupancy, the initial value then updated according to the evolution of the particle density function.
In one or more exemplary methods, the method further comprises assigning labels to the plurality of particles, each label indicating whether the corresponding particle represents an object or free space.
In one or more example methods, the method further includes updating the labels according to the sensor data.
In one or more exemplary methods, the method further comprises assigning successive weights to the plurality of particles, each weight representing a contribution of the individual particle to the cumulative density function.
In one or more exemplary methods, determining an update to the time-varying particle density function comprises using an Eulerian solver or a Lagrangian solver to determine the at least one parameter associated with a state of the at least one particle. determining solutions of one or more differential equations or one or more gradient operators defined for
In one or more exemplary methods, the state of the at least one particle includes at least one velocity associated with the at least one particle.
In one or more exemplary methods, the state of the at least one particle includes (i) multiple velocities along corresponding directions and (ii) covariances associated with the multiple velocities.
In one or more exemplary methods, the state of the at least one particle includes a force acting on the at least one particle.
In one or more exemplary methods, each of the one or more models represents a sensor model that has been trained to provide information about occupancy probabilities of various cells in a discretized representation, the probabilities conditioned on corresponding sensor data.
In one or more exemplary methods, each of the one or more models represents one of a known probability distribution, a mixture of two or more probability distributions, or an empirical distribution.
In one or more example methods, generating a prediction of occupancy of the at least one cell comprises determining the occupancy probability of the at least one cell: (i) a conditional probability generated based on the sensor data and a conditional probability generated based on the one or more sensor models. determining as the product of the ratio of the conditional probabilities and (ii) the updated time-varying particle density function for the at least one cell.
In one or more example methods, the conditional probability is a ratio of (i) a first probability of receiving sensor data conditioned on the at least one cell being occupied and (ii) the sensor data conditioned on the at least one cell being unoccupied. is a ratio of a second probability of receiving , wherein the first probability and the second probability are determined using one or more sensor models.
In one or more example methods, the method includes determining that sensor data corresponding to a particular sensor is outside a threshold range of expected values determined using a corresponding model associated with the particular sensor; and in response to determining that sensor data corresponding to the particular sensor is outside the threshold range, identifying a fault condition for the particular sensor.
In one or more example methods, the method includes determining that the sensor data is missing at least one parameter indicative of a state of the at least one particle; and in response to determining that the sensor data is missing the at least one parameter, using the previous value of the at least one parameter to determine an update to the time-varying particle density function.
In one or more example methods, at least a portion of the one or more processing devices is disposed in a cognitive circuit of a vehicle.
The vehicle is started. The vehicle may include one or more computer processors; and one or more non-transitory storage media storing instructions that, when executed by one or more computer processors, cause to perform operations, the operations comprising: generating a discretized representation of an environment, wherein the discretized representation comprises a plurality of cells wherein each cell of the plurality of cells is occupied by a plurality of particles representing at least one of an object or free space in the environment, from one or more sensors of the vehicle, at least one of the plurality of particles in the environment receiving sensor data indicative of a state of particles of generating a prediction of occupancy of the at least one cell in the discretized representation based on the updated particle density function, and operating the vehicle based at least in part on the prediction.
One or more non-transitory storage media are disclosed that store instructions that, when executed by one or more computing devices, cause performing any one of the one or more exemplary methods described above.
A method is disclosed. The method comprises: instantiating, using processing circuitry mounted on one or more sensors, a set of interacting software components representing the contents of cells of a discretized representation of an environment; receiving, from the one or more sensors, a sensor observation vector associated with an interactive software component of a set of interactive software components, the sensor observation vector comprising one or more parameters associated with the interactive software component at a given location; determining, using the processing circuitry, an update from the sensor observation vector to a time-varying particle density function associated with a cell of the cells of the discretized representation corresponding to the given location; generating, using the processing circuitry, a prediction of occupancy of the cell based on the updated particle density function; and using the processing circuitry to augment the operation of the vehicle based on the prediction.
In one or more exemplary methods, the discretized representation comprises a grid defined using a structured mesh, a block structured mesh, or an unstructured mesh.
In one or more exemplary methods, a grid is defined using a Cartesian or polar coordinate system, or a graphical model.
In one or more exemplary methods, the method further comprises assigning, for each cell of the plurality of cells, an initial value of occupancy, the initial value then updated according to the evolution of the particle density function.
In one or more exemplary methods, the method further comprises assigning labels to the plurality of particles, each label indicating whether the corresponding particle represents an object or free space.
In one or more example methods, the method further includes updating the labels according to the sensor data.
In one or more exemplary methods, the method further comprises assigning successive weights to the plurality of particles, each weight representing a contribution of the individual particle to the cumulative density function.
In one or more exemplary methods, determining an update to the time-varying particle density function comprises using an Eulerian solver or a Lagrangian solver to determine the at least one parameter associated with a state of the at least one particle. determining solutions of one or more differential equations defined for
In one or more exemplary methods, the state of the at least one particle includes at least one velocity associated with the at least one particle.
In one or more exemplary methods, the state of the at least one particle includes (i) multiple velocities along corresponding directions and (ii) covariances associated with the multiple velocities.
In one or more exemplary methods, the state of the at least one particle includes a force acting on the at least one particle.
In one or more exemplary methods, each of the one or more models represents a sensor model that has been trained to provide information about occupancy probabilities of various cells in a discretized representation, the probabilities conditioned on corresponding sensor data.
In one or more example methods, generating a prediction of occupancy of the at least one cell comprises determining the occupancy probability of the at least one cell: (i) a conditional probability generated based on the sensor data and a conditional probability generated based on the one or more sensor models. determining as the product of the ratio of the conditional probabilities and (ii) the updated time-varying particle density function for the at least one cell.
In one or more example methods, the conditional probability is a ratio of (i) a first probability of receiving sensor data conditioned on the at least one cell being occupied and (ii) the sensor data conditioned on the at least one cell being unoccupied. is a ratio of a second probability of receiving , wherein the first probability and the second probability are determined using one or more sensor models.
In one or more example methods, the method includes: determining that sensor data corresponding to a particular sensor is outside a threshold range of expected values determined using a corresponding model associated with the particular sensor; and in response to determining that sensor data corresponding to the particular sensor is outside the threshold range, identifying a fault condition for the particular sensor.
In one or more example methods, the method includes determining that the sensor data is missing at least one parameter indicative of a state of the at least one particle; and in response to determining that the sensor data is missing the at least one parameter, using the previous value of the at least one parameter to determine an update to the time-varying particle density function.
In one or more example methods, at least a portion of the processing device is disposed in a cognitive circuit of a vehicle.
The vehicle is started. The vehicle may include one or more computer processors; and one or more non-transitory storage media storing instructions that, when executed by one or more computer processors, cause to perform operations, the operations comprising: a set of interactive software components representing the contents of cells of a discrete representation of an environment; instantiating, receiving, from one or more sensors, a sensor observation vector associated with an interactive software component of a set of interactive software components, the sensor observation vector comprising one or more parameters associated with the interactive software component at a given location; ; determining an update to a time-varying particle density function associated with a cell of the cells of the discretized representation corresponding to a given location from a vector of sensor observations based on one or more models associated with the sensor; generating a prediction of occupancy of the cell based on the updated particle density function; and augmenting the operation of the vehicle based on the prediction.
One or more non-transitory storage media are disclosed that store instructions that, when executed by one or more computing devices, cause performing any of the above-described exemplary methods.

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

delete

In the foregoing description, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the detailed description and drawings are to be viewed in an illustrative rather than a restrictive sense. The only exclusive indication of the scope of the invention, and what Applicants intend to be of the invention, is the literal equivalent of a series of claims appearing in specific forms from this application, wherein the specific forms in which such claims appear are subject to any subsequent amendments. includes Any definitions expressly set forth herein for terms contained in such claims determine the meaning of such terms as used in the claims. Additionally, when the term “further comprising” is used in the preceding description and in the claims below, what follows this phrase is an additional step or entity, or a sub-step/sub-entity of a previously mentioned step or entity. can

Claims (39)

A method for operating a vehicle performed by a vehicle, the method comprising:
generating a discretized representation of the environment, using one or more processing devices of a vehicle operating in the environment, the discretized representation comprising a plurality of cells, each cell of the plurality of cells being an object in the environment or occupied by particles representing at least one of free space, labels assigned to a plurality of particles, each label indicating whether the corresponding particle represents an object or free space;
receiving, from one or more sensors of the vehicle, sensor data indicative of a state of at least one of a plurality of the particles in the environment, wherein the state of the at least one particle is a force acting on the at least one particle. including -;
determining, using the one or more processing devices, an update to a time-varying particle density function associated with a position of the at least one particle in a dynamic occupancy grating from the sensor data and one or more models associated with the one or more sensors;
generating, using the one or more processing devices, a prediction of the occupancy of at least one cell in the discretized representation based on the updated particle density function; and
operating the vehicle based, at least in part, on the prediction, using the vehicle's controller circuitry;
A method comprising The method of claim 1 , wherein the discretized representation comprises a grid defined using a structured mesh, a block structured mesh, or an unstructured mesh. 3. The method of claim 2, wherein the grid is defined using a Cartesian or polar coordinate system, or a graphical model. 4. A method according to any one of the preceding claims, wherein the discretized representation comprises a graph, wherein each cell corresponds to a node and two adjacent cells are characterized by an edge in the graph. In, way. 4. The method of any one of claims 1 to 3, further comprising, for each cell of the plurality of cells, assigning an initial value of occupancy, the initial value being then updated according to the evolution of the particle density function.
Further comprising, a method. delete 4. The method of any one of claims 1 to 3, further comprising: updating the labels according to the sensor data.
Further comprising, a method. 4. The method of any preceding claim, further comprising assigning successive weights to a plurality of said particles, each weight representing the contribution of the individual particle to the cumulative density function.
Further comprising, a method. 4. The method of any one of claims 1 to 3, wherein determining the update to the time-varying particle density function comprises: using an Eulerian solver or a Lagrangian solver; determining solutions of one or more differential equations or one or more gradient operators defined for one or more parameters associated with the state of particles of 4. The method of any preceding claim, wherein the state of the at least one particle comprises at least one velocity associated with the at least one particle. The method of claim 10 , wherein the state of the at least one particle comprises (i) a plurality of velocities along corresponding directions and (ii) covariances associated with the plurality of velocities. 4. The sensor according to any one of the preceding claims, wherein each of the one or more models represents a sensor model trained to provide information about occupancy probabilities of various cells of the discretized representation, wherein the probabilities are corresponding to a corresponding sensor. conditioned on the data. 4. The method of any preceding claim, wherein each of the one or more models represents one of a known probability distribution, a mixture of two or more probability distributions, or an empirical distribution. 4. The method according to any one of claims 1 to 3, wherein generating the prediction of the occupancy of the at least one cell determines the occupancy probability of the at least one cell by (i) a conditional generated based on the sensor data. determining as the product of a ratio of a probability to a conditional probability generated based on the one or more sensor models and (ii) an updated time-varying particle density function for the at least one cell. 15. The method of claim 14, wherein the conditional probability is a ratio of (i) a first probability of receiving the sensor data conditioned on the at least one cell being occupied and (ii) the condition that the at least one cell being unoccupied. ratio of a second probability of receiving the sensor data, wherein the first probability and the second probability are determined using the one or more sensor models. 4. The method according to any one of claims 1 to 3,
determining that the sensor data corresponding to a particular sensor is outside a threshold range of expected values determined using a corresponding model associated with the particular sensor; and
in response to determining that the sensor data corresponding to the particular sensor is outside a threshold range, identifying a fault condition for the particular sensor;
Further comprising, a method. 4. The method according to any one of claims 1 to 3,
determining that the sensor data is missing at least one parameter indicative of a state of the at least one particle; and
in response to determining that the sensor data is missing the at least one parameter, using a previous value of the at least one parameter to determine the update to the time-varying particle density function;
Further comprising, a method. 4. The method according to any one of claims 1 to 3,
and at least a portion of the one or more processing devices is disposed in a cognitive circuit of the vehicle. As a vehicle,
one or more computer processors; and
One or more non-transitory storage media storing instructions that, when executed by the one or more computer processors, cause them to perform operations
A method comprising:
generating a discretized representation of an environment, wherein the discretized representation comprises a plurality of cells, each cell of the plurality of cells occupied by a plurality of particles representing at least one of an object or free space in the environment and labels are assigned to a plurality of particles, each label indicating whether the corresponding particle represents an object or free space;
receiving, from one or more sensors of the vehicle, sensor data indicative of a state of at least one of a plurality of the particles in the environment, wherein the state of the at least one particle is a force acting on the at least one particle. including -,
determining an update to a time-varying particle density function associated with a position of the at least one particle in a dynamic occupancy grating from the sensor data and one or more models associated with the one or more sensors;
generating a prediction of the occupancy of at least one cell in the discretized representation based on the updated particle density function; and
operating the vehicle based at least in part on the prediction. One or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause the method as recited in claim 1 to be performed. A method performed by a vehicle, comprising:
instantiating, using processing circuitry mounted on one or more sensors, a set of interacting software components representing the contents of the cells of the discretized representation of the environment, labels are assigned to a plurality of particles; each label indicates whether the corresponding particle represents an object or free space;
receiving, from the one or more sensors, a vector of sensor observations associated with an interactive software component of the set of interactive software components, the vector of sensor observations comprising one or more parameters associated with the interactive software component at a given location. Ham -;
determining, using the processing circuitry, an update from the vector of sensor observations to a time-varying particle density function associated with a cell of the cells of the discretized representation corresponding to the given location;
generating, using the processing circuitry, a prediction of the occupancy of the cell based on the updated particle density function; and
augmenting, using the processing circuitry, the operation of a vehicle based on the prediction;
A method comprising The method of claim 21 , wherein the discretized representation comprises a grid defined using a structured mesh, a block structured mesh, or an unstructured mesh. 23. The method of claim 22, wherein the grid is defined using a Cartesian or polar coordinate system, or a graphical model. 24. The method according to any one of claims 21 to 23, further comprising, for each cell of a plurality of said cells, assigning an initial value of occupancy, said initial value being then updated according to the evolution of said particle density function.
Further comprising, a method. delete 24. The method of any one of claims 21 to 23, further comprising: updating the labels according to the sensor observations.
Further comprising, a method. 24. A method according to any one of claims 21 to 23, comprising assigning successive weights to a plurality of particles, each weight representing a contribution of the individual particle to the cumulative density function.
Further comprising, a method. 24. The method of any one of claims 21 to 23, wherein determining the update to the time-varying particle density function comprises using an Eulerian solver or a Lagrangian solver at least one determining solutions of one or more differential equations defined for one or more parameters associated with the state of the particle. 29. The method of claim 28, wherein the state of the at least one said particle comprises at least one velocity associated with the at least one said particle. 30. The method of claim 29, wherein at least one state of the particle comprises (i) a plurality of velocities along corresponding directions and (ii) covariances associated with the plurality of velocities. 24. The method according to any one of claims 21 to 23, wherein the state of the at least one said particle comprises a force acting on the at least one said particle. 24. The sensor according to any one of claims 21 to 23, wherein each of the one or more models represents a sensor model trained to provide information about occupancy probabilities of various cells of the discretized representation, wherein the probabilities are corresponding to a corresponding sensor. conditioned on observations. 24. The method according to any one of claims 21 to 23, wherein generating a prediction of the occupancy of at least one said cell determines the probability of occupancy of at least one said cell by (i) a conditional generated based on the sensor observation. determining as the product of a ratio of a probability to a conditional probability generated based on the one or more sensor models and (ii) an updated time-varying particle density function for at least one said cell. 34. The method of claim 33, wherein the conditional probabilities are ratios of (i) a first probability of receiving the sensor observation subject to at least one said cell being occupied and (ii) at least one said cell being unoccupied. ratio of a second probability of receiving the sensor observation, wherein the first probability and the second probability are determined using the one or more sensor models. 24. The method according to any one of claims 21 to 23,
determining that the sensor observation corresponding to a particular sensor is outside a threshold range of expected values determined using a corresponding model associated with the particular sensor; and
in response to determining that the sensor observation corresponding to the particular sensor is outside a threshold range, identifying a fault condition for the particular sensor;
Further comprising, a method. 24. The method according to any one of claims 21 to 23,
determining that the sensor observation is missing at least one parameter indicative of the state of the at least one particle; and
in response to determining that the sensor observation is missing the at least one parameter, using a previous value of the at least one parameter to determine the update to the time-varying particle density function;
Further comprising, a method. 24. The method according to any one of claims 21 to 23,
and at least a portion of the processing circuitry is disposed in a cognitive circuitry of the vehicle. As a vehicle,
one or more computer processors; and
One or more non-transitory storage media storing instructions that, when executed by the one or more computer processors, cause them to perform operations
A method comprising:
instantiating a set of interactive software components representing the contents of cells of the discretized representation of the environment, labels being assigned to the set of interactive software components, each label indicating whether a corresponding interactive software component represents an object or Indicate whether it represents free space -,
receiving, from the one or more sensors, a vector of sensor observations associated with an interactive software component of the set of interactive software components, the vector of sensor observations comprising one or more parameters associated with the interactive software component at a given location. Ham -;
determining an update to a time-varying particle density function associated with a cell of the cells of the discretized representation corresponding to the given location from the vector of sensor observations based on one or more models associated with the sensor;
generating a prediction of occupancy of the cell based on the updated particle density function; and
and augmenting the operation of the vehicle based on the prediction. One or more non-transitory storage media storing instructions that, when executed by one or more computing devices, cause the method of any one of claims 21-23 to be performed.

Images