WO2014183953A1

WO2014183953A1 - Arrangement and method for sensor fusion and production method for developing a fusion model

Info

Publication number: WO2014183953A1
Application number: PCT/EP2014/057867
Authority: WO
Inventors: Wendelin Feiten; Michael Fiegert
Original assignee: Siemens Aktiengesellschaft
Priority date: 2013-05-16
Filing date: 2014-04-17
Publication date: 2014-11-20
Also published as: WO2014183949A1; DE102013212710A1; WO2014183944A1; DE102013213807A1; WO2014183945A1; DE102013218678A1; DE102013215115A1; WO2014183948A2; DE102013215032A1

Abstract

A calculation rule which, by means of sensor data (4, 40), permits suitable modifications to be determined in an environment representation (6), i.e., in the internal evaluation of the environment perceived by a driver-assist application, is introduced as a fusion model (8). The fusion model (8) is thus, for example, a calculation rule which indicates how the sensor data (4, 40) are to be interpreted in relation to an environment representation type (for example, a grid chart or object list) and according to which the sensor data (4, 40) can be converted into the environment representation (6). The actual modifications in the environment representation (6), however, are made by a modularly separate inference engine (80) which interprets the fusion model (8) for that purpose. The modular separation of the sensor fusion into fusion model (8) and inference engine (80) reduces dependencies during the development of driver-assist systems and thus enables the work to be divided more flexibly. In addition, expenditure and effort during design and test cycles are reduced. As a result of modularization, design tasks can be divided between suppliers to a greater extent. Furthermore, the fusion model (8) can now be developed by machine leaning in that it can be formulated as a factor graph, for example, and parameterized by solving a mixed discrete-continuous optimization problem.

Description

description

Arrangement and method for sensor fusion and manufacturing method for creating a fusion model

For the following explanations, the terms "model" and "representation" must be defined differently. In the following, data structures are referred to as models which largely largely accurately model aspects of a real environment and can therefore be used as reliable input and / or specification for algorithms for sensor fusion as well as for their training. Such models are usually created by experts with great care and a lot of time. Deviating from this, the term "representation" is intended to denote those estimated assumptions and opinions that software, such as a sensor fusion algorithm, itself must form over an environment, such a representation being the result of a computer program flow and must be compared to the models previously mentioned necessarily remain both in their level of detail and in their reliability.

Accordingly, in the following an environment model denotes a largely exact model of a real environment, which an expert has created, for example, by exact manual measurement of a test environment and subsequent construction of a finely resolved polygon model. In contrast, in the following, an environment representation denotes an internal representation of an environment, which is represented by a

Algorithm for sensor fusion as a grid map is generated and necessarily must remain both in their level of detail and in their reliability compared to the environment model. The terms environment representation and internal environment representation are to be understood as synonymous. The terms measured values, sensor data and sensor signals are also used synonymously. Analogously, in the following a vehicle model refers to a largely exact model of a vehicle, for example in the form of a high-resolution 3D model or blueprint. On the other hand, in the following, a vehicle representation denotes estimated states of the vehicle, for example, whether or not there is a skid.

Furthermore, in the following, a driver model designates a largely exact model of a driver, for example in the form of an animated humanoid 3D model, which shows realistic head, eye and eyelid movements. On the other hand, in the following, a driver's representation denotes estimated states of the driver, for example, whether he is tired or not.

Often the environment will be an environment of a vehicle. As a special case, however, the environment can also be the vehicle itself or its driver. Unless specified otherwise, the term environment model should therefore also include the special cases vehicle model and driver model below. Similarly, unless otherwise specified, the term environmental representation should also include the special cases of vehicle representation and driver representation below.

However, the algorithms described below can also use an environment model, a vehicle model and / or a driver model and an environment representation, a vehicle representation and / or a driver representation side by side, which then exist in separate modules and work together in a suitable manner.

An essential step in the design of driver assistance systems is the derivation of internal representations of the environment, the vehicle and the driver from the measured values of one or more sensors, referred to below as a fusion algorithm. In a vehicle today several driver assistance applications are integrated. The determination of the fusion algorithm as well as the implementation of the actual application due to the environment representation used are conventionally in the hands of the respective developer.

The concrete merger algorithm is left to the experience and the tact of the respective developer and is usually developed by hand. A crucial element of this development is the way in which sensor data can be used to deduce the environmental representation, ie the inverse sensor model. However, the manual design or manual parameterization of this inverse sensor model again requires expertise on all other systems, models and representations used. The respective developer must therefore know exactly the characteristics of the sensors used and the environmental representation type. He also needs expert knowledge about the techniques of insecure closure. If a new type of sensor is to be integrated into the system, the entire fusion algorithm must be reopened. Then the fusion algorithm has to be re-formulated taking into account the new sensor type. Here, too, the developer must again know exactly the peculiarities of this new type of sensor.

According to the current state of the art, it is customary in the design of driver assistance systems to design the fusion algorithm by hand and then experimentally or in simulation to check how well the resulting environmental representation meets the requirements of the driver assistance application.

It is an object of the invention to provide an arrangement and a method for sensor fusion and a production method for producing a fusion model which increase the reusability of the corresponding arrangements and methods and / or simplify development work in the field of sensor fusion. This object is achieved by an arrangement for sensor fusion,

comprising an interface for receiving real sensor data or virtual sensor data,

comprising a memory, on which an environment representation is stored, which corresponds to a predetermined environment representation type,

marked by

an inference engine in which a computer program is executed which is set up to make changes in the environmental representation which fuse the real sensor data or virtual sensor data with the environmental representation, using a fusion model that is modularly separated from the inference engine.

On the computer-readable medium, a fusion model is stored, which is for use by the

Inferenzmaschine the arrangement is established.

Furthermore, the object is achieved according to the invention by a method for sensor fusion,

where real sensor data or virtual sensor data is received,

in which an environment representation corresponding to a given environmental representation type is accessed,

characterized in that

an inference machine computer-based on the basis of a modularly separated from the inference engine fusion model makes changes in the environmental representation, which merge the real sensor data or virtual sensor data with the environmental representation. In addition, according to the invention, the object is achieved by a production method for creating a fusion model, in which computer-based simulations of virtual measurements in an environment model based on a sensor model are carried out, and virtual sensor data that at least partially maps an environment model,

in which a fusion model specifies at least one calculation rule, at least one data structure, at least one function and / or at least one algorithm which receives the virtual sensor data as input and prescribes as output corresponding changes in an environment representation with a predetermined environment representation type,

in which the fusion model is created by computer-aided machine learning, wherein

deriving a target environment representation from the environment model and the environment representation type that specifies a machine learning training goal,

the fusion model by solving a continuous

Optimization problem is parametrized,

one modularly separate from the fusion model

Computer-based inference machine based on the fusion model (8) and the real sensor data or virtual sensor data makes changes in the environment representation, which merge the real sensor data or virtual sensor data with the environmental representation,

the environment representation is compared with the target environment representation, resulting in an error minimized in solving the continuous optimization problem.

On the computer-readable medium, a computer program is stored, which executes one of the methods, when it is processed in a microprocessor. The computer program is executed in a microprocessor and executes one of the methods. The advantages and explanations given below do not necessarily relate to the subject matters of the independent patent claims. Rather, these may also be advantages or aspects which relate only to individual embodiments, variants or developments.

At least one calculation rule, at least one data structure, at least one function and / or at least one algorithm is understood as a fusion model, which allows (s) to determine suitable changes in an environmental representation on the basis of sensor data. As defined above, the environment representation here is an internal estimate of the environment perceived by a driver assistance application. Thus, for example, the fusion model is an arithmetic rule that translates the sensor data into the environmental representation-in other words, a conversion rule that specifies how the sensor data is to be interpreted with respect to an environmental representation type (eg, grid map or object list), and according to which the sensor data can be converted into the environment representation.

The actual changes in the environmental representation are not from the fusion model, but from the

Inferenzmaschine made which interprets the fusion model. Consequently, at least functionally there is a modular separation between the inference engine and the fusion model. The modular separation here means that the fusion model is interchangeable, trainable or changeable, without an adaptation of the inference engine would have to take place, which would go beyond the mere provision of the new fusion model. Thus, the fusion model may be located outside or inside the inference engine. Furthermore, the inference engine can be implemented as a microprocessor, software program or virtual machine. The modular separation is thus a functional or logical separation. In addition, optionally a spatial separation can be provided by the location of the fusion model is separated from the inference engine.

The functional separation of the sensor model makes it possible to provide a designer of driver assistance functions with a design library with a large number of highly sophisticated fusion models, without the developer having to have complete expertise on the respective algorithms.

For example, in the manufacturing method for the fusion model, one of the following is given as the environmental representation type: 2D lattice map, 3D lattice or cube map, polygon model, object lists (list of object center points or rectangle envelopes with size specification), or simple lists of states ,

The modular division of the sensor fusion into a fusion model and inference engine reduces dependencies on the development of driver assistance systems and thus enables a more flexible division of labor. Furthermore, an effort in design and test cycles is reduced. Modularization means that development tasks can be more widely shared among suppliers.

Another advantage is the ability to progressively develop driver assistance systems through the increased modularity created. Instead of equipping each application with its own sensors as usual, existing sensors and inference machines for sensor fusion can be reused by appropriately modifying their fusion models. The additional cost of a new driver assistance application is then based solely on the optionally additionally installed sensors and the costs for new soft- ware.

The inference engine as well as other required computation units, simulators, etc. may be hardware and / or soft- be implemented technically. In a hardware implementation, it can be designed as a device or as part of a device, for example as a computer or as a microprocessor or as a control computer of a vehicle. In a software implementation, the respective unit may be designed as a computer program product, as a function, as a routine, as part of a program code or as an executable object. The computer-readable data carrier is, for example, a memory card, a USB stick, a CD-ROM, a DVD or even a data carrier of a server, from which a file with the computer program in a network is provided or delivered. This can be done, for example, in a wireless communication network by transmitting the appropriate file.

In one embodiment, the inference engine is a graph based generic probabilistic

Inference apparatus. The fusion model includes one

Factor graphs.

Furthermore, there is a further development

in which both the environment representation and the virtual sensor data contain random variables,

where the factor diagram of the fusion model maps each of these random variables into a variable node, and where the factor graph contains factor nodes connecting the variable nodes and describing conditional probabilities between the variable nodes.

According to one embodiment, a production method results,

in which the error is determined by means of an error function, which takes into account the requirements of a driver assistance function.

In a further development, a production method results, wherein the environment model models an environment of a vehicle, a vehicle with states and / or a driver of a vehicle, and

wherein the sensor model is a 2D or 3D camera, an ultrasonic sensor, a 2D or 3D laser scanner, a 2D or 3D radar, a lidar, a wheel rotation sensor, an inertial sensor, an acceleration sensor, a rotation rate sensor, a temperature sensor, a Humidity sensor, a position sensor for determining at least one parameter of the driving dynamics of a vehicle, a seat occupancy sensor or a distance sensor modeled, and wherein the predetermined environment representation type is a 2D or 3D grid map, an object list or a list of states.

According to one embodiment, a production method results,

where the fusion model is selected and parameterized by solving a mixed discrete-continuous optimization problem from a set of fusion models.

In a further development, a production method results in which the inference engine is a probabilistic one

Inference apparatus is,

wherein the fusion model includes a factor graph in which both the environmental representation and the real sensor data or the virtual sensor data contain random variables,

wherein the factor graph of the fusion model maps each of these random variables into a variable node, and wherein the factor graph contains factor nodes connecting the variable nodes and describing conditional probabilities between the variable nodes,

where the factor nodes and the links each

Factor can be determined by the discrete part of the optimization problem, whereby the fusion model is selected from the set of possible factor graphs, and in the continuous parameter, the factor nodes are determined by the continuous part of the optimization problem, thereby parameterizing the fusion model.

According to one embodiment, a production method results,

using the mixed discrete-continuous optimization problem solved

- of genetic algorithms,

from ant algorithms, or

a non-linear optimization, in particular by means of a branch-and-bound algorithm. In a further development, a production method results in which the environment representation type (7) is a 2D or 3D lattice map,

in which the random variables in the environment representation (6) are entered and updated for cells or cubes, respectively, and

in which the random variables express an uncertainty of information, in particular an occupancy probability for the respective cell or the respective cube.

In the following, embodiments of the invention will be explained in more detail with reference to figures. In the figures, identical or functionally identical elements are provided with the same reference numerals, unless stated otherwise. Show it:

FIG. 1 shows an architecture of a driver assistance system,

FIG. 2 shows a machine learning method for a fusion model,

FIG. 3 shows an arrangement or a method for sensor fusion. 1 shows an architecture of a driver assistance system and in particular the construction of an environmental representation 6 of real sensor data 4. A sensor 1 performs a real measurement 3 in a real environment 2 and generates the real sensor data 4. For the translation of the real sensor data 4 in the Environment representation 6 is basically always a sensor data fusion 5 required. Because from the second measurement, a temporal merger must already be made. Multiple sensors at different locations require a local fusion of the real sensor data 4. Furthermore, different sensor types, such as ultrasound and camera, must be fused by the sensor data fusion 5 with particular regard to their properties. Without limiting the generality, the embodiment described here can be understood as a design process and design system for driver assistance systems. The generic architecture of a driver assistance system shown in FIG. 1 is therefore not limited to a formalization of the design process, but is also suitable as a system architecture for sensor fusion and for implementing a driver assistance application 9, which controls a vehicle behavior 100. Sensor data fusion 5 plays a central role in this architecture. It is responsible for deriving the environmental representation 6 from the sequence of the real sensor data 4 of different sensors 1, ie, as defined above, an internal representation of the surroundings, of the vehicle condition and / or of the condition of the driver. The entries in the environment representation 6 are therefore from observations, ie the real sensor data 4.

In most cases, an environment representation type 7, ie the type of environmental representation 6, is predefined by the developer. The environment representation type 7 can be, for example, an occupancy grid map or an object list, it can define parameters of the vehicle state or it can generate a large amount of information. rather pretend conditions for the driver ("tired", "awake",

"equanimous", "tense", ...).

The state-dependent entries in the environmental representation 6 are modified, depending on what the sensors 1 measure. For example, For example, the occupancy probability in a cell of an occupancy lattice map may be increased if the sensor 1 measures a distance to an obstacle at a corresponding distance. The entries in the environment representation 6, e.g. An occupancy probability usually expresses an uncertainty of the information. Therefore, they usually correspond to one of the known uncertainty calculus, in particular from the fuzzy theory, the Dempster-Shafer calculus or the probability calculus. Mostly, probability calculus is used and the entries are random variables. For reasons of efficiency, a parametric probability density distribution is often used as the basis, but it is also possible to choose a sample-based representation, as is customary in the field of autonomous robots.

The calculation rule according to which the entries in the environmental representation 6 are modified on the basis of the real sensor data 4 is entered in FIG. 1 as a fusion model 8. An example of the driver assistance application 9 is a

Driver assistance function for automatic parking. This example is based on the representations in Thrun, Burgard and Fox: "Probabilistic Robotics", MIT Press 2005, chap. 6. The driver assistance application 9 for automatic parking needs as environment representation 6 a map of the environment in which a parking space can be identified and then a path can be planned into this parking space. The environmental representation type 7 of this card will typically be an occupancy lattice card. For this purpose, the plane in which the vehicle is located is usually divided into square cells of the same size, for which it is recorded whether this cell is occupied or not. The edge length of a cell is usually between 0.05 m undl.Om, but can also be smaller or bigger. Such a grid map is an example of an environmental representation 6 of the environment.

Thus, the environmental representation 6 should basically be static over a period of time, i. in it the parts of the real environment 2 are to be modeled, which do not change over a certain time. The measurement actually depends on the real environment 2. In deriving the environment representation 6, this is often taken synonymously in the literature to the information in the environment representation 6, ie the map of the environment.

FIG. 2 shows a separate and explicit modeling of the sensor properties with a sensor model 10, the environmental properties with an environment model 20, the environment representation type 7 and the environment representation 6 itself. In the design process, the required expertise about the sensor hardware becomes the expertise decoupled across application areas.

The sensor model 10, a not-shown vehicle model and the environment model 20 in this case allow the simulation of virtual measurements 30, resulting in virtual sensor data 40. The above-mentioned decoupling is achieved by physically modeling the sensor hardware in the sensor model 10 in such a way that the virtual measurements 30 for the sensor can be simulated by means of a suitable simulation and due to the correspondingly detailed environment model 20. In this case, not a deterministic, ie always the same measured value is determined, but the virtual sensor data 40 are subject to the same random fluctuations as the real sensor data determined in experiments (see FIG. The sensor model 10 thus contains the physical properties of the sensor. Which these are, of course, depends on the respective sensor types. The environment model 20 contains those physical properties of the environment needed to simulate or derive the ground truth of the internal environment representation: For each sensor model 10 to be used, the environment model 20 must include the corresponding physical properties of the environment.

For a video camera as a sensor, these properties include the geometry and textures (optical reflection properties) of the objects. Numerous very well-developed examples of such environment models 20 can be found in the areas of computer animation for movies, computer games, architectural simulation, etc. In these environment models 20, there are currently. also simulated properties of the transmission medium (fog, rain, snow, ...) as well as properties of lighting.

For the simulation of an ultrasonic sensor, an approximation of the geometry together with the acoustic reflection properties is sufficient as the environmental model 20. Depending on the simulation principle, the objects in the environment model 20 are modeled so that sections with rays can be calculated, or that the reflection of waves on the surfaces can be calculated. For this purpose the normals on the surfaces are needed.

The formats in which, on the one hand, the sensor model 10 and, on the other hand, the environment model 20 are set up, must match the corresponding simulators. For this purpose, a standardization is recommended, but should leave a certain amount of space. Most promising candidate for this is also a probabilistic formulation, ie the corresponding parameters are considered as random variables. In this context, the vehicle model describes where the sensors are relative to the vehicle. First and foremost, the geometric relationships are relevant here. But it can also be electrical conditions relevant if, for example, loud Sensor model 10 fluctuations in the supply voltage can lead to increased noise in the measured values. In this case, the supply voltage (possibly as a random variable) is part of the vehicle model.

The sensor simulation creates the same virtual sensor data 40 from the sensor model 10 and the environment model 20 as would the real sensors in a real use environment. The simulation can process random variables as input values, and also the virtual sensor data 40 are random variables, and their probability density distribution is also simulated.

The simulation can be carried out at very different accuracy levels, with the higher the rule

Space requirements and computing time requirements arise, the more accurate the simulation is.

A very simple simulation is based on the fact that for a discrete, often very small number of rays emanating from the sensor, the intersections of these rays are formed with objects in the environment. For this purpose, the intersection points of rays with objects must be able to be formed in the environment model 20. This simulation neglects the reflec- tion of acoustic waves on surfaces and thus provides more measured values than the physical sensor.

In another simulation method, in addition to the intersection of the rays with the surfaces, the normal vectors on the surfaces at the intersections are also taken into account. This makes it possible to judge whether a sound wave is reflected towards a receiver (this can also be the transmitter) and thus an echo is measured, or whether no echo is measured.

Another simulation method is markedly more strongly influenced by the environmental representation type 7 of the environmental representation 6. Specifically, here the volume of the internal environment decomposed into grid cells and the course of the pressure distribution over time simulated. This method (depending on the resolution, ie size of the grid cells and time intervals) simulates the echoes of very jagged obstacles, and also simulates echoes that reach a receiver via several reflectors, so-called multi-path echoes.

For example, the ultrasonic sensor emits a cone-shaped signal. An echo can be described by a circle segment (2D) or a spherical shell section (3D) within this cone whose radius corresponds to the measured distance. Accordingly, a likelihood of an obstacle is increased in all cells cut by the circle segment and lowered in all cells passed on the way there from the beam.

By agreeing formats, the sensor model 10, the environment model 20, and the vehicle model may come from different suppliers and still fit together. Similar de facto standards already exist in the area of CAD (eg the STEP format). In the context of the embodiments, standards are needed in which the probability density functions can also be described. In the case of the explicit separation of the modeling, the sensor model 10 can come from the sensor manufacturer. The environment model 20 may be from a company that specializes in 3D modeling. The simulator can come from a company specializing in simulation. In the future, the vehicle model will be contributed by the system integrator. The models and the simulator are usually used by the system integrator, ie an automobile manufacturer or designer. Due to the separate modeling (in the sensor model 10) and physical simulation (in the environment model 20), the various contributions of physical sensor hardware and sensor signal conditioning (sensor model 10) on the Level of system integration separate from the influences from the deployment environment (environment model 20). In this case, the system integrator no longer needs expertise on the sensor hardware, since this is detected in the sensor model 10. By simulating the overall system (sensor model 10 + vehicle model 15 + environment model 20), the forward sensor model relevant for the given application is developed.

The possibility of validating the simulated forward sensor model on real data or estimating parameters in this model remains unaffected.

In the following, an exemplary embodiment for the derivation of a target environmental representation 60 for preparing a machine learning of the fusion model 8 will be explained, still in the context of FIG.

In order to support the design of the fusion model 8, a simulation of all essential aspects of all the modules in the system is performed as shown in FIG. 2 for many reasons. FIG. 2 shows on the one hand a section of the driver assistance system in which the physical sensors 1 and the real environment 2 from FIG. 1 are replaced by corresponding models, and in which the real measurements 3 from FIG. 1 are replaced by virtual measurements 30. The section extends to the environmental representation 6, which in this case is derived from the virtual sensor data 40. In addition, in FIG. 2, a desired perceived environmental representation is shown, the target environmental representation 60. This is of the same environmental representation type 7 as the environmental representation 6.

On the basis of the intended driver assistance function, it is possible within a direct derivation 70 to decide which information from the environment model 20 is to be estimated in the environment representation 6. For example, an image of the parking space is needed for automatic parking, that is, the parking space present in a certain area. static obstacles. In this environment representation 6 moving objects should not be registered, especially when they leave the parking space again. So the goal is an occupancy grid map in which every cell that corresponds to a static object is marked with a high probability of being occupied, and every cell that does not correspond to a static object is marked with a high probability as free. These probabilities may be weighted according to the requirements of the application, eg parking. Thus, it may be required that the correct estimate of the occupied cells takes precedence over the proper estimation of the free cells in the immediate vicinity of the occupied cells. As a result, a planned path into the parking space is likely to be mobile, while it may happen that a physically physically possible path into the parking space is not found because of the inaccurate environment representation 6.

Likewise, peculiarities of the sensors are taken into account. As a result, sensors of driver assistance systems generally can not image the interior of objects. Therefore, the occupancy state of occupancy grid map cells associated with interior points of static objects in environment model 20 may be specified as "unknown" (regardless of how this state is associated with a specific probabilistic model of "busy" or "free" in FIG Individual case is coded).

All of these aspects are taken into account in order to design the target environment representation 60 as part of the direct derivation 70. In the following, still in the context of Figure 2, a

Embodiment for the machine learning of the fusion model 8 explained. The fusion model 8 is now judged according to how exactly the environmental representation 6 created from the virtual sensor data 40 by means of this fusion model 8 agrees with the target environmental representation 60. In this case, as indicated above, this error function can also be matched to the requirements of the driver assistance function. So it may be z. For example, if a cell in the environment representation 6 contains a higher rating for "free" than in the target environment representation 60, and vice versa, a higher rating for "occupied" in the

Environmental representation 6 is rated with a lower error value.

Without restricting generality, a comparatively simple approach is illustrated below and adapted for machine learning of the fusion model 8 using an example from Thrun, Burgard and Fox: "Probabilistic Robotics", MIT Press 2005, p. 294 ff. The aim is to estimate the environmental representation 6 from the vehicle state (assumed to be known here for simplicity) and the virtual sensor data 40, more precisely from a sequence of vehicle states x _{1: t} and a sequence of sensor data z _{1: t} . We are looking for the occupancy probability p (m | z _{1: t} , x _{1: t} ). In practice, because of limitations in memory space and computing time, this is preferably incrementally calculated, that is, p (m _t \ m _t _ ₁ , z _t , x _t ). This probability is to be determined by the role of the sensor data fusion 5 shown in FIG.

The sensor data fusion 5 is in this case of a

Inferenzmaschine 80 carried out using the fusion model 8. A detailed explanation of these relationships will be given in the context of FIG. 3 as well as in the context of the factor graphs described below.

As described in Thrun, Burgard and Fox: "Probabilistic Robotics", MIT Press 2005, p. 294 et seq Assignment grid map assigned a binary (ie bivalent) random variable, with the two values free and occ, for "free" and "occupied". Each cell must have p (free) + p (occ) = 1, so only one number needs to be stored, eg p (occ). If p (occ) = 1, then the corresponding cell is safely occupied, and if p (occ) = 0, then that cell is certainly free. In the following example, 0 <p (occ) <1, which is quite sufficient in practice. It is further assumed that this occupancy probability for a cell is independent of the occupancy probability of other cells (which does not agree with reality). This will allow each cell to be updated on its own.

Even updating a single cell can still be done in very different ways. Thrun, Burgard and Fox: "Probabilistic Robotics", MIT Press 2005, p. 94 provides a simple example using a Bayesian static state filter that can be used for this purpose. Due to advantages in the numerical calculation (in particular when approximating to the interval limits 0 and 1), not p (occ) is entered in the cell, but the quotient l (occ): =

loa -, the "log odds ratio" or the logarithmic quo-p (occ)

ten ratio. Since this will be a different value for each cell, the probability of occupancy ni as a random variable; cell i selected.

This computes the new value of a cell as l _ti = l _t - _1: i + log, called the inverse sensor

model, an increase or decrease in the occupancy probability of cell ί and l _n = loa ^{p (} - ^m ^ _{e is a} general a-priori occupancy probability (without knowledge of the vehicle condition or a measurement, ie only dependent on the environment ). In this example, the exact interpretation of the fusion model 8 is how the term loa - ^■ consists of the virtual

^ lp (mi \ z _t , x _t )

Sensor data 40 is to be interpreted. In a simple case, different constants are chosen, and this choice corresponds more to an empirical approach than to a formally strict derivation. Thus, the fusion model 8 is given by

• log i _{occ f} if the cell is within the field of view of the sensor

sors (for example, opening angle of an ultrasonic lobe) and the measured value matches the distance between cell and sensor,

• if the cell is in the field of vision of the cell

Sensor is (for example, opening angle of the ultrasonic lobe) and the measured value is smaller than the distance between the cell and sensor, or

• lo probably-

It does not change.

In order to determine the numerical values of l _{o cc} and lf _ree , precise knowledge of the sensor is required. In order to determine I ₀ , knowledge of the environment as well as of the intended use of the card is required.

However, in the proposed development environment, this knowledge has already been incorporated in the sensor models 10, the environment models 20 and the simulation algorithms, as well as in the direct derivation 70 of the target environment representation 60.

Therefore, the parameters l _{o cc} , lfree ^and d Ό can also be found by an optimization algorithm that optimizes the quality of the environment representation 6 produced in the simulation. This is a continuous optimization in this case. mierungsproblem, which is solved in Figure 2 by a fusion model - optimization 50.

Developer support can also be extended to selecting a suitable class of fusion model 8, with the choice of class as a discrete variable. This extends the problem to a mixed discrete-continuous optimization problem. Significant progress has recently been made in solving these problems.

One possible implementation of a fusion model 8, which is configured by discrete and / or continuous parameters for a specific sensor data fusion task, consists in a factor graph, as described in the document "Factor graph", available on the Internet on July 11, 2013 at http : //en.wikipedia.org/wiki/Factor_graph. In this formalism, random variables that characterize the current state of the environmental representation are considered as variable nodes of a graph. Likewise, the

Random variables that correspond to the real sensor data or the virtual sensor data 40, understood as a variable node. The variable nodes are associated with factor nodes that describe the conditional probabilities between these random variables. These conditional probabilities may belong to different parametric classes of distributions, e.g. to exponential distributions.

For many practical applications there are efficient algorithms for implementing the inference engine 80, which determine the probabilities of the variable nodes in the factor graph. A corresponding implementation of these algorithms can be found for example in the GTSAM software package of the Georgia Institute of Technology, documented in the document "The Borg Lab", available on the Internet on 11.07.13 at https://collab.cc.gatech.edu/ borg /. Another implementation is G20 of the University of Freiburg or of the OpenSLAM consortium, documented in the document "g2o: A General Framework for Graph Optimization ", available on the Internet on 11.07.13 at https://openslam.org/g2o.html.

With the inference engine 80 in conjunction with the

Factor graphs as a fusion model 8, it is possible to determine the state of the environmental representation 6 from real sensor data or the virtual sensor data 40. In this respect, these algorithms provide an example of a generic one

The fusion model 8 consists of a factor graph whose structure can be described by discrete parameters (variable nodes and factor nodes by type and connection structure), while the parameters for defining the factor nodes - also part of the fusion model 8 - are usually continuous parameters, eg Mean and covariance in a normal distribution.

Through this characterization of the corresponding

Factor graphs will be finding an expression of the fusion model 8, here by defining the factor graph, a task of mixed discrete-continuous optimization with the discrete and continuous parameters as variables. This type of problem has long been the subject of active and highly successful research under the name MINLP (Mixed Integer and Non Linear Programming). In addition to the well-known genetic algorithms (as an example of stochastic optimization), ant-algorithms were also given as solutions to the MINLP problem. The latter are among others from the document "ant algorithm", available on the Internet on 11.07.13 under

https://en.wikipedia.org/wiki/Ameisenalgorithmus, known.

Nonlinear optimization principles are based on NLP-based "branch and bound" methods and many others that are also suitable. An overview can be found in the document P. Bonami, M. Kilinc, and J. Linderoth: "Algorithms and Software for Convex Mixed Integer Nonlinear Programs", in Jon Lee and Sven Leyffer editors, Mixed Integer Nonlinear Pro- gramming, The IMA Volumes in Mathematics and Applications, Volume 154, 2012, pages 1-39.

An overview of other methods of machine learning applicable here, such as Bayesian networks can be found in D. Koller and N. Friedman: "Probabilistic Graphical Models:

Principles and Techniques ", MIT Press, 2009, pages 134-141.

FIG. 3 shows an exemplary embodiment of an arrangement 105 and a method for sensor fusion. Via an interface 104, for example a connection to a vehicle bus system, a USB, WLAN, Ethernet or Bluetooth port, real sensor data 4 or virtual sensor data 40 are received. In a memory 106, for example a hard disk, a flash memory or a RAM module, an environment representation 6 is stored, which corresponds to a predetermined environment representation type.

In an inference engine 80, for example a suitably programmed microprocessor, a computer program is executed, which is set up to make changes in the environmental representation 6, which include the real sensor data 4 or virtual sensor data 40, using a fusion model 8 that is modularly different from the inference engine 80 merge with the environmental representation 6.

In this case, the modular separation means that the fusion model 8 can be exchanged or changed without having to adapt the inference engine 80, which would go beyond the mere provision of the new fusion model 8. Thus, the fusion model 8 may optionally be stored in memory 106 as shown in FIG. 3 or on a computer-readable medium located in inference engine 80 itself, such as an SD card, USB stick, RAM, or microchip , Furthermore, the inference engine 80 can also be implemented as a software program or virtual machine, which in turn are then processed, for example, in a microprocessor represented in FIG. 3 by the arrangement 105.

The modular separation is thus a functional or logical separation. In addition, a spatial separation may optionally be provided by the storage location of the fusion model 8 is separated from the inference engine.

The described embodiments, embodiments, variants and developments can be combined freely with each other. In particular, arbitrary sub-aspects of the exemplary embodiments can be included in the context of FIG. 1 and FIG. 2 in the exemplary embodiment described in the context of FIG.

Claims

claims

1. arrangement for sensor fusion,

comprising an interface (104) for receiving real sensor data (4) or virtual sensor data (40),

comprising a memory (106) on which is stored an environmental representation (6) corresponding to a given environmental representation type (7), characterized by

an inference engine (80) in which a computer program is executed, which is set up to make changes in the environmental representation (6) using a fusion model (8) that is modularly separated from the inference engine (80), which the real sensor data (4) or virtual Sensor data (40) with the environment representation (6) merge.

2. Arrangement according to claim 1,

in which the fusion model (8) specifies at least one calculation rule, at least one data structure, at least one function and / or at least one algorithm which receives the real sensor data (4) or the virtual sensor data (40) as input and corresponding changes in the output Environment representation (6) dictates.

3. Arrangement according to claim 2,

wherein the inference engine (80) is a graph-based generic probabilistic inference apparatus, and wherein the fusion model (8) includes a factor graph.

4. Arrangement according to claim 3,

in which both the environment representation (6) and the virtual sensor data (40) contain random variables, - in which the factor graph of the fusion model (8) maps each of these random variables in a variable node, and where the factor graph contains factor nodes connecting the variable nodes and describing conditional probabilities between the variable nodes.

5. Computer-readable data medium,

on which is stored a fusion model (8) adapted for use by the inference engine (80) of the arrangement according to any one of claims 1 to 4.

6. Computer-readable data carrier according to claim 5,

where the fusion model (8) contains a factor graph,

wherein both the environment representation (6) and the virtual sensor data (40) contain random variables in which the factor graph maps each of these random variables into a variable node, and

where the factor graph contains factor nodes connecting the variable nodes and describing conditional probabilities between the variable nodes.

7. Method of sensor fusion,

in which real sensor data (4) or virtual sensor data (40) are received,

in which an environment representation (6) is used that corresponds to a given environment representation type (7),

characterized in that

an inference engine (80) computer-based on a modular basis of the inference engine (80) fusion model (8) makes changes in the environment representation (6), the real sensor data (4) or virtual sensor data (40) with the environmental representation (6) merge.

8. The method according to claim 7,

in which the fusion model (8) specifies at least one calculation rule, at least one data structure, at least one function and / or at least one algorithm, which the real sensor data (4) or the virtual sensor data (40) receives as input and as output prescribes appropriate changes in the environment representation (6).

, Method according to claim 8,

0. Method according to claim 9,

wherein both the environmental representation (6) and the real sensor data (4) or virtual sensor data (40) contain random variables,

wherein the factor graph of the fusion model (8) maps each of these random variables into a variable node, and wherein the factor graph contains factor nodes connecting the variable nodes and describing conditional probabilities between the variable nodes.

1. Manufacturing method for creating a fusion model (8),

in which virtual measurements (30) in an environment model (20) are simulated on the basis of a sensor model (10) and virtual sensor data (40) are generated based on the virtual measurements (30), which at least partially map an environment model (20),

in which a fusion model (8) specifies at least one calculation rule, at least one data structure, at least one function and / or at least one algorithm which receives the virtual sensor data (40) as input and changes corresponding to output in an environment representation (6) prescribes a given environment representation type (7),

in which the fusion model (8) is created by computer-aided machine learning, wherein

a target environment representation (60) from the environment model (20) and the environmental representation type (7) which specifies a training target for machine learning,

the fusion model (8) is parameterized by solving a continuous optimization problem,

- one of the fusion model (8) modularly separate

Inferenzmaschine (80) computer-based on the basis of the fusion model (8) and the real sensor data (4) or virtual sensor data (40) makes changes in the environment representation (6), the real sensor data (4) or virtual sensor data (40) with the Environment representation (6) merge,

the environment representation (6) is compared with the target environmental representation (60), resulting in an error which is minimized in the solution of the continuous optimization problem.

12. Production method according to claim 11,

in which the fusion model (8) specifies at least one calculation rule, at least one data structure, at least one function and / or at least one algorithm which receives the real sensor data (4) or the virtual sensor data (40) as input and corresponding changes in the output Environment representation (6) dictates. 13. A manufacturing method according to claim 11 or 12,

in which the error is determined by means of an error function, which takes into account the requirements of a driver assistance function. 14. A manufacturing method according to any one of claims 11 to 13, wherein the environment model (20) models an environment of a vehicle, a vehicle with conditions and / or a driver of a vehicle, and

in which the sensor model (10) a 2D or 3D camera, an ultrasonic sensor, a 2D or 3D laser scanner, a

2D or 3D radar, a lidar, a wheel rotation sensor, an inertial sensor, an acceleration sensor, a rotation rate sensor, a temperature sensor, an air humidifier tesensor, a position sensor for determining at least one parameter of the driving dynamics of a vehicle, a seat occupancy sensor or a distance sensor modeled, and

wherein the predetermined environment representation type (7) is a 2D or 3D grid map, an object list, or a list of states.

5. A manufacturing method according to any one of claims 11 to 14, wherein the fusion model (8) is selected and parameterized by solving a mixed discrete-continuous optimization problem from a set of fusion models (8).

6. Production method according to claim 15,

wherein the inference engine (80) is a probabilistic inference apparatus,

where the fusion model (8) contains a factor graph,

in which both the environmental representation (6) and the real sensor data (4) or the virtual sensor data (40) contain random variables,

wherein the factor graph of the fusion model (8) maps each of these random variables into a variable node, and wherein the factor graph contains factor nodes connecting the variable nodes and describing conditional probabilities between the variable nodes,

where the factor nodes and the links each

Factor are determined by the discrete part of the optimization problem, whereby the fusion model is selected from the set of possible factor graphs and in which continuous parameter the factor node is determined by the continuous part of the optimization problem, thereby parameterizing the fusion model (8).

17. Manufacturing method according to claim 16, where the mixed discrete-continuous optimization problem is solved by using

of genetic algorithms,

from ant algorithms, or

a non-linear optimization, in particular by means of a branch-and-bound algorithm.

18. A manufacturing method according to claim 16 or 17,

wherein the environment representation type (7) is a 2D or 3D grid map,

in which the random variables in the environment representation (6) are respectively entered and updated for cells or cubes, and

in which the random variables express an uncertainty of an information, in particular an occupancy probability for the respective cell or the respective cube.

19. computer-readable data carrier,

- On which a computer program is stored, which carries out the method according to one of claims 7 to 10 or the manufacturing method according to one of claims 11 to 18, when it is processed in a microprocessor. 20. computer program,

which is processed in a microprocessor and thereby carries out the method according to one of claims 7 to 10 or the manufacturing method according to one of claims 11 to 18.