WO2003051671A1

WO2003051671A1 - Electronic rearview system for vehicles

Info

Publication number: WO2003051671A1
Application number: PCT/EP2002/014484
Authority: WO
Inventors: Johann Stark
Original assignee: Magna Auteca Ag
Priority date: 2001-12-19
Filing date: 2002-12-18
Publication date: 2003-06-26
Also published as: AU2002361009A1; DE10164516A1

Abstract

Disclosed is an electronic rearview system for vehicles, comprising a device and a method for generating a panoramic rear view by means of at least two video cameras (2, 3) disposed directly next to each other at fixed locations. Image processing means combine the video images in such a way that a seamless, continuous, and instantaneous panoramic view of the rear area of the vehicle (5) is generated.

Description

Electronic vehicle rearview system

The invention relates to an electronic vehicle rear view system that transmits image information from at least two video cameras to an image display unit and provides a panoramic view of the vehicle rear area.

Known vehicle rearview systems, which usually consist of a left and a right exterior mirror and an interior vehicle mirror, have various disadvantages: For example, the view is often restricted by passengers in the rear seats of the vehicle, in the case of a conventional car through the B and C pillars of the vehicle Vehicle, or through rear neck supports. The infamous "blind spot" stands prominently here.

In order to be able to see areas that are not or only very difficult to see through mirror systems NEN, video surveillance systems are used, for example for monitoring the rear area of trucks, a video image of the non-visible location being transmitted to a display device that can be monitored by the driver or other persons.

However, these systems have disadvantages: If areas are to be displayed with a high degree of detail, the displayable viewing angle and thus the displayable image area is relatively small. If the viewing angle of the camera is greatly expanded, such as by using a "fisheye" lens, the display in the peripheral areas of the field of view is very poor in detail.

The invention is based on the object of providing a device and a method which provide an overall panoramic image of the vehicle rear area which has a wide viewing angle, that is to say the viewing area recorded at the same time is very large and in particular detects the "blind spots" of the vehicle rear area and has a high level of detail in all areas of the image.

This object is achieved according to the invention by a vehicle rear view system as a device according to claim 1 and a method according to claim 17.

By using two or more video cameras, which are each rotated about the same axis with respect to one another, so that the vehicle rear space to be displayed is captured in neighboring images that overlap each other, a wide field of view is captured and a uniformly good image resolution is achieved in the process.

Under video camera there is any image sensor, which can be equipped with a lens system, for example.

The digital image processing means provided make it possible to generate an overall panorama image from these adjacent and overlapping video images and to display them on the image display unit, the overall panorama image being a time-continuous and quasi-delay-free overall representation of what is happening in the rear of the vehicle. With these properties, the system can be a complete replacement for conventional vehicle rear-view systems, so that not only the disadvantages described at the outset are eliminated, but also greater freedom is available in the design of the vehicle, so that the aerodynamic properties improve Fuel consumption can be reduced. Furthermore, a considerably improved clarity is achieved in that there is a representation as an overall picture, which provides an even better viewing area compared to a conventional vehicle rear-view mirror, and at the same time has the advantage of free placement of the image display unit.

The fact that the video cameras are arranged in a fixed spatial assignment significantly reduces the computational effort required for the image processing means; the same applies to the feature that the video cameras are arranged in close proximity to one another and that the video cameras are each rotated about the same axis.

This fact is of particular importance for the advantageous embodiments of the invention. hend that the image processing means are designed so that they seamlessly join the individual images of the mechanically coupled video cameras to a panoramic image. The same applies if the image processing means are advantageously designed such that they correct the image distortions of the mechanically firmly coupled video cameras before the individual images of the video cameras are joined together. These falsifications include, on the one hand, camera-related, for example lens distortions, and on the other hand arrangement-related, for example position shifts.

Furthermore, the fixed spatial allocation of the video cameras is important if the image processing means are advantageously further developed in such a way that they have means for calibrating the electronic vehicle rear view system, i.e. Provide functions to permanently assign object points of the area to be displayed to pixels in the display. The functionality for calibrating the system is also significant and advantageous, especially with regard to the aging of the components involved, in particular the video cameras, since a change in the properties of the components to a changed display geometry, i.e. leads to a changed mapping of object points in the area to be displayed to pixels in the display. A calibration, ie a reassignment of object points to pixels, compensates for these, for example age-related, changes.

The fact that means for selecting and / or enlarging sections of the overall image are provided in the image processing means, that is to say an image section of the vehicle rear area is selected and closer can be fetched due to the functionality provided by the image processing means, mechanical components, such as those required for adjusting the camera position and / or zooming in by an optical lens system, can be dispensed with.

To further reduce the computational effort for the image processing means, and thus particularly advantageous, it is to rotate the spatially assigned cameras in such a way that the overlap of the adjacent video individual images is minimal.

In practical use, it also appears to be particularly favorable to provide the image processing means and / or the video camera with means that adjust the provided image to environmental lighting conditions, such as twilight, daylight, or to glare, such as those e.g. can be caused by vehicles approaching from behind with the high beam switched on. This can include, for example, a very large dynamic range of over 100 dB, so that good detection properties are always guaranteed in the case of very strong brightness and / or darkness.

An advantageous further development of the vehicle rear view system provides for the image processing means to be designed in such a way that they display and / or incorporate additional information and / or images from other video cameras, for example special views for reverse parking, into the overall display.

In practical use of such a vehicle review system, the system often becomes cost-saving and simplifies technical implementation as a subsystem of a comprehensive vehicle monitoring or Vehicle multimedia system.

To further develop the image processing means, it is inexpensive and also technically advantageous to provide the first processing means in the image processing means which are particularly suitable for processing the image data quickly and thus performing the data- and computation-intensive work, and providing second processor means for the rest Perform tasks, including the actual merging of the images, but also parts of or the full user interaction. DSP processors (Digital Signal JProcessing) are particularly suitable for use as the first processor means for data and compute-intensive image processing, and a standard PC processor as a second processor means due to the high level of distribution and easy availability.

In order to provide a particularly suitable connection between video cameras, image processing means and image display unit, a reliable high-speed data bus, which can be designed, for example, as a FireWire or CameraLink bus, is particularly suitable in order to meet the high demands on availability and display speed, e.g. in traffic, to be able to do justice.

Depending on the size and shape of the camera housing and / or the camera, it is particularly advantageous for realizing the property that the video cameras located in a fixed spatial association with one another are located in the immediate vicinity, that is to say as close as possible to one another, the cameras arranged around one another about the same axis to shift vertically to each other by which they are rotated. It becomes one Arranged in which the cameras are (quasi) one above the other and are rotated so that only a small proportion of the respective camera images overlap.

The method for providing the panoramic view of the rear area unfolds its advantages precisely in connection with the above-described devices of the vehicle rear-view system: In that two video images are continuously captured in such a way that the perspective of the images deviates from one another as described in the device part and thereby changes If a small proportion of the images of the video cameras overlap, the image data are collected from the same viewing position in different viewing directions so that they can be preprocessed in the second method step with as little effort as possible, the preprocessing being able to include a correction of the respective camera-related geometric distortions of the video images / or a cylindrical projection of the video images, and that in a third step the video images are combined in such a way that a seamless, continuous and instantaneous real-time panorama image from the video image rn is put together and finally presented.

It is particularly advantageous that the digital preprocessing of the video images includes identifying those parts of the individual video images which represent one and the same object and those of different parts of the image in the respective images, that is to say those which do not represent the same objects differ.

This enables the image processing to separate data and computation-intensive image parts from the less computation-intensive image parts so that they can be processed further in accordance with this property, for example by giving the overlapping image areas more computing capacity or processing priority than the non-overlapping ones.

Separate further processing of the overlapping parts of the video images and the non-overlapping parts of the video images is particularly advantageous, these parts of the image being reassembled later when the non-overlapping parts are subjected to a fixed offset correction, i.e. they are corrected in a predefined manner in accordance with the fixed camera arrangement and only the overlapping image contents are matched to one another, e.g. by estimating the image contents by pattern recognition, matching them and joining them together.

Here the advantage of a separate further processing becomes clear, because the computational effort required can be significantly reduced if only the overlapping image contents are subjected to the latter computationally intensive process and the rest are carried out by a low-cost offset correction.

Furthermore, the computational effort can be reduced in that the method uses previously calculated results, quasi as an intermediate result, in a look-up table and thus avoids unnecessary recalculations. These previously calculated results can be from a previous calibration process come .

In addition, the preprocessing, that is to say the acquisition of the image, the correction of the distortions caused by the camera and the cylinder projection, and the identification of the overlapping image areas can be carried out separately and simultaneously for different cameras, as a result of which the results of the image preprocessing can be made available more quickly overall , It is necessary for the respective image preprocessing processes to communicate with one another.

Overall, one advantage of using digital image processing means, or of a method which digitally preprocesses the video images, is that the selection of the image section or the enlargement or reduction of the displayed image is entirely electronic, that is, by means of digital image processing without mechanically moving parts. Furthermore, in comparison to a conventional vehicle rear view system, you can choose between a mirrored or a realistic representation.

In order to demonstrate the feasibility of the invention by way of example, a possible structure is described below with reference to the accompanying drawings. However, alternative embodiments are conceivable in particular for industrial series production or for broad practical use.

It shows:

1 shows a positioning for the two video cameras capturing the vehicle review and for the image display device (display) representing the overall picture of the review,

2 shows the basic representation and division of areas that are captured by the cameras,

3 the representation of the areas captured by the cameras if they are spatially and spatially offset according to the device,

4 shows a schematic representation of a possible structure of the electronic vehicle rear view system,

5 shows a possible variant of the method for acquiring the video images, their processing and the representation,

6 shows an overview of the image processing means in connection with the method in one possible embodiment for series production.

Fig. 1 illustrates the practical use of the vehicle rearview system when it is used in a passenger car 1. The video cameras 2 and 3 are positioned in the area of the rear window 6. These cameras are oriented so that they capture the entire rear space 5 of the vehicle 1, including the blind spots. For this purpose, cameras 2 and 3 are arranged offset from one another in terms of angle and / or height. The image information, which is processed by image processing equipment in the vehicle and combined to form an overall panorama image, are shown on an image display unit 4, which is located in the area 7 which is clearly visible to the driver. The situation in the rear of the vehicle 5 is shown in the display device 4 free of distortions and without perceptible delays. A flowing transverse movement through the vehicle rear area 5, that is to say transversely to the orientation of the vehicle, is seamlessly displayed on the display device 4 as an equally continuous movement, even and especially when it changes from the viewing area of the camera 2 to the viewing area of the camera 3.

If there are changes in the brightness conditions in the vehicle rear area 5, at dusk or in glare situations due to headlights from other vehicles, the display 4 is automatically adapted to the lighting conditions.

The cameras 2 and 3 are spatially assigned to each other, i.e. fixed immobile during operation. This allows the calibration of the vehicle rear view system, which means to adjust the system so that a fixed assignment of object points in the vehicle rear space 5 to be displayed to pixels in the display on the display unit 4 can be achieved. Such a calibration is possible, similar to the setting of the headlights of a motor vehicle, by using a uniformly patterned panel provided for this purpose in the rear area 5 of the vehicle.

If a specific area is to be selected from the vehicle rear area 5 for display on the display unit 4, as it were the adjustment of the rear view mirror in conventional vehicle rear view systems, this is done by the provided image processing equipment without mechanically moving parts. Likewise, part of the scenery in the vehicle rear area 5 can be zoomed in or reduced. Here too, mechanically moving parts can be dispensed with thanks to the image processing means. The overall panorama image is displayed on the display unit 4 either mirror-inverted or sideways. In addition to displaying the overall panorama image, animations to inform the driver, text and symbols, can be shown, as well as additional camera perspectives for certain driving conditions. The area near the bumper and the wheels are considered for parking.

FIG. 2 shows which areas basically arise when an image is captured by two video cameras 2 and 3. There are areas 11 and 12, each of which is only captured by a camera 2 or 3. Another area 10 is captured by both cameras 2 and 3 at the same time; this is the area where the two video images overlap, the image overlap area. The area 13 is not captured by either of the two cameras 2 and 3.

In order to display the vehicle rear space as completely as possible, the video cameras 2 and 3 are positioned such that the area 13, which is not covered by either of the two video cameras, is virtually non-existent. Furthermore, the cameras 2 and 3 are positioned in such a way that the area 10, which is captured by both cameras, is small, in order to reduce computing effort for digital image processing and thus to enable a very fast display, ie a quasi-delay-free real-time display , Such an arrangement of the cameras 2 and 3 and the resulting areas 10, 11 and 12 are shown accordingly in FIG. 3. Since the cameras themselves have a certain spatial extent, it is necessary to do not just the cameras by one

Rotate the axis apart, but also to move them around the same axis so that the cameras can be arranged in close proximity to each other. This again considerably reduces the computing effort for image processing. Since the spatial assignment of the

Cameras 2 and 3 is fixed, only the image information from the overlap area 10 has to be correlated in the course of the image processing, i.e. be brought into agreement and put together, while the height offset of the cameras to one another is taken into account and set in the aforementioned calibration process and therefore has to be corrected with little effort by the image processing means.

The basic structure of the electronic vehicle rear view system is shown in FIG. 4. The cameras 2 and 3 are each connected to a unit consisting of a processor for digital image processing and memories 21 and 31. These units 21 and 31 communicate with one another and with additional memory and a graphics unit 41, which represents the overall image as a composite on the image display unit 4, via the FireWire or IEEE 1394 bus as a reliable high-speed bus.

5 shows the mode of operation of the image processing. The images of the video cameras 2 and 3 are first acquired and preprocessed. This preprocessing takes place here, as already shown in FIG. 4, for each of the video cameras 2 and 3 on a separate connected unit 21 and 31 from digital signal processing processor and memory, instead; for camera 2 these are steps 22, 23, 24, 25 and 26, for camera 3 steps 32, 33, 34, 35 and 36.

First, the video images from cameras 2 and 3 are captured in steps 22 and 32. Through the communication of the units 31 and 21 in FIG. 4, the image overlap area 10 is immediately identified and represented as an ROI parameter (region

Of Interest) 50. Then, in steps 23 and 33, the static image errors are corrected by lens distortions or camera geometry. Previously calculated results in Lokup tables 28 and 38 are used to save computing effort. In the next step 24 or 34, a cylinder projection of the video images is carried out.

Based on the image overlap / region of interest

Parameter 50 is now differentiated between the image overlap area 10 (FIG. 3) and the non-overlapping areas 11 and 12 (FIG. 3) and in separate ways 25 and 35 for the overlap area and 26 and 36 for the non-overlapping area , further processed.

The non-overlapping areas are shifted in their image position by a simple fixed correction in 27 or 37 and are available for display in 46 after a memory shift.

The overlapping image areas are first correlated in 42, interpolated in 43 by triangulation, ie the overlapping image contents are estimated to be identified and 44 are identified assigned the same image content and corrected for misalignment and put it together in 45. Steps 43 to 45 again use a lookup table 51 which contains previously calculated results. In step 46, the resulting overall image is displayed on the image display unit 4.

By integrating additional sensors such as infrared or radar, the depth of field properties of the electronic are not shown in detail

Vehicle rearview system improved. Furthermore, the image information is stored for a certain period before the airbag is deployed, and image sequences can thus be reproduced immediately before an accident.

An overview of the structure and sequence of image processing in one possible embodiment, which is suitable for industrial series production and / or wide practical use, is shown in FIG. 6. The image processing system can be roughly divided into the subsystems image preprocessing 71, image main processing 72 and display and communication 73. The subsystem for the image preprocessing 71 includes the image acquisition components 64 and the scaling components 65, the subsystem for the main image processing 72 includes the correlation component 66 and the merging component 67, and the subsystem for display and communication 73 includes the display component 68, the system control component 69 and the network communication component 70.

The communication of the individual processing components takes place via the reliable high-speed bus 74, characterized here by the dic- ken arrow connections, while the thin arrow connections represent the data flow schematically.

In a manner not shown in more detail, the images from the video cameras are transmitted via the data flow 61 and 62 to a component 64 for image acquisition and locomotive table processing. Via the high-speed bus 74, which is designed as a FireWire bus, the data is transmitted to the scaling component 65, which carries out an image size adjustment. Again, the overlapping part of the image data is transmitted to the correlation component 66 via the high-speed bus 74, while the non-overlapping part of the image data is transmitted directly to the assembling component 67 via the high-speed bus 74. After the correlation process, the correlation component 66 also transmits the result data to the assembly component 67 via the bus 74. The data processed by the assembly component 67 now arrive via the bus 74 in the display component 68, which contains the image data and images of further video cameras 63, whose communication link to the image processing system is not shown in detail here, merges with the image and makes it available for display, as it should finally be displayed on the image display device.

User inputs and settings 77 are processed by the system control component 69 and forwarded via the high-speed bus 74 to the display component 68, which controls the actual image display unit 4 in FIG. 1 via the output data stream 75.

Also commute via the high-speed bus 74 identifies the network communication component 70 with the other components of the image processing system. This network communication component creates the connection to the vehicle bus with which the information of the data stream 76 is to be exchanged.

Claims

claims

1. Vehicle rear view system for providing a rear (5) panoramic view (4) with at least two video cameras (2, 3) and at least one image display unit (4), the video cameras (2, 3) with the image display unit (4) are connected for the transmission of the image information, characterized in that at least two of the video cameras (2, 3) are arranged in a fixed spatial association and in close proximity to one another and are each rotated about the same axis so that the delivered images overlap , and that digital image processing means (21, 31,

41) between these video cameras on the one hand and the image display unit (4) on the other hand are provided for the continuous generation of an overall image from the images of these video cameras.

2. Vehicle rear view system according to claim 1, characterized in that the cameras arranged next to one another are shifted relative to one another about the same aforementioned axis.

3. Vehicle rear view system according to one of claims 1 or 2, characterized in that the overall image (4) generated by the image processing means (21, 31, 41) is a panorama image from a seamless juxtaposition of the images of the spatially assigned video cameras (2, 3).

4. Vehicle rear view system according to one of claims 1 to 3, characterized in that the overall image generated by the image processing means (21, 31, 41) is a distortion-corrected joining of the images of the spatially assigned video cameras (2, 3).

5. Vehicle rear view system according to claim 4, characterized in that means for calibrating the system in connection with the fixed spatial allocation of the video cameras (2, 3) are provided in the image processing means (21, 31, 41).

6. Vehicle rear view system according to one of claims 1 to 5, characterized in that means for selecting and / or enlarging sections of the overall image are provided in the image processing means (21, 31, 41).

7. Vehicle rear view system according to one of claims 1 to 6, characterized in that the spatially assigned cameras are rotated so that the overlap of the images of these video cameras is minimal in connection with the opening angle of these cameras.

8. Vehicle rear view system according to one of claims 1 to 7, characterized in that video cameras are positioned so that the provided rear panorama view (4) has a uniformly high image resolution with respect to

Has to be shown (5).

9. Vehicle rear view system according to one of claims 1 to 8, characterized in that means are provided in the image processing means (21, 31, 41) and / or in the video cameras (2, 3) for adapting the provided image to ambient light conditions and glare.

10. Vehicle rear view system according to one of claims 1 to 9, characterized in that means for integrating additional information and / or further images into the representation (4) are provided in the image processing means (21, 31, 41).

11. Vehicle rear view system according to one of claims 1 to 10, characterized in that the system and / or the image processing means (21,

31, 41) of the system are a subsystem of a broader monitoring and / or multimedia system.

12. Vehicle rear view system according to one of claims 1 to 11, characterized in that the image processing means (21, 31, 41) comprise at least a first processor means (21) for fast digital processing of the camera images and at least one second processor means for combining the camera images.

13. Vehicle rearview system according to claim 12, characterized in that the first processor means is a DSP processor.

14. Vehicle review system according to one of claims 12 or 13, characterized in that the second processor means is a standard PC processor.

15. Vehicle rear view system according to one of claims 1 to 14, characterized in that the connection between cameras and image processing means and image display unit is designed as a reliable high-speed data bus.

16. Vehicle rear view system according to one of the preceding claims, characterized in that the image display unit (4) is positioned in the driver's field of vision and / or projects the image into the driver's field of vision.

17. A method for providing a rear panoramic view of a vehicle, characterized in that at least two video images are continuously captured (22, 32) in such a way that the viewing direction of the images deviates from one another and a small proportion of the images overlap, and the images are digitally preprocessed (22 to 26, 32 to 36) and the video images are finally combined (27, 37, 42 to 46) in such a way that a seamless, continuous and instantaneous panorama image is created from the video images.

18. The method according to claim 17, characterized in that the preprocessing comprises a correction (23, 33) of the camera-related geometric distortions of the video images.

19. The method according to any one of claims 17 or 18, characterized in that the preprocessing comprises a cylinder projection (24, 34) of the video images.

20. The method according to any one of claims 17 to 19, characterized in that the overlapping parts of the video images are identified (50) and distinguished from the non-overlapping parts.

21. The method according to claim 20, characterized in that the overlapping and the non-overlapping image parts are processed separately (25, 26, 35, 36) and later reassembled (46).

22. The method according to claim 21, characterized in that the non-overlapping parts are subjected to a fixed offset correction (27, 37) in accordance with the fixed camera arrangement.

23. The method according to any one of claims 21 or 22, characterized in that the overlapping image contents are identified (43) and brought into agreement (44) and are then suitably combined (45).

24. The method according to any one of claims 17 to 23, characterized in that a table (28, 38, 51) is used to use the results already calculated.

25. The method according to any one of claims 17 to 24, characterized in that the preprocessing and / or the identification of the overlapping image areas of the video cameras is carried out separately and simultaneously for each camera and that the processes are continuously coordinated with one another.