KR20200068045A - Around view monitoring system and method for detecting parking line with deep learning - Google Patents

Abstract

Disclosed are an around view monitoring (AVM) system and a parking line detection method using deep learning. The AVM system comprises: an image processing unit generating an AVM image in a top view form by using photographed images generated by a plurality of cameras; a parking line detection unit detecting a parking line using a corner point of a baseline and a parking line detected using each of a plurality of detection techniques previously designated for the AVM image; and an output unit outputting the AVM image overlaid with a parking line guideline corresponding to the parking line. The AVM system of the present invention can quickly and accurately detect the parking line, detect the parking line even when an obstacle exists inside the parking area, and overlay and display the parking line guideline to the AVM image, so that a driver can precisely recognize a parking target area.

Description

Around view monitoring system and method for detecting parking line with deep learning}

The present invention relates to a parking line detection method using an around view monitoring system and deep learning.

The Around View Monitoring (AVM) system captures the surrounding environment through cameras installed on the front, rear, left, and right sides of the vehicle, and corrects the overlapping area to appear naturally based on the captured image. The surrounding environment is displayed on the screen as a top view AVM image. The driver can accurately recognize the surroundings of the vehicle by referring to the displayed AVM image, and can conveniently park without looking in the side mirror or rearview mirror.

1 schematically illustrates the concept of an AVM system.

1(a) and 1(b), cameras 10, 20, 30, and 40 are installed on front, rear, and left and right sides of the vehicle, respectively. Each of the cameras 10, 20, 30, and 40 captures the images A, B, C, and D of the front, rear, and left and right sides of the vehicle, and reconstructs the captured image into a top view AVM image looking down at the vehicle. And outputs it to the display device mounted on the vehicle.

Recently, a technology for recognizing a parking line using a camera has been actively developed to recognize a parking area of a parking lot, and there are a method of using a hough transform and a method of template matching as a main development method. .

However, the method using the Hough transform has the advantage of being able to detect the parking line by simple calculation, but there is a problem that the parking line cannot be effectively detected in a specific surrounding environment.

In addition, the method using template matching has an advantage in that parking lines can be detected regardless of the surrounding environment, but there is a problem in that the detection speed is very slow.

Korean Patent Publication No. 10-2017-0124299 (released on November 10, 2017)

J. S. Lee and S. C. Kee, "Empty Parking Space Detection Method using Around View Monitoring System", in Institute of Control, Robotics and Systems, vol. 23, no. 6, pp. 455-461, 2017 J. K. Suhr and H. G. Jung, "Automatic parking space detection and tracking for underground and indoor environments," IEEE Transactions on Industrial Electronics, vol. 63, no. 9, pp. 5687-5698, 2016 M. Fan, Z. Hu, K. Hamada, and H. Chen, "Line Filter-Based Parking Slot Detection for Intelligent Parking Assistance System," in Proceedings of SAE-China Congress 2014: Selected Papers, 2015, pp. 175-181 L. Zhang, J. Huang, X. Li, L. Xiong. "Vision-Based Parking-Slot Detection: A DCNN_Based Approach and a Large-Scale Benchmark Dataset," IEEE Transactions on Image Processing, vol. 27, no. 11, pp.5350-5364, 2018

The present invention is to provide a parking line detection method using deep learning and an around view monitoring system capable of quickly and accurately detecting a parking line and detecting a parking line even when an obstacle exists inside the parking area.

The present invention is to provide a parking line detecting method using deep learning and an around view monitoring system that enables a driver to clearly recognize a parking target area by generating and displaying a parking line guideline corresponding to the detected parking line.

Objects other than the present invention will be easily understood through the following description.

According to an aspect of the present invention, an image processing unit generating an AVM image in a top view form using a captured image generated by a plurality of cameras; A parking line detection unit that detects a parking line using a corner point of a baseline and a parking line that are independently detected by using each of a plurality of detection techniques previously specified for the AVM image; And an output unit for outputting an AVM image overlaid with a parking line guideline corresponding to the parking line.

The parking line detection unit generates a superimposed feature map by applying a line filter to the AVM image, determines a baseline of the parking line by applying a Huff transform to the superimposed feature map, and a predetermined angle to the baseline A first corner detector configured to detect a straight line connected to and detect an intersection of the base line and the straight line as a corner point of the parking line; A second corner detection unit that detects a corner point of the parking line by applying an object detection model previously learned by a deep learning technique to the AVM image using a previously stored parking line data set; A verification unit that selects only a corner point existing in the same position as a valid corner point by comparing the corner point detected by the first corner detection unit and the corner point detected by the second corner detection unit; And a parking line detection unit that determines a parking line based on the base line and the effective corner point.

The first corner detection unit may determine the straight line having the largest calculated value using the length of each of the straight lines detected by Huff conversion and the density of pixels constituting the straight line as a baseline.

The parking line detection unit determines a straight line connected to the base line at a predetermined angle from the effective corner point among the straight lines detected by Huff conversion as a side line forming a parking line, and uses the base line and the side line. To determine the parking line.

According to another aspect of the present invention, a computer program stored in a computer-readable medium for detecting a parking line, the computer program causing the computer to perform the following steps, the steps comprising: (a) AVM image Generating an overlapping feature map by applying a line filter; (b) Applying a Hough transform to the overlapped feature map to determine the baseline of the parking line, and detecting a straight line connected at a predetermined angle to the baseline to cross the intersection of the baseline and the straight line to the corner point of the parking line Detecting by; (c) detecting a corner point of a parking line by applying a previously learned object detection model to the AVM image using a deep learning technique using a previously stored parking line dataset; (d) comparing the corner points detected in step (b) with the corner points detected in step (d) to select only corner points present in the same position as valid corner points; And (e) determining a parking line based on the baseline and the effective corner point.

The baseline may be a straight line having the largest calculated value using the length of the straight line and the density of pixels constituting the straight line among the straight lines detected by the Huff transform of the overlapped feature map.

In the step (e), a straight line connected to the base line at a predetermined angle from the effective corner point among the straight lines detected by Huff conversion is determined as a side line forming a parking line, and the base line and the side line are determined. The parking line may be determined using.

Other aspects, features, and advantages other than those described above will become apparent from the following drawings, claims, and detailed description of the invention.

According to an embodiment of the present invention, it is possible to quickly and accurately detect a parking line, and it is possible to detect a parking line even when an obstacle exists inside the parking area.

There is also an effect of overlapping the AVM image on the display device provided inside the vehicle so that the parking line guideline is displayed so that the driver can clearly recognize the parking target area.

1 schematically illustrates the concept of an AVM system.
2 is a block diagram of an AVM system according to an embodiment of the present invention.
3 and 4 are views for explaining the operation of the first corner detection unit according to an embodiment of the present invention.
5 is a view for explaining a deep learning reason of the second corner detection unit according to an embodiment of the present invention.
6 is a view showing the actual experimental results of the AVM system according to an embodiment of the present invention.
7 is a flowchart illustrating a parking line detection method using deep learning according to an embodiment of the present invention.

The present invention can be applied to various changes and can have various embodiments, and specific embodiments will be illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

When an element is said to be "connected" to or "connected" to another component, it is understood that other components may be directly connected to or connected to the other component, but may exist in the middle. It should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that no other component exists in the middle.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. The terms are used only for the purpose of distinguishing one component from other components.

The terms used herein are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, terms such as “include” or “have” are intended to indicate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, and that one or more other features are present. It should be understood that the existence or addition possibilities of fields or numbers, steps, operations, components, parts or combinations thereof are not excluded in advance.

In addition, terms such as “…unit”, “…unit”, “…module”, and “…group” described in the specification mean a unit that processes at least one function or operation, which is hardware or software or hardware and software. It can be implemented as a combination of.

In addition, in the description with reference to the accompanying drawings, the same components, regardless of reference numerals, are assigned the same or related reference numerals, and redundant description thereof will be omitted. In the description of the present invention, when it is determined that detailed descriptions of related known technologies may unnecessarily obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

2 is a block diagram of an AVM system according to an embodiment of the present invention, and FIGS. 3 and 4 are views for explaining the operation of the first corner detection unit according to an embodiment of the present invention. 5 is a diagram for explaining a deep learning reason of a second corner detection unit according to an embodiment of the present invention, and FIG. 6 is a view showing actual experimental results of the AVM system according to an embodiment of the present invention.

Referring to FIG. 2, the AVM system may include a camera unit 200, an image processing unit 210, an overlay unit 220, an output unit 230, and a parking line detection unit 240.

The camera unit 200 includes four or more cameras for AVM systems. Each camera is installed on the front and rear and right and left of the vehicle so as to obtain an image around the vehicle, and may be implemented as, for example, a wide-angle camera including a fish eye lens. Here, the wide-angle camera may further include an electrical device such as an image sensor that converts an optical signal into an electrical signal, as well as a simple optical device.

The image processing unit 210 may convert a plurality of captured images input from the camera unit 200 into a single AVM image. Here, the AVM image is an image that is captured in front, rear, left, and right of a vehicle and is synthesized as a top-view image perpendicular to the ground and displayed on one screen.

At this time, since the positions of the cameras installed in the front, rear, left, and right of the vehicle are different from each other, and the distance from the ground, the image processing unit 210 may generate AVM images by correcting distortion of each of the captured images through image processing. . To this end, the image processing unit 210 may, for example, correct a luminance and color deviation of a captured image, compensate for lens distortion to a fish-eye lens, compensate for a deviation in perspective, and convert a plurality of captured images into one AVM. You will be able to convert it to video. In the AVM image, a virtual vehicle generated using a pre-registered size of a vehicle on which the AVM system is installed may be displayed together (see (b) of FIG. 1 ).

The overlay unit 220 may process the AVM image generated by the image processing unit 210 so that the parking line guideline corresponding to the parking line detected by the parking line detection unit 240 to be described later is overlaid and displayed.

The AVM image overlaid with the parking line guideline may be output through, for example, the output unit 230 which is a display device provided in the vehicle. Of course, when the detected parking line does not exist, it is natural that the AVM image in which the parking line guideline is not overlaid may be output through the output unit 230.

The parking line detection unit 240 may include a first corner detection unit 231, a second corner detection unit 233, a verification unit 235, a learning DB 237, and a parking line detection unit 239.

The first corner detection unit 241 detects a baseline and a corner point of the parking line using a first predetermined detection technique. Here, the predetermined first detection technique may be a technique using a line filter and a hough transform.

First, the first corner detector 241 detects a parking line using a line filter. In general, the parking line is made of a light colored line having a certain thickness to increase the driver's perception, but the first corner detection unit 241 of this embodiment is conventional to enable the parking line to be detected more effectively in various environments. Compared to edge detection filters, such as Sobel filters or canny filters, line filters with relatively strong changes in lighting and shadows are applied.

The line filter is {-1,… , -1, 0, 1,… , 1} has a mask structure, and when the AVM image is calculated with a line filter in the X and Y directions, respectively, the part where the parking line starts (ie, the area that becomes relatively bright) has a rising edge. In addition, it is recognized that the part where the parking line ends (that is, a region that is relatively dark) has a falling edge. Using this feature, once again, there is a characteristic that the highest response value appears at the center position of the parking line.

For reference, FIG. 3(b) shows a feature map calculated in the X direction with a line filter for an AVM image (see FIG. 3(a)) of a situation in which a vehicle is parked in a parking line. Fig. 3(c) shows the feature map calculated in the Y direction with the line filter for the same AVM image, and Fig. 3(d) shows the overlapped feature map of the result of binarizing the two directions.

When referring to each feature map illustrated in FIG. 3, it can be seen that a portion (for example, a parking line, a plant, etc.) having a relatively bright value than surrounding values has a relatively large feature. Also, when referring to each feature map, it can be also seen that the parking line is displayed to have a straight line shape.

The first corner detection unit 241 performs Hough transformation on an overlapped feature map (see FIG. 3D) calculated with a line filter. The Hough transform is generally used to find features such as straight lines, curves, and circles in an image, and detailed descriptions thereof are omitted.

When the Hough transformation is performed on the overlapped feature map, as illustrated in FIG. 4(a), the parking line displayed in the form of a straight line in the overlapped feature map, as well as parts having other large features, may be linearly formed. It is detected in the form.

At this time, considering that the first corner detection unit 241 is for the purpose of detecting the parking line, lines other than the parking line need to be removed as unnecessary.

To this end, the first corner detector 241 can selectively detect only the parking line. The line having the largest calculated value using the length of the detected line and the density of the pixels in the line is selected as a baseline. At this time, there are various methods for calculating the calculated values for each line and determining the superiority of the calculated values for the selection of the baseline, such as adding weights to each of the calculated line length and pixel density.

That is, the first corner detector 241 calculates the length of the line and the density of the pixels constituting the line for each of the lines detected by the Hough transform, and then selects the line having the largest value as a baseline (FIG. 4) (B) see the green circle).

Thereafter, the first corner detection unit 241 detects a corner point of the parking area based on the selected baseline. In general, since the Hough transform detects a line based on two points, the equation of each straight line can be calculated using each of the two points.

Therefore, the first corner detector 241 calculates the slope using the linear equation of the baseline, calculates the angle difference between the slopes of the straight line extracted as a result of the Huff transform, and has a line perpendicular to the slope of the baseline. If present, the location of the intersection can be detected as a corner point of the parking line (see yellow circles in FIG. 4(c)). At this time, the detected corner points are located on the baseline. Here, it is natural that a corner point may be detected by detecting a line having a corresponding inclination angle when the sideline dividing the parking line is connected to the baseline to have a vertical or arbitrary inclination angle.

However, in the case of the Hough conversion, since various lines are detected, there is a problem in that the wrong position can be detected as a corner point. The parking line detection unit 240 according to the present embodiment includes a second corner detection unit 243 and a verification unit ( 245) is used to perform verification for the detection of an accurate corner point. For reference, in FIG. 4(c), four points indicated by yellow circles are detected as corner points, but the second position from the left is a case where the corner points are incorrectly detected.

The second corner detector 243 detects a corner point of the parking line in the AVM image using a second predetermined detection technique using a learned object detection model. Here, the object detection model may be, for example, a You Only Look Once (YOLO) v2 model, Regions with Convolutional Neural Networks (R-CNN), Region based Convolutional Neural Networks (R-CNN) model, fast R-CNN model, or the like.

In general, an object detection model such as YOLOv2 may incorrectly detect an arrow road marking of the parking lot as a corner point of the parking line (see FIG. 5(a)), or the light reflected on the road surface connects with the actual parking line, and The case of incorrect detection as a corner point (refer to (b) of FIG. 5) frequently occurs.

In order to improve detection accuracy, the second corner detection unit 243 pre-trains the object detection model to be applied by using a deep learning technique using the parking line dataset stored in the learning DB 237, so that the corner point of the parking line is detected. Can be used for detection. The parking line dataset stored in advance in the learning DB 237 is, for example, a training dataset consisting of 9,827 images published in the article "Vision-Based Parking-Slot Detection: A DCNN_Based Approach and a Large-Scale Benchmark Dataset" It may be a test data set consisting of 2,338 images.

However, even if the second corner detection unit 243 detects a corner point of the parking line using an object detection model previously learned, the detection accuracy cannot be perfect. This is the same as if the first corner detection unit 241 described above detects the corner point of the parking line by the first detection technique using the line filter and the Hough transform, and the detection accuracy cannot be perfect.

Therefore, the parking line detection unit 240 according to the present embodiment uses the second corner detection unit 243 and the disadvantages of the first detection technique used by the first corner detection unit 241 using the verification unit 245 described below. The disadvantages of the second detection technique are complemented with each other, so that the corner points of the parking line can be detected more accurately.

The verification unit 245 compares the corner points detected by the first corner detection unit 241 and the corner points detected by the second corner detection unit 243, and only the corner points present in the same position are valid corner points. Select.

The parking line detection unit 249 determines the parking line based on the baseline determined by the first corner detection unit 241 and a valid corner point selected by the verification unit 245. At this time, a parking line (ie, a side line) connected to a base line at a predetermined angle (for example, a vertical or diagonal direction) using a valid corner point and a line detected in the Huff transformation by the first corner detector 241. This may be determined (see FIG. 4(d)).

The parking line detection unit 249 may provide a parking line guideline corresponding to the determined parking line to the overlay unit 220 so that the AVM image overlaid with the parking line guideline is output through the output unit 230.

FIG. 6 shows the experimental results of the inventor installing the four cameras in the vehicle and constructing the AVM environment, and then testing the parking line detection performance in the outdoor parking lot. At this time, the size of the input AVM image was 520x688, and the entire algorithm was performed in a Intel i7-7700 3.60GHz CPU environment and a desktop environment equipped with NVIDIA's RTX2070 graphics card.

As a result of the experiment, it was proved that the parking line was effectively detected as illustrated in FIG. 6. For ease of understanding, in the experiment results, the baseline was marked with a red line, the corner point of the parking line was marked with a blue circle, and the sidelines detected based on the baseline and corner points were marked with a green line. When referring to the experimental results, it is confirmed that the parking line detection unit 240 according to the present embodiment can not only detect the parking line regardless of the position of the vehicle, but also detect the parking line even when there is an obstacle. Can be.

7 is a flowchart illustrating a parking line detection method using deep learning according to an embodiment of the present invention.

Referring to FIG. 7, in step 710, the parking line detection unit 240 receives an AVM image in which a captured image generated by a plurality of cameras is synthesized in a top view form.

In step 720, the parking line detection unit 240 detects a corner point of the parking line according to each detection technique by using a plurality of detection techniques previously specified.

For example, the first detection technique may be a technique of detecting a baseline and a corner point of a parking line using a line filter and a Hough transform.

That is, the AVM image is computed in the X and Y directions respectively using a line filter, an overlapped feature map is generated by superimposing the results of the two directions, and a straight line detected by performing Hough transformation on the overlapped feature map. The line with the highest line length and the pixel density of the line is selected as the baseline.

Subsequently, a straight line connected to the slope of the linear equation of the baseline and a predetermined angle (for example, an orthogonal angle or a predetermined slope angle) is detected, and an intersection with the straight line is detected as a corner point of the parking line.

Also, the second detection technique may be a technique of detecting a corner point of a parking line in an AVM image using an object detection model trained by a deep learning technique. Here, an object detection model such as a YOLOv2 model may be pre-learned by a deep learning method using a parking line dataset previously stored in the learning DB 237 to reduce the probability of false detection. Since the practical method of detecting the corner point of the parking line using the learned object detection model is obvious to a person skilled in the art, a description thereof will be omitted.

In step 730, the parking line detection unit 240 compares the corner points of the parking line detected in each of a plurality of predetermined detection techniques to select only the corner points present in the same position as valid corner points. This is because each of the plurality of detection techniques used to detect the corner point of the parking line has a disadvantage, and the disadvantages of each detection technique are complemented with each other so that the corner points of the parking line can be detected more accurately.

In step 740, the parking line detection unit 240 determines the parking line based on the baseline determined in step 720 and the valid corner points selected in step 730. At this time, a parking line (ie, a side line) connected to the base line in a vertical or diagonal direction may be determined using a valid corner point and a line detected in the aforementioned Hough transformation, and a parking line guide corresponding to the determined parking line The line may be overlaid on the AVM image and output through the output unit 230.

As described above, the parking line detection method using the around view monitoring system and deep learning according to the present embodiment can quickly and accurately detect the parking line and detect the parking line even when an obstacle exists inside the parking area. The parking line guideline is overlaid and displayed on the AVM image, so that the driver can clearly recognize the parking target area.

It is natural that the above-described method for detecting a parking line using deep learning may be implemented as an automated procedure according to a time series sequence as it is implemented as a software program or application built into a digital processing device. The codes and code segments constituting the program and the like can be easily inferred by a computer programmer in the field. In addition, the program is stored in a computer readable media, and read and executed by a computer to implement the method.

Although described above with reference to embodiments of the present invention, those skilled in the art variously modify and change the present invention without departing from the spirit and scope of the present invention as set forth in the claims below. You can understand that you can.

10, 20, 30, 40: Camera 200: Camera unit
210: image processing unit 220: overlay unit
230: output unit 240: parking line detection unit
241: first corner detector 243: second corner detector
245: verification unit 247: learning DB
249: Parking line detection unit

Claims (7)

An image processing unit generating an AVM image in a top view form by using captured images generated by a plurality of cameras;
A parking line detection unit that detects a parking line using a corner point of the baseline and the parking line, which are independently detected by using each of a plurality of predefined detection techniques for the AVM image; And
An AVM system including an output unit for outputting an AVM image overlaid with a parking line guideline corresponding to the parking line.
According to claim 1,
The parking line detection unit,
A line filter is applied to the AVM image to generate an overlapping feature map, a Huff transform is applied to the overlapping feature map to determine a baseline of the parking line, and a straight line connected at a predetermined angle to the baseline is detected. A first corner detection unit detecting an intersection of the base line and the straight line as a corner point of the parking line;
A second corner detection unit that detects a corner point of the parking line by applying an object detection model previously learned by a deep learning technique to the AVM image using a previously stored parking line data set;
A verification unit that selects a corner point existing in the same position as a valid corner point by comparing the corner point detected by the first corner detection unit and the corner point detected by the second corner detection unit; And
And a parking line detector configured to determine a parking line based on the baseline and the effective corner point.
According to claim 2,
The first corner detection unit AVM system, characterized in that for determining the straight line having the largest calculated value using the length of each of the detected Huff transformed and the density of the pixels constituting the straight line as a baseline.
According to claim 2,
The parking line detection unit determines a straight line connected to the base line at a predetermined angle from the effective corner point among the straight lines detected by Huff conversion as a side line forming a parking line, and uses the base line and the side line. AVM system characterized in that to determine the parking line.
A computer program stored in a computer-readable medium for detecting a parking line, the computer program causing the computer to perform the following steps, the steps comprising:
(a) generating an overlapped feature map by applying a line filter to the AVM image;
(b) Applying a Hough transform to the overlapped feature map to determine a baseline of the parking line, and detecting a straight line connected to the baseline at a predetermined angle to cross the intersection of the baseline and the straight line to a corner point of the parking line Detecting by;
(c) detecting a corner point of the parking line by applying a previously learned object detection model to the AVM image using a deep learning technique using a previously stored parking line dataset;
(d) comparing the corner points detected in step (b) with the corner points detected in step (d) to select only corner points present in the same position as valid corner points; And
(e) determining a parking line based on the baseline and the valid corner point, a computer program stored on a computer-readable medium.
The method of claim 5,
The baseline is stored in a computer-readable medium, characterized in that the superimposed feature map is a straight line having the largest calculated value using the length of the straight line and the density of pixels constituting the straight line among the detected straight lines. Computer program.
The method of claim 5,
In the step (e), a straight line connected to the base line at a predetermined angle from the effective corner point among the straight lines detected by Huff conversion is determined as a side line forming a parking line, and the base line and the side line are determined. Computer program stored in a computer-readable medium, characterized in that the parking line is determined by using.

Images