CN114723785A

CN114723785A - Lane line key point tracking method, device, equipment and computer readable medium

Info

Publication number: CN114723785A
Application number: CN202210424741.7A
Authority: CN
Inventors: 胡禹超
Original assignee: HoloMatic Technology Beijing Co Ltd
Current assignee: HoloMatic Technology Beijing Co Ltd
Priority date: 2022-04-22
Filing date: 2022-04-22
Publication date: 2022-07-08

Abstract

The embodiment of the disclosure discloses a method, a device, equipment and a computer readable medium for tracking key points of a lane line. One embodiment of the method comprises: tracking each key point coordinate of the lane line in the key point coordinate set of the lane line to obtain a tracking result; updating the counter value of the tracker to obtain an updated counter value; determining a secondary tracker corresponding to the key point coordinate set of the lane line; matching the secondary tracker with trackers in the tracker list to obtain a matching result; updating the matching count value to obtain a target count value; and in response to the fact that the target count value meets a preset target condition, combining the auxiliary tracker and the tracker to obtain a target tracker, and adding the target tracker to the tracker list so as to continue to perform the track line key point tracking operation. The method and the device can improve the accuracy of the lane line key point tracking result.

Description

Lane line key point tracking method, device, equipment and computer readable medium

Technical Field

The embodiment of the disclosure relates to the technical field of computers, in particular to a method, a device, equipment and a computer readable medium for tracking key points of a lane line.

Background

The key point tracking of the lane line is an important technical direction in the field of unmanned driving. In highway and urban scenes, vehicles need to determine a running track according to the indication of a lane line. At present, when the key points of the lane line are tracked, the following methods are generally adopted: and (3) converting two-dimensional lane line key points in the image into three-dimensional lane line key points, and tracking the key points based on an optimization method of multi-frame image feature point matching.

However, when the above-mentioned method is adopted to track the key points of the lane line, the following technical problems often exist:

firstly, as the key points of the lane lines are the angular points of a section of the lane lines, the similarity between the points is large, and the situation of matching errors is easy to occur, so that the tracking result of the key points of the lane lines is not accurate;

secondly, the pixel positions of the key points of two adjacent frames are far, which easily causes the condition of missing matching, thereby reducing the accuracy of the tracking result of the key points of the lane line.

Disclosure of Invention

This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Some embodiments of the present disclosure propose lane line keypoint tracking methods, apparatuses, devices and computer readable media to address one or more of the technical problems mentioned in the background section above.

In a first aspect, some embodiments of the present disclosure provide a method for tracking a lane line key point, the method including: tracking each lane line key point coordinate in a lane line key point coordinate set based on a tracker corresponding to a pre-generated lane line key point coordinate set in a preset tracker list to obtain a tracking result, wherein the tracker comprises a tracker count value; in response to determining that the tracking result does not meet a preset tracking condition, updating the count value of the tracker to obtain an updated count value; in response to determining that the updated count value meets a preset count condition, determining a secondary tracker corresponding to the set of lane line key point coordinates, wherein the secondary tracker comprises a matching count value; matching the secondary tracker with trackers in the tracker list to obtain a matching result; updating the matching count value by using the matching result to obtain a target count value; and in response to the fact that the target count value meets a preset target condition, combining the auxiliary tracker and the tracker to obtain a target tracker, and adding the target tracker to the tracker list so as to continue to perform the track line key point tracking operation.

In a second aspect, some embodiments of the present disclosure provide a lane line keypoint tracking apparatus, the apparatus comprising: the tracking processing unit is configured to track and process each key point coordinate of the lane line in a key point coordinate set of the lane line based on a tracker corresponding to the key point coordinate set of the lane line generated in advance in a preset tracker list to obtain a tracking result, wherein the tracker comprises a tracker count value; a first updating unit configured to update the tracker count value to obtain an updated count value in response to determining that the tracking result does not satisfy a preset tracking condition; a determining unit configured to determine a sub-tracker corresponding to the set of lane line key point coordinates in response to determining that the updated count value satisfies a preset count condition, wherein the sub-tracker includes a matching count value; the matching processing unit is configured to match the secondary tracker with trackers in the tracker list to obtain a matching result; a second updating unit configured to update the matching count value by using the matching result to obtain a target count value; and the merging processing unit is configured to perform merging processing on the secondary tracker and the tracker to obtain a target tracker in response to the fact that the target count value meets a preset target condition, and add the target tracker to the tracker list so as to continue to perform track line key point tracking operation.

In a third aspect, some embodiments of the present disclosure provide an electronic device, comprising: one or more processors; a storage device having one or more programs stored thereon, which when executed by one or more processors, cause the one or more processors to implement the method described in any of the implementations of the first aspect.

In a fourth aspect, some embodiments of the present disclosure provide a computer readable medium on which a computer program is stored, wherein the program, when executed by a processor, implements the method described in any of the implementations of the first aspect.

The above embodiments of the present disclosure have the following advantages: by the lane line key point tracking method of some embodiments of the present disclosure, the accuracy of the lane line key point tracking result can be improved. Specifically, the reason for causing the inaccuracy of the tracking structure of the key points of the lane line is as follows: since the key points of the lane lines are the corner points of a section of the lane lines, the similarity between the points is large, and the situation of matching errors is easy to occur. Based on this, in the disclosed method for tracking the key points of the lane line in some embodiments, first, based on a tracker in a preset tracker list corresponding to a pre-generated coordinate set of key points of the lane line, the coordinates of the key points of each lane line in the coordinate set of key points of the lane line are tracked, and a tracking result is obtained. By introducing a tracker list, the coordinates of the key points of the lane lines can be tracked. And then, in response to the fact that the tracking result does not meet the preset tracking condition, updating the counter value of the tracker to obtain an updated counter value. And then, in response to determining that the updated count value meets a preset count condition, determining a secondary tracker corresponding to the set of key point coordinates of the lane line. Wherein the secondary tracker includes a match count value. By introducing the tracking condition and the preset counting condition, the method can be used for improving the accuracy of the tracking of the key points of the lane lines. In addition, the secondary tracker can be established to continuously track the key points of the lane lines, so that the matching error is avoided. And then, matching the auxiliary tracker with trackers in the tracker list to obtain a matching result. Through the matching process, it can be used to determine the correspondence between the tracker and the sub-tracker. And then, updating the matching count value by using the matching result to obtain a target count value. And finally, in response to the fact that the target count value meets a preset target condition, combining the auxiliary tracker and the tracker to obtain a target tracker, and adding the target tracker to the tracker list so as to continuously perform the track line key point tracking operation. Through the merging processing, the method can be used for fusing the key points of the lane lines tracked by the tracker and the auxiliary tracker, so that the accuracy of the tracking result of the key points of the lane lines can be improved.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and elements are not necessarily drawn to scale.

FIG. 1 is a flow diagram of some embodiments of a lane line keypoint tracking method according to the present disclosure;

FIG. 2 is a flow diagram of further embodiments of a lane line keypoint tracking method according to the present disclosure;

FIG. 3 is a schematic structural diagram of some embodiments of a lane line keypoint tracking device according to the present disclosure;

FIG. 4 is a schematic block diagram of an electronic device suitable for use in implementing some embodiments of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be noted that, for convenience of description, only the portions related to the related invention are shown in the drawings. The embodiments and features of the embodiments in the present disclosure may be combined with each other without conflict.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

The names of messages or information exchanged between devices in the embodiments of the present disclosure are for illustrative purposes only, and are not intended to limit the scope of the messages or information.

The present disclosure will be described in detail below with reference to the accompanying drawings in conjunction with embodiments.

Fig. 1 illustrates a flow 100 of some embodiments of a lane line keypoint tracking method according to the present disclosure. The process 100 of the method for tracking the key points of the lane line comprises the following steps:

step 101, tracking each lane line key point coordinate in the lane line key point coordinate set based on a tracker corresponding to a pre-generated lane line key point coordinate set in a preset tracker list to obtain a tracking result.

In some embodiments, the executing body of the lane line key point tracking method may track, based on a tracker in a preset tracker list corresponding to a pre-generated lane line key point coordinate set, each lane line key point coordinate in the lane line key point coordinate set to obtain a tracking result. Wherein the tracker includes a tracker count value. The lane line key points in the lane line key point coordinate set may be key point coordinates of a section of a lane line extracted from one road image. The coordinates of the key points of the lane lines can represent the positions of the corner points of a section of lane line in the virtual lane lines on the road in the road image. The tracker may be a container for storing the key points of the lane line. Each tracker may correspond to a lane line. Thus, an identification of the tracked segment of lane line may be included in the tracker. The identification of the lane line may be used to characterize the location of the lane line segment in the road image. The trackers closest in distance between the coordinates of the lane line key points in the lane line key point coordinate set and the positions corresponding to the identifications of the lane lines may be in a corresponding relationship. The tracking processing may be to match each lane line key point in the lane line key point coordinate set with each tracker in the tracker list by a key point matching algorithm to obtain a tracking result. The tracking result may include matching information. The matching information may characterize whether each of the lane line keypoint coordinates matches one of the trackers in the tracker list. For example, the matching information may be: "match successful" or "match failed". In addition, the tracker count value may characterize the number of times the tracker failed to match the lane line keypoint coordinates. Each tracker may correspond to a tracking count value. "match success" may be used to characterize that there is a tracker in the tracker list that matches the coordinates of the key points of the lane line in the set of key point coordinates of the lane line, indicating that the tracking was successful. "failure to match" may be used to characterize that there is no tracker in the tracker list that matches the set of the key point coordinates of the lane line, indicating a tracking failure.

By way of example, the container may be, but is not limited to, at least one of: dictionary containers, list containers, containers in the application container engine, etc. The above-mentioned keypoint matching algorithm may include, but is not limited to, at least one of: scale-invariant Feature Transform (SIFT-invariant Feature Transform) algorithm, euclidean distance value, Harris corner detection, FAST corner detection, and the like.

In some optional implementations of some embodiments, the set of lane line keypoint coordinates may be generated by:

the method comprises the following steps of firstly, acquiring a road image, a camera transformation matrix and a camera internal parameter matrix. Wherein the road image may be acquired by an onboard camera. The camera transformation matrix and the camera internal reference matrix stored in advance can be acquired from the vehicle memory.

And secondly, extracting key points of the road image to obtain an extracted key point coordinate set. The key point extraction can be performed on the road image through a preset key point extraction algorithm to obtain an extracted key point coordinate set.

As an example, the above-mentioned keypoint extraction algorithm may include, but is not limited to, at least one of: ORB (organized Fast and Rotated Brief, feature extraction algorithm), superfint (feature extraction neural network), and the like.

And thirdly, based on the camera transformation matrix and the camera internal reference matrix, back projecting each extracted key point coordinate in the extracted key point coordinate set to a vehicle coordinate system to obtain a lane line key point coordinate set. The extracted key point coordinates in the extracted key point coordinate set can be back-projected from the image coordinate system to the vehicle coordinate system by an inverse perspective transformation method. The lane line key point coordinates may be three-dimensional coordinates.

In some optional implementations of some embodiments, the tracker in the tracker list may further include: tracking the key point coordinate set, the tracking area corresponding to each tracking key point coordinate, and the tracking processing of each lane line key point coordinate in the lane line key point coordinate set by the execution main body based on the tracker corresponding to the pre-generated lane line key point coordinate set in the preset tracker list to obtain the tracking result, which may include the following steps:

the method comprises the steps of firstly, determining the position relationship between each lane line key point coordinate in the lane line key point coordinate group and a tracking area corresponding to each tracking key point coordinate in the tracking key point coordinate group to generate position relationship information, and obtaining a position relationship information set. The tracking area may be a minimum bounding rectangle of a lane line area in the road image. The set of tracking keypoint coordinates may be the four corner coordinates of a section of lane line tracked by the tracker. It is possible to determine whether the coordinates of the key points of the lane lines are in the tracking area, thereby generating the positional relationship information.

As an example, the positional relationship information may be: "the keypoint coordinates are inside the tracking area", "the keypoint coordinates are outside the tracking area", etc. In addition, the position relation information may further include the coordinates of the key points of the lane lines and the corresponding tracking areas.

And secondly, generating a tracking result based on the position relation information set. Wherein, if the position relation information is: and if the key point coordinates are in the tracking area, determining that the corresponding lane line key point coordinates are successfully tracked. If the position relation information is: if the key point coordinates are outside the tracking area, it can be determined that the corresponding lane line key point coordinates are failed to track.

As an example, the tracking result may include corresponding tracking results of key points of each lane line, i.e., tracking failure or tracking success.

In some optional implementation manners of some embodiments, the executing unit may perform tracking processing on each key point coordinate of the lane line in the key point coordinate set of the lane line based on a tracker corresponding to a pre-generated key point coordinate set of the lane line in a preset tracker list to obtain a tracking result, and may further include the following steps:

the method comprises the following steps of firstly, fitting the coordinates of all tracking key points in a tracking key point coordinate set in each tracker list in the tracker list to generate a fitted lane line equation, and obtaining a fitted lane line equation set. Wherein each fitted lane line equation may correspond to a segment of a lane line.

And secondly, sending the fitted lane line equation set to a display terminal for displaying. Wherein the display can be used for the reference of the driver.

In some optional implementations of some embodiments, the tracking area corresponding to each tracking keypoint coordinate is generated by:

first, a camera transformation matrix is acquired. The camera transformation matrix can be acquired from the inertial measurement unit in a wired mode or a wireless mode. The camera transformation matrix described above may be used to characterize a transformation matrix of the camera coordinate system relative to the vehicle coordinate system at the current time. In addition, the camera transformation matrix may include a vehicle attitude matrix and a vehicle translation vector. Specifically, the vehicle pose matrix can be used for representing the posture transformation of the vehicle at the current moment relative to the previous moment. The vehicle translation vector may be used to characterize a displacement distance of the vehicle at the current time relative to the previous time.

And secondly, splitting the camera transformation to obtain a pitch angle transformation matrix and a yaw angle transformation matrix. The splitting process may be to represent the vehicle attitude matrix by an euler angle. Since the roll angle fluctuation of the vehicle body is relatively slight, only the pitch angle and the yaw angle are considered here. And a pitch angle transformation matrix and a yaw angle transformation matrix can be obtained through Euler angle representation.

And thirdly, determining a pitch angle in the pitch angle transformation matrix and a yaw angle in the yaw angle transformation matrix.

And fourthly, carrying out back projection processing on the tracking key point coordinates to obtain back projection key point coordinates. First, the coordinates of the tracking key points may be converted into a three-dimensional vector. The first data of the three-dimensional vector may be the abscissa value of the above-mentioned tracking keypoint coordinates. The second data of the three-dimensional vector may be a vertical coordinate value of the coordinate of the tracking key point. The second data of the three-dimensional vector may be 1. Then, a product of an inverse matrix of the camera internal reference matrix, a preset scale truth value, and the three-dimensional vector may be determined as a back projection key point coordinate. The true scale value may be a variable for setting a vertical coordinate value of the backprojection key point coordinate to 1. Specifically, the back projection processing may be performed by an inverse perspective transformation method.

And fifthly, generating an area coordinate set based on the back projection key point coordinates, the pitch angle, the yaw angle, the pitch angle transformation matrix, the yaw angle transformation matrix, the tracking key point coordinates, the camera transformation matrix, the camera internal reference matrix and preset standard deviation information. The standard deviation information may include a priori error value, a priori error standard deviation, a pitch angle error standard deviation, a yaw angle error standard deviation, and a displacement error standard deviation. The prior error value may be a predetermined back projection error, and is used to characterize an error value generated by back projection. The a priori error standard deviation described above may be used to characterize the standard deviation of the backprojection error. The pitch angle error standard deviation, yaw angle error standard deviation, and displacement error standard deviation may also be pre-calibrated or defined by the manufacturer of the device. The standard deviation of the displacement error can be a three-dimensional vector and can comprise three data of a transverse standard deviation of the displacement error, a longitudinal standard deviation of the displacement error and a vertical standard deviation of the displacement error.

The region coordinate set may be generated by the following formula:

。

wherein,

representing the set of region coordinates.

Representing the three-dimensional region coordinates.

Representing an argument, i.e. the region coordinates.

Representing the camera reference matrix described above.

A camera transformation matrix, i.e., a transformation matrix of the camera coordinate system with respect to the vehicle body coordinate system, is represented.

Representing the pitch transformation matrix described above.

Representing the yaw angle transformation matrix described above.

Is shown onThe yaw angle.

The pitch angle is shown.

Representing the vehicle translation vector.

Representing a first parameter.

Representing the second parameter.

Representing a third parameter.

Representing a fourth parameter.

A fifth parameter is indicated.

Indicating a sixth parameter.

The conversion parameters are expressed for shortening the formula length.

Representing the backprojected keypoint coordinates.

The 3 rd element of the vector in parentheses is taken.

The representation takes the 1 st to 2 nd elements of the vector in parentheses.

The above standard deviation of pitch angle error is indicated.

The standard deviation of the yaw angle error described above is indicated.

The above standard deviation of the lateral displacement error is shown.

The standard deviation of the longitudinal displacement error described above is indicated.

The above-mentioned standard deviation of vertical displacement error is indicated.

Representing the a priori error values.

The standard deviation of the prior error mentioned above is indicated.

And sixthly, determining the circumscribed rectangle of each area coordinate in the area coordinate set. The circumscribed rectangle of each region coordinate in the region coordinate set may be determined by an IOU (Intersection over Union) algorithm.

And seventhly, determining the circumscribed rectangle as a tracking area. The tracking area can be used for representing the position of a section of lane line in a road image.

The above formula and its related content are used as an invention point of the embodiment of the present disclosure, and further solve the technical problem mentioned in the background art, i.e. the key point of the lane line is the corner point of a segment of the lane line, and the similarity between these points is large, so that the situation of matching error is easy to occur, and thus, the tracking result of the key point of the lane line is not accurate. Factors that lead to a reduction in the accuracy of lane line keypoint tracking results tend to be as follows: since the key points of the lane lines are the corner points of a section of the lane lines, the similarity between the points is large, and the situation of matching errors is easy to occur. If the factors are solved, the accuracy of the tracking result of the key points of the lane line can be improved. To achieve this, an area coordinate set is generated by the above formula and its related content, and a tracking area is obtained. With this, a tracking range corresponding to each tracking keypoint coordinate can be determined. In addition, preset standard deviation information is introduced to offset errors to a certain extent, so that the accuracy of the generated key point coordinates of the lane lines can be improved. Therefore, the influence of the similarity of the key points of the lane lines on the tracking result can be greatly avoided, and the condition of mismatching is effectively reduced. Furthermore, the accuracy of tracking the key points of the lane lines can be improved.

And 102, in response to the fact that the tracking result does not meet the preset tracking condition, updating the count value of the tracker to obtain an updated count value.

In some embodiments, the execution subject may update the tracker counter value to obtain an updated counter value in response to determining that the tracking result does not satisfy a preset tracking condition. Wherein, the preset tracking condition may be: the matching information in the tracking result is "matching failure". The updating of the tracker count value may be: and updating the tracking count value of the tracker. The tracker may be a tracker corresponding to the set of the key point coordinates of the lane line in the tracker list.

As an example, the updating may be to add 1 to the tracking count value.

And 103, in response to the fact that the updated count value meets the preset counting condition, determining the secondary tracker corresponding to the key point coordinate set of the lane line.

In some embodiments, the executing entity may determine the sub-tracker corresponding to the set of key point coordinates of the lane line in various ways in response to determining that the updated count value satisfies a preset count condition. Wherein the secondary tracker may include a match count value. The preset count condition may be that the update count value is greater than a preset update threshold (e.g., 5). The matching count value may be the number of times of matching between the sub-tracker and the tracker with the highest success number, and is used to characterize the correspondence between the sub-tracker and the tracker with the highest success number of matching. In addition, if the tracker with the highest matching success frequency with the sub-tracker changes, the corresponding relation also changes. The secondary tracker may further include a set of coordinates of key points of the lane line to be confirmed. The set of the coordinates of the key points of the lane line to be confirmed may be coordinates of key points of the lane line that do not match each of the trackers in the tracker list.

And 104, matching the secondary tracker with trackers in the tracker list to obtain a matching result.

In some embodiments, the executing entity may perform matching processing on the sub tracker and trackers in the tracker list to obtain a matching result. Whether the secondary tracker is matched with the trackers in the tracker list or not can be determined through the key point matching algorithm, and therefore a matching result is obtained. The matching result may include identification and matching result information between the sub-tracker and the matched tracker. The matching result may further include a matching result and matching result information between the sub-tracker and the tracker in the tracker list. The match result information may be "the sub-tracker matches with the tracker" or "the sub-tracker does not match with the tracker". "Secondary tracker to tracker match" may be used to characterize that the secondary tracker corresponds to the same lane line as the tracker. "Secondary tracker to tracker mismatch" can be used to characterize that a secondary tracker does not correspond to a same lane line. Additionally, the identification between the secondary tracker and the matched tracker may be used to characterize the correspondence between the secondary tracker and the tracker.

And 105, updating the matching count value by using the matching result to obtain a target count value.

In some embodiments, the execution body may update the matching count value by using the matching result to obtain a target count value. Wherein the matching technical value can be updated in the following way: first, the history matching count value between the secondary tracker and each tracker in the cache may be obtained. Then, the history matching count value corresponding to each tracker that matches the sub-tracker in the matching result may be incremented (e.g., by 1), resulting in an incremented matching count value set. Finally, the largest incremented match count value in the set of incremented match count values described above may be determined as the target count value.

In practice, there are cases where a plurality of trackers match the sub-tracker after the matching process. Therefore, by generating the target count value, the tracker that matches the sub-tracker the largest number of times is selected. Therefore, the corresponding relation between the secondary tracker and the tracked section of lane line can be more accurately determined. Thus, the accuracy of tracking the key points of the lane lines can be improved. In addition, the characteristic of large similarity of the corner points of the lane lines is considered, so that the direct matching between the corner points of the lane lines is avoided, and the key points of the lane lines are integrally matched in a tracker mode. Thereby improving the identification degree of the corner points of the lane lines. Thus, the method can be used for improving the tracking accuracy of the key points of the lane lines.

And 106, in response to the fact that the target count value meets the preset target condition, combining the secondary tracker and the tracker to obtain a target tracker, and adding the target tracker to a tracker list so as to continuously perform the track line key point tracking operation.

In some embodiments, the executing agent may, in response to determining that the target count value satisfies a preset target condition, perform a merging process on the secondary tracker and the tracker to obtain a target tracker, and add the target tracker to the tracker list, so as to continue to perform the track line key point tracking operation. The merging process may be to add the coordinates of the to-be-confirmed lane line key points in the to-be-confirmed lane line key point coordinate set in the secondary tracker to the tracker, so as to obtain the target tracker. In addition, the secondary tracker can be deleted to reduce the memory consumption.

The above embodiments of the present disclosure have the following advantages: by the lane line key point tracking method of some embodiments of the present disclosure, the accuracy of the lane line key point tracking result can be improved. Specifically, the reason for causing the inaccuracy of the tracking structure of the key points of the lane line is as follows: since the key points of the lane lines are the corner points of a section of the lane lines, the similarity between the points is large, and the situation of matching errors is easy to occur. Based on this, in the disclosed method for tracking the key points of the lane line in some embodiments, first, based on a tracker in a preset tracker list corresponding to a pre-generated coordinate set of key points of the lane line, the coordinates of the key points of each lane line in the coordinate set of key points of the lane line are tracked, and a tracking result is obtained. By introducing a tracker list, the coordinates of the key points of the lane lines can be tracked. And then, in response to the fact that the tracking result does not meet the preset tracking condition, updating the counter value of the tracker to obtain an updated counter value. Then, in response to determining that the updated count value satisfies a preset count condition, a secondary tracker corresponding to the set of lane line key point coordinates is determined. Wherein the sub-tracker includes a match count value. By introducing the tracking condition and the preset counting condition, the method can be used for improving the accuracy of the tracking of the key points of the lane lines. In addition, the method can be used for continuously tracking the key points of the lane lines by establishing the secondary tracker, so that the matching error is avoided. And then, matching the auxiliary tracker with trackers in the tracker list to obtain a matching result. Through the matching process, it can be used to determine the correspondence between the tracker and the sub-tracker. And then, updating the matching count value by using the matching result to obtain a target count value. And finally, in response to the fact that the target count value meets a preset target condition, combining the auxiliary tracker and the tracker to obtain a target tracker, and adding the target tracker to the tracker list so as to continuously perform the track line key point tracking operation. The merging processing can be used for fusing the key points of the lane lines tracked by the tracker and the auxiliary tracker, so that the accuracy of the tracking result of the key points of the lane lines can be improved.

With further reference to fig. 2, a flow 200 of further embodiments of a lane line keypoint tracking method is illustrated. The process 200 of the method for tracking the key points of the lane line comprises the following steps:

step 201, tracking each lane line key point coordinate in the lane line key point coordinate set based on a tracker corresponding to a pre-generated lane line key point coordinate set in a preset tracker list to obtain a tracking result.

Step 202, in response to determining that the tracking result does not satisfy the preset tracking condition, updating the count value of the tracker to obtain an updated count value.

In some embodiments, the specific implementation manner and technical effects of the steps 201-202 can refer to the steps 101-102 in the embodiments corresponding to fig. 1, which are not described herein again.

Step 203, resetting the tracker count value in response to determining that the tracking result satisfies the preset tracking condition.

In some embodiments, the main body of executing the lane line key point tracking method may reset the tracker count value in response to determining that the tracking result satisfies the preset tracking condition. The tracking result meets the preset tracking condition, and can be used for representing that the tracker successfully tracks the key point coordinates of each lane line in the key point coordinate set of the lane line. The reset may be to zero the tracker count value. Therefore, the tracking failure times of the tracker can be eliminated by resetting the tracker count value. And influence on subsequent tracking is avoided.

And step 204, in response to the fact that the updated count value does not meet the preset counting condition, constructing a secondary tracker corresponding to the key point coordinate set of the lane line.

In some embodiments, the execution subject may construct a sub-tracker corresponding to the set of key point coordinates of the lane line in response to determining that the updated count value does not satisfy the preset count condition. And creating a secondary tracker corresponding to the set of the key point coordinates of the lane line in a preset memory area. Thus, the tracking of the key point coordinates of the lane line in the key point coordinate set of the lane line can be continued.

And step 205, matching the secondary tracker with trackers in the tracker list to obtain a matching result.

And step 206, updating the matching count value by using the matching result to obtain a target count value.

And step 207, in response to the fact that the target count value meets the preset target condition, combining the secondary tracker and the tracker to obtain a target tracker, and adding the target tracker to a tracker list so as to continue to perform the track line key point tracking operation.

In some embodiments, the specific implementation manner and technical effects of steps 205-207 can refer to steps 104-106 in those embodiments corresponding to fig. 1, and are not repeated herein.

In practice, if the number of times that the tracker fails to track the key point coordinates of the lane line of a segment of the lane line is greater than a certain threshold (e.g., 5 times), it may indicate that the tracker has failed to track. However, in consideration of the situation that the sensor is blocked by an overlarge or short-time obstacle in a short time, the pixel positions of the key point coordinates of two adjacent frames are far, so that the matching is missed, and the matching failure of the key points of the lane line is reduced. If the direct definition tracking fails, the accuracy of the tracking result of the key points of the lane line is reduced. Therefore, by constructing the secondary tracker, the coordinates of the key points of the lane lines can be continuously tracked after the failure of the tracker is confirmed. And meanwhile, after each tracking is successful, the tracker is matched with the tracker. To confirm whether the secondary tracker tracks the same lane line as the tracker tracks. If the two types of the target trackers are the same, the target trackers are obtained in a merging processing mode, namely, the trackers are updated and can continue to track the key points of the lane lines. If the lane line tracked by the tracker is not the same as the lane line tracked by the sub-tracker, the lane line tracked by the sub-tracker may be a new lane line. Therefore, it is possible to continue the lane line key point tracking by using the sub-tracker as a tracker. Therefore, the situation of the second technical problem is greatly avoided. And furthermore, the accuracy of the tracking result of the key points of the lane line is improved.

As can be seen from fig. 2, compared to the description of some embodiments corresponding to fig. 1, the flow 200 of the lane line key point tracking method in some embodiments corresponding to fig. 2 embodies the steps of constructing the secondary tracker. The secondary tracker is constructed, so that the accuracy of the tracking result of the key points of the lane lines can be improved.

With further reference to fig. 3, as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a lane line keypoint tracking apparatus, which correspond to those shown in fig. 2, and which may be applied in various electronic devices.

As shown in fig. 3, the lane line key point tracking apparatus 300 of some embodiments includes: a tracking processing unit 301, a first updating unit 302, a determining unit 303, a matching processing unit 304, a second updating unit 305, and a merging processing unit 306. The tracking processing unit 301 is configured to perform tracking processing on each lane line key point coordinate in a lane line key point coordinate set based on a tracker corresponding to a pre-generated lane line key point coordinate set in a preset tracker list to obtain a tracking result, where the tracker includes a tracker count value; a first updating unit 302, configured to update the tracker counter value in response to determining that the tracking result does not satisfy a preset tracking condition, to obtain an updated counter value; a determining unit 303 configured to determine a sub-tracker corresponding to the set of key point coordinates of the lane line in response to determining that the updated count value satisfies a preset count condition, wherein the sub-tracker includes a matching count value; a matching processing unit 304, configured to perform matching processing on the sub tracker and trackers in the tracker list to obtain a matching result; a second updating unit 305 configured to update the matching count value using the matching result to obtain a target count value; and the merging processing unit 306 is configured to, in response to determining that the target count value meets a preset target condition, merge the sub-tracker and the tracker to obtain a target tracker, and add the target tracker to the tracker list, so as to continue to perform a track line key point tracking operation.

It will be understood that the units described in the apparatus 300 correspond to the various steps in the method described with reference to fig. 1. Thus, the operations, features and resulting advantages described above with respect to the method are also applicable to the apparatus 300 and the units included therein, and are not described herein again.

Referring now to fig. 4, a block diagram of an electronic device 400 suitable for use in implementing some embodiments of the present disclosure is shown. The electronic device shown in fig. 4 is only an example, and should not bring any limitation to the functions and the scope of use of the embodiments of the present disclosure.

As shown in fig. 4, electronic device 400 may include a processing device (e.g., central processing unit, graphics processor, etc.) 401 that may perform various appropriate actions and processes in accordance with a program stored in a Read Only Memory (ROM) 402 or a program loaded from a storage device 408 into a Random Access Memory (RAM) 403. In the RAM 403, various programs and data necessary for the operation of the electronic apparatus 400 are also stored. The processing device 401, the ROM 402, and the RAM 403 are connected to each other via a bus 404. An input/output (I/O) interface 405 is also connected to bus 404.

Generally, the following devices may be connected to the I/O interface 405: input devices 406 including, for example, a touch screen, touch pad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; an output device 407 including, for example, a Liquid Crystal Display (LCD), a speaker, a vibrator, and the like; storage 408 including, for example, tape, hard disk, etc.; and a communication device 409. The communication means 409 may allow the electronic device 400 to communicate wirelessly or by wire with other devices to exchange data. While fig. 4 illustrates an electronic device 400 having various means, it is to be understood that not all illustrated means are required to be implemented or provided. More or fewer devices may alternatively be implemented or provided. Each block shown in fig. 4 may represent one device or may represent multiple devices as desired.

In particular, according to some embodiments of the present disclosure, the processes described above with reference to the flow diagrams may be implemented as computer software programs. For example, some embodiments of the present disclosure include a computer program product comprising a computer program embodied on a computer readable medium, the computer program comprising program code for performing the method illustrated in the flow chart. In some such embodiments, the computer program may be downloaded and installed from a network through the communication device 409, or from the storage device 408, or from the ROM 402. The computer program, when executed by the processing apparatus 401, performs the above-described functions defined in the methods of some embodiments of the present disclosure.

It should be noted that the computer readable medium described above in some embodiments of the present disclosure may be a computer readable signal medium or a computer readable storage medium or any combination of the two. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the foregoing. More specific examples of the computer readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In some embodiments of the disclosure, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. In some embodiments of the present disclosure, however, a computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated data signal may take many forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to: electrical wires, optical cables, RF (radio frequency), etc., or any suitable combination of the foregoing.

In some embodiments, the clients, servers may communicate using any currently known or future developed network Protocol, such as HTTP (HyperText Transfer Protocol), and may interconnect with any form or medium of digital data communication (e.g., a communications network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), the Internet (e.g., the Internet), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future developed network.

The computer readable medium may be embodied in the apparatus described above; or may exist separately without being assembled into the electronic device. The computer readable medium carries one or more programs which, when executed by the electronic device, cause the electronic device to: tracking each lane line key point coordinate in a lane line key point coordinate set based on a tracker corresponding to a pre-generated lane line key point coordinate set in a preset tracker list to obtain a tracking result, wherein the tracker comprises a tracker count value; in response to determining that the tracking result does not meet a preset tracking condition, updating the count value of the tracker to obtain an updated count value; in response to determining that the updated count value meets a preset count condition, determining a secondary tracker corresponding to the set of lane line key point coordinates, wherein the secondary tracker comprises a matching count value; matching the secondary tracker with trackers in the tracker list to obtain a matching result; updating the matching count value by using the matching result to obtain a target count value; and in response to the fact that the target count value meets a preset target condition, combining the auxiliary tracker and the tracker to obtain a target tracker, and adding the target tracker to the tracker list so as to continue to perform the track line key point tracking operation.

Computer program code for carrying out operations for embodiments of the present disclosure may be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C + +, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider).

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

The units described in some embodiments of the present disclosure may be implemented by software, and may also be implemented by hardware. The described units may also be provided in a processor, and may be described as: a processor includes a trace processing unit, a first update unit, a determination unit, a matching processing unit, a second update unit, and a merge processing unit. Here, the names of these units do not constitute a limitation to the unit itself in some cases, and for example, the matching processing unit may also be described as a "unit that performs matching processing on the above-described sub tracker and trackers in the above-described tracker list".

The functions described herein above may be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), systems on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), and the like.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the invention in the embodiments of the present disclosure is not limited to the specific combination of the above-mentioned features, but also encompasses other embodiments in which any combination of the above-mentioned features or their equivalents is made without departing from the inventive concept as defined above. For example, the above features and (but not limited to) the features with similar functions disclosed in the embodiments of the present disclosure are mutually replaced to form the technical solution.

Claims

1. A method for tracking key points of a lane line comprises the following steps:

tracking each lane line key point coordinate in a lane line key point coordinate set based on a tracker corresponding to a pre-generated lane line key point coordinate set in a preset tracker list to obtain a tracking result, wherein the tracker comprises a tracker count value;

in response to the fact that the tracking result does not meet the preset tracking condition, updating the count value of the tracker to obtain an updated count value;

in response to determining that the updated count value meets a preset count condition, determining a secondary tracker corresponding to the set of lane line key point coordinates, wherein the secondary tracker includes a matching count value;

matching the secondary tracker with trackers in the tracker list to obtain a matching result;

updating the matching count value by using the matching result to obtain a target count value;

and in response to the fact that the target count value meets a preset target condition, combining the auxiliary tracker and the tracker to obtain a target tracker, and adding the target tracker to the tracker list so as to continuously perform the track line key point tracking operation.

2. The method of claim 1, wherein the set of lane line keypoint coordinates is generated by:

acquiring a road image, a camera transformation matrix and a camera internal parameter matrix;

extracting key points of the road image to obtain an extracted key point coordinate set;

and performing back projection on each extracted key point coordinate in the extracted key point coordinate set to a vehicle coordinate system based on the camera transformation matrix and the camera internal reference matrix to obtain a lane line key point coordinate set.

3. The method of claim 1, wherein the trackers in the tracker list further comprise: tracking key point coordinate sets and tracking areas corresponding to the coordinates of each tracking key point; and

the method comprises the following steps of tracking each lane line key point coordinate in a lane line key point coordinate set based on a tracker corresponding to a pre-generated lane line key point coordinate set in a preset tracker list to obtain a tracking result, wherein the tracking result comprises the following steps:

determining the position relationship between each lane line key point coordinate in the lane line key point coordinate group and the tracking area corresponding to each tracking key point coordinate in the tracking key point coordinate group to generate position relationship information, and obtaining a position relationship information set;

and generating a tracking result based on the position relation information set.

4. The method of claim 3, wherein the method further comprises:

fitting the coordinates of each tracking key point in the tracking key point coordinate set in each tracker list in the tracker lists to generate a fitted lane line equation to obtain a fitted lane line equation set;

and sending the fitted lane line equation set to a display terminal for display.

5. The method of claim 1, wherein the method further comprises:

resetting the tracker count value in response to determining that the tracking result satisfies the preset tracking condition.

6. The method of claim 5, wherein said determining a secondary tracker corresponding to the set of lane line keypoint coordinates comprises:

and constructing a secondary tracker corresponding to the key point coordinate set of the lane line.

7. The method of claim 3, wherein the tracking area corresponding to each tracking keypoint coordinate is generated by:

acquiring a camera transformation matrix;

splitting the camera transformation to obtain a pitch angle transformation matrix and a yaw angle transformation matrix;

determining a pitch angle in the pitch angle transformation matrix and a yaw angle in the yaw angle transformation matrix;

carrying out back projection processing on the tracking key point coordinates to obtain back projection key point coordinates;

generating an area coordinate set based on the back projection key point coordinates, the pitch angle, the yaw angle, the pitch angle transformation matrix, the yaw angle transformation matrix, the tracking key point coordinates, the camera transformation matrix, the camera internal reference matrix and preset standard deviation information;

determining a circumscribed rectangle of each region coordinate in the region coordinate set;

and determining the circumscribed rectangle as a tracking area.

8. A lane line keypoint tracking apparatus, comprising:

the tracking processing unit is configured to track and process each key point coordinate of the lane line in a key point coordinate set of the lane line based on a tracker corresponding to the key point coordinate set of the lane line generated in advance in a preset tracker list to obtain a tracking result, wherein the tracker comprises a tracker count value;

a first updating unit configured to update the tracker count value to obtain an updated count value in response to determining that the tracking result does not satisfy a preset tracking condition;

a determining unit configured to determine a sub-tracker corresponding to the set of lane line key point coordinates in response to determining that the updated count value satisfies a preset count condition, wherein the sub-tracker includes a matching count value;

the matching processing unit is configured to match the secondary tracker with trackers in the tracker list to obtain a matching result;

a second updating unit configured to update the matching count value by using the matching result to obtain a target count value;

and the merging processing unit is configured to merge the secondary tracker and the tracker to obtain a target tracker in response to the fact that the target count value meets a preset target condition, and add the target tracker to the tracker list so as to continue to perform the track line key point tracking operation.

9. An electronic device, comprising:

one or more processors;

a storage device having one or more programs stored thereon,

when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

10. A computer-readable medium, on which a computer program is stored which, when being executed by a processor, carries out the method according to any one of claims 1-7.