KR101996167B1 - Apparatus for generating image used for learning of unmanned autonomous vehicle and method thereof - Google Patents

Abstract

The present invention relates to an apparatus for generating a learning image for learning of an unmanned autonomous vehicle and a method thereof. According to the present invention, the method for generating the learning image for learning of the unmanned autonomous vehicle comprises the steps of: detecting an edge portion of a road or a lane from each of multiple frames with respect to an image photographed while an unmanned vehicle is moving; searching for coordinates of pixels caught in the edge portion among multiple pixels in the frame and detecting coordinates of n pixels with higher frequency caught in the edge portions with respect to all the multiple frames; and determining at least one rectangular region used for neural network learning in the frame using at least two among the n pixels and resizing or combining the determined at least one rectangular region to generate the learning image with a preset standard. The learning image is used as input data for the neural network learning. The present invention can automatically extract the learning image, from the image photographed by the vehicle, used for the neural network learning for autonomous running of the vehicle, and extract only a meaningful region with features from the image to be used for the learning image, thereby reducing the data amount and burden on memory for the neural network learning and simultaneously enhancing learning performance.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus and method for generating a learning image for learning an autonomous unmanned moving object,

The present invention relates to an apparatus and method for generating a learning image for learning an autonomous unmanned moving object and, more particularly, to an apparatus and method for generating a learning image for learning an autonomous unmanned moving object, The present invention relates to an apparatus and method for generating a learning image for learning an autonomous unattended moving object. Particularly, the present invention is a patent application filed by the same cultural scholarship foundation research fund support project in 2017.

Autonomous driving cars are vehicles that drivers drive themselves without operating the car, and it is a technology that is attracting attention as a next-generation automobile industry.

There is an END-To-END technique as a method for learning autonomous vehicles. The END-To-END technique is a method of learning images captured by an autonomous vehicle and control signals that control the vehicle at the time of image capture through a CNN (Convolutional Neural Network).

After the learning is completed, the control signal corresponding to the image photographed in the vehicle can be easily estimated using the learned CNN, and the running of the vehicle can be autonomously controlled using the estimated control signal.

However, memory problems arise when inputting a high-resolution image into a neural network during autonomous driving learning using the END-To-END technique. Here, although a high-resolution image can be reduced to a small size without a memory burden, it may be reduced to an actual feature region, thereby distorting or offsetting the feature, and the learning can not be properly performed.

In addition, there is a method of extracting only the characteristic portion from the image and learning, but since the extraction portion is directly determined by the person, the extracted portion may not be a necessary characteristic for driving the autonomous vehicle. Therefore, there is a need for a technique that can automatically extract only the feature region by analyzing the feature portion of the image.

The technology of the background of the present invention is disclosed in Korean Patent Laid-Open Publication No. 2017-0133149 (published on Dec. 21, 2017).

An object of the present invention is to provide an apparatus and method for generating a learning image for learning an autonomous unmanned moving object capable of automatically deriving a learning image used for learning a neural network for autonomous travel of a moving object from an image of a moving object have.

There is provided a learning image generating method for learning an autonomous unmanned moving body using a learning image generating apparatus, the method comprising: detecting an edge portion of a road or a lane from each of a plurality of frames of an image photographed during movement of an unmanned moving body; Searching coordinates of pixels hung on the edge portion among a plurality of pixels in the frame and detecting coordinates of upper n pixels having a high frequency of occurrence of the edge portion based on the entirety of the plurality of frames, Determining at least one rectangular area to be used for neural network learning in the frame using at least two of the upper n pixels and resizing or combining the determined at least one rectangular area, Wherein the learning image comprises at least one of a neural network There is provided a method of generating a learning image for learning an autonomous unattended moving object used as input data for humans.

The generating of the learning image may include determining a minimum rectangular area covering all the coordinates of the upper two pixels among the upper n pixels, resizing the determined minimum rectangular area into the setting standard, Lt; / RTI >

In addition, the learning image is divided into m sub-areas (m is an integer of 2 or more) according to a predetermined rule, and the step of generating the learning image comprises: dividing the upper n pixels into m groups Determining a minimum rectangular area covering all the coordinates of the pixels in the group for each of the groups, and individually resizing the determined m minimum rectangular areas to m sub-areas corresponding thereto, And finally inserting the training image into the box of the training image.

The preset rule may include dividing the learning image at least once into two, and dividing one of the sub-regions immediately before when m is 3 or more.

The learning image may be divided into m sub-areas having a uniform size of m (m is an integer of 2 or more), and the generating of the learning image may include generating sub-sub- Determining a position of a first rectangular area which is arranged to cover at least one pixel of the higher order n pixels while covering at least one pixel of the higher order n pixels, Determining the positions of the remaining m-1 second through m-th rectangular areas by repeating the same operation principle on the remaining pixels of the m remaining sub-areas, And finally creating the learning image through inserting each of the training images.

The step of detecting the edge portion may include extracting at least one color of interest from a predetermined region of interest set in the frame and extracting at least one region of interest corresponding to the region of interest, And detecting the edge portion.

An edge detecting unit for detecting an edge portion of a road or a lane from each of a plurality of frames for an image photographed during movement of the unmanned moving vehicle; A pixel detector for detecting the coordinates of the top n pixels having a high frequency of occurrence of the edge portion based on the whole of the plurality of frames, And an image generating unit that determines at least one rectangular area to be used for learning and generates a learning image of a set specification by resizing or combining the determined at least one rectangular area, For learning autonomous unmanned mobile objects used as input data for It provides for an image-generating device.

According to the present invention, it is possible to automatically derive a learning image used for learning a neural network for autonomous travel of a moving object from a moving image of a moving object, and using only a significant region having a characteristic in the image as a learning image, There is an advantage that it is possible to reduce the burden of the data amount and the memory and increase the learning performance.

FIG. 1 is a diagram illustrating a configuration of a learning image generating apparatus for learning an autonomous unattended moving object according to an embodiment of the present invention.
Fig. 2 is a diagram for explaining a learning image generating method using Fig. 1. Fig.
3 is a diagram illustrating an edge detection result according to step S210 of FIG.
FIG. 4 is a diagram illustrating a result of analyzing coordinates of pixels having a high frequency at an edge portion using the results of FIG. 3. FIG.
FIG. 5 is a view for explaining how an image generated through FIG. 1 is utilized as input data of a neural network learning device.
6 to 8 are views showing first through third embodiments for generating a learning image using the apparatus of FIG.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention.

The present invention relates to an apparatus and method for generating a learning image for learning an autonomous unmanned moving object, which automatically analyzes and derives a significant region having a characteristic in an image photographed by a moving object, We propose a method to reduce the memory burden of the learning machine and improve the learning performance by using it as the input data of neural network learning machine for driving learning.

In an embodiment of the present invention, the unmanned vehicle may include a traveling means capable of autonomous traveling or unmanned traveling, such as a vehicle, robot, drones, airplanes, and the like. Hereinafter, for convenience of explanation, it is assumed that the moving object is a vehicle.

FIG. 1 is a diagram illustrating a configuration of a learning image generating apparatus for learning an autonomous unattended moving object according to an embodiment of the present invention. FIG. 2 is a diagram for explaining a learning image generating method using FIG.

1 and 2, a learning image generating apparatus 100 for learning an autonomous unmanned moving body according to an exemplary embodiment of the present invention includes an edge detecting unit 110, a pixel detecting unit 120, and an image generating unit 130 .

First, the edge detection unit 110 detects an edge portion of a road or a lane from each of a plurality of frames of an image photographed during movement of the unmanned vehicle (S210). Here, edge detection can use an RGB color-based detection method.

3 is a diagram illustrating an edge detection result according to step S210 of FIG.

3, first, the edge detecting unit 110 extracts at least one color of interest (ex, gray, white, yellow) in the predetermined region of interest A in the frame. In case of frames # 1, # 2, and #L, gray and white series are extracted as the color of interest in the area of interest (A). In case of # 3, gray and yellow series are extracted as the color of interest.

Here, the position of the area of interest A with respect to the frame may be predetermined or manually set by a user or an administrator. Also, the region of interest A may be set to include a central region on the frame. The reason for this is that the color of the edge is mainly detected at the center of the image, while the edge is highly likely to be detected by the surrounding environment which is not related to the road characteristic.

After extracting the color of interest, the edge detecting unit 110 derives an area corresponding to the color of interest in the frame and detects an edge (boundary) part of the derived area. The area corresponding to the color of interest may substantially correspond to a road or a lane area. Of course, the color of the same series can be recognized as the same block area in the area derivation process. The edge detection result refers to the outline shown in the lower figure of Fig.

In this way, the edge portions of roads and lanes can be detected for each of L frames. After the edge detection is completed, the pixel detector 120 analyzes the coordinates of the pixels on the edge among the plurality of pixels in the frame on a frame-by-frame basis, and then analyzes the pixels having the number of times (frequency) So that they can be sorted in the order of frequency.

That is, the pixel detecting unit 120 searches for the coordinates of the pixels attached to the edge portion of the plurality of pixels in the frame (S220), and calculates the coordinates of the top n pixels (S230).

For example, when the resolution of a frame to be photographed is 1280 x 720, it is composed of 921600 pixels per frame. At this time, when the coordinates of the pixel on the edge portion of each frame are checked and the number of times of the edge is accumulated for all L frames, the number of the top n pixels having a high frequency of the edge portion of 921600 pixels You can check the coordinates.

FIG. 4 is a diagram illustrating a result of analyzing coordinates of pixels having a high frequency at an edge portion using the results of FIG. 3. FIG.

FIG. 4 shows the edges detected in each frame of FIG. 3 in a superimposed manner on one screen, for convenience of explanation. If the resolution of the frame is 1280 x 720, the lower left coordinate of the frame is (0, 0) and the upper right coordinate is (1280, 720).

The four coordinate points P1, P2, P3, and P4 shown in FIG. 4 illustratively show the top four pixels having the highest frequency of the edges for all L frames. As described above, the position of a pixel having a high frequency in an edge of the image may mean a significant position or area that reveals a main feature of an actual road or a lane in a forward or backward image photographed during driving of the vehicle.

In addition, by using only these significant n pixels and extracting only a significant region of the frame as an input image of the learning apparatus for autonomous driving learning, it is possible to reduce the amount of data and calculation amount input to the learning apparatus, At the same time.

For this, in the embodiment of the present invention, the image generating unit 130 determines at least one rectangular area in the frame using at least two of the n number of upper pixels (S240), and resets the determined at least one rectangular area (S250), or generates a learning image of the set specification.

In general, since the size specification (A x B) of the learning image input to the neural network learning device is smaller than the video standard (ex, 1280 x 720) taken in the actual vehicle, The size of the image is reduced and resizing is carried out.

Thereafter, the generated learning image is used as input data of the neural network learning device 10 together with the control signal, and data learning is performed in the neural network learning device 10 (S260).

FIG. 5 is a view for explaining how an image generated through FIG. 1 is utilized as input data of a neural network learning device.

Generally, in order to learn data, a control signal of an adjusted vehicle should be collected in correspondence with a shooting time of an image and an image taken during traveling (running) of the vehicle. The corresponding pairs of collected images and control signals all correspond to the data sets necessary for neural network learning.

However, in the embodiment of the present invention, the image input to the neural network learning device 10 corresponds to the image newly processed by the image generating unit 130, not the original image captured by the vehicle.

That is, the neural network learner 10 receives the processed learning image and the control signal as input data, and when a predetermined image is input to the learned neural network, the control signal corresponding thereto is automatically output, Can be controlled. Various embodiments for generating a learning image will be described below.

FIG. 6 is a diagram illustrating a first embodiment for generating a learning image using the apparatus of FIG. 1; The first embodiment shows a method of generating a learning image using the coordinates of the upper four pixels out of n pixels.

Referring to FIG. 6, the image generator 130 determines a minimum rectangular area covering all the coordinates of the upper four pixels P1, P2, P3, and P4 in the frame among the n pixels. That is, the minimum rectangular area for covering the four coordinate points with a minimum size like the dotted rectangle in Fig. 6 is determined in the frame.

Then, the image generating unit 130 may generate a learning image by resizing the determined minimum rectangular area into a preset standard. Here, of course, the setting standard refers to a predetermined size specification (A x B) of the learning image used at the time of neural network learning. In the first embodiment, resizing mainly means reduction in size, but size enlargement may be applied in some cases.

Of course, the image generating unit 130 may determine a minimum rectangular area covering the coordinates of the upper two pixels P1 and P2 among the n pixels, and may generate a learning image by resizing the minimum rectangular area to a predetermined setting standard . This is because at least two pixels can be used to determine one rectangular area. In the example of FIG. 6, a minimum rectangular area may be set such that P1 and P2 are positioned inside the upper left corner of the square and the lower right corner, respectively.

However, in the method of the first embodiment, for example, if the positions of the upper four pixels are all located at the outer periphery of the frame, the minimum rectangular area becomes almost similar to the original size of the frame. In this case, The objective of extracting only a minimum area can not be achieved.

The second and third embodiments are complementary to the first embodiment and generate a learning image in a manner of combining m square regions determined in a frame. Of course, in the second and third embodiments, the size of the learning image has a setting standard (A x B).

However, in the second and third embodiments, it is assumed that the learning image is divided into m sub-areas (m is an integer of 2 or more). In the second and the second embodiments, the image generator 130 inserts m rectangular areas determined using n pixels into the corresponding m sub-area cells to complete a learning image.

FIG. 7 is a diagram showing a second embodiment for generating a learning image using FIG. 1. FIG.

In the second embodiment, the learning image is partitioned into m sub-areas (m is an integer of 2 or more) according to a predetermined rule. In the case of FIG. 7, for the sake of explanation, it is exemplified that the learning image is divided into three sub-regions using predetermined rules.

Here, the predetermined rule may be a rule for dividing the learning image into m regions of equal size (divided by 1 / m) as shown in Fig. 7 (a) May be at least once divided into two, and when m is 3 or more, it may be a rule of dividing one of sub regions immediately before into two again.

In the second embodiment, the image generator 130 divides the top n pixels into m groups according to ranking, and determines a minimum rectangular area covering all the coordinates of the pixels in the group for each group. FIG. 7 shows an example in which the top 15 pixels are divided into three groups according to the order, and three minimum rectangular regions (b1, b2, b3) covering each group are determined.

Then, the image generating unit 130 individually generates the training images by individually resizing the determined m minimum rectangular areas to m sub-area sizes corresponding thereto, and inserting them into the m sub-area, respectively.

That is, the image generation unit 130 individually reduces the minimum rectangular areas b1, b2, and b3 to the sizes of the first, second, and third sub-areas, respectively, And generates a training image.

In FIG. 7 (a), since the sizes of the three sub-areas are all the same, b1, b2, and b3 are finally reduced to the same size. In the case of FIG. 7 (b), the sizes of the three sub-areas are not uniform, and b1, b2, and b3 are reduced according to the size of each sub-area.

FIG. 7B shows an example where m = 3, and the second and third sub-areas have a half size of the first sub-area. If m = 4, the third sub-region is divided into 2 and divided into 3 and 4 sub-regions. In this case, the second sub-region has a half size of the first sub-region and the third and fourth sub- Lt; th > sub-area.

FIG. 8 is a diagram showing a third embodiment for generating a learning image using FIG. 1. FIG.

In the third embodiment, the learning images are partitioned into m sub-areas (m is an integer of 2 or more) having mutually uniform sizes. In the case of Fig. 8, it is exemplified that the learning image is divided into six uniform sub-regions for convenience of explanation.

Unlike the second embodiment, the third embodiment does not require a separate resizing process before inserting each rectangular area into the corresponding box in the learning image.

First, the image generating unit 130 determines a position on a frame of m subareas and m rectangular areas of the same size. That is, the image generating unit 130 arranges six sub-areas on the frame as shown in FIG. 8, and determines the arrangement positions of the six sub-areas using the actual size of the sub-area and the positions of the top n pixels .

Then, the image generating unit 130 generates at least one pixel (at least one of the pixels following p1) capable of covering (covering) among the pixels of the following order while covering at least one pixel p1 of the upper n pixels, The first rectangular region being arranged to sequentially cover the first rectangular region. At this time, it is apparent that a process of rotating the rectangular area is performed so that the pixels of the first order and the pixels of the at least one order are contained in the first square area.

8, a first rectangular area covering p1 and p2 is determined first, and the determined first rectangular area is inserted into a space of a first sub-area in a future learning image. Here, of course, if p3 is close to p1 or p2, the first rectangular region may cover p3 additionally.

Thereafter, the remaining m-1 (5) second to mth rectangular areas are determined using the same method as above. That is, the image generating unit 130 repeats the same operation principle on the remaining pixels (p3 and below) to determine the positions of the remaining m-1 rectangular areas.

In the example of FIG. 8, p4 is much spaced apart from p3. In this case, it is difficult for p4 to be included in the area even if the rectangular area is rotated in any direction with reference to p3. Therefore, when determining the second rectangular area, the second rectangular area is arranged so as to pass p4 and cover p5 and p6 together with p3. The determined second rectangular area is inserted in the second sub-area of the learning image.

The third rectangular area is arranged to cover p7 and p8, and the fourth rectangular area is arranged to cover p9 and p10. The determined third and fourth rectangular regions are respectively inserted into the third and fourth sub-regions of the learning image.

In the case of the fifth rectangular area, p1, p12 and p13 are arranged so as to cover p1, p12 and p13 which are not previously covered in position, and the sixth rectangular area is arranged to cover p14 and p15. The determined fifth and sixth rectangular regions are respectively inserted into the fifth and sixth subareas of the learning image.

As described above, in the third embodiment, the image generating unit 130 finally generates the training image by inserting the determined m square regions into the corresponding m sub-regions.

In this embodiment of the present invention, input data for neural network learning for autonomous autonomous navigation of an autonomous unmanned mobile object can be generated, and can be effectively used for extracting a feature portion in an application to which an end-to-end technique is applied to an image.

According to the present invention as described above, it is possible to automatically derive a learning image used for learning a neural network for autonomous travel of a moving object from a moving image of a moving object, extract only a significant region having a characteristic in the image to a minimum size, As an image, there is an advantage that it is possible to reduce the data amount and memory burden for neural network learning, and to improve the learning performance.

While the present invention has been described with reference to exemplary embodiments, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. Accordingly, the true scope of the present invention should be determined by the technical idea of the appended claims.

100: learning image generation device 110: edge detection part
120: Pixel detection unit 130:

Claims (12)

A learning image generating method for learning an autonomous unattended moving object using a learning image generating apparatus,
Detecting an edge portion of a road or a lane from each of a plurality of frames for an image photographed during movement of the unmanned vehicle;
Searching coordinates of pixels caught in the edge portion among a plurality of pixels in the frame and detecting coordinates of the top n pixels having a high frequency of occurrence in the edge portion based on the entirety of the plurality of frames; And
Determining at least one rectangular area to be used for neural network learning in the frame by using at least two of the upper n pixels and resizing or combining the determined at least one rectangular area, And generating,
Wherein the learning image is used as input data for learning the neural network. The method according to claim 1,
Wherein the generating the learning image comprises:
A training image for learning an autonomous unmanned moving body for determining one minimum rectangular area covering all the coordinates of the upper two pixels among the n pixels and generating the learning image by resizing the determined minimum rectangular area to the setting standard Generation method. The method according to claim 1,
The learning image is divided into m sub-areas (m is an integer of 2 or more) according to a preset rule,
Wherein the generating the learning image comprises:
Dividing the upper n pixels into m groups according to a ranking and determining a minimum rectangular area covering all the coordinates of pixels in the group for each of the groups; And
And finally generating the learning image by individually resizing the m minimum rectangular areas determined to correspond to the sizes of the m subareas and inserting them into the cells of the m subareas respectively Learning image generation method for learning. The method of claim 3,
The pre-
Dividing the learning image at least once into two, and dividing one of the sub-areas immediately before when the m is equal to or greater than 3, into two halves. The method according to claim 1,
The learning image is divided into m sub-areas having m uniform size (m is an integer of 2 or more)
Wherein the generating the learning image comprises:
Determining at least one of m pixels arranged in the sub-area as a sub-area on the frame, arranging at least one pixel among the sub-pixels, Determining a position of a first rectangular area arranged to cover the remaining pixels and repeating the same operation principle with respect to remaining remaining pixels to determine positions of remaining m-1 second through mth rectangular areas; And
And finally generating the learning image by inserting the determined m square areas into the corresponding m sub-area squares respectively. The method according to claim 1,
Wherein the step of detecting the edge portion comprises:
Extracting at least one color of interest from a predetermined region of interest within the frame; And
And deriving an area corresponding to the color of interest in the frame, and detecting an edge part of the derived area. An edge detector for detecting an edge portion of a road or a lane from each of a plurality of frames for an image photographed during movement of the unmanned moving body;
A pixel detecting unit for searching coordinates of pixels hung on the edge portion among a plurality of pixels in the frame and detecting coordinates of the top n pixels having a high frequency of occurrence of the edge portion based on the entirety of the plurality of frames; And
Determining at least one rectangular area to be used for neural network learning in the frame by using at least two of the upper n pixels and resizing or combining the determined at least one rectangular area, And an image generating unit for generating an image,
Wherein the learning image is used as input data for learning the neural network. The method of claim 7,
The image generation unit may include:
Learning for learning an autonomous unmanned moving body that determines one minimum rectangular area covering all the coordinates of the upper two pixels among the upper n pixels and generates the learning image by resizing the determined minimum rectangular area to the setting standard Image generating device. The method of claim 7,
The learning image is divided into m sub-areas (m is an integer of 2 or more) according to a preset rule,
The image generation unit may include:
And dividing the upper n pixels into m groups according to a rank, determining a minimum rectangular area covering all the coordinates of pixels in the group for each of the groups,
And a learning unit for learning the autonomous unmanned moving body which finally generates the learning image by individually resizing the m minimum rectangular areas determined to correspond to the sizes of the m sub areas and then inserting them into the cells of the m sub areas respectively Image generating device. The method of claim 9,
The pre-
Wherein the learning image is divided at least once into two, and when m is equal to or more than 3, the autonomous unattended mobile body learns an autonomous unattended mobile body that divides one of the divided sub regions immediately before into two. The method of claim 7,
The learning image is divided into m sub-areas having m uniform size (m is an integer of 2 or more)
The image generation unit may include:
Determining at least one of m pixels arranged in the sub-area as a sub-area on the frame, arranging at least one pixel among the sub-pixels, And determines the positions of the remaining m-1 second through mth rectangular areas by repeating the same operation principle with respect to the remaining remaining pixels,
And finally inserting the determined m square regions into the corresponding cells of the m subareas, thereby learning the autonomous unmanned moving object. The method of claim 7,
Wherein the edge detecting unit comprises:
Learning for learning an autonomous unmanned moving body that extracts at least one color of interest from a predetermined region of interest set in the frame and then derives an area corresponding to the color of interest in the frame and detects an edge portion for the derived region Image generating device.

Images