JP2015133634A - image selection device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image selection device which selects a frame image in consideration of the content of a moving image.SOLUTION: An image selection device 1 comprises: an acquisition unit 11 which acquires a moving image; a division unit 12 which divides each of plural frame images constituting the moving image acquired by the acquisition unit 11, into areas of each object included in the frame image; a quantization unit 13 which performs quantization of areas obtained by division by the division unit 12, by regarding the matching or similar areas among plural frames as one class; and a selection unit 14 which selects as a representative frame image a frame image including the largest number of classes obtained by the quantization unit 13.

Description

  The present invention relates to an image selection device that selects one or a plurality of frame images from a moving image.

  Conventionally, there is an apparatus that extracts a representative frame image representing the content of a moving image from the moving image. The simplest method for extracting a representative frame image from a moving image is a method of selecting one frame from a predetermined portion of the moving image to obtain a representative frame image. The method is, for example, a method of selecting one frame at the beginning of the moving image or selecting one frame at the temporal intermediate point of the moving image. Here, the method described in Patent Document 1 selects one frame that has passed a predetermined time from the beginning of the moving image as a representative frame image.

Japanese Patent Application Laid-Open No. 2005-151085

However, in the conventional method described above, the content of the moving image is not considered at all.
Further, in the method described in Patent Document 1, when the representative frame image is not appropriate, one frame is selected based on the user's operation and used as the representative frame image. However, in the method of setting the representative frame image based on the user's operation, the user is troublesome. Further, with the method described in Patent Document 1, for example, in the case of a long-time moving image, a burden is placed on the user.

  An object of the present invention is to provide an image selection device that selects a frame image in consideration of the content of a moving image.

  The image selection device includes an acquisition unit that acquires a moving image, a dividing unit that divides each of a plurality of frame images constituting the moving image acquired by the acquisition unit into regions for each object reflected in the frame image, and the dividing unit. With respect to the divided region, a quantization unit that quantizes a plurality of frames that match or are similar to each other as a class, and a frame image in which more classes obtained by the quantization unit are reflected, And a selection unit that selects a representative frame image from a plurality of frame images.

  The selection unit selects, as a representative frame image, a frame image in which more classes are shown among the plurality of classes obtained by the quantization unit, and a representative frame image in which a part of the plurality of classes is shown It is preferable to cover all of the plurality of classes by selecting a plurality of classes.

  In addition, the image selection device includes a moving image acquisition unit that acquires a moving image, and a plurality of frame images that form the moving image acquired by the moving image acquisition unit. A change amount acquisition unit that acquires a change amount between frames, a storage unit that stores a cumulative sum of the change amounts acquired by the change amount acquisition unit, and a cumulative sum stored in the storage unit reaches a predetermined threshold value A selection unit that selects a frame image at the time of arrival as a representative frame image from the plurality of frame images.

  ADVANTAGE OF THE INVENTION According to this invention, the image selection apparatus which selects the frame image in consideration of the content of a moving image can be provided.

It is a block diagram for demonstrating the image selection apparatus of 1st Embodiment. It is a figure for demonstrating operation | movement of the image selection apparatus of 1st Embodiment. It is a flowchart for demonstrating operation | movement of the image selection apparatus of 1st Embodiment. It is a block diagram for demonstrating the image selection apparatus of 2nd Embodiment. It is a figure for demonstrating operation | movement of the image selection apparatus of 2nd Embodiment. It is a flowchart for demonstrating operation | movement of the image selection apparatus of 2nd Embodiment. It is a block diagram for demonstrating the image selection apparatus of 3rd Embodiment. It is a flowchart for setting a threshold value. It is a figure for demonstrating operation | movement of the image selection apparatus of 3rd Embodiment.

[First Embodiment]
First, the first embodiment will be described. FIG. 1 is a block diagram for explaining an image selection apparatus 1 according to the first embodiment.
The image selection device 1 according to the first embodiment includes an acquisition unit 11, a division unit 12, a quantization unit 13, a selection unit 14, a first storage unit 15, and a second storage unit 16.

  The acquisition unit 11 acquires a moving image. A moving image is composed of a plurality of frame images. The moving image may be the entire television program or a part of the moving image (for example, a shot). The acquisition unit 11 sequentially extracts frame images from the moving image. When the resolution of the frame image is larger than a preset threshold value, the acquisition unit 11 may resize the frame image.

  The dividing unit 12 divides each of a plurality of frame images constituting the moving image acquired by the acquiring unit 11 into regions for each object shown in the frame image. That is, the dividing unit 12 divides the frame image acquired from the acquiring unit 11 into image areas. Various techniques can be used for the division into image regions. As an example, the division method can use a technique such as an area expansion method that connects adjacent areas having similar colors, textures, or the like, or can use a technique that estimates areas in units of objects. . The dividing unit 12 performs a process of dividing all frame images into image regions. The dividing unit 12 stores the divided area (image area) in the first storage unit 15.

  The quantization unit 13 quantizes the regions divided by the dividing unit 12 by making a region that matches or is similar between a plurality of frames into one class. In other words, the quantization unit 13 quantizes the image area stored in the first storage unit 15. Here, quantization refers to replacing a plurality of image areas with limited types of representative data. More specifically, the quantizing unit 13 reduces the types of image areas by grouping image areas that match or are similar into one class. As a result, the image selection device 1 can reduce the number of types of data included in the set and reduce the processing load for selecting a representative frame image by the selection unit 14 described later.

  As a quantization method, a method using clustering such as a k-average method or a method of assigning to limited types of representative data obtained in advance can be used. The representative data in the method based on the representative data is effective when the target video genre (for example, natural program, historical drama, etc.) is limited. For example, when a moving image is limited to a natural program, representative texture data appearing in a natural program such as a forest or a river is obtained in advance and set as representative data.

In either case of the method based on clustering or the method based on representative data, it is necessary to define the distance between image regions. For example, a method based on clustering is necessary when similar image regions are integrated into the same cluster. In the method based on representative data, it is necessary to determine to which representative data each image area is assigned. As a method for defining the distance, a method of simply using the sum of squared differences of pixel values as shown in the following equation can be given.

  As a method for defining the distance, there is a method for measuring the similarity of textures based on a known local binary pattern or the like. Further, as a method for defining the distance, there is a method of measuring the distance between textures using a known classification technique.

  The selection unit 14 selects a frame image showing more classes obtained by the quantization unit 13 as a representative frame image. The selection unit 14 selects, as a representative frame image, a frame image in which a larger number of classes appear among the plurality of classes obtained by the quantization unit 13, and a representative frame image in which a part of the plurality of classes is reflected. By selecting multiple, all of multiple classes are covered. The selection unit 14 stores the selected representative frame image in the second storage unit 16.

  That is, the selection unit 14 determines a set of representative frame images by obtaining a set of frame images that include a set of image regions quantized by the quantization unit 13. Specifically, the selection unit 14 targets a set of quantized image areas obtained from all frames of the input moving image, and obtains a set of frame images covering the image area set from the input moving image. The selection unit 14 sets the obtained result as a representative frame image.

Note that the selection unit 14 uses, for example, a method of obtaining an approximate solution using a greedy method, a method of using a genetic algorithm, a method of combining a Lagrange and a greedy method, or the like when selecting a representative frame image. It is possible.
The greedy method is a method of repeatedly selecting, for example, a frame image including more classes among a plurality of classes quantized by the quantization unit 13 as a representative frame image. This repetition is executed until all of the plurality of classes are included in any of the plurality of representative frame images.

  Next, the operation of the image selection device 1 will be described. FIG. 2 is a diagram for explaining the operation of the image selection device 1 according to the first embodiment. FIG. 3 is a flowchart for explaining the operation of the image selection apparatus 1 according to the first embodiment.

  In step ST <b> 1 shown in FIG. 3, the acquisition unit 11 determines whether all the frame images constituting the moving image have been acquired. If all frame images have not been acquired (NO), the process proceeds to step ST2. If all frame images have been acquired (YES), the process proceeds to step ST4.

  In step ST2, the acquisition unit 11 acquires a moving image. Here, the moving image includes a plurality of frame images 20 as shown in FIG.

  In step ST3, the dividing unit 12 divides the frame image of the moving image acquired in step ST2 into regions (image regions). That is, the dividing unit 12 divides the frame image into regions based on colors, textures, objects, and the like, and converts the input moving image into a set of image regions. For example, as shown in FIG. 2B, if “mountains”, “cars”, “sun”, and “clouds” are reflected in one frame image 20, the dividing unit 12 displays As shown in FIG. 5, the image is divided into image areas corresponding to “mountain”, “car”, “sun”, and “cloud”. After step ST3, the process returns to step ST2.

Step ST4 is reached when “NO” is determined in step ST1. In step ST4, the quantization unit 13 quantizes the image region obtained from all frames.
That is, the quantization unit 13 quantizes the image regions extracted from all frame images by classifying them into a limited number of classes using a clustering method or the like. For example, all cars are classified into the “car” class regardless of the type of car. Thereby, each frame image is also expressed by a set of quantized image area data.
As a specific example, as shown in FIG. 2D, the quantization unit 13 performs quantization on the image regions obtained from all the frames, as shown in FIG. 2D, “computer”, “animal”, “bird”, Obtain classes of “human”, “car”, “sun”, “mountain” and “cloud”.

In step ST <b> 5, the selection unit 14 selects a frame image in which more classes among a plurality of classes are shown as a representative frame image. The selection unit 14 can select a plurality of representative frame images. The selection unit 14 covers all classes by selecting a plurality of representative frame images in which some of the plurality of classes are reflected.
In other words, the selection unit 14 approximately obtains a combination of frame images that can cover a set of image regions acquired from the entire moving image with the minimum number of frames, and sets these as representative frame images of the video.
For example, as illustrated in FIG. 2E, the selection unit 14 includes a first frame in which “clouds”, “mountains”, and “computers” are reflected, “mountain”, “sun”, “human”, and “car”. Each of the second frame in which “I” is reflected and the third frame in which “animal” and “bird” are reflected is selected as a representative frame image.

  As described above, the image selection device 1 performs quantization after dividing each of the plurality of frame images into image regions based on colors, textures, objects, etc., and covers all classes resulting from the quantization. As described above, one or a plurality of frame images are selected. The selected frame image becomes the representative frame image. A representative frame image including more classes can be considered to represent the content of a moving image. Therefore, the image selection device 1 can select a frame image in consideration of the content of the moving image.

  Since moving images are media with a time axis, it is difficult to grasp the entire contents in a short time. In this regard, the image selection device 1 selects a frame image (representative frame image) that is considered to represent the content of the moving image, so that the user can grasp the content of the moving image in a short time by allowing the user to confirm the representative frame image. It becomes possible to make it.

[Second Embodiment]
Next, a second embodiment will be described. FIG. 4 is a block diagram for explaining an image selection apparatus 1A according to the second embodiment.
The image selection device 1A of the second embodiment includes a moving image acquisition unit 31, a change amount acquisition unit 32, a first storage unit 33, a selection unit 34, and a second storage unit 35.

  The moving image acquisition unit 31 acquires a moving image. The moving image acquisition unit 31 sequentially extracts frame images from the input moving image. When the resolution of the frame image is larger than a preset threshold value, the moving image acquisition unit 31 may resize the frame image.

  The change amount acquisition unit 32 acquires the amount of change between frames of the object shown in the frame image by comparing the plurality of frame images in the plurality of frame images constituting the moving image acquired by the moving image acquisition unit 31. That is, the change amount acquisition unit 32 calculates the amount of change between frames by comparing the immediately preceding frame image with the current frame image. Various methods can be considered as a method of calculating the amount of change. When it is important that there is little leakage, a method (first method) that reacts sensitively to changes is adopted. When it is important that extra representative frames are not extracted as much as possible, a method (second method) in which motion information between frames can be taken into consideration is adopted.

  As a first method, for example, there is a method of using an absolute difference sum of pixel values, a sum of squared differences, an absolute difference sum of histograms, and a cosine between images.

ここで、ｆｉ（ｘ，ｙ）は、ｉフレームにおける座標（ｘ，ｙ）の画素値を表す。 The absolute difference sum of pixel values is obtained by the following equation.

Here, fi (x, y) represents the pixel value of the coordinates (x, y) in the i frame.

Further, the sum of squared differences is obtained by the following equation.

ここで、ｈｉ（ｂ）は、ｉフレームについての輝度値の頻度ヒストグラムにおけるｂ番目のビンの値を表す。Ｂは、ヒストグラムのビンの総数である。 Further, the absolute difference sum of the histogram is obtained by the following equation.

Here, hi (b) represents the value of the b-th bin in the luminance value frequency histogram for the i frame. B is the total number of histogram bins.

Also, the following formula is used as a method of using a cosine between images.

ここで、ｖｎは、ｎ番目のブロックにおける動きベクトルを表す。Ｎは、ブロックの総数を表す。 The second method includes a motion estimation method using block matching. This method is a method in which a motion vector between frames is obtained and the amount of change is defined by the length of the motion vector as in the following equation.

Here, vn represents a motion vector in the nth block. N represents the total number of blocks.

ここで、ｗ（ｘ，ｙ）は、座標（ｘ，ｙ）の重みを表す。また、ｗ（ｘ，ｙ）の総和は１である。 In addition, when calculating the amount of change, the amount-of-change acquisition unit 32 can increase the weight of a more important part in the frame image when obtaining the amount of change in the frame image. As an example, the change amount acquisition unit 32 can increase the weight near the center of the frame image and decrease the weight of the edge region of the frame image. Alternatively, the change amount acquisition unit 32 can use a saliency map or the like to increase the weights of locations where edges are concentrated and locations where changes in luminance and color are large. A calculation formula in the case where weighting is performed on the above-described absolute difference sum of pixel values is illustrated below.

Here, w (x, y) represents the weight of coordinates (x, y). The sum of w (x, y) is 1.

  The change amount acquisition unit 32 adds the value calculated as described above to the already calculated value and stores the sum in the first storage unit 33.

  The first storage unit 33 corresponds to an embodiment of the “storage unit” of the present invention. The first storage unit 33 stores the cumulative sum of the change amounts acquired by the change amount acquisition unit 32.

  When the cumulative sum stored in the first storage unit 33 reaches a predetermined threshold value, the selection unit 34 selects the frame image at the time of arrival as the representative frame image. That is, the selection unit 34 determines whether to select the current frame image as the representative frame image by comparing the cumulative sum of the amount of change between frames and the threshold value. Specifically, the selection unit 34 determines that the frame is a representative frame image when the cumulative sum of changes stored in the first storage unit 33 is equal to or greater than a threshold value. When determining that the frame is a representative frame image, the selection unit 34 resets the value of the cumulative sum stored in the first storage unit 33 to zero. In addition, the selection unit 34 stores the selected representative frame image in the second storage unit 35.

  Next, the operation of the image selection device 1A will be described. FIG. 5 is a diagram for explaining the operation of the image selection apparatus 1A according to the second embodiment. FIG. 6 is a flowchart for explaining the operation of the image selection apparatus 1A of the second embodiment.

  In step ST11 shown in FIG. 6, the moving image acquisition unit 31 determines whether all frames constituting the moving image have been acquired. If all frames have not been acquired (NO), the process proceeds to step ST12. If all frames have been acquired (YES), the process ends.

  In step ST12, the moving image acquisition unit 31 acquires a moving image. That is, the moving image acquisition unit 31 extracts a frame image from the moving image.

  In step ST13, the change amount acquisition unit 32 acquires the change amount from the previous frame image and the current frame image among the frames obtained in step ST12.

  In step ST14, the change amount acquisition unit 32 adds the change amount obtained in step ST13 to the accumulated sum of the already obtained change amounts. Here, in FIG. 5, the vertical axis indicates the cumulative sum of the change amounts, and the horizontal axis indicates the frame number. As shown in FIG. 5, as the moving image progresses, that is, as the frame number increases, the cumulative sum of changes increases.

  In step ST15, the selection unit 34 determines whether the cumulative sum of the change amounts has reached a preset threshold value. If the cumulative sum has reached the threshold (YES), the process proceeds to step ST16. If the cumulative sum has not reached the threshold value (NO), the process returns to step ST11.

  In step ST16, the selection unit 34 selects the frame image (current frame image) when the cumulative sum reaches the threshold value as the representative frame image. That is, the selection unit 34 selects the frame images A, B, C, and D when the cumulative sum reaches the threshold value TH as the representative frame image (see FIG. 5).

  In step ST17, the selection unit 34 sets the cumulative sum to zero.

  As described above, when the cumulative sum of the amount of change between frames reaches the threshold value, the image selection device 1A selects the frame image at the time of arrival as the representative frame image. In such an image selection device 1A, representative frame images are extracted at a long time interval from a section with little change between frames, and many representative frame images are extracted at a short time interval from a section with many changes between frames. Will be. This is because there are many cases in which the content of the video has hardly changed, such as when the camera is stationary in a section where there is little change, and conversely, in the section where there is a lot of change, the camera is pointing in a completely different direction or is reflected. This is based on the knowledge that the subject is often changed. As a result, the image selection device 1A selects a few representative frame images from a section with little change, and selects many representative frames from a section with many changes. Thus, it can be considered that the representative frame image considering the change between frames represents the content of the moving image. Therefore, the image selection device 1A can select a frame image in consideration of the content of the moving image.

[Third Embodiment]
Next, a third embodiment will be described. The image selection device 1B of the third embodiment has the same configuration as the image selection device 1A of the second embodiment. For this reason, in 3rd Embodiment, description is abbreviate | omitted about the structure similar to 2nd Embodiment.

FIG. 7 is a block diagram for explaining an image selection device 1B of the third embodiment.
The image selection device 1B includes a moving image acquisition unit 41, a change amount acquisition unit 42, a first storage unit 43, a second storage unit 45, and a selection unit 44. The moving image acquisition unit 41, the change amount acquisition unit 42, the first storage unit 43, and the second storage unit 45 have the same configuration as each unit of the second embodiment.

  When the cumulative sum stored in the first storage unit 43 reaches a predetermined threshold value, the selection unit 44 selects a frame at the time of arrival as a representative frame image. That is, the selection unit 44 determines whether or not to select the current frame image as the representative frame image by comparing the cumulative sum of the amount of change between frames and the threshold value. Specifically, the selection unit 44 determines that the frame is a representative frame image when the cumulative sum of changes stored in the first storage unit 43 is equal to or greater than a threshold value. When determining that the frame is a representative frame image, the selection unit 44 resets the value of the cumulative sum stored in the first storage unit 43 to zero. The selection unit 44 stores the selected representative frame image in the second storage unit 45.

  The threshold is set in advance. When setting the threshold, first, the image selection device 1B applies the processing of the moving image acquisition unit 41 and the processing of the change amount acquisition unit 42 to the entire moving image, thereby calculating the cumulative sum of the change amounts of the entire moving image. Ask. The selection unit 44 uses a value obtained by equally dividing the cumulative sum of the amount of change of the entire moving image into a predetermined value as a threshold value. The predetermined value is the number of representative frames to be selected.

  Specifically, the image selection device 1B sets the threshold value as follows. FIG. 8 is a flowchart for setting the threshold.

  In step ST21, the moving image acquisition unit 41 determines whether all the frame images constituting the moving image have been acquired. If all frames have not been acquired (NO), the process proceeds to step ST22. If all frames have been acquired (YES), the process proceeds to step ST25.

  In step ST22, the moving image acquisition unit 41 acquires a moving image. That is, the moving image acquisition unit 41 extracts a frame image from the moving image.

  In step ST23, the change amount acquisition unit 42 acquires the change amount from the previous frame image and the current frame image among the frames obtained in step ST22.

  In step ST24, the change amount acquisition unit 42 adds the change amount obtained in step ST23 to the cumulative sum of the already obtained change amounts. After step ST24, the process returns to step ST21.

  In step ST25, the selection unit 44 sets (cumulative sum / number of designated representative frame images) as a threshold value. That is, the selection unit 44 divides the cumulative sum of the change amounts of the entire moving image by the number of representative frame images to be acquired, and sets a value that is a result of the division as a threshold value.

  Based on the threshold value thus obtained, the image selection device 1B selects a representative frame image. FIG. 9 is a diagram for explaining the operation of the image selection device 1B of the third embodiment. That is, the image selection device 1B acquires the moving image again, and every time the cumulative sum of the changes reaches the threshold value TH, the frame images E, F, G, H, and I at that time are represented as the representative frame images. (See FIG. 9).

  As described above, when the cumulative sum of the amount of change between frames reaches the threshold value, the image selection device 1B selects the frame image at the time of arrival as the representative frame image. Therefore, the image selection device 1B can select a frame image in consideration of the content of the moving image, as in the second embodiment.

DESCRIPTION OF SYMBOLS 1 Image selection apparatus 11 Acquisition part 12 Division | segmentation part 13 Quantization part 14 Selection part 1A, 1B Image selection apparatus 31, 41 Movie acquisition part 32, 42 Change amount acquisition part 33, 43 1st memory | storage part (memory | storage part)
34,44 selection part

Claims (3)

An acquisition unit for acquiring videos;
For each of a plurality of frame images constituting the moving image acquired by the acquisition unit, a dividing unit that divides into regions for each object shown in the frame image;
For the region divided by the dividing unit, a quantization unit that quantizes by making a class that matches or is similar between a plurality of frames, and
A selection unit that selects a frame image in which more classes obtained by the quantization unit are reflected as a representative frame image from the plurality of frame images; and
An image selection device comprising: The selection unit selects, as a representative frame image, a frame image in which more classes are shown among the plurality of classes obtained by the quantization unit, and a representative frame image in which a part of the plurality of classes is shown The image selection apparatus according to claim 1, wherein all of the plurality of classes are covered by selecting a plurality of classes. A video acquisition unit that acquires videos;
In a plurality of frame images constituting the moving image acquired by the moving image acquisition unit, a change amount acquisition unit that acquires a change amount between frames of an object reflected in the frame image by comparing the plurality of frame images;
A storage unit for storing a cumulative sum of the change amounts acquired by the change amount acquisition unit;
A selection unit that selects a frame image when the accumulated sum stored in the storage unit reaches a predetermined threshold as a representative frame image from the plurality of frame images;
An image selection device comprising:

Images