JP7295053B2 - Scene extraction method, device and program - Google Patents

Description

The present invention relates to a method, apparatus, and program for extracting a scene of interest from a moving image. The present invention relates to a scene extracting method, device and program capable of automatically extracting a scene of interest by going back to a related posture scene that was a strategy or a trigger.

Techniques for extracting an interesting scene of interest from a moving image are disclosed in Patent Documents 1 to 3.

Patent Document 1 discloses a technique for identifying a referee from a sports video, estimating the posture of the referee, estimating the motion of the referee from the posture estimation result, and accurately extracting a play section based on the motion estimation result. disclosed.

Patent Literature 2 discloses a technique of extracting a specific telop included in a sports video and extracting a temporal section before and after the change in the telop as an event section. Patent Literature 2 also discloses a technique of extracting the time when a specific telop display last disappears in an event period, and performing correction by setting the time that is a certain amount of time before that time as the starting point of the event period. .

Japanese Patent Application Laid-Open No. 2004-200002 discloses a technique for switching a sub-screen to a main screen when the feature amount of content displayed on the sub-screen exceeds a predetermined threshold. In the case of a live soccer program, when a goal is scored, the sound output of the audience cheering is louder, and the sound volume and audio signals of a predetermined frequency are louder. can be displayed on the screen.

Patent application No. 2016-556198 Patent application No. 2006-98340 Patent application No. 2005-208839

Z. Cao, T. Simon, S. Wei and Y. Sheikh, "Realtime Multi-person 2D Pose Estimation Using Part Affinity Fields," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp 1302-1310.

In Patent Literature 1, a specific important play section can be extracted from the referee's movement. In Patent Documents 2 and 3, it is possible to extract a section including before and after a telop or an audio signal as an important play section in conjunction with the magnitude of the telop or audio signal. However, in sports broadcasts, there may be a related scene that serves as the foundation for the important play section before the important play section. be

For example, shooting scenes and goal scenes can be important scenes in a soccer broadcast, but before these important scenes, there are scenes related to assists such as passing and centering, which are the foundation for those important scenes. There are many things.

However, the prior art described above focuses only on a specific important scene, and does not extract the target scene by going back to the relevant scene that was the foundation for that.

SUMMARY OF THE INVENTION An object of the present invention is to solve the above-described technical problems, and to automatically detect a scene of interest by going back to scenes related to the scene that serves as a basis for extracting a scene of interest based on the result of estimating the player's posture. The object is to provide a scene extraction method, device, and program capable of extracting scenes in a realistic manner.

In order to achieve the above object, the present invention is characterized by having the following configuration in a method, apparatus and program for extracting a scene of interest from a moving image.

(1) Estimating the pose of a person extracted from a moving image, detecting a related posture scene in which the result of posture estimation is a related posture and a specific posture scene in which a specific posture is detected. A video section up to the specific posture scene is extracted as a target scene by going back to the related posture scene. Here, the related posture is a posture that serves as a foundation for the specific posture.

(2) We tracked objects in moving images, and detected specific pose scenes and related pose scenes based on the results of human pose estimation and object tracking.

(3) A scene from a related posture scene whose elapsed time to a specific posture scene is within a predetermined time to the specific posture scene is extracted as a target scene.

(4) When multiple related posture scenes are detected for which the elapsed time to the specific posture scene is within a predetermined period of time, a list of multiple target scene candidates from each related posture scene to the specific posture scene is displayed, and the operator is prompted to select one. I made it so that you can select the attention scene.

According to the present invention, the following effects are achieved.

(1) When a specific posture scene is detected based on a person's posture, it is traced back to the related posture scene that was the foundation for the scene, and the video section from the related posture scene to the specific posture scene is extracted as the target scene. Scenes can be continuously and objectively extracted from the scene that triggered them.

(2) Tracking objects in moving images and detecting specific pose scenes and related pose scenes based on the results of human pose estimation and object tracking. Pose scenes and related pose scenes can be detected accurately.

(3) Since it is possible to extract, as a scene of interest, a section from a related posture scene in which the elapsed time to the specific posture scene is within a predetermined time to the specific posture scene, the target scene can be extracted based on the specific posture scene and the related posture scene that are not related to each other. can be prevented from being extracted.

(4) If multiple related posture scenes are detected before a specific posture scene, a list of multiple target scene candidates from each related posture scene to the specific posture scene can be displayed and the operator can select one. Subjective scene extraction through human judgment becomes possible.

1 is a functional block diagram of a scene extraction device according to a first embodiment of the present invention; FIG. FIG. 4 is a diagram showing an example of a skeleton to be extracted in posture estimation; 4 is a flow chart showing the operation of the first embodiment; FIG. 4 is a diagram schematically showing a method of determining a scene of interest; FIG. 5 is a functional block diagram of a scene extraction device according to a second embodiment of the present invention; FIG. 3 illustrates a scene detection method that considers object tracking results; FIG. 10 is a diagram showing another method of extracting a scene of interest; FIG. 10 is a diagram showing another example of reproduction of a scene of interest;

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a scene extraction device 1 according to the first embodiment of the present invention. Here, a scene of interest including a goal scene is extracted from a camera image of a soccer match. A case will be described as an example.

Such a scene extraction device 1 is implemented by installing an application (program) for realizing each function described later in at least one general-purpose computer or mobile terminal equipped with a CPU, a memory, an interface, and a bus connecting them. can be configured with Alternatively, a part of the application can be configured as a dedicated machine or a single-function machine that is hardware or programmed.

A camera image acquisition unit 101 acquires camera images from a plurality of cameras that capture images of a competition field. A camera video is recorded in a video database (DB) 103, and a frame image acquisition unit 102 acquires a frame image from the camera video. At this time, after the camera video is once recorded in the video database (DB) 103, the frame image acquisition unit 102 reads the camera video again from the video database 103, and the frame image is acquired from the read camera video. good. The posture estimation unit 104 includes a person region extraction unit 104a and a skeleton information extraction unit 104b, and estimates the posture of a person extracted from each frame image for each camera.

The person area extraction unit 104a extracts a person area from each frame image of the camera video. An SSD (Single Shot Multibox Detector), for example, can be used to extract the person area.

The skeleton information extraction unit 104b extracts a skeleton registered in advance as an extraction target from the person region of the frame image, and registers the position information and the connection state with other skeletons as skeleton information. Existing skeleton extraction technology (Cascaded Pyramid Network) can be used to extract skeleton information.

FIG. 2 is a diagram showing a skeleton to be extracted by the skeleton information extraction unit 104b. Left and right elbow joints P3 and P6, left and right wrist joints P4 and P7, left and right knee joints P9 and P12, and left and right ankles Joints P10 and P13 and bones connecting these joints are extracted.

Note that the skeleton extraction method is not limited to the method targeting the human region extracted in advance as described above. For example, as disclosed in Non-Patent Document 1, a Confidence Map that encodes the positions of body parts and Part Affinity Fields (PAFs) that encode connectivity between body parts are added to a feature map extracted from a frame image. A skeleton model is constructed by sequentially applying the two sequential prediction processes used to estimate the positions and connectivity of body parts of human objects (users) extracted from frame images in a single inference using a bottom-up approach. You may

At this time, by implementing a process that excludes the connectivity of body parts extracted from different partial regions from the target of estimation, the position and connectivity of body parts are estimated for each partial region, that is, for each user, and the skeleton model of the object is estimated. become able to.

The scene detection unit 105 includes a specific posture scene detection unit 105a, a related posture scene detection unit 105b, a learning model 105c, and a scene registration unit 105d. A related posture scene Rs in which a person is in a related posture is detected.

A specific posture is a representative posture of a player that serves as a basis for extracting a scene of interest, and corresponds to, for example, a player's posture in a goal scene or a shoot scene. A related posture is a representative posture of a player related to a specific posture, and corresponds to, for example, a player's posture such as pass, centering, free kick, corner kick, or heading. Therefore, the related posture scene Rs is detected before the specific posture scene Qs, and can be said to be a video scene that serves as a foundation for the specific posture scene Qs. The learning model 105c stores a prediction model M that has been learned in advance for estimating whether the player's posture estimated by the posture estimation unit 104 corresponds to a specific posture or a related posture.

The specific posture scene detection unit 105a detects a specific posture scene Qs in which the player is in a specific posture by applying the posture estimation result to the prediction model M. FIG. The related posture scene detection unit 105b detects a related posture scene Rs in which the player is in a related posture by applying the posture estimation result to the prediction model M. FIG. The playback time of the specific posture scene Qs and the playback time of the related posture scene Rs are registered in the scene registration unit 105d.

The attention scene determination unit 106 includes an attention scene candidate presentation unit 106a and an attention scene selection unit 106b, and determines an attention scene based on the reproduction time of the specific posture scene Qs and the reproduction time of the related posture scene Rs. The attention scene reproducing unit 107 reproduces the determined attention scene.

Note that if the related posture scene Rs is detected based on the player's heading posture, for example, the scene of interest is a scene slightly before the time when the heading posture was detected. It would be desirable to include an example scene up to a related posture, such as running to the position where the ball reaches, and then jumping to compete with an opponent player in the air. Therefore, in the present embodiment, the start time of the scene of interest may be set to a time that precedes the playback time of the related posture scene Rs by a predetermined time Δt. For the same reason, the end time of the scene of interest may also be set to a time after the playback time of the specific posture scene Qs by a predetermined time Δt. The predetermined time Δt may be a fixed value, or may be set in advance for each type of related posture or specific posture.

FIG. 3 is a flow chart showing the operation of this embodiment, and FIG. 4 is a diagram schematically showing a method of determining a scene of interest by the scene-of-interest determining unit 106. As shown in FIG.

In FIG. 3, in step S1, a camera image is obtained in real time and recorded in the image DB 103. FIG. In step S2, the frame image acquisition unit 102 acquires a frame image from the acquired or recorded camera video. In step S3, the person area is extracted from the frame image by the person area extraction unit 104a. In step S4, the skeleton information of each player is extracted by the skeleton information extraction unit 104b, and the posture of the player is estimated.

In step S5, the related posture scene detection unit 105b determines whether or not the estimated player posture corresponds to a predefined related posture. In this embodiment, many related postures are defined, and if the estimated posture corresponds to any of the related postures, the process proceeds to step S6. In step S6, the playback time t1 (t2, t3) of the related posture is registered as the timing of the related posture scene Rs by the scene registration unit 105d. After that, the process returns to step S2, and the registration of the related posture scene Rs is repeated.

On the other hand, if it is determined in step S5 that the player's posture is not a related posture, the process proceeds to step S7, where the player's posture estimated in step S4 is specified in advance by the specific posture scene detection unit 105a. It is determined whether or not it corresponds to the posture. In this embodiment, a plurality of specific orientations are defined, and if the estimated orientation does not correspond to any of the specific orientations, the process returns to step S2, and similar processing is repeated for the next frame image.

On the other hand, if the estimated player's posture corresponds to any of the specific postures, the process proceeds to step S8, and the playback time t4 is registered as the timing of the specific posture scene Qs by the scene registration unit 105d. In step S9, the elapsed time ΔT (=t4-t1, t4-t2, t4-t3) up to time t4 among the related posture scenes Rs registered up to time t4 at which the particular posture was estimated is specified. All relevant pose scenes Rs within the time ΔTref are extracted.

In step S10, each video section from each extracted related posture scene Rs to the specific posture scene Qs is displayed as a list of target scene candidates by the target scene candidate presenting unit 106a.

FIG. 4 is a diagram showing a presentation example of target scene candidates when three related pose scenes Rs1, Rs2, and _Rs3 are extracted in the order with respect to the specific pose scene Qs. Thumbnails of the scene Qs and the three related pose scenes Rs1, Rs2, and Rs3 are displayed at their respective playback times t1, t2, t3, and t4, and below them, three candidate scenes of interest are listed together with their playback times. there is

In the present embodiment, the first scene of interest from the first related posture scene Rs1 to the specific posture scene Qs, the second scene of interest from the second related posture scene Rs2 to the specific posture scene Qs, and the third related posture scene Rs3 are specified. A list of the third scenes of interest up to the posture scene Qs is displayed. Thumbnails of related posture scenes Rs1, Rs2, and Rs3 are displayed at the start positions of the respective scenes of interest, and thumbnails of the specific posture scene Qs are displayed at the end positions.

The operator selects one of the attention scene candidates as the attention scene, and when this is detected by the attention scene selection unit 106b in step S11, the process proceeds to step S12. In step S12, the selected attention scene is reproduced from the related posture scene Rs to the specific posture scene Qs. For example, when the second scene of interest is selected, the video period from time t2 or time t2-Δt, which is a predetermined time Δt before, to time t4 or time t4+Δt, which is a predetermined time Δt after t4, is reproduced. .

In the above embodiment, it is assumed that all related posture scenes Rs within the predetermined time ΔTref are extracted when the specific posture scene Qs is detected, but the present invention is not limited to this. , a related posture scene Rs is associated with each type of specific posture Qs (a goal scene or a shooting scene), and when a specific posture scene Qs is detected, a related posture scene Rs within a predetermined time ΔTref corresponding to the type of the specific posture scene Qs is determined. Only the posture scene Rs may be extracted.

FIG. 5 is a functional block diagram showing the configuration of the main parts of a scene extraction device 1 according to the second embodiment of the present invention. omitted.

In this embodiment, an object tracking unit 108 that tracks objects between frame images is provided, and a scene detection unit 105 detects a specific posture scene Qs and a related posture scene Rs based on the tracking result of each object and the result of estimating the posture of each player. It is characterized in that it is designed to detect

In the object tracking section 108, the object detection section 108a detects an object from each frame image. In this embodiment, a player and a soccer ball are objects to be detected. The ID assigning unit 108b tracks each object between frame images based on similarity in shape, size and/or texture (e.g., uniform design) and position estimation based on a movement vector. By assigning the same ID (object identifier) to each object, object tracking is realized between frames.

In the above-described first embodiment, since each scene is detected based on the results of estimating the player's posture, attributes such as the team to which each player belongs and the position of each player (for example, whether or not the player is a goalkeeper) cannot be identified. Also, the whereabouts of the ball could not be reflected in the scene detection. In contrast, in the present embodiment, each player and ball can be identified and tracked, so each scene can be detected more finely and accurately.

FIG. 6 is a diagram schematically showing a method for detecting the related posture scene Rs and the specific posture scene Qs in this embodiment. , the ball 30 is passed to another player a2 of the same team A, and the ball 30 is passed from the said player a2 to the player a3 of the same team A, and the said player a3 dodges the goalkeeper bg of the other team (team B). It shows an example flow from shooting the ball 30 to scoring a goal.

In such a case, when the ball 30 moves between the players, it can be determined as a pass between players of the same team, and an interception or a cut between players of different teams, so that scene detection can be performed accurately. Become.

In this embodiment, the ball 30 shot by the player a3 and the ball 30 that hit the goal immediately after that are the same (that is, the ball 30 can be tracked; the same applies hereinafter). , and player a3 can be recognized as the scorer.

Furthermore, since player a3 and player a2 are on the same team and the ball 30 kicked by player a2 is the same as the ball 30 received by player a3 immediately after that, the act is a pass from player a2 to player a3. At the same time, it is found that player a2 is an assist. Therefore, the scene can be detected as a related posture (assist) scene.

Furthermore, since player a2 and player a1 are on the same team, and the ball 30 kicked by player a1 is the same as the ball 30 received by player a2 immediately after that, the act is a pass from player a1 to player a2. , and it can be seen that player a1 may also be an assist. Therefore, this scene can also be detected as a related posture (assist) scene.

Thus, according to this embodiment, each player can be identified and tracked, and the ball can be tracked. In addition, by referring to the tracking results of the player and the ball up to the specific posture scene Qs of shooting, the related posture scene Rs of assist and pass can also be accurately detected.

In other words, it becomes possible to identify similar scenes that could not be identified based on the player's posture alone, and to accurately detect a wide variety of scenes that were difficult to accurately detect based on the player's posture alone.

Note that, in the above-described embodiment, related posture scenes that are likely to be detected in the future and that serve as a basis for a specific posture are detected and registered in advance. It has been explained that the attention scene is determined by going back to the relevant posture scene. However, the present invention is not limited to this, and when a specific posture is detected without detecting a related posture scene in advance, the camera image is traced back and the related posture scene corresponding to the specific posture is detected. A scene may be determined.

Furthermore, in the above embodiment, there is no mention of the difference between the camera cam that captured the camera image from which the related posture scene was detected and the camera cam that captured the camera image from which the specific posture scene was detected. There is already established technology for tracking each object not only between frames but also between cameras when shooting a competition field with multiple cameras as in . Therefore, when a specific posture scene is detected in a camera image taken with one camera cam, the related posture scene detected not only in the camera image taken by the same camera but also in the camera image taken by a different camera is used. A video section up to a specific posture scene may be set as a target scene (candidate).

In the example shown in FIG. 7, at time t1, the related posture scene Rs1 is detected from the camera video taken by the camera cam1, at time t2, the related posture scene Rs2 is detected from the camera video taken by the camera cam2, and at time t3. , the related posture scene Rs3 is detected from the camera video taken by the camera cam3. After that, at time t4, when specific posture scene Rs is detected from the camera video taken by camera cam4, the video segment from time t1 to t2 captured by camera cam2, the video segment from time t2 to t3 captured by camera cam3, and A scene of interest is extracted by connecting video sections from times t3 to t4 captured by camera cam4.

It should be noted that if the target scene includes a video section of a predetermined time Δt up to the related posture scene Rs1 in the same manner as described above, as shown in FIG. to t1 may be connected before the scene of interest.

Reference Signs List 1 scene extraction device 30 ball 101 camera image acquisition unit 102 frame image acquisition unit 103 image DB 104 posture estimation unit 105 scene detection unit 106 attention scene determination unit 107 Attention scene reproducing unit 108... Object tracking unit

Claims (9)

In a scene extraction device for extracting a scene of interest from a moving image,
posture estimation means for estimating the posture of a person extracted from a moving image;
a means for detecting a related pose scene whose result of pose estimation is a related pose;
means for detecting a specific posture scene in which the result of posture estimation is a specific posture;
means for, when the specific posture scene is detected, extracting as a scene of interest up to the specific posture scene by going back to related posture scenes that have already been detected and for which the elapsed time up to the specific posture scene is within a predetermined period of time ;
means for displaying a list of a plurality of target scene candidates from each related posture scene to the specific posture scene when a plurality of related posture scenes having elapsed time up to the specific posture scene within a predetermined time are detected ,
The scene extracting apparatus, wherein the extracting means selects a separately designated target scene candidate as the target scene . further comprising means for tracking an object in the moving image;
2. The scene extracting apparatus according to claim 1, wherein said means for detecting a specific pose scene and said means for detecting a related pose scene detect each scene based on a result of pose estimation and a result of object tracking. . 3. The scene extracting apparatus according to claim 1 , wherein the related posture is a posture that serves as a foundation for the specific posture. comprising means for associating a related posture with each type of the specific posture;
4. When a specific posture scene is detected, the extracting means traces back to a related posture scene corresponding to a type of the specific posture scene and extracts the scene up to the specific posture scene as the target scene. The scene extraction device according to any one of 1. 5. The scene extracting apparatus according to claim 1, wherein said scene of interest is configured by connecting video sections extracted from a plurality of moving pictures from different viewpoints. In a scene extraction method in which a computer extracts a scene of interest from a moving image,
Estimates the pose of a person extracted from a moving image,
Detecting a related pose scene whose pose estimation result is a related pose,
Detecting a specific posture scene in which the result of posture estimation is a specific posture,
when the specific posture scene is detected, extracting as a scene of interest up to the specific posture scene by going back to related posture scenes that have already been detected and for which the elapsed time up to the specific posture scene is within a predetermined time , and
if a plurality of related posture scenes having elapsed time up to the specific posture scene within a predetermined time are detected, displaying a list of a plurality of target scene candidates from each related posture scene to the specific posture scene;
2. A scene extracting method, characterized in that, in extracting the scene of interest, a separately specified one candidate of the scene of interest is set as the scene of interest . 7. The scene extraction method according to claim 6 , further comprising tracking an object in a moving image, and detecting the specific pose scene and the related pose scene based on the result of pose estimation and the result of object tracking. In a scene extraction program for extracting attention scenes from moving images,
a procedure for estimating the pose of a person extracted from a moving image;
a procedure for detecting a related pose scene whose result of pose estimation is a related pose;
a procedure for detecting a specific posture scene in which the result of posture estimation is the specific posture;
a step of, when the specific posture scene is detected, extracting as a scene of interest up to the specific posture scene by going back to related posture scenes that have already been detected and for which the elapsed time up to the specific posture scene is within a predetermined time ;
If a plurality of related posture scenes whose elapsed time to the specific posture scene is within a predetermined time are detected, a procedure for displaying a list of a plurality of target scene candidates from each related posture scene to the specific posture scene is caused to be executed by the computer. ,
A scene extraction program in which the extracting procedure uses a separately designated target scene candidate as a target scene . further comprising tracking an object in the moving image;
9. The scene extraction program according to claim 8 , wherein the specific pose scene and the related pose scene are detected based on the pose estimation result and the object tracking result.

Images