WO2018123202A1

WO2018123202A1 - Moving-image processing device, display device, moving-image processing method, and control program

Info

Publication number: WO2018123202A1
Application number: PCT/JP2017/036763
Authority: WO
Inventors: 直大北城
Original assignee: シャープ株式会社
Priority date: 2016-12-28
Filing date: 2017-10-11
Publication date: 2018-07-05

Abstract

The texture of an object represented in a moving-image is heightened. A signal processing unit (10) is provided with a texture detection unit (13) for analyzing the motion vector of a moving-image and thereby determining the texture of an object represented in the moving-image, and a texture correction unit (14) for performing a moving-image process that corresponds to the determination result of the texture detection unit (13).

Description

Moving image processing device, display device, moving image processing method, and control program

The following disclosure relates to a moving image processing apparatus and the like.

Conventionally, various moving image processing techniques have been proposed in order to provide a moving image that looks like a real thing to a person (user, viewer). As an example, Patent Literature 1 discloses a technique for imparting a predetermined texture to a predetermined area in a moving image. Specifically, Patent Document 1 discloses an image texture manipulation method for dynamically manipulating an image area (second image area) that gives a transparent layer texture.

Japanese Patent Publication “Japanese Unexamined Patent Publication No. 2016-126641 (Released on July 11, 2016)”

However, in the invention disclosed in Patent Document 1, it is not possible to perform moving image processing according to the texture of individual objects (objects) expressed in the moving image.

One aspect of the present disclosure has been made in view of the above-described problems, and an object thereof is to realize a moving image processing apparatus or the like that can enhance the texture of an object expressed in a moving image. .

In order to solve the above-described problem, a moving image processing apparatus according to an aspect of the present disclosure analyzes a motion vector of a moving image to determine a texture of an object expressed in the moving image moving image. And a moving image processing unit that processes the moving image according to the determination result of the texture determining unit.

In order to solve the above-described problem, a moving image processing method according to an aspect of the present disclosure analyzes a motion vector of a moving image to determine a texture of an object expressed in the moving image. And a moving image processing step for processing the moving image in accordance with the determination result in the texture determination step.

The moving image processing apparatus according to one aspect of the present disclosure has an effect that the texture of the object expressed in the moving image can be enhanced.

Also, the same effect can be achieved by the moving image processing method according to an aspect of the present disclosure.

3 is a functional block diagram illustrating a configuration of a main part of the display device according to the first embodiment. FIG. It is the schematic for demonstrating a motion vector. It is a figure which shows an example of a 2nd motion vector set. (A) And (b) is a figure for demonstrating the relationship between the viscosity and the texture of a liquid, respectively. It is a figure which shows an example of HMM. (A) And (b) is a figure for demonstrating the moving image process in the display apparatus of FIG. 1, respectively. It is a functional block diagram which shows roughly the signal processing part which concerns on Embodiment 2, and the structure of the periphery. It is a functional block diagram which shows roughly the structure of the signal processing part which concerns on Embodiment 3, and its periphery. It is a functional block diagram which shows roughly the structure of the signal processing part which concerns on one modification, and its periphery. It is a functional block diagram which shows roughly the signal processing part which concerns on Embodiment 4, and the structure of the periphery.

Embodiment 1
Hereinafter, Embodiment 1 will be described in detail with reference to FIGS. First, an overview of the display device 1 according to the first embodiment will be described with reference to FIG. FIG. 1 is a functional block diagram illustrating a configuration of a main part of the display device 1.

(Outline of display device 1)
The display device 1 includes a signal processing unit 10 (moving image processing device), a receiving unit 60, a decoding unit 61, a display unit 70, and a storage unit 90. As an example, the display device 1 may be a television or a PC (Personal Computer). Alternatively, the display device 1 may be a portable information terminal such as a multifunction mobile phone (smartphone) or a tablet.

As described below, in the display device 1, the signal processing unit 10 processes the moving image (input image image) and outputs the processed moving image (output moving image) to the display unit 70. The display unit 70 displays a moving image. The display unit 70 may be, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display.

The signal processing unit 10 is provided as a part of a control unit (not shown) that comprehensively controls each unit of the display device 1. The function of the control unit may be realized by a CPU (Central Processing Unit) executing a program stored in the storage unit 90. The function of each part of the signal processing unit 10 will be described in detail later.

The storage unit 90 stores various programs executed by the signal processing unit 10 and data used by the programs. As an example, the storage unit 90 stores a feature pattern model DB (DataBase) 91. The feature pattern model DB 91 is a DB in which various feature pattern models (described later) are stored.

In Embodiment 1, the receiving unit 60 receives broadcast waves (radio waves). The decoding unit 61 acquires compressed moving image data (moving image data compressed by a predetermined encoding method) included in the broadcast wave received by the receiving unit 60. Subsequently, the decoding unit 61 acquires an input moving image (input video signal) by decoding the compressed moving image data. Then, the decoding unit 61 supplies the acquired input moving image to the signal processing unit 10 (more specifically, the first correction unit 11 described later).

Hereinafter, for the sake of simplicity, an input moving image (moving image input to the signal processing unit 10) is also referred to as a moving image A. The moving image A is a moving image to be processed in the signal processing unit 10. As an example, the moving image A may have a resolution of 4K2K (resolution of 3840 horizontal pixels × 2160 vertical pixels). However, the resolution of each moving image described in the first embodiment is not limited to the above, and may be set as appropriate.

Note that the receiving unit 60 and the decoding unit 61 may be provided as an integrated functional unit. For example, a known tuner can be used as the receiving unit 60 and the decoding unit 61.

Further, when the moving image A is stored in the storage unit 90 in advance, the signal processing unit 10 may acquire the moving image A from the storage unit 90. Alternatively, the signal processing unit 10 may acquire the moving image A from an external device (for example, a digital movie camera) connected to the display device 1.

As described below, the signal processing unit 10 processes the moving image A supplied from the receiving unit 60 to generate an output moving image (output video signal). Then, the signal processing unit 10 (more specifically, a texture correction unit 14 described later) supplies the output moving image to the display unit 70. According to this configuration, the output moving image can be displayed on the display unit 70. A display control unit (not shown) that controls the operation of the display unit 70 may be provided in the signal processing unit 10 or may be provided in the display unit 70 itself.

(Signal processing unit 10)
Next, a specific configuration of the signal processing unit 10 will be described. As shown in FIG. 1, the signal processing unit 10 includes a first correction unit 11, a frame rate conversion unit 12, a texture detection unit 13 (texture determination unit), and a texture correction unit 14 (moving image processing unit). Yes.

Note that, as described below, the texture detection unit 13, the texture correction unit 14, and the feature pattern model DB 91 are main parts of the moving image processing apparatus according to an aspect of the present disclosure. The texture detection unit 13, the texture correction unit 14, and the feature pattern model DB 91 may be collectively referred to as “texture processing unit”. In FIG. 1 and each drawing to be described later, the texture processing unit is indicated by a dotted line for convenience of explanation.

The first correction unit 11 processes the moving image A described above. Hereinafter, the moving image after processing in the first correction unit 11 is also referred to as a moving image B. The process in the first correction unit 11 may be a known image quality correction process. As an example, the first correction unit 11 may perform scaling (resolution change) on the moving image A. In this case, the resolution of the moving image displayed on the display unit 70 can be converted into a resolution according to the performance specifications of the display unit 70.

However, it should be noted that the first correction unit 11 is not an essential component in the signal processing unit 10 as shown in FIG. For example, if the resolution of the moving image A already conforms to the performance specifications of the display unit 70, the first correction unit 11 does not need to generate the moving image B (convert the resolution).

Note that the first correction unit 11 sets image quality parameters of the moving image A (eg, parameters indicating the degree of brightness, contrast, color density, peaking, outline enhancement, etc.) according to the user's operation. Also good. In this case, the first correction unit 11 processes the moving image A using the set image quality parameter. For example, when the image quality parameter is arbitrarily selected by the user according to the usage mode of the user, the first correction unit 11 may be operated as described above.

The first correction unit 11 supplies the moving image B to the frame rate conversion unit 12 (more specifically, each of the interpolation image generation unit 121 and the motion vector calculation unit 122 described below). When the first correction unit 11 is excluded from the signal processing unit 10, the moving image A may be supplied from the decoding unit 61 to the frame rate conversion unit 12.

The frame rate conversion unit 12 includes an interpolated image generation unit 121 and a motion vector calculation unit 122. The interpolated image generation unit 121 performs processing for increasing the frame rate of the moving image B. Specifically, the interpolated image generation unit 121 extracts each of a plurality of frames constituting the moving image B from the moving image B. Each frame extracted by the interpolated image generation unit 121 may be stored, for example, in a frame memory (not shown). The frame memory may be provided in the frame rate conversion unit 12 or may be provided outside the frame rate conversion unit 12.

Subsequently, the interpolated image generation unit 121 generates an interpolation frame (intermediate frame) based on the frame using a known algorithm. For example, the interpolated image generation unit 121 may generate an interpolation frame using a motion vector described below. Then, the interpolated image generation unit 121 increases the frame rate of the moving image B by inserting interpolation frames into the moving image B at predetermined frame intervals.

Hereinafter, the processed moving image in the interpolated image generation unit 121 is also referred to as a moving image C. As an example, the frame rate conversion unit 12 may increase the frame rate of the moving image B by a factor of two. For example, when the frame rate of the moving image B is 60 fps (frames per second), the interpolated image generating unit 121 generates a moving image C having a frame rate of 120 fps. Then, the interpolated image generation unit 121 supplies the moving image C to the texture correction unit 14 (more specifically, the second correction unit 142 described below).

Note that the conversion rate of the frame rate in the frame rate conversion unit 12 is not limited to the above, and may be set as appropriate. Further, the frame rate of each moving image described in the first embodiment is not limited to the above.

By providing the interpolated image generation unit 121, the frame rate of the moving image displayed on the display unit 70 can be converted into one according to the performance specifications of the display unit 70. However, it should be noted that the interpolated image generation unit 121 is not an essential component of the signal processing unit 10 as illustrated in FIG. For example, if the frame rate of the moving image B (moving image A) already conforms to the performance specifications of the display unit 70, the interpolated image generating unit 121 generates the moving image C (converts the frame rate). It is not necessary.

The motion vector calculation unit 122 calculates (detects) a motion vector by analyzing the moving image B (more specifically, each frame of the moving image B stored in the frame memory). A known algorithm may be used to calculate the motion vector in the motion vector calculation unit 122.

Note that when the interpolated image generation unit 121 is excluded from the frame rate conversion unit 12, a function of extracting each frame from the moving image B may be given to the motion vector calculation unit 122. Further, as shown in FIG. 8 and the like described later, the motion vector calculation unit 122 can be excluded from the signal processing unit 10. That is, it should be noted that the frame rate conversion unit 12 is also not an essential component in the signal processing unit 10.

Next, the motion vector will be described. First, consider a case where each frame constituting a moving image (eg, moving image B) is spatially divided into a plurality of blocks (regions). A motion vector is a block (more specifically, a virtual object located in a block) in one frame (eg, a reference frame) and another frame (eg, a reference frame) subsequent to the one frame. This is a vector indicating the positional deviation from the corresponding block in the next frame.

That is, a motion vector is a vector indicating to which position a block in one frame has moved in another subsequent frame. The motion vector is used as an index indicating the movement amount of the block.

FIG. 2 is a schematic diagram for explaining a motion vector. As shown in FIG. 2, each frame included in the moving image B is uniformly divided into blocks having a horizontal length (resolution) a and a vertical length b. Here, the horizontal pixel number of the moving image B is represented as H, and the vertical pixel number is represented as V.

In this case, each frame is divided in the horizontal direction (H / a) and in the vertical direction (V / b). That is, each frame is divided into (H / a) × (V / b) blocks. Note that the values of a, b, H, and V may be set arbitrarily.

Here, one of the blocks in FIG. 2 is represented as a block (x, y). x and y are indices (numbers) indicating horizontal and vertical positions in each frame, respectively. In addition, let the block located in the upper left among the blocks in FIG. 2 be a block (0, 0). In FIG. 2, (i) the block number in the horizontal direction increases from left to right, and (ii) the block number in the vertical direction increases by 1 from top to bottom. Is set. Therefore, “0 ≦ x ≦ H / a−1 and 0 ≦ y ≦ V / b−1”.

As shown in FIG. 2, the motion vector of the block (x, y) is represented as MV (x, y). The motion vector calculation unit 122 calculates a motion vector for each block in FIG.

Hereinafter, a set of motion vectors calculated by the motion vector calculation unit 122 for each block of one frame is referred to as a first motion vector set. Note that the first motion vector set is the following set MVSet1, that is,
MVSet1 = {MV (0, 0), MV (1, 0),..., MV (H / a-1, 0), MV (0, 1), MV (1, 1),. a-1, 1), ..., MV (H / a-1, V / b-1)}
Represented as:

The motion vector calculation unit 122 supplies the first motion vector set described above to the interpolation image generation unit 121 and the texture detection unit 13 (more specifically, the extraction unit 131 described below).

The texture detection unit 13 includes an extraction unit 131 and a collation unit 132. As will be described below, the texture detection unit 13 analyzes each motion vector of a moving image (for example, moving image B), so that each object represented in the moving image (more specifically, in the moving image) Is determined. Then, the texture detection unit 13 supplies the determination result (a texture ID described later) to the texture correction unit 14. Hereinafter, each part of the texture detection part 13 is demonstrated concretely.

The extraction unit 131 extracts (acquires) a part (subset) of the first motion vector set described above. Hereinafter, this subset is referred to as a second motion vector set.

FIG. 3 is a diagram illustrating an example of the second motion vector set. As illustrated in FIG. 3, the extraction unit 131 may extract an area composed of blocks (m, n) to (m + A−1, n + B−1) in each frame as a partial area. The values of m, n, A, and B may be arbitrary values as long as the partial area is set so as not to deviate spatially from each frame.

Subsequently, the extraction unit 131 acquires a motion vector of each block in the partial area. That is, the extraction unit 131 acquires, as the second motion vector set, the motion vector set of each block in the partial area in the first motion vector set. Then, the extraction unit 131 supplies the second motion vector set to the collation unit 132.

In the example of FIG. 3, the second motion vector set is the following set MVSet2, that is,
MVSet2 = {MV (m, n), MV (m + 1, n), ..., MV (m + A-1, n), MV (m, n + 1), MV (m + 1, n + 1), ..., MV (m + A-1, 1), ..., MV (m + A-1, n + B-1)}
Represented as:

The collation unit 132 compares (collates) the second motion vector set (motion vector) with various feature pattern models included in the feature pattern model DB 91 described above. Here, the “feature pattern model” may be a model representing a motion vector (more specifically, a set of motion vectors) representing the texture of an object represented in a moving image. Alternatively, the feature pattern model may be a model related to a motion vector (a set of motion vectors) representing the texture.

As an example, the feature pattern model is a set of motion vectors derived (set) by performing learning (automatic learning) with a prior pattern recognition technique on the second motion vector set. The feature pattern model may be set in advance in the display device 1. As will be described below, the matching unit 132 can determine the texture of an object based on the feature pattern model.

In the first embodiment, for convenience of explanation, a case where the collation unit 132 has a function as a function unit (pattern setting unit) for setting a feature pattern model is illustrated. However, the pattern setting unit may be provided as a functional unit separate from the matching unit 132.

The pattern setting unit performs the above learning using a known algorithm and sets a feature pattern model. The pattern setting unit stores the feature pattern model in the feature pattern model DB 91. By performing the above processing on various moving images (moving images obtained by shooting different objects), a feature pattern model corresponding to the texture of each object can be stored in the feature pattern model DB 91.

The “texture” in the present specification is a sensation perceived by a person (user, viewer), and among those sensations, such as a glossy feeling or a material feeling perceived by dynamically changing. Mean sense.

As an example, the sensation is expressed by onomatopoeia or mimetic words such as “sarasara”, “nebaba”, “yurayura”, “fluffy”, “kirakira”, and the like.

Thus, the texture in this specification does not necessarily have to directly specify the material of the object (eg, metal or paper). However, the texture in this specification may simulate visual texture in a general sense.

Note that FIG. 3 described above schematically shows an example of the second motion vector set when the object represented in the moving image is a liquid having high viscosity (eg, oil). The motion vector included in the second motion vector set is one of indices indicating the liquid flow velocity. In addition, the flow rate generally depends on the viscosity (viscosity) of the fluid. Therefore, the magnitude of the viscosity of the liquid can be distinguished by the motion vector. From this, the difference in the texture according to the viscosity can be distinguished by the motion vector.

4 (a) and 4 (b) are diagrams for explaining the relationship between the viscosity and the texture of the liquid. FIG. 4A shows a liquid having a low viscosity (for example, water). When a person sees a low-viscosity liquid flowing (moving), the liquid generally flows at a relatively high flow rate, and thus it is common for a person to perceive a “free” sensation.

On the other hand, a liquid (eg oil) having a high viscosity is shown in FIG. When a user sees a liquid with a high viscosity flowing, the liquid flows at a relatively low flow rate, and thus it is common for a person to perceive a “sticky” sensation.

Furthermore, the liquid generally moves smoothly (is easily deformed) in a natural state as compared with a solid. That is, the movement pattern differs greatly between liquid and solid. Even if it is a solid, the movement pattern varies depending on, for example, the difference in rigidity. From this, it can be said that each object has a peculiar movement pattern according to the texture.

Also, when light is incident on an object, the surface of the object reflects the light, so that a luminance distribution of reflected light is formed on the surface. A person perceives the glossiness (sensation of “glitter”) of an object by the luminance distribution. In addition, the person perceives the glossiness of the object more specifically by the movement (change) of the reflected light (specular reflection component or diffuse reflection component) accompanying the change of the viewpoint or the movement of the object.

From this, it can be said that the pattern of movement of the reflected light is peculiar to the texture of each object. Therefore, a motion vector indicating the pattern of reflected light movement can also be used as one index indicating the glossiness of the object.

As described above, the motion vector of the moving image can be used as one of indices indicating the texture of the object (particularly the surface of the object) expressed in the moving image. Based on the above points, the inventor of the present application “discriminates (detects and estimates) the texture of the object represented in the moving image based on the motion vector of the moving image, and the moving image corresponding to the determination result”. We have discovered a novel technical idea (knowledge) that “process”. Each component of the texture processing unit described above (for example, the texture detection unit 13 and the texture correction unit 14 included in the signal processing unit 10) is conceived based on the technical idea.

The collation unit 132 calculates (evaluates) the relevance (matching degree) of the second motion vector set (motion vector) to the feature pattern model. That is, the collation unit 132 acquires information indicating how much the texture of the object indicated by the second motion vector set matches the texture of the object indicated by the feature pattern model.

Hereinafter, this information is also referred to as a texture ID (texture discrimination information). The texture ID may be understood as information indicating a result of determining the texture of each object expressed in the moving image (eg, moving image B) by the matching unit 132. The texture detection unit 13 supplies the texture ID to the texture correction unit 14 (more specifically, the parameter setting unit 141 described below).

The collation unit 132 may acquire the texture ID using a known pattern recognition method for a two-dimensional vector set (vector sequence). Specific examples of the pattern recognition method include the following three.

(First method)
A correlation function between the second motion vector set and the feature pattern model is calculated. For example, the following equation (1):

Thus, the collation unit 132 calculates the correlation function ΦMV _in MV _database (x ′, y ′). The matching unit 132 may calculate the value of the correlation function as the texture ID. Note that MV _in represents a second motion vector set (observation series), and MV _database represents a feature pattern model.

According to the first method, the texture ID can be calculated by a simple calculation as compared with the second method and the third method described below. For this reason, the texture ID can be calculated even when the hardware resources are relatively limited (for example, when the processing performance of the processor is relatively low). Therefore, the signal processing unit 10 can be realized with a simple configuration.

(Second method)
A matching method using a probabilistic model is adopted. As the probability model, for example, an HMM (Hidden Markov Model) may be used. FIG. 5 is a diagram illustrating an example of the HMM.

In the HMM of FIG. 5, latent variables are represented as x (t) = {x1, x2, x3}, and observed values are represented as y (t) = {y1, y2, y3, y4}. Note that t is a symbol representing time. a _ij is a state transition probability, and b _ij is an output probability. When the second method is used, the pattern setting unit sets the values of a _ij and b _ij in advance by learning in advance. Then, the pattern setting unit stores the values of a _ij and b _{ij in} the feature pattern model DB 91.

And the following formula (2), that is,

Based on the above, the matching unit 132 is made to calculate the probability P (Y). The collation unit 132 may calculate the value of the probability P (Y) as the texture ID. The probability P (Y) is a probability that the observation series MV _in = Y = y (1), y (2), y (3), y (4) is obtained. Further, X in Expression (2) is the latent state series X = x1, x2, x3. Equation (2) indicates that the probability P (Y) is expressed as the sum of the probabilities of the latent state series X.

According to the second method, the texture ID can be calculated with higher accuracy than in the first method described above. For example, the texture ID can be appropriately calculated even when non-local deformation (that is, deformation with continuous expansion / contraction) occurs in the object (when the motion vector distribution includes expansion / contraction). When hardware resources are relatively abundant (for example, when the processing performance of the processor is relatively high), the second method may be adopted. When the second method is employed, an appropriate probability model needs to be set in advance by the designer of the display device 1.

(Third method)
Adopt deep learning technology. For example, using a neural network such as CNN (Convolutional Neural Network), the pattern setting unit is made to learn a feature pattern model in advance. Then, the collation unit 132 is made to compare the above-described observation series MV _in with the feature pattern model. In this case, the collation unit 132 may output the comparison result as a texture ID.

According to the third method, the texture ID can be calculated with higher accuracy than in the second method described above. In particular, when the feature setting model is learned by the pattern setting unit using sufficient hardware resources, it can be expected to calculate the texture ID with particularly high accuracy.

Further, according to the third method, the feature pattern can be obtained by appropriate learning even if the designer of the display device 1 does not specify in advance the individual features of the motion vector according to the type of texture (for example, glossiness). A model can be obtained. Therefore, it is expected that a texture ID corresponding to a wide range of textures can be acquired by a more flexible method. However, when the third method is adopted, it is necessary to prepare particularly abundant hardware resources.

Subsequently, the texture correction unit 14 will be described. The texture correction unit 14 includes a parameter setting unit 141 and a second correction unit 142 (correction unit). The parameter setting unit 141 selects (sets) an image quality parameter based on the texture ID acquired from the matching unit 132.

Specifically, in the parameter setting unit 141, a table of image quality parameters associated with the texture ID is set in advance. The parameter setting unit 141 refers to the table and selects an image quality parameter corresponding to the texture ID. Then, the parameter setting unit 141 supplies the selected image quality parameter to the second correction unit 142.

The second correction unit 142 may be a functional unit that performs a known image quality correction process, similarly to the first correction unit 11 described above. However, the second correction unit 142 may perform image quality correction processing (image quality correction processing mainly intended for texture reproduction) by a new method that is not known.

As described above, the moving image C is supplied from the interpolated image generation unit 121 to the second correction unit 142. The second correction unit 142 processes the moving image C using the image quality parameter selected by the parameter setting unit 141. Hereinafter, the moving image after processing in the second correction unit 142 is also referred to as a moving image D (output moving image). The second correction unit 142 supplies the moving image D to the display unit 70.

According to this configuration, the second correction unit 142 can perform processing according to the texture of the object expressed in the moving image C using the image quality parameter selected by the parameter setting unit 141. According to the image quality correction process, for example, it is possible to enhance the glossiness or texture of an object. As described above, the second correction unit 142 can generate the moving image D with a higher texture. Therefore, it is possible to provide the user with the moving image D in which the texture is more emphasized (expressed more effectively).

Note that the second correction unit 142 may process the moving image C using an image quality parameter corresponding to the texture ID of the object that occupies the most part in the moving image C. That is, the second correction unit 142 may process the moving image C based on one texture ID (main texture ID).

Alternatively, the second correction unit 142 may process the moving image C using image quality parameters corresponding to the texture IDs of the plurality of objects. For example, the second correction unit 142 may use an image quality parameter corresponding to the texture ID of an object that occupies the most part in each of the partial regions in FIG. In this case, the second correction unit 142 processes each area corresponding to each partial area in the moving image C.

The parameter setting unit 141 may also supply the selected image quality parameter to the first correction unit 11 described above. In this case, the first correction unit 11 can perform the same image quality correction process as that of the second correction unit 142 on the moving image A described above. In other words, each of the first correction unit 11 and the second correction unit 142 can perform processing according to the texture of the object, so that the texture can be more effectively emphasized.

In addition, the second correction unit 142 may further perform processing for enhancing the texture pattern of the object expressed in the moving image C. In this case, the second correction unit 142 may store in advance a table indicating the correspondence relationship between the texture ID and the texture pattern. The 2nd correction part 142 may give a texture pattern to an object according to texture ID. Alternatively, the second correction unit 142 may enhance the texture pattern of the object by performing a filter process.

Further, the second correction unit 142 may perform special processing such as HDR (High Dynamic Range) expansion on the object according to the texture ID. According to the HDR extension, the brightness of the object can be partially enhanced, so that a predetermined texture (eg, glossiness) of the object can be enhanced.

Further, the second correction unit 142 may perform a process of distorting the object and the area around the object according to the texture ID. Also by this processing, the texture of the object can be enhanced. As described above, the second correction unit 142 may be configured to perform an arbitrary process for enhancing the texture of an object according to the texture ID.

(effect)
The signal processing unit 10 in the display device 1 includes a texture detection unit 13 and a texture correction unit 14 (components of the above-described texture processing unit). The texture detecting unit 13 determines the texture of each object expressed in the moving image by analyzing the motion vector of the moving image. Then, the texture detection unit 13 supplies the texture correction unit 14 with the texture ID indicating the determination result.

Subsequently, the texture correction unit 14 processes the moving image based on the texture ID (according to the determination result of the texture detection unit 13). That is, the moving image can be processed by the texture correction unit 14 so as to more effectively express the texture of the object. Therefore, according to the signal processing unit 10, it is possible to improve the texture of the object expressed in the moving image.

Conventionally, in order to sufficiently express the texture of an object expressed in a moving image, it has been necessary to extremely increase the resolution of the moving image (for example, 8K level resolution). Alternatively, even when the resolution of the moving image is very high, when compressed moving image data generated by irreversible compression is provided, the moving image deteriorates during decoding in the decoding unit 61. . In this case, due to the deterioration, the expressiveness of the texture in the moving image is lowered. As described above, conventionally, there is a problem that it is not easy to effectively express the texture in the moving image.

On the other hand, according to the signal processing unit 10, even if (i) the resolution of the moving image is not necessarily high enough, or (ii) the moving image deteriorates during decoding in the decoding unit 61, The texture of the can be expressed effectively. That is, a moving image that can sufficiently express the texture of an object can be provided with a simpler configuration than in the past.

6A and 6B are diagrams for explaining the moving image processing in the texture correction unit 14, respectively. Specifically, FIGS. 6A and 6B show an object before the moving image processing and an object after the moving image processing are performed.

FIG. 6A illustrates a case where moving image processing (eg, contrast adjustment, brightness adjustment, HDR expansion) for enhancing the glossiness of an object is performed by moving image processing in the texture correction unit 14. . In the case of (a) in FIG. 6, it is understood that the motion vector is used as one of indexes indicating the glossiness of the object in the processing in the texture detection unit 13.

In FIG. 6B, when moving image processing (eg, contour correction) is performed by the moving image processing in the texture correction unit 14 to enhance the “fluffy” feeling of the object (material feeling representing lightness). Is illustrated. In the case of FIG. 6B, it is understood that the motion vector is used as one of indexes indicating the “fluffy” feeling of the object in the processing in the texture detection unit 13.

Note that the moving image processing method according to an aspect of the present disclosure can be expressed as follows. That is, the moving image processing method includes: (i) a texture determining step of determining a texture of an object represented in the moving image by analyzing a motion vector of the moving image; and (ii) a determination in the texture determining step. A moving image processing step for processing the moving image according to the result.

[Embodiment 2]
The second embodiment will be described below with reference to FIG. For convenience of explanation, members having the same functions as those described in the first embodiment are denoted by the same reference numerals and description thereof is omitted.

FIG. 7 is a functional block diagram schematically showing the configuration of the signal processing unit 20 (moving image processing apparatus) and its periphery according to the second embodiment. Note that the display device of Embodiment 2 is referred to as a display device 2. In FIG. 7, portions not shown are the same as those of the display device 1 of FIG. This also applies to each embodiment described below.

As shown in FIG. 7, the signal processing unit 20 has a configuration in which the first correction unit 11 and the interpolated image generation unit 121 are excluded from the signal processing unit 10 of the first embodiment. In the signal processing unit 20, the above-described moving image A (input moving image) is supplied from the decoding unit 61 to each of the motion vector calculation unit 122 and the texture correction unit 14.

Therefore, in the signal processing unit 20, the motion vector calculation unit 122 supplies the set of motion vectors of the moving image A to the extraction unit 131 of the texture detection unit 13 as the first motion vector set. Similar to the first embodiment, the extraction unit 131 extracts a second motion vector set from the first motion vector set. As in the first embodiment, the collation unit 132 compares the second motion vector set with various feature pattern models included in the feature pattern model DB 91. Since the specific processing of the texture detection unit 13 is the same, the description thereof is omitted.

Then, the texture correction unit 14 processes the moving image A based on the texture ID acquired from the matching unit 132 of the texture detection unit 13. That is, the texture correction unit 14 processes the moving image A to generate a moving image D (output moving image) and supplies the moving image D to the display unit 70.

As shown in the signal processing unit 20 of the second embodiment, in the moving image processing apparatus according to an aspect of the present disclosure, some of the components that are not included in the above-described texture processing unit may be omitted. For example, in the signal processing unit 20, the first correction unit 11 (image quality correction processing before the texture is determined) and the interpolated image generation unit 121 (frame) are compared with the signal processing unit 10 of the first embodiment. (Rate conversion processing) is omitted. According to the signal processing unit 20, the configuration of the moving image processing apparatus can be simplified as compared with the first embodiment.

[Embodiment 3]
The third embodiment will be described below with reference to FIG. FIG. 8 is a functional block diagram schematically showing the configuration of the signal processing unit 30 (moving image processing apparatus) and its periphery according to the third embodiment. Note that the display device of Embodiment 3 is referred to as a display device 3.

As described above, the decoding unit 61 acquires the compressed moving image data from the receiving unit 60. In the third embodiment, it is assumed that information (motion vector information) indicating a motion vector for compression is included in advance in the compressed moving image data. An example of a format of compressed moving image data including the motion vector information can be MPEG4.

As shown in FIG. 8, the signal processing unit 30 has a configuration in which the motion vector calculation unit 122 is excluded from the signal processing unit 20 of the second embodiment. That is, in the signal processing unit 30, the configuration of the moving image processing apparatus is further simplified as compared with the second embodiment described above.

In the signal processing unit 30, the moving image A (input moving image) is supplied from the decoding unit 61 to the texture correction unit 14. In the texture detection unit 13 of the third embodiment, the extraction unit 131 acquires motion vector information included in the above-described compressed moving image data from the decoding unit 61.

Specifically, the extraction unit 131 acquires a set of motion vectors indicated in the motion vector information as a first motion vector set. Then, the extraction unit 131 extracts a second vector set from the motion vector information (first motion vector set). And the collation part 132 compares a 2nd motion vector set with the various feature pattern models contained in feature pattern model DB91 similarly to each above-mentioned embodiment.

As described above, the texture detection unit 13 in the third embodiment performs the same processing as in each of the above-described embodiments using the motion vector information included in the compressed moving image data. That is, the texture detection unit 13 according to the third embodiment analyzes a motion vector included in advance in the compressed moving image data.

As described above, when the motion vector information is included in the compressed motion image data, the motion image processing apparatus according to an aspect of the present disclosure can omit the process of calculating the motion vector. Therefore, the configuration of the moving image processing apparatus is further simplified.

[Modification]
A modification of the present disclosure will be described with reference to FIG. FIG. 9 is a functional block diagram schematically showing the configuration of the signal processing unit 30v (moving image processing apparatus) and its periphery according to the present modification. Note that the display device of this modification is referred to as a display device 3v.

In FIG. 9, a signal processing unit 30v as an example of a variation of the signal processing unit 30 of the third embodiment is illustrated for convenience of explanation. However, the configuration of the present modification may be applied to the above-described first and second embodiments and the later-described fourth embodiment.

As shown in FIG. 9, auxiliary information may be further input to the texture detection unit 13 (more specifically, each of the extraction unit 131 and the collation unit 132). The auxiliary information means information other than the motion vector included in the moving image. As an example of the auxiliary information, information indicating an object boundary, a color, a texture pattern, and the like can be given.

In the signal processing unit 30v, the texture detection unit 13 may further determine the texture of the object expressed in the moving image by further using auxiliary information in addition to the motion vector information (motion vector). According to this configuration, the texture can be determined by further considering the shape and color of the object indicated by the auxiliary information, and it is expected that the texture can be determined with higher accuracy.

[Embodiment 4]
Embodiment 4 will be described below with reference to FIG. FIG. 10 is a functional block diagram schematically illustrating the configuration of the signal processing unit 40 (moving image processing apparatus) and its periphery according to the fourth embodiment. Note that the display device of Embodiment 4 is referred to as a display device 4.

In addition, in FIG. 10, the signal processing part 40 as an example of the variation of the signal processing part 30 of Embodiment 3 is illustrated for convenience of explanation. However, the configuration of the fourth embodiment may be applied to any of the above-described embodiments and modifications.

In each of the above-described embodiments, the case where the texture of an object is determined using a motion vector calculated for one frame (eg, a reference frame) included in a moving image has been exemplified. For example, when the frame (current frame) whose texture is to be determined is the Nth frame (N is an integer equal to or greater than 2), the texture of the object is determined using only the motion vector in the Nth frame. It was.

However, the total from the “motion vector in the first frame (the first frame of the moving image)” to the “motion vector in the (N−1) th frame (the frame immediately before the current frame)” (N− 1) The texture of an object may be determined by further using a single motion vector. That is, the texture of the object may be determined using the motion vector history in the past frame.

As shown in FIG. 10, the fourth embodiment exemplifies a configuration in which a motion vector history 92 described below is added to the above-described texture processing unit. As an example, the signal processing unit 40 of the fourth embodiment shows a configuration in which the motion vector history 92 is further stored in the storage unit 90 in the signal processing unit 30 of the third embodiment.

The motion vector history 92 stores a set of motion vectors in the first to N−1th frames. That is, the motion vector history 92 stores a history of a set of motion vectors (first motion vector set) in a past frame.

In the fourth embodiment, the decoding unit 61 stores motion vector information indicating the set of motion vectors described above in the motion vector history 92 for each frame of a moving image. That is, the motion vector history 92 stores motion vector information (hereinafter also referred to as “past motion vector information”) indicating a set of motion vectors in the first to (N−1) th frames.

In the texture detection unit 13 of the fourth embodiment, the extraction unit 131 acquires the first motion vector set of the current (Nth) frame as in the third embodiment. Specifically, the extraction unit 131 acquires, from the decoding unit 61, motion vector information indicating a set of motion vectors in the current frame as a first motion vector set.

In addition, the extraction unit 131 further acquires past motion vector information included in the motion vector history 92. According to this configuration, the texture detection unit 13 can further determine the texture of the object by further using the motion vector history in the past frame in addition to the motion vector in the current frame.

As described above, the main elements that characterize the texture of an object include the pattern of movement of the object or the pattern of movement of reflected light. Therefore, by paying attention to the motion vector history in the past frame, the temporal transition of each pattern can be further considered. Therefore, it is expected that the texture can be discriminated with higher accuracy.

[Example of software implementation]
The control blocks (particularly the signal processing units 10 to 40) of the display devices 1 to 4 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or a CPU (Central Processing Unit). It may be realized by software using

In the latter case, the display devices 1 to 4 include a CPU that executes instructions of a program that is software that realizes each function, and a ROM (Read Only Memory) in which the program and various data are recorded so as to be readable by the computer (or CPU). ) Or a storage device (these are referred to as “recording media”), a RAM (Random Access Memory) for expanding the program, and the like. And the objective of 1 aspect of this indication is achieved when a computer (or CPU) reads and runs the said program from the said recording medium. As the recording medium, a “non-temporary tangible medium” such as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one aspect of the present disclosure can also be realized in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[Summary]
A moving image processing apparatus (signal processing unit 10) according to aspect 1 of the present disclosure includes a texture determination unit (13) that determines a texture of an object represented in the moving image by analyzing a motion vector of the moving image. A moving image processing unit (texture correcting unit 14) that processes the moving image according to the determination result of the texture determining unit.

As described above, the inventor of the present application has proposed a novel technique of “determining the texture of an object expressed in a moving image based on the motion vector of the moving image and performing moving image processing according to the determination result”. I found a new technical idea. The above configuration has been conceived based on the technical idea.

According to this configuration, the texture determination unit can determine the texture of the object expressed in the moving image by analyzing the motion vector of the moving image. Then, the moving image processing unit performs moving image processing according to the determination result. That is, it is possible to cause the moving image processing unit to perform moving image processing so as to more effectively express the texture of the object. Therefore, it is possible to enhance the texture of the object expressed in the moving image.

In the moving image processing apparatus according to aspect 2 of the present disclosure, a feature pattern model that is a model representing the motion vector representing the texture or a model related to the motion vector representing the texture in the aspect 1 is set in advance. The texture determination unit preferably determines the texture by comparing the motion vector of the moving image with the feature pattern model.

According to the above configuration, it is possible to determine the texture based on a preset feature pattern model.

In the moving image processing apparatus according to aspect 3 of the present disclosure, in the aspect 2, the texture determination unit compares the motion vector of the moving image with the feature pattern model, thereby moving the moving image with respect to the feature pattern model. Material discrimination information indicating the degree of coincidence of the motion vectors of the image may be generated as the discrimination result.

According to the above configuration, it is possible to evaluate the texture based on the degree of coincidence (texture discrimination information) of the motion vector of the moving image with respect to the feature pattern model.

The moving image processing apparatus according to Aspect 4 of the present disclosure is any one of Aspects 1 to 3, wherein the texture determination unit is included in advance in the compressed moving image data that is data obtained by compressing the moving image. It is preferable to analyze the motion vector.

According to the above configuration, since the process for calculating the motion vector is not required in the moving image processing apparatus, the configuration of the moving image processing apparatus can be simplified.

The display device according to aspect 5 of the present disclosure preferably includes the moving image processing apparatus according to any one of aspects 1 to 4.

According to the above configuration, the same effect as the moving image processing apparatus according to an aspect of the present disclosure is obtained.

In the moving image processing method according to the aspect 6 of the present disclosure, the motion vector of the moving image is analyzed to determine the texture of the object represented in the moving image, and the determination result in the texture determining step. And a moving image processing step for processing the moving image accordingly.

The moving image processing apparatus according to each aspect of the present disclosure may be realized by a computer. In this case, the moving image processing apparatus is operated by causing the computer to operate as each unit (software element) included in the moving image processing apparatus. A control program for a moving image processing apparatus for realizing the above in a computer and a computer-readable recording medium on which the control program is recorded also fall within the category of one aspect of the present disclosure.

[Additional Notes]
One aspect of the present disclosure is not limited to the above-described embodiments, and various modifications can be made within the scope shown in the claims, and the technical means disclosed in different embodiments can be appropriately combined. Embodiments to be included are also included in the technical scope of one aspect of the present disclosure. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

(Cross-reference of related applications)
This application claims the benefit of priority over Japanese patent application: Japanese Patent Application No. 2016-256284 filed on Dec. 28, 2016. Included in this document.

1, 2, 3, 3v, 4

Display device

10, 20, 30, 30v, 40 Signal processing unit (moving image processing device)
13 Texture detection unit (material discrimination unit)
14 Texture correction unit (moving image processing unit)

Claims

A texture discriminating unit that discriminates the texture of an object represented in the moving image by analyzing a motion vector of the moving image;
A moving image processing apparatus comprising: a moving image processing unit that processes the moving image according to a determination result of the texture determining unit.
A feature pattern model, which is a model representing a motion vector representing the texture or a model related to the motion vector representing the texture, is preset,
The moving image processing apparatus according to claim 1, wherein the texture determining unit determines the texture by comparing the motion vector of the moving image with the feature pattern model.
The texture discriminating unit collates the motion vector of the moving image with the feature pattern model,
The moving image processing apparatus according to claim 2, wherein texture discrimination information indicating a degree of coincidence of the motion vector of the moving image with respect to the feature pattern model is generated as the discrimination result.
The moving image according to any one of claims 1 to 3, wherein the texture determination unit analyzes a motion vector included in advance in compressed moving image data, which is data obtained by compressing the moving image. Image processing device.
A display device comprising the moving image processing device according to any one of claims 1 to 4.
A texture determination step of determining a texture of an object represented in the moving image by analyzing a motion vector of the moving image;
A moving image processing method comprising: a moving image processing step for processing the moving image in accordance with a determination result in the texture determination step.
A control program for causing a computer to function as the moving image processing apparatus according to claim 1, wherein the control program causes the computer to function as the texture determination unit and the moving image processing unit.