KR102716679B1 - Method and device for providing customized cultural and art contents curation and for recommending mate playing the contents with user - Google Patents

Abstract

A method and device for providing customized cultural and artistic content curation and recommending a mate to view with the user are disclosed. A content providing server for providing customized cultural and artistic content curation and recommending a mate to view with the user includes: a user profile management unit for receiving user information corresponding to each of users of a plurality of user terminals, and generating and managing user profile information including the user information and usage history information of the users using cultural and artistic content; a cultural and artistic content management unit for generating and managing content information corresponding to each of a plurality of registered cultural and artistic contents; a content recommendation unit for determining a predicted preference of a first user for the cultural and artistic contents based on evaluation information corresponding to each of the cultural and artistic contents included in the content information, selecting a user-tailored content candidate group based on the predicted preference, and selecting a user-tailored recommended content for the first user from the user-tailored content candidate group; and a mate selection unit for selecting a mate to enjoy with the user-tailored recommended content from among the users.

Description

{METHOD AND DEVICE FOR PROVIDING CUSTOMIZED CULTURAL AND ART CONTENTS CURATION AND FOR RECOMMENDING MATE PLAYING THE CONTENTS WITH USER}

The present invention relates to a method and device for providing cultural and artistic content, and more specifically, to a method and device for providing customized cultural and artistic content curation and recommending mates to view together with the user.

As the cultural arts industry develops, various cultural arts content, including movies, dramas, plays, musicals, and art exhibitions, is becoming increasingly diverse. Therefore, it is becoming an important issue for users to choose the content they want to see and appreciate among the various cultural arts content.

In addition, since these cultural and artistic contents can be downloaded and viewed online or reserved online as various communication terminals including smartphones become widespread, there is an advantage in that users can easily obtain various information about each cultural and artistic content.

However, since users have no choice but to set search terms within the scope of their prior knowledge in order to individually check and search for various pieces of information on each piece of cultural and artistic content, it is difficult for users to select and receive cultural and artistic content based on information they are not exposed to or aware of.

To solve these problems, conventional content providers select and provide users with cultural and artistic content that is popular due to its high real-time views, even if the user did not directly enter a search term, or select and provide users with content similar to cultural and artistic content frequently used by the user.

However, since these conventional methods only provide similar content by simply referring to the usage history of a specific user or provide specific content by considering the frequency of use of the content itself, the user's feelings at the time of receiving the content are not considered at all.

In addition, since cultural and artistic content tends to be used by multiple people, a method is needed to enjoy richer cultural and artistic content by providing other users with whom one can enjoy cultural and artistic content together.

The purpose of the present invention to solve the above problems is to provide a method and device for providing customized cultural and artistic content curation and recommending a mate to view with the user.

One aspect of the present invention for achieving the above purpose is to provide a content provision server that provides customized cultural and artistic content curation and recommends mates to view together with the user.

The above content providing server includes a user profile management unit which receives user information corresponding to each user of a plurality of user terminals, and generates and manages user profile information including the user information and usage history information of the users using cultural and artistic content; a cultural and artistic content management unit which generates and manages content information corresponding to each of a plurality of registered cultural and artistic contents; a content recommendation unit which determines a predicted preference of a first user for the cultural and artistic contents based on evaluation information corresponding to each of the cultural and artistic contents included in the content information, selects a user-tailored content candidate group based on the predicted preference, and selects user-tailored recommended content for the first user from the user-tailored content candidate group; and a mate selection unit which selects a mate to enjoy the user-tailored recommended content together with among the users.

The above content information includes an appreciation tag composed of at least one of the atmosphere, genre, material, and award history for each of the above cultural and artistic contents, and a recommended emotion indicating the emotional state of a user who viewed each of the above cultural and artistic contents.

The above content recommendation unit searches for the recommended emotion corresponding to the user emotion indicating the current emotional state of the first user, and selects cultural and artistic content corresponding to the searched recommended emotion as the user-tailored recommended content from the user-tailored content candidate group.

The above content providing server further includes a user emotion tracking unit that receives a voice signal recorded during the most recent preset time interval from a user terminal of the first user and determines the user emotion of the first user based on the voice signal.

The above user emotion tracking unit obtains a spectrogram from the voice signal and determines the first emotion by analyzing the spectrogram using a convolutional neural network (CNN).

The above user emotion tracking unit converts the voice signal into STT (Speech to Text) to obtain a voice text corresponding to the voice signal, and analyzes the words constituting the voice text using a recurrent neural network (RNN) to determine a second emotion.

The above user emotion tracking unit determines the user emotion based on the first emotion and the second emotion.

The above evaluation information includes individual ratings according to multiple evaluation items for each of the above cultural and artistic contents and a comprehensive rating for the individual ratings.

The above content recommendation unit calculates the proximity between the cultural and artistic contents based on the comprehensive ratings left by the users for each of the cultural and artistic contents, and determines the predicted preference based on the calculated proximity.

When using the method and device for providing customized cultural and artistic content curation according to the present invention as described above and recommending a mate to view with the user, it is possible to predict the preference that a specific user will have for cultural and artistic content based on the evaluation information left by each user for cultural and artistic content, and thus select the most suitable cultural and artistic content.

In addition, since the recommended feelings of users who have viewed or experienced cultural and artistic content are collected and managed for each cultural and artistic content, cultural and artistic content can be selected by considering the feelings of users who will experience the cultural and artistic content.

In addition, it has the advantage of tracking the user's current emotional state based on the user's voice signal and providing cultural and artistic content that is appropriate for the tracked current emotional state as customized cultural and artistic content.

Figure 1 is a schematic diagram showing an environment in which a method for providing user-customized cultural and artistic content according to one embodiment is performed.
Figure 2 is a block diagram showing the functional configuration of a content provision server according to Figure 1.
Figure 3 is a diagram showing information managed by the user profile management unit according to Figure 2.
Figure 4 is a diagram showing information managed by the cultural and artistic content management department according to Figure 2.
Fig. 5 is an example diagram for explaining the operation of the content recommendation unit according to Fig. 2.
Figure 6 is a drawing showing tags of cultural and artistic content managed by the cultural and artistic content management department according to Figure 2.
Fig. 7 is a drawing for explaining the operation of the user emotion tracking unit according to Fig. 2.
Fig. 8 is a diagram exemplarily showing the structure of a convolutional neural network according to Fig. 7.
Figure 9 is a drawing for explaining the operation of the mate selection unit according to Figure 2.
FIG. 10 is a drawing exemplarily showing the hardware configuration of a content provision server according to FIG. 1.

The present invention can have various modifications and various embodiments, and specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include all modifications, equivalents, or substitutes included in the spirit and technical scope of the present invention. In describing each drawing, similar reference numerals are used for similar components.

The terms first, second, A, B, etc. may be used to describe various components, but the components should not be limited by the terms. The terms are only used to distinguish one component from another. For example, without departing from the scope of the present invention, the first component may be referred to as the second component, and similarly, the second component may also be referred to as the first component. The term and/or includes any combination of a plurality of related described items or any item among a plurality of related described items.

When it is said that a component is "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components in between. On the other hand, when it is said that a component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

The terminology used in this application is only used to describe specific embodiments and is not intended to limit the present invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, it should be understood that the terms "comprises" or "has" and the like are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but do not exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms defined in commonly used dictionaries, such as those defined in common dictionaries, should be interpreted as having a meaning consistent with the meaning they have in the context of the relevant art, and will not be interpreted in an idealized or overly formal sense unless expressly defined in this application.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a schematic diagram showing an environment in which a method for providing user-customized cultural and artistic content according to one embodiment is performed.

Referring to FIG. 1, a method for providing customized cultural and artistic content can be performed by a content providing server (100).

The content provision server (100) can register a number of cultural and artistic contents, select user-customized cultural and artistic contents from among the registered cultural and artistic contents, and provide them to the user terminal (200).

Here, cultural and artistic content may include movies, dramas, performances, exhibitions, classes, and experiential activities. Specifically, cultural and artistic content may refer to a ticketing service that allows online or offline viewing of movies, dramas, performances, and exhibitions, a class reservation service that allows online or offline taking classes such as pottery or leather workshops, and an experiential activity reservation service that allows travel tours or experiential activities to specific regions.

The content provision server (100) can collect user information from the user terminal (200) and manage the user profile based on the collected user information. In addition, the content provision server (100) can store content information for each registered cultural and artistic content. Here, the content information can include tag information that is provided by a user who has viewed or experienced each cultural and artistic content, or is provided by input from an administrator.

The content provision server (100) can select user-customized content based on tag information individually assigned to each cultural and artistic content, and provide the selected user-customized content to the user terminal (200). Here, the user-customized content can be provided to the user terminal (200) in the form of multiple lists.

Specifically, the content provision server (100) can predict the content preference of a user of a specific user terminal (200) based on the user's evaluation information for each cultural and artistic content, and select a plurality of user-customized content candidates according to the predicted content preference.

At this time, the content provision server (100) can determine the user emotion representing the user's emotional state at the current point in time by continuously tracking the emotional state of the user of a specific user terminal (200).

The content provision server (100) can select at least one user-customized content from cultural and artistic contents included in the user-customized content candidate group based on the determined user sentiment.

That is, the content provision server (100) according to one embodiment of the present invention predicts a specific user's predicted preference for cultural and artistic content based on the user's evaluation information, selects a user-customized content candidate group, and then selects at least one user-customized content based on the user's emotion among the cultural and artistic content included in the user-customized content candidate group.

Accordingly, the content provision server (100) provides the user terminal (200) with cultural and artistic content suitable for viewing or enjoying in the user's current emotional state as customized content, as well as the user's content preference, so there is an advantage in that it is possible to recommend customized content that simultaneously takes into account the user's emotional state and preference.

In addition, the content providing server (100) can compare users with each other based on evaluation information for each cultural and artistic content, and select a suitable mate to enjoy the same cultural and artistic content together based on the comparison result, and recommend the mate to each other. For example, the content providing server (100) can compare the first user with the remaining other users based on the first user's evaluation information for each cultural and artistic content, and select a mate to enjoy or view a specific cultural and artistic content together with the first user from among the remaining other users based on the comparison result, and recommend the mate to the first user.

The user terminal (200) can access the content provision server (100) and provide user information to the content provision server (100), and can make reservations or purchases for a number of cultural and artistic contents registered in the content provision server (100), or leave reviews after viewing the contents.

At this time, the user terminal (200) can receive recommendations for at least one customized cultural and artistic content from the content providing server (100). In addition, the user terminal (200) can receive recommendations for at least one mate from the content providing server (100).

Most of the users of the user terminal (200) may be consumers who consume and receive cultural and artistic content, but some of the users of the user terminal (200) may include artists who directly plan or produce cultural and artistic content.

Fig. 2 is a block diagram showing the functional configuration of the content provision server according to Fig. 1. Fig. 3 is a diagram showing information managed by the user profile management unit according to Fig. 2. Fig. 4 is a diagram showing information managed by the cultural and artistic content management unit according to Fig. 2.

Referring to FIGS. 2 and 3, the user profile management unit (101) can receive user information corresponding to each user of the user terminal (200), and create and manage user profile information including the received user information and usage history information using cultural and artistic content.

Specifically, the user information may include the gender, age, nickname, points, balance, etc. of each user of the user terminal (200). In addition, the usage record information may include the name of the performance used by the user of the user terminal (200), the type of ticket for the performance used, the date of the performance used, the number of mates who watched the cultural and artistic content together, the nickname, etc.

Meanwhile, users managed by the user profile management unit (101) may include not only users who participate in cultural and artistic content as participants, but also artists who actually compose and create cultural and artistic content. For example, artists may include performers in movies, plays, performances, etc., and teachers who plan and conduct classes or experiential activities.

Accordingly, the user profile management unit (101) can additionally create and store artist information. The artist information can include information corresponding to the user information of the user profile information, such as the artist's name, pen name, nationality, phone number, email, address, homepage address, gender, etc.

Referring to FIGS. 2 and 4, the cultural arts content management department (102) can create and manage content information for each of a plurality of registered cultural arts contents. For example, the content information can include the name of the cultural arts contents, performance area, type, genre, performance period, performance venue, audience rating, host name, performer name, representative name, etc.

In addition, the cultural arts content management department (102) can additionally create and manage art space information for each cultural arts content in addition to the content information. The art space information is information about the place where the cultural arts content is performed or opened, and can include the name of the place, the classification of the place, the address, contact information for the place, and email or web pages for the place.

Accordingly, the cultural arts content management department (102) can recommend geographically advantageous cultural arts content to the user terminal (200) using the art space information.

For example, the cultural arts content management department (102) may select at least one cultural arts content corresponding to the current location information and geographically close art space information based on the current location information of the user terminal (200) and provide the selected cultural arts content to the user terminal (200).

In addition, the cultural arts content management unit (102) provides art space information on all cultural arts content to the user terminal (200) using a geographical map, measures the total time that the user terminal (200) explores the geographical map, and the time that a specific range area of the geographical map is displayed (i.e., the time that the user's gaze is maintained), and if the time that the specific range area is displayed compared to the time that the geographical map is explored is greater than a preset ratio, the specific range area can be determined as an area of interest that the user of the user terminal (200) is interested in.

Accordingly, when recommending customized cultural and artistic content, the cultural and artistic content management department (102) can provide cultural and artistic content within the area of interest to the user terminal (200) as customized cultural and artistic content.

Fig. 5 is an example diagram for explaining the operation of the content recommendation unit according to Fig. 2.

The content information managed by the cultural arts content management department (102) may further include evaluation information corresponding to each cultural arts content. Here, the evaluation information may include ratings, reviews, etc. left by user terminals (200) who directly viewed or experienced the cultural arts content. Here, the rating may include individual ratings for each of a number of evaluation items (e.g., interest, probability, etc.) and a comprehensive rating expressed as an average or median of the individual ratings.

A content recommendation unit (104) according to one embodiment of the present invention calculates proximity between cultural and artistic contents based on comprehensive ratings left by users of multiple user terminals (200) for each cultural and artistic content, and determines a preference that a user of a specific user terminal (200) will have for a specific cultural and artistic content using the calculated proximity.

Specifically, the content recommendation unit (104) can calculate the proximity between cultural and artistic contents based on the following mathematical expression 1.

Referring to the above mathematical expression 1, w _i,j is the proximity between cultural and artistic content i and cultural and artistic content j, U is a set of all users of user terminals (200) who have left comprehensive ratings on both cultural and artistic content i and cultural and artistic content j, r _u,i is a comprehensive rating left by user u included in set U for cultural and artistic content i, and r _u,j is a comprehensive rating left by user u included in set U for cultural and artistic content i. is the average of the overall ratings left by all users on cultural and artistic content i, is the average of the comprehensive ratings left by all users for cultural and artistic content j.

Therefore, using mathematical expression 1, the proximity between two cultural and artistic contents can be calculated. The proximity according to mathematical expression 1 can be a value between -1 and +1.

The content recommendation unit (104) can use the calculated proximity to calculate a predicted preference that predicts the degree to which a user of a specific user terminal (200) prefers each cultural and artistic content.

For example, the content recommendation unit (104) can calculate the predicted preference for each cultural and artistic content according to the following mathematical expression 2.

In mathematical expression 2, P _a,i is a predicted preference that predicts the degree to which a user a of a specific user terminal (200) prefers cultural and artistic content i, N is a set of cultural and artistic contents excluding cultural and artistic content i, w _i,n is a proximity between cultural and artistic content i and cultural and artistic content n included in the set N, and r _a,n may be a comprehensive rating left by user a for cultural and artistic content n.

The content recommendation unit (104) can select cultural and artistic contents within a preset number in order of high predicted preference, and can compose a user-customized content candidate group with the selected cultural and artistic contents.

For example, referring to FIG. 5, the content recommendation unit (104) may determine that if a third user (203) to whom customized content is to be recommended has given a high overall rating to the first cultural and artistic content (C1) and the proximity between the first cultural and artistic content (C1) and the second cultural and artistic content (C2) is high, the predicted preference of the third user (203) for the second cultural and artistic content (C2) may be high. Accordingly, in this case, the content recommendation unit (104) may include the second cultural and artistic content (C2) in the candidate group of customized content.

Accordingly, the third user (203) can receive the second cultural and artistic content (C2) included in the user-customized content candidate group as user-customized content.

Meanwhile, the content recommendation unit (104) may select user-customized content based on the user's emotions, which indicate the user's current emotional state, from among the cultural and artistic contents included in the user-customized content candidate group, and provide the selected user-customized content to the user terminal (200), thereby providing content that takes the user's current emotional state into maximum consideration.

Figure 6 is a drawing showing tags of cultural and artistic content managed by the cultural and artistic content management department according to Figure 2.

The cultural and artistic content management department (102) can collect appreciation tags corresponding to each cultural and artistic content as content information. Here, the appreciation tags can be various texts in which users record their impressions or information regarding each cultural and artistic content. Specifically, for example, the appreciation tags can include the atmosphere, genre, evaluation, material, award history, etc. corresponding to the cultural and artistic content.

Appreciation tags can be collected from a user terminal (200) of a user who directly experienced or viewed cultural and artistic content, and at least some of the appreciation tags can be collected by being input by an administrator of a content providing server (100).

The cultural and artistic content management unit (102) can receive a search word from a user terminal (200), search for an appreciation tag corresponding to the received search word, and provide cultural and artistic content corresponding to the searched appreciation tag to the user terminal (200). In other words, the appreciation tag can be used as an index for cultural and artistic content.

Meanwhile, in order for the content recommendation unit (104) to select customized cultural and artistic content according to the user's current emotional state, the appreciation tag may include a recommended emotion corresponding to each cultural and artistic content. Here, the recommended emotion may be a text indicating the emotional state of a user who directly experienced or viewed the cultural and artistic content.

For example, the recommended emotion can be expressed as 'when you want to love' for cultural and artistic content that is suitable to enjoy when the user's emotion is to love, and 'when you want to laugh' for cultural and artistic content that is suitable to enjoy when the user's emotion is to laugh.

It is most desirable that the recommended emotions be collected only from the user terminal (200) of a user who directly experienced or viewed cultural and artistic content, but, like the appreciation tag, at least a portion of the recommended emotions may be collected by being input by the administrator of the content providing server (100).

The content recommendation unit (104) according to one embodiment of the present invention actively uses the recommended emotion to search for the recommended emotion corresponding to the user emotion indicating the current emotional state of the user, and selects the cultural and artistic content corresponding to the searched recommended emotion as a user-customized cultural and artistic content from a group of user-customized content candidates and provides it to the user terminal (200). At this time, the content recommendation unit (104) can receive the user emotion determined by the user emotion tracking unit (103) from the user emotion tracking unit (103).

Figure 7 is a drawing for explaining the operation of the user emotion tracking unit according to Figure 2.

The user emotions indicating the user's current emotional state may include anger, love, happiness, difficulty, thrill, laughter, etc. In order to determine the user's current emotional state in various ways, the user emotion tracking unit (103) may receive a voice signal recorded during the most recent preset time interval from the user terminal (200). In order to record the voice signal, a predetermined application may be installed in advance on the user terminal (200).

People express their current emotional state in various forms, but their emotional state is often particularly revealed through their voice. Therefore, the user emotion tracking unit (103) according to one embodiment of the present invention determines the user's emotion based on this voice signal.

Specifically, the user emotion tracking unit (103) can obtain a spectrogram using a voice signal received from the user terminal (200). The spectrogram is a tool that can visualize a voice signal in terms of frequency, amplitude, and time, with the time axis as the horizontal (x) axis, the frequency of the voice signal as the vertical (y) axis, and the amplitude size of the voice signal as a color value within a preset range at points corresponding to the horizontal and vertical axes.

For example, the user emotion tracking unit (103) can determine the number of window elements of a fast Fourier transform (FFT) according to the following mathematical expression 3 using the sampling rate of the voice signal.

In mathematical expression 3, sr is a sampling rate, and W is a window size of a fast Fourier transform (FFT), which can be a preset value.

The user emotion tracking unit (103) can obtain a normalized Hanning window by applying a Hanning window to the number of window elements and then normalizing the window.

The user emotion tracking unit (103) can obtain a spectrogram (SPT) according to the following mathematical expression 4 by using the result of performing a fast Fourier transform (FFT) on a voice signal truncated by a normalized Hanning window, a sampling rate, and a preset hop length for the voice signal.

In mathematical expression 4, VC is a speech signal, trunc(VC) is a speech signal truncated with a preset hop length, FFT is a function representing a fast Fourier transform, Wnh is a normalized Hanning window, sr is a sampling rate, and T may be a time interval of the truncated speech signal.

Meanwhile, when performing voice signal analysis through an artificial neural network from a spectrogram (SPT), a learning method can be used in which the MFCC (Mel-Frequency Cepstral Coefficient) corresponding to the feature vector is extracted from the result value of the spectrogram according to mathematical expression 4, and the extracted MFCC, etc. are input into the artificial neural network.

However, in this case, in addition to MFCC according to the spectrogram, feature vectors such as ZCR (zero crossing rete) are extracted and trained on the artificial neural network, and the feature learning model in the artificial neural network changes depending on these various feature vectors.

Meanwhile, convolutional neural networks (CNNs) have been studied extensively in classifying input images into classes, unlike the method of learning the aforementioned feature vectors as input values, and thus have high performance in classifying input images.

The present invention proposes a method of visualizing a spectrogram and learning and analyzing it through a convolutional neural network (CNN) as a means to solve the classification performance advantage of such a convolutional neural network and the problem of setting a feature vector according to a speech signal.

Specifically, the spectrogram (SPT) can be visualized as shown in Fig. 6 by displaying the time of the voice signal as the x-axis, the frequency of the voice signal as the y-axis, and the amplitude size of the voice signal as color values within a preset range at points corresponding to the x-axis and y-axis.

The user emotion tracking unit (103) divides an area defined by the x-axis and y-axis in the spectrogram visualized in this manner into blocks of a preset size, converts (or quantizes) a plurality of color values displayed in each of the divided blocks into pixel values within a predefined pixel range (for example, values between 0 and 255 when corresponding to 8-bit pixel values), and determines a pixel value corresponding to each of the blocks as an average value or median value of the converted plurality of pixel values, thereby generating an input image corresponding to the spectrogram.

The input image generated in this way may lose some of the detailed features of the voice signal because it simplifies some of the information by blocking and quantizing the visualized spectrogram. However, it can express the current emotional state of the voice signal very well because it contains more information than using a feature vector.

The user emotion tracking unit (103) inputs an input image corresponding to a spectrogram into a pre-supervised learning convolutional neural network (CNN), and can obtain a first emotion representing the user's current emotional state as an output of the convolutional neural network. To this end, the convolutional neural network (CNN) can be pre-learned using input images converted into images of spectrograms corresponding to each of a plurality of user emotions as training data.

Meanwhile, the user emotion tracking unit (103) according to one embodiment of the present invention can additionally obtain a second emotion representing the user's emotional state revealed from the words themselves that make up the text spoken by the user, as well as a first emotion representing the user's emotional state inherent in the user's voice signal.

Specifically, the user emotion tracking unit (103) can obtain voice text corresponding to a voice signal by converting the voice signal into STT (Speech to Text).

Next, the user emotion tracking unit (103) obtains word vectors corresponding to each word composing the acquired voice text using various vector conversion algorithms (word2Vec), inputs the obtained word vectors into a recurrent neural network (RNN), and obtains a second emotion as the output of the recurrent neural network.

A recurrent neural network (RNN) is an artificial neural network (ANN) that learns data that changes over time. Since spoken text is spoken sequentially according to the temporal flow of the speech signal, it is suitable for analyzing spoken text according to such temporal flow. A recurrent neural network (RNN) can be pre-trained to analyze each word in temporal order to determine the user's emotion (secondary emotion). For example, a recurrent neural network (RNN) can be implemented as a recurrent neural network using a long short-term memory (LSTM) method, or can be configured with various other known recurrent neural network structures.

The user emotion tracking unit (103) can determine the user emotion using the first emotion and the second emotion.

Specifically, the user emotion tracking unit (103) can preset and store selection information for selecting one of the first emotion and the second emotion for each of a plurality of user emotions.

For example, some of the user emotions, such as anger and love, may be more accurately identified through the spectrogram of the voice signal, and the rest of the user emotions may be better expressed through the voice text. Specifically, 'love' may be configured as selection information so that it is selected when it is output as the first emotion through the CNN rather than when it is output as the second emotion through the RNN, and 'anger' may be configured as selection information so that it is selected when it is output as the second emotion through the RNN.

Accordingly, considering these points, selection information is entered and stored in advance by the administrator, and the user emotion tracking unit (103) can select one of the first and second emotions based on this selection information and determine it as the user emotion.

Fig. 8 is a diagram exemplarily showing the structure of a convolutional neural network according to Fig. 7.

Referring to FIG. 8, a convolutional neural network (10) according to one embodiment of the present invention may include a convolutional layer (11) that receives an input image corresponding to a spectrogram of a preset size and extracts a feature map, an activation layer (12) that determines whether to activate an output using an activation function for the extracted feature, a pooling layer (13) that performs sampling on the output according to the activation layer (12), a fully connected layer (14) that performs classification according to a class, and an output layer (15) that finally outputs the output according to the fully connected layer (14).

The convolutional layer (11) may be a layer that extracts features of input data by convolving the input image and the filter with each other. Here, the filter is a function that detects a characteristic part of the input image, and may be a function that is generally expressed as a matrix and is determined by continuously learning by learning data. The features extracted by the convolutional layer (11) may be referred to as a feature map. In addition, the interval value for performing the convolution may be referred to as a stride, and feature maps of different sizes may be extracted depending on the stride value. At this time, if the size of the filter is smaller than that of the input image, the feature map has a smaller size than the existing input image, and a padding process may be additionally performed to prevent the feature from being lost through several steps. At this time, the padding process may be a process of maintaining the size of the input image and the size of the feature map the same by adding a preset value (for example, 0 or 1) to the periphery of the generated feature map.

Here, the convolutional layer (11) according to one embodiment of the present invention can use a structure in which a 1×1 convolutional layer and a 3×3 convolutional layer are sequentially and repeatedly connected.

The activation layer (12) is a layer that changes a feature extracted as a value (or matrix) into a nonlinear value according to an activation function to determine whether to activate it. A sigmoid function, a ReLU function, a softmax function, etc. can be used as an activation function. For example, the softmax function can be a function that normalizes all input values to values between 0 and 1 and has the characteristic that the sum of the output values is always 1.

The pooling layer (130) is a layer that selects a feature representing the feature map by performing subsampling or pooling on the output of the activation layer (12). Max pooling, which extracts the largest value for a certain area of the feature map, and average pooling, which extracts the average value, may be performed. At this time, the pooling layer is not necessarily performed after the activation function, but may be performed selectively.

In addition, the convolutional neural network (10) here may include multiple connection structures of a convolutional layer (11), an activation layer (12), and a pooling layer (13).

Fig. 9 is a drawing for explaining the operation of the mate selection unit according to Fig. 2.

Referring to Figure 9, the mate selection unit (105) can select a mate with whom to enjoy customized recommended content and provide the same to the user.

Specifically, the mate selection unit (105) can select n cultural and artistic contents for which user A and user B have left evaluation information in common. Here, the number of n cultural and artistic contents can be the same as the number of evaluation items of individual ratings corresponding to one cultural and artistic content.

Next, the mate selection unit (105) can determine evaluation item vectors (a1 to an) corresponding to the evaluation information left by user A for each of n cultural and artistic contents, and generate a first user matrix (MatA) having the determined evaluation item vectors (a1 to an) as columns.

In the same manner, the mate selection unit (105) can determine evaluation item vectors (b1 to bn) corresponding to the evaluation information left by user B for each of n cultural and artistic contents, and generate a second user matrix (MatB) having the determined evaluation item vectors (b1 to bn) as columns.

Here, each of the evaluation item vectors may include, as an element, each of the individual ratings corresponding to the cultural and artistic content. That is, the first user matrix (MatA) and the second user matrix (MatB) may be square matrices having a size of n×n.

Next, the mate selection unit (105) can determine the mate distance between users A and B using the first eigenvector (Aev) produced from the first user matrix (MatA) and the second eigenvector (Bev) produced from the second user matrix (MatB). At this time, if an eigenvector does not exist, the columns constituting each of the user matrices (MatA, MatB) can be exchanged to derive an eigenvector.

Specifically, the mate selection unit (105) can determine the mate distance (d(Aev, Bev)) using the first eigenvector (Aev) and the second eigenvector (Bev) according to the following mathematical expression 5.

According to mathematical expression 5, the closer the mate distance is to 0, the more similar the content evaluation tastes of users A and B are seen to be, and the greater the distance is to 0, the more different the evaluation tastes are seen to be.

The mate selection unit (105) can select a user who is closest to a specific user who will receive customized recommended content in the manner described above as a mate, and provide the selected mate to the user terminal (200) as a user with whom to enjoy customized recommended content together.

FIG. 10 is a drawing exemplarily showing the hardware configuration of a content provision server according to FIG. 1.

Referring to FIG. 10, the content provision server (100) may include at least one processor (110); and a memory that stores instructions that instruct at least one processor (110) to perform at least one operation.

Here, at least one operation may include at least some of the operations or functions of the content providing server (100) described with reference to FIGS. 1 to 9.

Here, at least one processor (110) may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. Each of the memory (120) and the storage device (160) may be configured with at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (120) may be one of a read only memory (ROM) and a random access memory (RAM), and the storage device (160) may be a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or various memory cards (e.g., a micro SD card).

In addition, the content providing server (100) may include a transceiver (130) that performs communication via a wireless network. In addition, the content providing server (100) may further include an input interface device (140), an output interface device (150), a storage device (160), etc. Each component included in the content providing server (100) may be connected by a bus (170) and communicate with each other.

Examples of the content providing server (100) may include a desktop computer, a laptop computer, a notebook, a smart phone, a tablet PC, a mobile phone, a smart watch, a smart glass, an e-book reader, a portable multimedia player (PMP), a portable game console, a navigation device, a digital camera, a digital multimedia broadcasting (DMB) player, a digital audio recorder, a digital audio player, a digital video recorder, a digital video player, a PDA (Personal Digital Assistant), etc.

The methods according to the present invention may be implemented in the form of program commands that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the computer-readable medium may be those specially designed and configured for the present invention or may be known and available to those skilled in the art of computer software.

Examples of computer-readable media may include hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions may include not only machine language codes generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter, and the like. The above-described hardware devices may be configured to operate with at least one software module to perform the operations of the present invention, and vice versa.

In addition, the above-described method or device may be implemented by combining all or part of its configuration or function, or may be implemented separately.

Although the present invention has been described above with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various modifications and changes may be made to the present invention without departing from the spirit and scope of the present invention as set forth in the claims below.

Claims (1)

As a content provision server that provides customized cultural and artistic content curation and recommends mates to view with the user,
A user profile management unit that receives user information corresponding to each user of a plurality of user terminals, and creates and manages user profile information including the user information and the user's usage history information using cultural and artistic content;
A cultural arts content management department that creates and manages content information corresponding to each of the many registered cultural arts contents;
A content recommendation unit that determines a predicted preference of a first user for the cultural and artistic contents based on evaluation information corresponding to each of the cultural and artistic contents included in the content information, selects a user-customized content candidate group based on the predicted preference, and selects a user-customized recommended content for the first user from the user-customized content candidate group;
A mate selection unit that selects a mate to enjoy with the customized recommended content among the above users; and
Including a user emotion tracking unit that receives a voice signal recorded during the most recent preset time interval from the user terminal of the first user and determines the user emotion of the first user based on the voice signal;
The above content information is:
It includes an appreciation tag composed of at least one of the atmosphere, genre, material, and award history for each of the above cultural and artistic contents, and a recommended emotion indicating the emotional state of the user who viewed each of the above cultural and artistic contents.
The above user emotion tracking part,
By visualizing the time of the voice signal as the x-axis, the frequency of the voice signal as the y-axis, and the amplitude size of the voice signal as a color within a preset range, a spectrogram is obtained, the spectrogram is divided into blocks of a preset size, and each of the divided blocks is converted into a pixel value corresponding to the pixel value to generate an input image, and the generated input image is input to a pre-trained convolutional neural network (CNN), and a first emotion is determined based on the output of the convolutional neural network.
The above speech signal is converted into STT (Speech to Text) to obtain a speech text corresponding to the speech signal, and the words constituting the speech text are analyzed using a recurrent neural network (RNN) to determine the second emotion.
Selecting one of the first emotion and the second emotion by referring to predefined selection information for each of a plurality of user emotions in advance, and determining the selected emotion as the user emotion,
The above mate selection department is,
Generate a first user matrix corresponding to the first user,
Generate second user matrices corresponding to second users who are distinct from the first user,
Calculate a predefined mate distance between the first user matrix and the second user matrix,
A content providing server that selects a mate to enjoy content with the first user among the second users based on the calculated mate distance.