KR102317742B1 - Server and control method thereof, and image processing apparatus and control method thereof - Google Patents

Abstract

A server according to an embodiment of the present invention includes: a communication interface for communicating with one or more image processing apparatuses; a storage in which data is stored; A processor configured to provide a processing result performed on the first event to the image processing apparatus when a first event caused by a user's speech is received from the image processing apparatus, and to store a history of the first event in a storage according to the processing of the first event includes, wherein the processor determines a relevance between the first event and the second event when a second event by a non-utterance user input is received from the image processing apparatus, and the first event and the second event are related to each other. If it is determined that there is, the second event is processed based on the history of the first event stored in the storage.

Description

Server and its control method, image processing apparatus and its control method

The present invention relates to a server in a system for processing an image signal to be displayed as an image and a control method thereof, an image processing apparatus and a control method thereof, and more particularly, to a function corresponding to a corresponding voice command by recognizing a user's voice command or A server having a structure capable of executing an operation and a method for controlling the same, and an image processing apparatus and a method for controlling the same.

The image processing apparatus processes image signals/image data received from the outside according to various image processing processes. The image processing apparatus may display the processed image signal as an image on a display panel having its own, or output the processed image signal to the corresponding display apparatus to be displayed as an image on another display apparatus having a panel. That is, if the image processing apparatus is an apparatus capable of processing an image signal, it may include both a case including a panel capable of displaying an image and a case not including a panel. An example of the former case is a TV, and the latter case An example of this is a set-top box.

In the image processing apparatus, the addition and expansion of various functions are continuously reflected according to the development of technology. According to this trend, the configuration of inputting a command reflecting the user's intention to the image processing apparatus in the image processing apparatus also has various structures or A method is being proposed. For example, in the prior art, when a key/button on a user's remote controller is pressed, the remote controller transmits a control signal to the image processing apparatus so that an operation desired by the user is executed. Various configurations have been proposed for controlling the image processing apparatus by reflecting the user's intention, such as detecting a motion or utterance by a user, analyzing the sensed content, and executing a corresponding action.

A server according to an embodiment of the present invention includes: a communication interface for communicating with one or more image processing apparatuses; a storage in which data is stored; When a first event caused by a user's utterance is received from the image processing apparatus, a processing result performed on the first event is provided to the image processing apparatus, and a history of the first event is displayed according to the processing of the first event. and a processor for storing in the storage, wherein the processor determines a relevance between the first event and the second event when a second event according to a non-utterance user input is received from the image processing apparatus, and When it is determined that the first event and the second event are related, the second event is processed based on a history of the first event stored in the storage.

Here, when the second event is received, the processor refers to the history of the first event stored in the storage to determine whether the second event can be processed independently of the first event, or a processing result of the first event It may be determined whether the second event is processed based on the .

In addition, when the first event is received, the processor performs a search in a preset database based on the first event, and receives the second event in a state in which a plurality of search results are derived from the first event. , one final result selected based on the second event among the plurality of search results for the first event may be transmitted to the image processing apparatus.

Here, when there are a plurality of search results for the first event, the processor transmits the plurality of search results and a request command for requesting selection of one of the plurality of search results to the image processing apparatus, , may receive the second event corresponding to the request command from the image processing apparatus.

Also, the second event may be converted into a form of a voice command according to an utterance by the image processing apparatus to be recognized by the server and received by the server.

In addition, a method of controlling a server according to an embodiment of the present invention includes: receiving a first event by a user's utterance from an image processing apparatus; providing a processing result performed on the first event to the image processing apparatus and storing a history of the first event according to the processing of the first event; receiving a second event according to a non-ignition type user input from the image processing apparatus; determining the relevance between the first event and the second event, and processing the second event based on the pre-stored history of the first event if it is determined that the first event and the second event are related It is characterized in that it includes.

Here, the step of determining the relevance between the first event and the second event may include whether the second event can be processed independently of the first event with reference to the pre-stored history of the first event, or the second event It may include determining whether to process the second event based on the processing result of the first event.

In addition, the step of providing the processing result performed on the first event to the image processing apparatus includes performing a search in a preset database based on the first event, and the stored first event The processing of the second event based on the history of The method may include transmitting one final result selected based on the second event to the image processing apparatus.

Here, in the receiving of the second event, if there are a plurality of search results for the first event, the plurality of search results and a request command for requesting selection of any one of the plurality of search results transmitting the image to an image processing apparatus; The method may include receiving the second event corresponding to the request command from the image processing apparatus.

Also, the second event may be converted into a form of a voice command according to an utterance by the image processing apparatus to be recognized by the server and received by the server.

In addition, an image processing apparatus according to an embodiment of the present invention includes a storage for storing data; A user interface comprising: a user interface including a speech input interface through which a user's speech is input, and a non-speech input interface through which a user input of a non-speech type is performed; a processor for processing an operation corresponding to a first event generated by the speech input interface to be executed; A history of the occurrence and processing of the first event is stored in the storage, and when a second event is generated by the non-ignition input interface after the occurrence of the first event, between the first event and the second event and a controller that determines the relevance and controls the second event to be processed based on the history of the first event stored in the storage when it is determined that the first event and the second event are related. .

Here, when the second event occurs, the controller refers to the history of the first event stored in the storage to determine whether the second event can be processed independently of the first event, or a processing result of the first event It may be determined whether the second event is processed based on the .

In addition, when a plurality of results are searched for an operation corresponding to the first event, the controller may selectably display the plurality of search results on a display, and the second event is displayed while the plurality of search results are displayed. When it occurs, it may be determined whether the second event indicates selection of any one of the plurality of search results.

Here, further comprising a communication interface communicating with a server that performs a search process for a voice command corresponding to the utterance, wherein the controller transmits the first event to the server and retrieves the search result for the first event. It can be received from the server and displayed on the display.

In addition, the controller may convert the second event into a form of a voice command according to an utterance so that the server can recognize the second event and transmit it to the server.

In addition, a control method of an image processing apparatus according to an embodiment of the present invention includes: generating a first event by a speech input interface to which a user's speech is input; and executing an operation corresponding to the first event, wherein the executing of the operation corresponding to the first event includes: storing a history of the generation and processing of the first event in the storage; generating a second event by a non-speech input interface through which a user input in a non-utterance manner is performed; determining the relevance between the first event and the second event, and controlling the second event to be processed based on the stored history of the first event if it is determined that the first event and the second event are related It is characterized in that it comprises a step.

Here, the determining of the relevance between the first event and the second event may include whether the second event can be processed independently of the first event with reference to the pre-stored history of the first event, or the second event It may include determining whether to process the second event based on the processing result of the first event.

In addition, the step of executing the operation corresponding to the first event further includes the step of selectively displaying the plurality of search results when a plurality of results are searched for the operation corresponding to the first event, The determining of the relevance between the first event and the second event may include determining whether the second event indicates selection of any one of the plurality of search results.

Here, the image processing apparatus communicates with a server that performs a search process for a voice command corresponding to an utterance, and transmits the first event to the server to receive and display a search result for the first event from the server can do.

Also, the image processing apparatus may convert the second event into a form of a voice command according to an utterance so that the server can recognize the second event, and transmit it to the server.

In addition, an image processing apparatus according to an embodiment of the present invention includes: a user interface including a speech input interface through which a user's speech is input and a non-speech input interface through which a user input in a non-speech manner is performed; a communication interface for communicating with a server that searches for an operation corresponding to a voice command by utterance; a processor for receiving a search result for a first event generated by the speech input interface from the server and processing the search result to be executed; When a second event instructing selection of any one of the plurality of search results occurs by the non-utterance input interface in a state in which a plurality of the search results are derived with respect to the first event by the server, the server A controller that converts the second event into a form of a voice command recognizable by the server and transmits it to the server so as to provide a final search result by selecting any one of the plurality of search results based on the second event It is characterized in that it includes.

1 is a block diagram of an image processing apparatus included in a system according to a first embodiment of the present invention;
2 is a block diagram of a server included in the system according to the first embodiment of the present invention;
3 and 4 are flowcharts showing a method in which the interactive server of FIG. 2 provides a search result corresponding to an event to a display device;
5 is a block diagram of the server-side processor of the interactive server of FIG. 2 subdivided by function;
6 and 7 are exemplary views showing the data transmission relationship performed between the display device and the interactive server of FIG. 2 and each component in the interactive server;
8 and 9 are flowcharts showing a control method of a display apparatus according to a second embodiment of the present invention;
10 is a block diagram of a processor of a display apparatus according to a third embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments, only the components directly related to the spirit of the present invention will be described, and descriptions of other components will be omitted. However, in implementing the device or system to which the spirit of the present invention is applied, it is revealed that this does not mean that the configuration in which the description is omitted is unnecessary.

1 is a block diagram of an image processing apparatus 100 included in a system according to a first embodiment of the present invention.

Hereinafter, a case in which the image processing apparatus 100 is implemented as a display device having a structure capable of displaying an image by itself, for example, a TV will be described. However, the spirit of the present invention is applicable even to a case in which the image processing apparatus 100 has a structure capable of outputting image data or control information to another display apparatus without displaying an image by itself. It is not limited to the embodiment described below. That is, the spirit of the present invention may be variously changed and applied, and may be implemented in various types of devices.

As shown in FIG. 1 , the image processing apparatus 100 or the display apparatus 100 according to the present embodiment receives an image signal from an external image supply source (not shown). The type or characteristic of the image signal that the display apparatus 100 can receive is not limited. For example, the display apparatus 100 receives a broadcast signal transmitted from a transmission device (not shown) of a broadcasting station, and the corresponding broadcast signal. can be tuned to display the broadcast image.

The display apparatus 100 includes a communication interface 110 that communicates with the outside to transmit and receive data/signals, and a processor that processes data received through the communication interface 110 according to a preset process. When the data processed by the processor 120 is image data, a display 130 for displaying the image data as an image, and a user interface 140 through which an input operation by a user is performed; , a storage 150 for storing data/information, and a controller 160 for controlling general operations of the display apparatus 100 .

The communication interface 110 transmits/receives data so that the display apparatus 100 can bi-directionally communicate with an external device (not shown) such as a server. The communication interface 110 connects to an external device (not shown) through a wired/wireless wide area/local area network or a local access method according to a preset communication protocol.

The communication interface 110 may be implemented by a collection of connection ports or connection modules for each device, and a protocol for connection or an external device to be connected is not limited to one type or type. does not The communication interface 110 may be built-in to the display apparatus 100 , and all or part of the configuration is additionally installed in the display apparatus 100 in the form of an add-on or a dongle. is also possible

The communication interface 110 transmits/receives signals according to a protocol designated for each connected device, and may transmit/receive signals based on an individual access protocol for each connected device. For example, in the case of image data, the communication interface 110 may include a radio frequency (RF) signal, a composite/component video, a super video, a SCART, a high definition multimedia interface (HDMI), and a display. It is possible to transmit and receive signals based on various standards such as a port (DisplayPort), a unified display interface (UDI), or a wireless HD.

The processor 120 performs various processes on data/signals received through the communication interface 110 . When the image data is received by the communication interface 110 , the processor 120 performs an image processing process on the image data, and outputs the image data on which the process is performed to the display 130 to display the corresponding image on the display 130 . Allows an image based on data to be displayed. When the signal received by the communication interface 110 is a broadcast signal, the processor 120 extracts video, audio, and additional data from the broadcast signal tuned to a specific channel, and adjusts the video to a preset resolution to display 130 to be displayed in

The type of the image processing process performed by the processor 120 is not limited, for example, decoding corresponding to the image format of the image data, and converting image data of an interlace method into a progressive method. De-interlacing, scaling for adjusting image data to a preset resolution, noise reduction for image quality improvement, detail enhancement, frame refresh rate conversion, etc. may include.

Since the processor 120 can perform various processes according to the type and characteristics of data, the process that the processor 120 can perform cannot be limited to an image processing process. In addition, data that the processor 120 can process cannot be limited to only being received by the communication interface 110 . For example, when a user's utterance is input through the user interface 140 , the processor 120 may process the corresponding utterance according to a preset voice processing process.

The processor 120 is an image processing board in which a system-on-chip (SOC) integrating these various functions or individual chipsets capable of independently performing each of these processes are mounted on a printed circuit board. (not shown), and is embedded in the display device 100 .

The display 130 displays the image signal/image data processed by the processor 120 as an image. The implementation method of the display 130 is not limited, but liquid crystal, plasma, light-emitting diode, organic light-emitting diode, surface-conducting electron gun (surface-) conduction electron-emitter), carbon nano-tube, and nano-crystal can be implemented in various display methods.

The display 130 may additionally include an additional configuration according to an implementation method thereof. For example, in the case of the liquid crystal type, the display 130 includes a liquid crystal display panel (not shown), a backlight unit (not shown) for supplying light thereto, and a panel driving substrate (not shown) for driving the panel (not shown). city), etc.

The user interface 140 transmits various preset control commands or information to the controller 160 according to a user's manipulation or input. The user interface 140 informs various events generated by the user according to the user's intention and transmits the information to the controller 160 . Here, various types of events generated by the user are possible, for example, the user's manipulation, utterance, gesture, and the like are possible.

The user interface 140 corresponds to a method in which a user inputs information, and is provided to detect information input in a corresponding manner. Accordingly, the user interface 140 may be divided into an interactive input interface 141 and a non-interactive input interface 142 .

The interactive input interface 141 is provided to input a user's utterance. That is, the interactive input interface 141 is implemented as a microphone, and detects various sounds generated in the external environment of the display apparatus 100 . The interactive input interface 141 may generally detect a utterance by a user, but may also detect sounds generated by various environmental factors other than the user.

The non-interactive input interface 142 is provided so that a user's input other than an utterance method is performed. In this case, the implementation form of the non-interactive input interface 142 is various, for example, a remote controller separated from the display device 100 or a menu key installed outside the display device 100 . -key) or an input panel, or a motion sensor or camera that detects a user's gesture is possible.

Alternatively, the non-interactive input interface 142 may be implemented as a touch-screen installed on the display 130 . In this case, the user may transmit a preset command or information to the controller 160 by touching the input menu or UI image displayed on the display 130 .

The storage 150 stores various data according to the control of the controller 160 . The storage 150 is implemented as a non-volatile memory such as a flash-memory or a hard-disc drive to preserve data regardless of whether or not system power is provided. As the storage 150 is accessed by the controller 160 , reading, writing, modifying, deleting, updating, etc. of pre-stored data are performed.

The controller 160 is implemented as a CPU, and controls the operation of all components of the display apparatus 100 including the processor 120 according to the occurrence of a predetermined event. In the case of a user's speech event, for example, when a user's speech is input through the interactive input interface 141 , the controller 160 controls the processor 120 to process the inputted speech. If the user utters the channel number, the controller 160 controls the broadcast image to be switched and displayed with the uttered channel number.

Here, the process of processing the input utterance may be performed by the display apparatus 100 alone, or may be performed by sharing it with an external apparatus in consideration of various factors, such as a system load and processing capability. This configuration will be described below.

2 is a configuration block diagram of the servers 200 and 300 included in the system of the present embodiment.

As shown in FIG. 2 , the display apparatus 100 communicatively connects to one or more servers 200 and 300 through a network. Here, the display apparatus 100 includes a speech-to-text (STT) server 200 that converts the user's utterance into a voice command, and an action/action/result corresponding to the voice command by analyzing the voice command. Each is connected to the interactive server 300 to determine the. Here, the names of each of the servers 200 and 300 are applied only for mutual identification in consideration of their respective functions, and the corresponding names do not limit the structural characteristics of each of the servers 200 and 300 .

The STT server 200 generates the content of the voice signal as text by analyzing the waveform of the voice signal. Accordingly, when the STT server 200 receives a voice signal from the display device 100, it converts it into a text voice command. Here, the utterance refers to an action made by a user, and the voice command refers to the content of the utterance being data in a text format.

The interactive server 300 searches for operations of various display apparatuses 100 corresponding to voice commands. The interactive server 300 analyzes the voice command received from the display device 100, and, according to the analysis result, provides a control signal for performing an operation corresponding to the voice command or information searched based on the voice command. It is transmitted to the display device 100 .

The interactive server 300 includes a server-side communication interface 310 for communicating with the display device 100 and a server-side processor 320 for processing a voice command received through the server-side communication interface 310 . includes

An operation performed when the user of the display apparatus 100 speaks under the system by the display apparatus 100 , the STT server 200 and the interactive server 300 will be described below.

When the user's utterance is input, the display apparatus 100 transmits a voice signal according to the corresponding utterance to the STT server 200 . The STT server 200 converts a voice signal received from the display device 100 into a data-formatted voice command, and transmits the voice command to the display device 100 . The display device 100 transmits the voice command received from the STT server 200 to the interactive server 300 .

The interactive server 300 analyzes the content of the voice command, derives a result corresponding to the voice command, and returns it to the display device 100 . For example, when the content of the voice command is "Who is the performer of this week's 'program ABC'?", the interactive server 300 may search for "this week", "program ABC", A search is performed with the keyword of "performer" and the result is obtained.

Here, since the database is structured for each category of keywords, a desired result can be searched for with a specific keyword. The structured database can be implemented in various ways depending on the design and construction method of the database, and a detailed description thereof will be omitted.

However, when the interactive server 300 performs a search with one or more keywords derived from a voice command, a plurality of search results may appear. As an example of the preceding voice command, it is conceivable that the broadcast program "program ABC" airing this week is broadcast at different times or on different broadcast channels.

In this case, the interactive server 300 needs to select any one search result that meets the user's intention from among a plurality of search results for a voice command. As a corresponding method for the interactive server 300 to select one search result, for example, the following two methods are possible.

In the first method, any one search result is selected according to a specific rule established by the client of the interactive server 300 , that is, the display apparatus 100 itself. For example, if the voice command is a request for a broadcast time of a specific program, and there are a plurality of programs satisfying these conditions, the most recently broadcast program may be selected. In the second method, when the user of the display apparatus 100 utters a utterance that selects a search result, the interactive server 300 connects the utterance with the previous utterance history to obtain a search result that meets the user's original intention. You can choose.

However, since environments of various input methods are provided to the user in such a system, even if the display apparatus 100 supports a voice recognition input environment, the user selects and inputs the method that is most convenient in each situation using various input methods. will perform In an environment of a system supporting such a multi-input mode, if history management is performed individually among multiple inputs of a user, there may be a limitation in selecting or deriving a result reflecting the user's original intention. If the user does not provide a result originally intended by the user even though the user initially performs the speech input and then changes the input method to another method, the user may feel uncomfortable about the use environment.

Therefore, in the interactive system environment by the interactive server 300 and the display apparatus 100, there are a plurality of results initially searched for by the user's utterance, and then, the user selects any one search result through non-utterance input. is selected, the interactive system needs to provide a search result that matches the intention of the user's initial utterance.

Here, the system according to the prior art does not integrally support history management for each of these multiple input modes. That is, in the interactive server 300 according to the prior art, the input by the user's speech of the display apparatus 100, the input by the user's gesture, and the input by the operation of the user's remote controller (not shown) are individually performed. , and do not manage by linking each of these inputs to each other.

In the system according to the prior art, a state in which the display apparatus 100 displays a plurality of search results received from the interactive server 300 according to the second method is considered. In this case, if the user is not uttering, the input may be performed in a non-speech method through the non-interactive input interface 142. Because the history of the data is not referenced in relation to each other, results that do not match the user's intention are produced.

For example, consider a case in which a user utters "record 'program ABC'" and a plurality of contents related to 'program ABC' are displayed on the display apparatus 100 . In the conventional case, when the user selects one of the utterances, recording is performed by selecting one of the contents normally, whereas when the user selects any one by a separate non-utterance input method, only the selection of the content is performed. Recording is not performed. This is because the interactive server 300 cannot take an action after the utterance because information by the non-utterance input by the user is not transmitted to the interactive server 300 .

In consideration of this point, the following method is proposed according to the present embodiment.

An event generated by the interactive input interface 141 of the display apparatus 100 according to a user input is referred to as a first event, and an event generated by the non-interactive input interface 142 is referred to as a second event for convenience. . When a first event occurs from the display device 100 , the interactive server 300 processes the first event and stores the history of the first event, and then, when a second event occurs, the first event stored in advance is stored. The second event is processed based on the processing history. Here, the display apparatus 100 converts the second event into the form of a voice command so that the interactive server 300 provided to recognize the voice command recognizes the second event when the second event occurs. It is transmitted to the brother server 300 .

The interactive server 300 may refer to a history of previously occurring events when determining an action for a predetermined event. The interactive server 300 manages the events generated by each of the interactive input interface 141 and the non-interactive input interface 142 of the display device 100 in an integrated manner, rather than individually, to respond to a newly generated event. When determining a corresponding action, it is possible to refer to an event that has previously occurred regardless of whether the interactive input interface 141 or the non-interactive input interface 142 is present.

For example, if there are a plurality of search results corresponding to the user's initial utterance, the interactive server 300 provides the plurality of search results for the user to select. At this time, even if the user inputs an instruction to select one of the search results in a non-verbal manner, the interactive server 300 communicates between the two even if the new event is different from the previous event through the integrated management of the event input. can be correlated. Accordingly, a search result desired by the user may be derived and provided to the user.

3 and 4 are flowcharts illustrating a method in which the interactive server 300 provides a search result corresponding to an event to the display apparatus 100 .

As shown in FIG. 3 , in step S100 , the interactive server 300 receives a first event that is a voice command by utterance from the display apparatus 100 .

In step S110, the interactive server 300 derives a search keyword by analyzing the first event.

In step S120, the interactive server 300 performs a search with the derived search keyword, and stores the occurrence and processing history of the first event.

In step S130 , the interactive server 300 determines whether there are multiple search results.

If it is determined that there are not a plurality of search results, in step S140 , the interactive server 300 transmits the search results to the display apparatus 100 and ends.

On the other hand, if it is determined that there are a plurality of search results, in step S150 , the interactive server 300 transmits the plurality of search results to the display apparatus 100 and prompts the display apparatus 100 to select one of the plurality of search results. ) to request

4 , in step S160 , the interactive server 300 receives a second event including a selection instruction from the display apparatus 100 . Here, the second event is generated by a non-ignition type user input in the display apparatus 100 .

In step S170, the interactive server 300 refers to the pre-stored occurrence and processing history of the first event to determine the relevance of the first event and the second event, that is, whether the second event should be processed dependently on the first event. judge For example, the interactive server 300 determines whether the second event is a selection instruction command for a plurality of search results searched for by the first event.

If it is determined in step S180 that the second event is not related to the first event, the interactive server 300 analyzes and processes the second event independently of the first event in step S190.

On the other hand, if it is determined in step S180 that the second event is a selection instruction for a plurality of search results searched for by the first event, the interactive server 300 selects a search result according to the second event in step S200. In this process, the interactive server 300 may perform an additional search according to necessity.

In step S200 , the interactive server 300 transmits the final search results according to the first event and the second event to the display apparatus 100 .

Hereinafter, a detailed internal configuration of the interactive server 300 will be described with reference to FIG. 5 .

5 is a block diagram of the server-side processor 320 of the interactive server 300 subdivided by function.

As shown in FIG. 5 , the server-side processor 320 of the interactive server 300 includes a dialog engine 321 that analyzes a voice command received through the server-side communication interface 310 . and a content manager 322 (content manager) that performs a search based on the search keyword received from the conversation engine 321 .

The dialog engine 321 includes a dialog engine manager 323 , a dialog agent 324 , an action agent 325 , and a database 326 . The division of these components with respect to the server-side processor 320 or the dialogue engine 321 is applied for convenience by function, and does not mean that the server-side processor 320 or the dialogue engine 321 must have such a division form. .

In addition, the above components may be a hardware configuration, may be a software configuration, or hardware and software may be mixed and applied.

The dialog engine manager 323 transmits the event in the form of a voice command transmitted to the dialog engine 321 to the dialog agent 324 . The dialog agent 324 analyzes the contents of the corresponding event, generates the analysis result as structured data, and returns it to the dialog engine manager 323 .

Here, the structured data generated by the dialog agent 324 is hereinafter referred to as a dialog frame. The dialog frame includes the contents of the voice command event, but the voice command is structured by category so that other components such as the dialog engine manager 323, the action agent 325, and the content manager 322 can decode and refer to it. . The type of category and structure may be variously determined according to a design method of the server-side processor 320 .

The dialog frame includes information about the user's purpose in the voice command, attributes necessary to achieve the purpose, and values of the attributes. A specific example of the dialog frame will be described later.

The dialog engine manager 323 transfers the dialog frame received from the dialog agent 324 to the action agent 325 . The action agent 325 determines an action corresponding to the dialog frame received from the dialog engine manager 323 , and returns a determination result to the dialog engine manager 323 . That is, the action agent 325 determines an action corresponding to the event.

When information retrieval is required in the determination process, the action agent 325 may transmit a retrieval keyword extracted from the conversation frame to the content manager 322 to request information retrieval. The content manager 322 performs a search on various types of search databases inside or outside the interactive server 300 with the received keyword, and returns the search result to the action agent 325 .

When the dialog engine manager 323 receives the action or search result corresponding to the event from the action agent 325 , the dialog engine manager 323 transmits the received result to the server-side communication interface 310 to be transmitted to the display device 100 .

The database 326 includes general information that can be referenced by the dialog engine manager 323 , the dialog agent 324 , and the action agent 325 in the process described above. For example, the database 326 may include context information for the dialog agent 324 to generate a dialog frame, event history information, dialog model information, and the action agent 325 to determine an action corresponding to the event. It may include action model information for

Under the above structure, a detailed operation example of the case in which the interactive server 300 returns a result corresponding to the voice command received from the display apparatus 100 to the display apparatus 100 will be described.

6 and 7 are exemplary diagrams illustrating a data transmission relationship performed between the display apparatus 100 and the interactive server 300 and each component in the interactive server 300 .

As shown in FIG. 6 , when the user performs an utterance such as "Who is the performer of this week's 'program ABC'?", the display apparatus 100 acquires a voice command corresponding to the utterance. A voice command corresponding to this utterance is referred to as a first event. The display apparatus 100 transmits the first event to the interactive server 300 ( 410 ).

The dialog engine manager 323 transfers the first event received in the server-side communication interface 310 to the dialog agent 324 ( 420 ).

The dialog agent 324 creates a dialog frame corresponding to the first event by analyzing the first event. When the first event is the content of "Who is the performer of this week's 'program ABC'?", the dialog frame may be expressed as follows.

(1) Purpose: Inquiry of performers of broadcast programs

(2) Title: program ABC

(3) Time: This week

In the dialogue frame above, “purpose”, “title”, and “time” are conditional categories of the dialogue frame, and for each conditional category, the attribute values of “inquiry of broadcasters”, “program ABC”, and “this week” are included. do. By analyzing the contents of the first event, the dialog agent 324 may derive a category of conditions including each keyword included in the first event and attribute values for each category.

Upon receiving the dialog frame generated as described above from the dialog agent 324 ( 430 ), the dialog engine manager 323 transmits the dialog frame to the action agent 325 ( 440 ).

Based on the received dialog frame, the action agent 325 creates the following keywords and transmits them to the content manager 322 ( 450 ).

(1) Title: program ABC

(2) Time: 2013/06/03~2013/06/09

In the case of the attribute value of the time condition category, a specific date must be specified for search. Accordingly, the action agent 325 determines the specific date and time section of this week based on the current time based on the attribute value of “this week” of the time category of the conversation frame.

The content manager 322 performs a search on a search service of various contents and metadata using keywords received from the action agent 325 . A search target may be a database, a search service, and a content service that can be accessed by the content manager 322 .

The content manager 322 transmits the search result obtained through the search to the conversation agent 324 (460). For example, the search result may include a plurality as follows.

(1) program ABC / 2013.06.04 17:00 / Channel 4

(2) program ABC / 2013.06.08 10:00 / Channel 20

(3) program ABC / 2013.06.09 20:00 / Channel 35

Each search result includes information on a program title, broadcast time, and broadcast channel, respectively, and further includes information about performers, although not indicated above. According to the search result above, there are three cases of "program ABC" being broadcast on "this week".

The action agent 325 transmits the search result to the dialog engine manager 323 ( 470 ). In this process, the action agent 325 adds a command to allow the display apparatus 100 to selectively display a plurality of search results because there are a plurality of search results.

The conversation engine manager 323 transmits the search result and the command to the display apparatus 100 ( 480 ).

As shown in FIG. 7 , the display apparatus 100 displays a plurality of search results in a user-selectable manner according to a command received from the conversation engine manager 323, and displays the selection of any one of the displayed search results. The user is notified by the system utterance of the device 100 .

Accordingly, the user inputs, for example, a selection instruction for the first search result through a non-utterance method such as a button operation or a gesture of a remote controller (not shown). This input is referred to as a second event.

The display apparatus 100 generates a fictitious voice command including contents of the second event in response to the second event. Here, the display device 100 displays the second event similarly to the first event so that the interactive server 300 provided to recognize the event in the form of a voice command by utterance can recognize the second event. It is converted into a form and transmitted to the interactive server 300 . The second event may include, for example, "No. 1".

The display apparatus 100 transmits the second event to the dialog engine manager 323 (operation 510).

The dialog engine manager 323 transmits the second event to the dialog agent 324 ( 520 ).

Since the second event only includes the content of “No. 1”, it is difficult for the dialog agent 324 to process the second event only with the content of the second event. Accordingly, the dialog agent 324 calls processing state information about the first event from the previously performed event processing history, and generates the following dialog frame based on the called information.

(1) Purpose: Inquiry of performers of broadcast programs

(2) Order: 1

Although the object category and its attribute value are not included in the second event in this dialog frame, the dialog agent 324 may generate a dialog frame of the above content by referring to the first event related to the second event.

The dialog agent 324 transmits the generated dialog frame to the dialog engine manager 323 ( 530 ), and the dialog engine manager 323 delivers the dialog frame to the action agent 325 ( 540 ).

The action agent 325 selects a first search result from among three search results searched for by the preceding first event based on the attribute value of the order condition category in the dialog frame. The action agent 325 transmits the selection result of “program ABC / 2013.06.04 17:00 / Channel 4” to the dialog engine manager 323 ( 550 ).

The dialog engine manager 323 transmits the received selection result to the display apparatus 100 ( 560 ). In this case, the dialog engine manager 323 may additionally transmit the system utterance command written by the dialog agent 324 to the display apparatus 100 with reference to the selection result.

The display apparatus 100 outputs the performer of "program ABC" broadcast on channel 4 at 17:00 on June 4, 2013 through system utterance according to the received command.

According to the above embodiment, when there are a plurality of results that satisfy the target and condition input by the user through utterance in the system implemented by the display apparatus 100 and the interactive server 300, the user may use the remote controller or gesture Even if a non-utterance input such as such is mixed, the user's convenience can be improved by providing a result that meets the user's original intention.

Meanwhile, in the above embodiment, the case in which the interactive server 300 stores the history of events by the user's utterance input and non-utterance input of the display apparatus 100 has been described, but the spirit of the present invention This is not limited thereto. For example, a configuration in which the display apparatus 100 stores an event history is also possible, which will be described below with reference to FIGS. 8 and 9 .

8 and 9 are flowcharts illustrating a control method of the display apparatus 100 according to the second embodiment of the present invention.

As shown in FIG. 8 , in step S300 , the display apparatus 100 receives the user's utterance. In step S310 , the display apparatus 100 transmits a voice command corresponding to the user's utterance to the interactive server 300 .

In step S320 , the display apparatus 100 determines whether there are a plurality of corresponding results of the voice command received from the interactive server 300 .

If there is one corresponding result of the voice command received from the interactive server 300, in step S330, the display device 100 displays the corresponding result.

On the other hand, if there are a plurality of corresponding results of the voice command received from the interactive server 300 , in step S340 , the display apparatus 100 displays the plurality of corresponding results for the user to select.

In step S350 , the display apparatus 100 receives a selection input for a plurality of corresponding results from the user. While the above process is in progress, the display apparatus 100 stores the history of the occurrence and processing of the event of the voice command by the utterance.

As shown in FIG. 9 , in step S360 , the display apparatus 100 determines whether the received selection input is an utterance method or a non-utterance method.

In this process, the display apparatus 100 determines whether the selection input received in step S350 is related to the utterance generated in step S300 with reference to the previously stored event history. The display apparatus 100 performs step S360 when it is determined that there is a relationship between the two as a result of the determination.

If the received selection input is an utterance method, in step S370 , the display apparatus 100 transmits a voice command corresponding to the selection input to the interactive server 300 and proceeds to step S400 .

On the other hand, if the received selection input is a non-speech type, in step S380 , the display apparatus 100 generates a fictitious voice command corresponding to the content of the non-utterance type selection input. In step S390 , the display device 100 transmits the generated fictitious voice command to the interactive server 300 .

In step S400 , the display apparatus 100 receives and displays a result corresponding to the transmitted selection input from the interactive server 300 .

Meanwhile, in the above embodiment, a case in which the interactive server 300 analyzes and searches a result corresponding to an event occurring in the display apparatus 100 has been described. However, since the spirit of the present invention is not limited thereto, an embodiment in which the display device 100 performs at least one function of the STT server 200 and the interactive server 300 in the first embodiment is also possible. .

10 is a block diagram of the processor 620 of the display device 600 according to the third embodiment of the present invention.

As shown in FIG. 10 , the display apparatus 600 according to the present embodiment includes a communication interface 610 and a processor 620 . As for other components included in the display apparatus 600 , the first embodiment can be applied, and detailed descriptions thereof will be omitted.

The processor 620 receives the voice processing engine 621 for processing the utterance input to the display device 600 , the conversation engine 622 for analyzing the voice command corresponding to the utterance, and the conversation engine 622 . and a content manager 623 that performs a search based on the received search keyword. In addition, the dialog engine 622 includes a dialog engine manager 624 , a dialog agent 625 , an action agent 626 , and a database 627 . In these drawings and embodiments, only the components directly related to the spirit of the invention are described, and the illustration and description of other components are omitted.

When a user's utterance is input, the voice processing engine 621 transmits a voice signal of the corresponding utterance to the STT server 200, receives a voice command corresponding to the utterance from the STT server 200, and receives the conversation engine manager 624. pass it on to

Hereinafter, specific operations by the dialog engine manager 624 , the dialog agent 625 , the action agent 626 , and the content manager 623 are substantially the same as the components of the same name described in the previous embodiment. can be applied, a detailed description thereof will be omitted.

Alternatively, the display device 600 may perform a function of the STT server 200 by the voice processing engine 621, ie, converting utterances into voice commands. In this case, when a user's utterance is input, the voice processing engine 621 converts the utterance into a voice command and transmits it to the conversation engine manager 624 .

The above-described embodiment is merely exemplary, and various modifications and equivalent other embodiments are possible by those of ordinary skill in the art. Accordingly, the true technical protection scope of the present invention should be determined by the technical spirit of the invention described in the claims below.

100: display device
110: communication interface
120: processor
130: display
140 : user interface
141: interactive input interface
142: non-interactive input interface
150 : storage
160: controller
200: STT server
300 : interactive server
310: server-side communication interface
320: server-side processor

Claims (17)

In an electronic device,
display unit;
communication interface;
user interface; and
controlling the communication interface to transmit a first search request to a server based on a user voice input received through the user interface;
Controls the display unit to display first information received through the communication interface, corresponding to the user voice input,
a processor for controlling the display unit to display second information received through the communication interface, corresponding to the user input, based on the user input received through the user interface while displaying the first information; including,
The second information is information related to the first information or information independent of the first information according to whether the user input is related to the user voice input. According to claim 1,
The first information includes at least one search result searched based on the user's voice input. According to claim 1,
The processor is configured to identify whether the user input is related to the first information. According to claim 1,
The user input includes a touch input for retrieving the second information through an input menu or a UI image. According to claim 1,
further comprising a storage unit,
The processor is configured to store, in the storage unit, history information of a function execution corresponding to the user's voice input. According to claim 1,
The processor is configured to determine whether a second search request for searching for the second information is a search request that can be processed in relation to the first search request. According to claim 1,
The processor is configured to receive information on whether a second search request for searching the second information determined by the server is a search request that can be processed in relation to the first search request. According to claim 1,
The processor is
at least one first search result and keyword related to the user's voice input are received from the server and displayed on the display unit,
The electronic device receiving the user input for selecting one of the displayed at least one keyword. 9. The method of claim 8,
The user input includes a touch input for retrieving the second information through an input menu or a UI image. In the control method of an electronic device,
transmitting a first search request to a server based on the user's voice input received through the user interface;
displaying first information received through a communication interface, corresponding to the user voice input; and
while displaying the first information, based on the user input received through the user interface, displaying second information received through the communication interface, corresponding to the user input,
The method of controlling an electronic device, wherein the second information is information related to the first information or information independent of the first information according to whether the user input is related to the user voice input. 11. The method of claim 10,
The first information includes at least one search result searched based on the user's voice input. 11. The method of claim 10,
and identifying whether the user input is related to the first information. 11. The method of claim 10,
The user input includes a touch input for retrieving the second information through an input menu or a UI image. 11. The method of claim 10,
and storing, in a storage unit, history information of function execution corresponding to the user's voice input. 11. The method of claim 10,
and determining whether a second search request for searching the second information is a search request that can be processed in relation to the first search request. 11. The method of claim 10,
and receiving information on whether the second search request for searching the second information determined by the server is a search request that can be processed in relation to the first search request. 11. The method of claim 10,
receiving and displaying at least one first search result and keyword related to the user's voice input from the server; and
and receiving the user input for selecting one of the displayed at least one keyword.

Images