KR102661376B1 - Server and control method thereof, and image processing apparatus and control method thereof - Google Patents

Abstract

A server according to an embodiment of the present invention includes a communication interface for communicating with one or more image processing devices; Storage where data is stored; When a first event resulting from a user's speech is received from the image processing device, a processor provides the result of processing for the first event to the image processing device and stores the history of the first event in storage according to the processing of the first event. It includes, wherein the processor determines the relationship between the first event and the second event when a second event by a non-verbal user input is received from the image processing device, and determines the relationship between the first event and the second event. If it is determined that there is a second event, the second event is processed based on the history of the first event stored in the storage.

Description

Server and its control method, image processing device and its control method {SERVER AND CONTROL METHOD THEREOF, AND IMAGE PROCESSING APPARATUS AND CONTROL METHOD THEREOF}

The present invention relates to a server and a control method thereof in a system that processes video signals to be displayed as images, and a video processing device and a control method thereof. Specifically, the present invention relates to a function or function corresponding to the voice command by recognizing the user's voice command. It relates to a server structured to execute operations and its control method, and an image processing device and its control method.

An image processing device processes image signals/image data received from the outside according to various image processing processes. The image processing device can display the processed video signal as an image on a display panel provided by itself, or output the processed video signal to the display device so that it can be displayed as an image on another display device equipped with a panel. In other words, an image processing device can include both a panel capable of displaying images and a case without a panel as long as it is capable of processing video signals. An example of the former is a TV, and in the latter case, a TV is an example. An example of this is a set-top box.

Image processing devices continue to reflect the addition and expansion of various functions according to the development of technology, and according to this trend, the configuration of inputting commands that reflect the user's intention into the image processing device is also various. A method is being proposed. For example, conventionally, when the user presses a key/button on a remote controller, the remote controller transmits a control signal to the image processing device to execute the user's desired action, but recently, the image processing device Various configurations have been proposed to control the image processing device by reflecting the user's intention, such as detecting motion or speech, analyzing the detected content, and executing a corresponding action.

A server according to an embodiment of the present invention includes a communication interface for communicating with one or more image processing devices; Storage where data is stored; When a first event caused by a user's speech is received from the image processing device, the result of processing performed on the first event is provided to the image processing device, and the history of the first event is stored according to the processing of the first event. and a processor storing the data in the storage, wherein when a second event based on a non-verbal user input is received from the image processing device, the processor determines a relationship between the first event and the second event, and determines the relationship between the first event and the second event. If it is determined that the first event and the second event are related, the second event is processed based on the history of the first event stored in the storage.

Here, when the second event is received, the processor refers to the history of the first event stored in the storage and determines whether the second event can be processed independently of the first event or the processing result of the first event. Based on this, it can be determined whether the second event is being processed.

In addition, when the first event is received, the processor performs a search in a preset database based on the first event, and the second event is received while a plurality of search results are derived from the first event. If so, one final result selected based on the second event from among the plurality of search results for the first event can be transmitted to the image processing device.

Here, if there are a plurality of search results for the first event, the processor transmits the plurality of search results and a request command requesting selection of one of the plurality of search results to the image processing device, and , the second event corresponding to the request command may be received from the image processing device.

Additionally, the second event may be converted into a voice command according to speech by the image processing device and received by the server so that the server can recognize it.

Additionally, a server control method according to an embodiment of the present invention includes receiving a first event caused by a user's speech from an image processing device; providing a processing result performed on the first event to the image processing device and storing a history of the first event according to processing of the first event; receiving a second event based on a non-verbal user input from the image processing device; Determining the relationship between the first event and the second event, and if it is determined that the first event and the second event are related, processing the second event based on the previously stored history of the first event. It is characterized by including.

Here, the step of determining the relationship between the first event and the second event refers to the pre-stored history of the first event and determines whether the second event can be processed independently of the first event or whether the second event can be processed independently of the first event. It may include determining whether to process the second event based on the processing result of the first event.

In addition, the step of providing a result of processing performed on the first event to the image processing device includes performing a search in a preset database based on the first event, wherein the pre-stored first event The step of processing the second event based on the history includes, when the second event is received while a plurality of search results are derived from the first event, among the plurality of search results for the first event It may include transmitting one final result selected based on the second event to the image processing device.

Here, the step of receiving the second event includes, if there are a plurality of search results for the first event, sending the plurality of search results and a request command requesting selection of one of the plurality of search results. transmitting to an image processing device; It may include receiving the second event corresponding to the request command from the image processing device.

Additionally, the second event may be converted into a voice command according to speech by the image processing device and received by the server so that the server can recognize it.

Additionally, an image processing device according to an embodiment of the present invention includes storage for storing data; a user interface including a speech input interface through which a user's speech is input, and a non-speech input interface through which a user input in a non-speech manner is performed; a processor configured to execute an operation corresponding to a first event generated by the speech input interface; The history of the occurrence and processing of the first event is stored in the storage, and when a second event occurs by the non-igniting input interface after the occurrence of the first event, a connection between the first event and the second event is stored. and a controller that determines relevance and controls processing of the second event based on the history of the first event stored in the storage when it is determined that the first event and the second event are related. .

Here, when the second event occurs, the controller refers to the history of the first event stored in the storage and determines whether the second event can be processed independently of the first event or the processing result of the first event. Based on this, it can be determined whether the second event is being processed.

In addition, when a plurality of results are searched for an operation corresponding to the first event, the controller displays the plurality of search results on the display so that the plurality of search results can be selected, and when the second event is displayed while the plurality of search results are displayed, the controller When it occurs, it can be determined whether the second event indicates selection of one of the plurality of search results.

Here, it further includes a communication interface for communicating with a server that performs search processing for a voice command corresponding to the utterance, wherein the controller transmits the first event to the server and provides a search result for the first event. It can be received from the server and displayed on the display.

Additionally, the controller may convert the second event into a voice command according to speech and transmit it to the server so that the server can recognize the second event.

Additionally, a method of controlling an image processing device according to an embodiment of the present invention includes generating a first event by a speech input interface through which a user's speech is input; A step of executing an operation corresponding to the first event, wherein executing an operation corresponding to the first event includes: storing a history of occurrence and processing of the first event in the storage; generating a second event through a non-speech input interface where a non-speech user input is performed; Determining the relationship between the first event and the second event, and controlling the processing of the second event based on the stored history of the first event if it is determined that the first event and the second event are related It is characterized by including steps.

Here, the step of determining the relationship between the first event and the second event refers to the pre-stored history of the first event and determines whether the second event can be processed independently of the first event or whether the second event can be processed independently of the first event. It may include determining whether to process the second event based on the processing result of the first event.

In addition, the executing the operation corresponding to the first event further includes displaying the plurality of search results to be selectable when a plurality of results are searched for the operation corresponding to the first event, Determining the relationship between the first event and the second event may include determining whether the second event indicates selection of one of the plurality of search results.

Here, the image processing device communicates with a server that performs search processing for voice commands corresponding to speech, and receives and displays search results for the first event from the server by transmitting the first event to the server. can do.

Additionally, the image processing device may convert the second event into a voice command according to speech and transmit it to the server so that the server can recognize the second event.

Additionally, an image processing device according to an embodiment of the present invention includes a user interface including a speech input interface through which a user's speech is input, and a non-speech input interface through which a user input in a non-speech manner is performed; a communication interface that communicates with a server that searches for an action corresponding to a voice command by speech; a processor that receives search results for a first event generated by the speech input interface from the server and processes the search results to be executed; When a second event indicating selection of one of the plurality of search results occurs through the non-utterance input interface while a plurality of search results are derived for the first event by the server, the server A controller that converts the second event into a voice command recognizable by the server and transmits it to the server so that one of the plurality of search results can be selected based on the second event and provide a final search result. It is characterized by including.

1 is a block diagram of an image processing device included in a system according to a first embodiment of the present invention;
Figure 2 is a block diagram of the server included in the system according to the first embodiment of the present invention;
FIGS. 3 and 4 are flowcharts showing how the interactive server of FIG. 2 provides search results corresponding to events to a display device;
Figure 5 is a block diagram showing the server-side processor of the interactive server of Figure 2 subdivided by function;
6 and 7 are exemplary diagrams showing the data transmission relationship between the display device and the interactive server of FIG. 2 and each component within the interactive server;
8 and 9 are flowcharts showing a control method of a display device according to a second embodiment of the present invention;
Figure 10 is a block diagram of the processor of the display device according to the third embodiment of the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments, only configurations directly related to the spirit of the present invention will be described, and descriptions of other configurations will be omitted. However, in implementing a device or system to which the spirit of the present invention is applied, this does not mean that the configuration omitted from the description is unnecessary.

Figure 1 is a block diagram of an image processing device 100 included in a system according to a first embodiment of the present invention.

The following embodiment will describe a case where the image processing device 100 is implemented as a display device capable of displaying images on its own, for example, a TV. However, the idea of the present invention can be applied even when the image processing device 100 does not display images on its own but is a device capable of outputting image data or control information to another display device, and the implementation form of the present invention is It is not limited to the embodiments described below. In other words, the idea of the present invention can be implemented in various types of devices by changing and applying it in various ways.

As shown in FIG. 1, the image processing device 100 or the display device 100 according to this embodiment receives an image signal from an external image source (not shown). The type or characteristics of the video signal that the display device 100 can receive are not limited. For example, the display device 100 receives a broadcast signal transmitted from a broadcasting station's transmission equipment (not shown), and transmits the corresponding broadcast signal. You can display broadcast video by tuning.

The display device 100 includes a communication interface 110 that communicates to transmit and receive data/signals with the outside, and a processor that processes data received in the communication interface 110 according to a preset process. (120), when the data processed by the processor 120 is image data, a display (130) that displays the image data as an image, and a user interface (140) that performs an input operation by the user, and , a storage 150 in which data/information is stored, and a controller 160 that controls overall operations of the display device 100.

The communication interface 110 transmits and receives data to enable two-way communication between the display device 100 and an external device (not shown) such as a server. The communication interface 110 connects to an external device (not shown) through a wired/wireless wide area/local area network or local connection method according to a preset communication protocol.

The communication interface 110 can be implemented by a collection of connection ports or connection modules for each device, and the protocol for connection or the external device to be connected is not limited to one type or format. No. The communication interface 110 may be built into the display device 100, and all or part of the configuration may be additionally installed on the display device 100 in the form of an add-on or dongle. It is also possible.

The communication interface 110 transmits and receives signals according to a protocol designated for each connected device, and can transmit and receive signals based on an individual connection protocol for each connected device. For example, in the case of video data, the communication interface 110 is a radio frequency (RF) signal, composite/component video, super video, SCART, high definition multimedia interface (HDMI), and display. Signals can be transmitted and received based on various standards such as DisplayPort (DisplayPort), unified display interface (UDI), or wireless HD.

The processor 120 performs various processes on data/signals received through the communication interface 110. When image data is received in the communication interface 110, the processor 120 performs an image processing process on the image data and outputs the image data on which this process has been performed to the display 130, thereby displaying the corresponding image on the display 130. Allows an image based on data to be displayed. When the signal received at the communication interface 110 is a broadcast signal, the processor 120 extracts video, audio, and additional data from the broadcast signal tuned to a specific channel, adjusts the video to a preset resolution, and displays the display 130. It is displayed in .

The type of image processing process performed by the processor 120 is not limited, for example, decoding corresponding to the video format of the video data, converting interlace video data to progressive method. De-interlacing, scaling to adjust video data to a preset resolution, noise reduction to improve video quality, detail enhancement, frame refresh rate conversion, etc. may include.

Since the processor 120 can perform various processes depending on the type and characteristics of data, the processes that the processor 120 can perform cannot be limited to image processing processes. Additionally, data that can be processed by the processor 120 cannot be limited to being received through the communication interface 110. For example, when a user's utterance is input through the user interface 140, the processor 120 may process the utterance according to a preset speech processing process.

The processor 120 is an image processing board in which a SOC (system-on-chip) that integrates these various functions or individual chipsets that can independently perform each of these processes are mounted on a printed circuit board. It is implemented as (not shown) and is built into the display device 100.

The display 130 displays the image signal/image data processed by the processor 120 as an image. The implementation method of the display 130 is not limited, and may include liquid crystal, plasma, light-emitting diode, organic light-emitting diode, surface conduction electron gun, etc. It can be implemented in various display methods such as conduction electron-emitter, carbon nano-tube, and nano-crystal.

The display 130 may additionally include additional components depending on its implementation method. For example, in the case of a liquid crystal display, the display 130 includes a liquid crystal display panel (not shown), a backlight unit (not shown) that supplies light to the panel, and a panel driving substrate (not shown) that drives the panel (not shown). poetry), etc.

The user interface 140 transmits various preset control commands or information to the controller 160 according to the user's manipulation or input. The user interface 140 converts various events generated by the user into information according to the user's intention and transmits it to the controller 160. Here, the event generated by the user can take various forms, for example, the user's manipulation, speech, or gesture.

The user interface 140 is provided to respond to the method in which the user inputs information and to detect information input in that method. Accordingly, the user interface 140 can be divided into an interactive input interface 141 and a non-interactive input interface 142.

The interactive input interface 141 is provided to input the user's speech. That is, the interactive input interface 141 is implemented with a microphone and detects various sounds occurring in the external environment of the display device 100. The interactive input interface 141 can generally detect utterances made by a user, but can also detect sounds generated by various environmental factors other than the user.

The non-interactive input interface 142 is provided to allow user input other than speech. In this case, the non-interactive input interface 142 can be implemented in various forms, for example, a remote controller separated from the display device 100 or a menu key installed outside the display device 100. -key) or an input panel, or a motion sensor or camera that detects the user's gestures.

Alternatively, the non-interactive input interface 142 may be implemented as a touch-screen installed on the display 130. In this case, the user can transmit a preset command or information to the controller 160 by touching the input menu or UI image displayed on the display 130.

The storage 150 stores various data under the control of the controller 160. The storage 150 is implemented with non-volatile memory such as flash-memory or hard-disc drive so that data can be preserved regardless of whether system power is provided or not. The storage 150 is accessed by the controller 160 to read, record, modify, delete, update, etc. previously stored data.

The controller 160 is implemented as a CPU and controls the operation of various components of the display device 100, including the processor 120, according to the occurrence of a predetermined event. For example, in the case of a user speech event, when a user's speech is input through the interactive input interface 141, the controller 160 controls the processor 120 to process the input speech. If the user utters a channel number, the controller 160 controls the broadcast video to be switched and displayed according to the uttered channel number.

Here, the process of processing the input utterance may be performed by the display device 100 alone, but may also be performed by sharing it with an external device, taking into consideration various factors such as system load and processing capacity. This configuration will be described below.

Figure 2 is a block diagram of servers 200 and 300 included in the system of this embodiment.

As shown in FIG. 2, the display device 100 is communicatively connected to one or more servers 200 and 300 through a network. Here, the display device 100 includes an STT (speech-to-text) server 200 that converts the user's utterance into a voice command, and an operation/action/result corresponding to the voice command by analyzing the voice command. Each connects to the interactive server 300 that determines . Here, the names of each server (200, 300) are applied only for mutual distinction in consideration of their respective functions, and the names do not limit the structural characteristics of each server (200, 300).

The STT server 200 generates the content of the voice signal as text by analyzing the waveform of the voice signal. Accordingly, when the STT server 200 receives a voice signal from the display device 100, it converts it into a text voice command. Here, an utterance refers to an action performed by the user, and a voice command refers to the content of the utterance converted into data in text format.

The interactive server 300 searches for operations of various display devices 100 corresponding to voice commands. The interactive server 300 analyzes the voice command received from the display device 100, and according to the analysis result, provides a control signal for performing an operation corresponding to the voice command or search result information based on the voice command. Transmitted to the display device 100.

The interactive server 300 includes a server-side communication interface 310 for communicating with the display device 100, and a server-side processor 320 that processes voice commands received through the server-side communication interface 310. Includes.

Under the system using the display device 100, the STT server 200, and the interactive server 300, operations performed when the user of the display device 100 speaks are described below.

When a user's speech is input, the display device 100 transmits a voice signal resulting from the speech to the STT server 200. The STT server 200 converts the voice signal received from the display device 100 into a data voice command and transmits this voice command to the display device 100. The display device 100 transmits the voice command received from the STT server 200 to the interactive server 300.

The interactive server 300 analyzes the contents of the voice command, derives a result corresponding to the voice command, and returns it to the display device 100. For example, when the content of the voice command is "Who is the cast of this week's 'program ABC'?", the interactive server 300 may retrieve "this week", "program ABC", etc. from the internal or external database. A search is performed using the keyword “performer” and the results are obtained.

Here, since the database is structured by keyword category, desired results can be searched using specific keywords. A structured database can be implemented in a variety of ways depending on the design and construction method of the database, so a detailed description of this will be omitted.

However, when the interactive server 300 performs a search using one or more keywords derived from voice commands, multiple search results may appear. As an example of the previous voice command, you can consider the case where this week's broadcast program "program ABC" is broadcast at different times or on different broadcast channels.

In this case, the interactive server 300 needs to select one search result that matches the user's intention from among a plurality of search results for voice commands. As a corresponding method for the interactive server 300 to select one search result, for example, the following two methods are possible.

In the first method, a search result is selected according to a specific rule established by the client of the interactive server 300, that is, the display device 100 itself. For example, if the voice command inquires about the broadcast time of a specific program, and there are a plurality of programs that satisfy this condition, the program to be broadcast most recently may be selected. In the second method, when the user of the display device 100 makes an utterance to select a search result, the interactive server 300 connects the utterance with the previous utterance history to provide a search result that matches the user's original intention. You can choose.

However, in such a system, the user is provided with a variety of input method environments, so even if the display device 100 supports a voice recognition input environment, the user can input by selecting the method that is considered most convenient in each situation among the various input methods. will be carried out. In the environment of a system that supports such multiple input modes, if history is individually managed among the user's multiple inputs, there will be limitations in selecting or deriving a result that reflects the user's original intention. If the user initially inputs a speech and later changes the input method to a different method, but the result that the user originally intended is not provided, the user will feel uncomfortable in the usage environment.

Therefore, in an interactive system environment using the interactive server 300 and the display device 100, there are a plurality of results initially searched by the user's speech, and then the user selects one search result through non-utterance input. If selected, the interactive system needs to provide search results that match the intent of the user's initial utterance.

Here, the system according to the prior art does not comprehensively support history management for each of these multiple input modes. That is, in the interactive server 300 according to the prior art, input by the user's speech of the display device 100, input by the user's gesture, and input by the user's operation of the remote controller (not shown) are individually , and each of these inputs is not managed by linking them to each other.

In this system according to the prior art, consider a state in which the display device 100 displays a plurality of search results received from the interactive server 300 according to the second method. At this time, the user may perform input in a non-verbal manner through the non-interactive input interface 142 instead of speech, but the interactive server 300 according to the prior art provides input between the speech input method and the non-interactive input method. Since the history is not cross-referenced, results that do not match the user's intent are produced.

For example, consider a case where a user utters “Record ‘program ABC’” and a plurality of contents related to ‘program ABC’ are displayed on the display device 100. In the conventional case, when a user selects one utterance, a certain content is normally selected and recording is performed. However, when one is selected through a separate non-utterance input method instead of a utterance, only the content is selected. This is performed, but recording is not performed. This is because the information generated by the user's non-utterance input is not transmitted to the interactive server 300, so the interactive server 300 cannot take action after speech.

Considering this, the following method is proposed according to this embodiment.

According to the user's input, the event generated by the interactive input interface 141 of the display device 100 is referred to as a first event, and the event generated by the non-interactive input interface 142 is referred to as a second event for convenience. . When a first event occurs from the display device 100, the interactive server 300 processes the first event and stores the history of the first event. When a second event occurs, the interactive server 300 processes the first event and stores the history of the first event. Processing of the second event is performed based on the processing history. Here, when the second event occurs, the display device 100 converts the second event into the form of a voice command so that the interactive server 300, which is equipped to recognize the voice command, can recognize the second event and conducts a conversation. It is transmitted to the type server (300).

The interactive server 300 may refer to the history of previously occurring events when determining an action for a predetermined event. The interactive server 300 manages events generated by each of the interactive input interface 141 and the non-interactive input interface 142 of the display device 100 in an integrated manner rather than individually, thereby responding to newly generated events. When determining a corresponding action, previously occurring events may be referred to regardless of whether the interactive input interface 141 or the non-interactive input interface 142 exists.

For example, if there are a plurality of search results corresponding to the user's initial utterance, the interactive server 300 provides the plurality of search results for the user to select. At this time, even if the user inputs an instruction to select one search result in a non-verbal manner, the interactive server 300 manages the integrated management of event input to differentiate between the two even if the new event has a different input method from the previous event. can be interrelated. As a result, the search results desired by the user can be derived and provided to the user.

3 and 4 are flowcharts showing a method by which the interactive server 300 provides search results corresponding to an event to the display device 100.

As shown in FIG. 3, in step S100, the interactive server 300 receives a first event, which is a voice command by speech, from the display device 100.

In step S110, the interactive server 300 analyzes the first event and derives a search keyword.

In step S120, the interactive server 300 performs a search using the derived search keyword and stores the occurrence and processing history of the first event.

In step S130, the interactive server 300 determines whether there are multiple search results.

If it is determined that there are not multiple search results, in step S140, the interactive server 300 transmits the search results to the display device 100 and ends.

On the other hand, if it is determined that there are multiple search results, in step S150, the interactive server 300 transmits the multiple search results to the display device 100 and instructs the display device 100 to select one of the multiple search results. ) to request.

As shown in FIG. 4, in step S160, the interactive server 300 receives a second event including a selection instruction from the display device 100. Here, the second event is generated by a non-verbal user input in the display device 100.

In step S170, the interactive server 300 refers to the previously stored occurrence and processing history of the first event to determine the relationship between the first event and the second event, that is, whether the second event should be processed dependently on the first event. judge. For example, the interactive server 300 determines whether the second event is a selection instruction command for a plurality of search results retrieved by the first event.

If it is determined that the second event is not related to the first event in step S180, the interactive server 300 analyzes and processes the second event independently of the first event in step S190.

On the other hand, if it is determined in step S180 that the second event is a selection instruction for a plurality of search results retrieved by the first event, the interactive server 300 selects the search results according to the second event in step S200. In this process, the interactive server 300 may perform additional searches, depending on whether necessary.

In step S200, the interactive server 300 transmits the final search result according to the first event and the second event to the display device 100.

Hereinafter, the specific internal configuration of the interactive server 300 will be described with reference to FIG. 5.

Figure 5 is a block diagram showing the server-side processor 320 of the interactive server 300 subdivided by function.

As shown in FIG. 5, the server-side processor 320 of the interactive server 300 includes a dialog engine 321 that analyzes voice commands received through the server-side communication interface 310. and a content manager 322 that performs a search based on search keywords received from the conversation engine 321.

The dialogue engine 321 includes a dialogue engine manager 323, a dialogue agent 324, an action agent 325, and a database 326. The division of these components for the server-side processor 320 or the conversation engine 321 is applied for convenience by function, and does not necessarily mean that the server-side processor 320 or the conversation engine 321 must have this division. .

Additionally, the above-described components may be hardware-based, but may also be software-based, or hardware and software may be applied together.

The conversation engine manager 323 transmits an event in the form of a voice command delivered to the conversation engine 321 to the conversation agent 324. The conversation agent 324 analyzes the contents of the event, generates the analysis results as structured data, and returns it to the conversation engine manager 323.

Here, the structured data generated by the dialogue agent 324 is hereinafter referred to as a dialogue frame. The dialogue frame contains the contents of the voice command event, and is data that structures the voice command by category so that other components such as the dialogue engine manager 323, action agent 325, and content manager 322 can decode and refer to it. . Categories and structuring types may be determined in various ways depending on the design method of the server-side processor 320.

The dialog frame includes information about the user's purpose in the voice command, the attributes necessary to achieve that goal, and the values of the attributes. Specific examples of conversation frames will be described later.

The dialogue engine manager 323 transmits the dialogue frame received from the dialogue agent 324 to the action agent 325. The action agent 325 determines an action that matches the dialogue frame received from the dialogue engine manager 323 and returns the judgment result to the dialogue engine manager 323. That is, the action agent 325 determines the action corresponding to the event.

If information search is necessary during the decision process, the action agent 325 may request information search by transmitting the search keyword extracted from the conversation frame to the content manager 322. The content manager 322 performs a search on various types of search databases inside or outside the interactive server 300 using the received keywords and returns the search results to the action agent 325.

When the conversation engine manager 323 receives an action or search result corresponding to an event from the action agent 325, the received result is transmitted to the server-side communication interface 310 to be transmitted to the display device 100.

The database 326 includes all information that can be referenced by the conversation engine manager 323, conversation agent 324, and action agent 325 in the above process. For example, the database 326 provides context information, event history information, and conversation model information for the conversation agent 324 to create a conversation frame, and the action agent 325 determines the action corresponding to the event. It may include action model information, etc.

Under the above structure, a specific example of operation when the interactive server 300 returns a result corresponding to a voice command received from the display device 100 to the display device 100 will be described.

Figures 6 and 7 are exemplary diagrams showing the data transmission relationship between the display device 100 and the interactive server 300, and each component within the interactive server 300.

As shown in FIG. 6, when a user utters the question “Who is the cast of this week's 'Program ABC'?", the display device 100 acquires a voice command corresponding to the utterance. The voice command corresponding to this utterance is referred to as the first event. The display device 100 transmits the first event to the interactive server 300 (410).

The conversation engine manager 323 transmits the first event received through the server-side communication interface 310 to the conversation agent 324 (420).

The conversation agent 324 analyzes the first event and creates a conversation frame corresponding to the first event. When the first event is the content of "Who is on this week's 'program ABC'?", the conversation frame can be expressed as follows.

(1) Purpose: Inquiry about broadcast program performers

(2) Title: program ABC

(3) Time: This week

In the above dialogue frame, “Purpose”, “Title”, and “Time” are the condition categories of the dialogue frame, and each condition category includes the attribute values of “Broadcast Program Cast Inquiry”, “Program ABC”, and “This Week”. do. By analyzing the content of the first event, the conversation agent 324 can derive the category of conditions in which each keyword included in the first event is included and the attribute value for each category.

Upon receiving the dialogue frame created in this way from the dialogue agent 324 (430), the dialogue engine manager 323 transmits the dialogue frame to the action agent 325 (440).

The action agent 325 creates the following keywords based on the received dialogue frame and delivers them to the content manager 322 (450).

(1) Title: program ABC

(2) Time: 2013/06/03~2013/06/09

For attribute values in the time condition category, a specific date must be specified for search. Accordingly, the action agent 325 determines the specific date and time section of this week based on the current time, based on the attribute value of “this week” in the time category of the conversation frame.

The content manager 322 performs a search on a search service for various content and metadata using keywords received from the action agent 325. Search targets include various databases, search services, and content services that the content manager 322 can access.

The content manager 322 delivers the search results obtained through the search to the conversation agent 324 (460). For example, the search results may include a plurality of items as follows.

(1) program ABC / 2013.06.04 17:00 / Channel 4

(2) program ABC / 2013.06.08 10:00 / Channel 20

(3) program ABC / 2013.06.09 20:00 / Channel 35

Each search result includes information about the program title, broadcast time, and broadcast channel, and also includes performer information, although not indicated above. According to the above search results, there are three cases of “program ABC” airing “this week”.

The action agent 325 delivers the search results to the conversation engine manager 323 (470). In this process, the action agent 325 adds a command that causes the display device 100 to selectably display multiple search results since there are multiple search results.

The conversation engine manager 323 transmits search results and commands to the display device 100 (480).

As shown in FIG. 7, the display device 100 displays a plurality of search results so that the user can select them according to a command received from the conversation engine manager 323, and displays the user to select one of the displayed search results. The user is notified by system ignition of the device 100.

Accordingly, the user inputs an instruction to select the first search result, for example, by non-verbal means such as a button operation or a gesture on a remote controller (not shown). This input is referred to as the second event.

In response to the second event, the display device 100 generates a fictitious voice command including the content of the second event. Here, the display device 100 converts the second event into a voice command similar to the first event so that the interactive server 300, which is provided to recognize an event in the form of a voice command by utterance, can recognize the second event. It is converted into a form and transmitted to the interactive server 300. The second event may include, for example, the content “number 1.”

The display device 100 transmits the second event to the conversation engine manager 323 (510).

The conversation engine manager 323 transmits the second event to the conversation agent 324 (520).

Since the second event only contains the content “number 1,” it is difficult for the conversation agent 324 to process the second event based only on the content of the second event. Accordingly, the conversation agent 324 calls processing status information about the first event from the previously performed event processing history and creates the following conversation frame based on the called information.

(1) Purpose: Inquiry about broadcast program performers

(2) Order: 1

In this conversation frame, the target category and its attribute values are not included in the second event, but the conversation agent 324 can create a conversation frame with the above content by referring to the first event related to the second event.

The conversation agent 324 delivers the created conversation frame to the conversation engine manager 323 (530), and the conversation engine manager 323 delivers the conversation frame to the action agent 325 (540).

The action agent 325 selects the first search result among the three search results retrieved by the previous first event based on the attribute value of the order condition category within the conversation frame. The action agent 325 delivers the selection result of “program ABC / 2013.06.04 17:00 / Channel 4” to the conversation engine manager 323 (550).

The conversation engine manager 323 transmits the received selection result to the display device 100 (560). At this time, the dialogue engine manager 323 may additionally transmit the system speech command written by the dialogue agent 324 to the display device 100 with reference to the selection result.

According to the received command, the display device 100 outputs the performer of "program ABC", which will be broadcast on Channel 4 at 17:00 on June 4, 2013, through system speech.

According to the above embodiment, in the system implemented by the display device 100 and the interactive server 300, when there are multiple results that satisfy the goals and conditions input by the user through speech, the user uses a remote controller or a gesture. Even if non-verbal inputs such as the like are mixed, user convenience can be improved by providing results that match the user's original intention.

Meanwhile, in the above embodiment, the case where the interactive server 300 stores the history of events caused by each of the user's speech input and non-speech input of the display device 100 has been described, but the spirit of the present invention This is not limited to this. For example, the display device 100 can be configured to store event history, which will be described below with reference to FIGS. 8 and 9.

Figures 8 and 9 are flow charts showing a control method of the display device 100 according to the second embodiment of the present invention.

As shown in FIG. 8, in step S300, the display device 100 receives the user's speech. In step S310, the display device 100 transmits a voice command corresponding to the user's utterance to the interactive server 300.

In step S320, the display device 100 determines whether there are multiple corresponding results of the voice command received from the interactive server 300.

If there is only one corresponding result of the voice command received from the interactive server 300, the display device 100 displays the corresponding result in step S330.

On the other hand, if there are multiple corresponding results of the voice command received from the interactive server 300, in step S340, the display device 100 displays the plurality of corresponding results so that the user can select them.

In step S350, the display device 100 receives a selection input for a plurality of corresponding results from the user. While the above process is in progress, the display device 100 stores the history of the occurrence and processing of voice command events caused by speech.

As shown in FIG. 9, in step S360, the display device 100 determines whether the received selection input is a speaking method or a non-speaking method.

In this process, the display device 100 refers to the previously stored event history and determines whether the selection input received in step S350 is related to the utterance that occurred in step S300. The display device 100 performs step S360 when it is determined that there is a relationship between the two.

If the received selection input is a speech type, in step S370, the display device 100 transmits a voice command corresponding to the selection input to the interactive server 300 and proceeds to step S400.

On the other hand, if the received selection input is a non-speech type, in step S380, the display device 100 generates a processed voice command corresponding to the content of the non-speech type selection input. In step S390, the display device 100 transmits the created voice command to the interactive server 300.

In step S400, the display device 100 receives a result corresponding to the transmitted selection input from the interactive server 300 and displays it.

Meanwhile, in the above embodiment, a case where the interactive server 300 analyzes and searches for results corresponding to an event occurring in the display device 100 has been described. However, the spirit of the present invention is not limited to this, and an embodiment in which the display device 100 performs at least one function of the STT server 200 and the interactive server 300 in the first embodiment is also possible. .

Figure 10 is a block diagram of the processor 620 of the display device 600 according to the third embodiment of the present invention.

As shown in FIG. 10, the display device 600 according to this embodiment includes a communication interface 610 and a processor 620. As for other components included in the display device 600, the first embodiment can be applied, so detailed descriptions are omitted.

The processor 620 includes a voice processing engine 621 that processes speech input to the display device 600, a dialogue engine 622 that analyzes voice commands corresponding to the speech, and a speech engine 622 that processes speech input to the display device 600. It includes a content manager 623 that performs a search based on the received search keyword. Additionally, the conversation engine 622 includes a conversation engine manager 624, a conversation agent 625, an action agent 626, and a database 627. It should be noted that in the drawings and examples, only the configurations directly related to the spirit of the invention are described, and the illustration and description of other configurations are omitted.

When a user's utterance is input, the voice processing engine 621 transmits the voice signal of the utterance to the STT server 200, receives a voice command corresponding to the utterance from the STT server 200, and sends it to the conversation engine manager 624. deliver it to

Hereinafter, specific operations by the conversation engine manager 624, conversation agent 625, action agent 626, and content manager 623 are substantially the same as the components of the same name described in the previous embodiment, so the description of the previous embodiment Since it can be applied, detailed explanation is omitted.

Alternatively, the voice processing engine 621 of the display device 600 may perform the function of the STT server 200, that is, the function of converting speech into voice commands. In this case, when the user's utterance is input, the voice processing engine 621 converts the utterance into a voice command and transmits it to the conversation engine manager 624.

The above-described embodiments are merely illustrative, and various modifications and equivalent embodiments may be made by those skilled in the art. Therefore, the true technical protection scope of the present invention should be determined by the technical spirit of the invention described in the claims below.

100: display device
110: communication interface
120: processor
130: display
140: user interface
141: Interactive input interface
142: Non-interactive input interface
150: storage
160: controller
200: STT server
300: interactive server
310: Server-side communication interface
320: Server-side processor

Claims (20)

In electronic devices,
display unit;
Ministry of Communications;
An interactive input interface that receives voice commands;
A non-interactive input interface that accepts non-voice commands; and
When a first event related to the voice command is received through the interactive input interface, control the display unit to receive and output a plurality of search results retrieved by the first event from a server through the communication unit,
Receiving a second event related to a non-voice command for text input using the non-interactive input interface,
If the second event is a selection instruction command for selecting one of a plurality of search results retrieved by the first event, controlling the display unit to receive and output a screen corresponding to the selected search result from the server,
If the second event is not a selection instruction command for selecting one of the plurality of search results retrieved by the first event, the result of processing the second event not based on the plurality of search results retrieved by the first event An electronic device comprising a processor that controls the display unit to receive and output from the server. According to paragraph 1,
Whether the second event is the selection instruction command is related to whether the first event and the second event are related. According to paragraph 2,
When the first event and the second event are related, the electronic device is a case in which the second event is processed dependently on the first event. According to paragraph 1,
Based on data received from the server that identifies whether the second event is the selection instruction command, control the display unit to output either a screen corresponding to the selected search result or a result of processing the second event. an electronic device. According to paragraph 1,
The electronic device wherein the text input due to the received second event is a text input converted into the same form as the voice command due to the first event. According to paragraph 1,
The non-voice command is an electronic device generated by manipulation of an external remote controller or gesture input. According to paragraph 1,
Contains more memory,
An electronic device that stores in the memory a previous voice history generated based on a first event related to the voice command. In clause 7,
When the second event is the selection instruction command, the corresponding function cannot be performed with only the second event and the previous voice history is required. According to paragraph 1,
The processor is configured to receive from the server a result corresponding to a plurality of inputs using different input methods including the voice command and the non-voice command and output the display unit to receive and output a result corresponding to the connection between the plurality of input methods including the voice command and the non-voice command. Electronic device that controls. In a control method of an electronic device,
When a first event related to a voice command is received through an interactive input interface, receiving and outputting a plurality of search results retrieved by the first event from a server;
Receiving a second event related to a non-speech command for text input using a non-interactive input interface;
If the second event is a selection instruction command for selecting one of a plurality of search results retrieved by the first event, receiving and outputting a screen corresponding to the selected search result from the server; and
If the second event is not a selection instruction command for selecting one of the plurality of search results retrieved by the first event, the result of processing the second event not based on the plurality of search results retrieved by the first event A control method of an electronic device comprising receiving and outputting from the server. According to clause 10,
Whether the second event is the selection instruction command is related to whether the first event and the second event are related. According to clause 11,
When the first event and the second event are related, the second event is processed dependently on the first event. According to clause 10,
It further includes outputting one of a screen corresponding to the selected search result or a result of processing the second event, based on data received from the server that identifies whether the second event is the selection instruction command. A control method of an electronic device. According to clause 10,
A method of controlling an electronic device in which the text input by the received second event is a text input converted into the same form as the voice command by the first event. According to clause 10,
The non-voice command is a control method of an electronic device generated by manipulation of an external remote controller or gesture input. According to clause 10,
A method of controlling an electronic device further comprising storing a previous voice history generated based on a first event related to the voice command. According to clause 16,
When the second event is the selection instruction command, the corresponding function cannot be performed with only the second event and the previous voice history is required. A non-volatile recording medium readable by a computer on which a program of a method executable by a processor of an electronic device is recorded,
The method includes, when a first event related to a voice command is received through an interactive input interface, receiving and outputting a plurality of search results retrieved by the first event from a server;
Receiving a second event related to a non-speech command for text input using a non-interactive input interface;
If the second event is a selection instruction command for selecting one of a plurality of search results retrieved by the first event, receiving and outputting a screen corresponding to the selected search result from the server; and
If the second event is not a selection instruction command for selecting one of the plurality of search results retrieved by the first event, the result of processing the second event not based on the plurality of search results retrieved by the first event A recording medium comprising the step of receiving and outputting from the server. According to clause 18,
Whether the second event is the selection instruction command is related to whether the first event and the second event are related. According to clause 19,
When the first event and the second event are related, the recording medium is a case where the second event is processed dependently on the first event.

Images