KR102687797B1 - Quality assurance method for consultation service using artifical neural network, and computer program performing the method - Google Patents

Abstract

A method of operating a consultation quality assurance service server is disclosed. The operation method converts a voice file recording the conversation between a customer and a counselor into text, and determines whether the text is a counselor speech text corresponding to the counselor's speech or a customer speech text corresponding to the customer's speech, When the text is a text uttered by the counselor, it is determined whether the text uttered by the counselor contains content that must be uttered to the customer and whether it contains content that is inappropriate to utter to the customer, and according to the result of the judgment, the counselor A quality assurance analysis result for the counselor who uttered the sentence corresponding to the speech text is generated and fed back to the counselor.

Description

Consultation service quality assurance method using an artificial neural network and a computer program capable of performing the method {QUALITY ASSURANCE METHOD FOR CONSULTATION SERVICE USING ARTIFICAL NEURAL NETWORK, AND COMPUTER PROGRAM PERFORMING THE METHOD}

The present invention relates to a consultation service quality assurance method, and particularly to a consultation service quality assurance method using an artificial neural network and a computer program that performs the method.

Speech recognition refers to a process in which a computer or a computer program interprets the spoken language spoken by a person and converts the content of the spoken language into text data. The technology that converts voice into text is called STT (Speech-to-Text) or voice recognition. STT, or speech recognition, is attracting attention as a method of entering text using voice instead of using a keyboard.

Registered Patent Gazette: Registration No. 10-1662383 (announced on October 10, 2016) Public Patent Publication: Publication No. 10-2019-0024324 (published on March 8, 2019) Registered Patent Gazette: Registration No. 10-1970899 (announced on April 24, 2019)

The technical task to be achieved by the present invention is to convert a voice file recording the conversation between a customer and a counselor into text, and to use an artificial neural network to make it necessary for the counselor to utter (or speak) the sentences included in the text to the customer. ) Not only does it determine whether it contains content that needs to be done and whether it contains inappropriate content to utter to the customer, it not only feeds back to the counselor the quality assurance analysis results of the sentence uttered by the counselor to the customer, but also provides feedback to the customer's inquiry. To provide a counseling service quality assurance method and device that can feed back an answer appropriate to a question to the counselor, and a computer program that performs the method.

The operating method of the consultation quality assurance service server according to the present invention includes the steps of converting a voice file recording the conversation between a customer and a counselor into text, and determining whether the text is a counselor speech text corresponding to the counselor's speech or the customer's speech. A step of determining whether the text is a customer speech text corresponding to the text, and when the text is the counselor speech text, whether or not the counselor speech text includes content that must be uttered to the customer and content that is inappropriate to utter to the customer. A step of determining presence or absence of inclusion, and according to the result of the determination of presence or absence, an analysis result for quality assurance for the counselor who uttered the sentence corresponding to the counselor's utterance text is generated, and the analysis result is transmitted to the counselor's computing device. and feeding back the analysis results to the counselor's computing device in real time so that they are displayed on the monitor.

A computer program is recorded on the computer-readable medium according to the present invention to execute the operation method of the consultation quality assurance service server.

The consultation quality assurance service server according to the present invention includes a processor and a storage medium recording a computer program readable by the processor, wherein the storage medium converts a voice file recording a conversation between a customer and a counselor into text. and determining whether the text is a counselor speech text corresponding to the counselor's speech or a customer speech text corresponding to the customer's speech, and when the text is the counselor speech text, sending the counselor speech text to the customer. A step of determining whether content that must be uttered is included and whether content that is inappropriate to utter to the customer is included, and according to the result of the judgment on presence or absence, the counselor who uttered the sentence corresponding to the text uttered by the counselor The computer program generates analysis results for quality assurance and feeds back the analysis results to the counselor's computing device in real time so that the analysis results are displayed on the monitor of the counselor's computing device.

According to the present invention, a computer program combined with hardware and stored in a medium for providing a consultation quality assurance service includes the steps of converting a voice file recording a conversation between a customer and a counselor into text, and converting the text to the counselor's utterance. A step of determining whether the text is a corresponding counselor speech text or a customer speech text corresponding to the customer's speech, and when the text is the counselor speech text, whether or not the counselor speech text includes content that must be uttered to the customer; A step of determining whether content inappropriate to utter to the customer is included, and according to the result of the determination of presence or absence, generating an analysis result for quality assurance for the counselor who uttered a sentence corresponding to the text uttered by the counselor; Feeding back the analysis results to the counselor's computing device in real time so that the analysis results are displayed on the monitor of the counselor's computing device is performed.

According to the present invention, a method of providing a consultation quality assurance service using ARS and a consultation quality assurance service server includes the steps of the consultation quality assurance service server receiving a voice file recording a conversation between a customer and a counselor from the ARS and converting it into text; , the consultation quality assurance service server determining whether the text is a counselor utterance text corresponding to the counselor's utterance or a customer utterance text corresponding to the customer's utterance, and the consultation quality assurance service server determining whether the text is the text uttered by the customer. When the counselor's speech text is a text, determining whether the counselor's speech text contains content that must be uttered to the customer and whether it contains content that is inappropriate to utter to the customer; and the consultation quality assurance service server determines the presence or absence of the text. According to the result of the judgment, an analysis result for quality assurance for the counselor who uttered the sentence corresponding to the text of the counselor's speech is generated, and the analysis result is sent to the counselor so that the analysis result is displayed on the monitor of the counselor's computing device. It includes real-time feedback to the computing device.

A counseling service quality assurance method and device according to an embodiment of the present invention, and a computer program for performing the method convert a voice file recording a conversation between a customer and a counselor into text, and use an artificial neural network to convert the text into text. Determine whether the included sentence contains content that the counselor must utter to the customer and whether it contains content that is inappropriate to utter to the customer, and determine the quality assurance analysis results for the sentence uttered by the counselor to the customer. In addition to the effect of providing feedback to the counselor, it is determined whether the customer's inquiry included in the text is atypical, and based on the judgment result, the answer existing within the quality assurance service server or the answer transmitted from an external database is sent to the counselor. It has the effect of providing feedback to

In order to more fully understand the drawings cited in the detailed description of the present invention, a detailed description of each drawing is provided.
1 is a conceptual diagram illustrating a method for classifying text according to an embodiment of the present invention.
Figure 2 shows a block diagram of a quality assurance service system that performs a consultation service quality assurance method using an artificial neural network according to an embodiment of the present invention.
FIG. 3 is a diagram for explaining the operation of the computer program shown in FIG. 2.

1 is a conceptual diagram illustrating a method for classifying text according to an embodiment of the present invention.

Referring to FIG. 1, the common controller 110 that performs an interfacing (or communication) function with an external device is a voice recording file related to product or service sales (for example, insurance product sales) converted into a script. A text file (also referred to as 'text') is received, and each sentence included in the text file is divided and transmitted to the preprocessor 122 for each sentence.

The REST API interface 120, which performs the function of a handler, includes a pre-processor 122 and a post-processor 124.

The pre-processor 122 receives each sentence included in the text file from the common controller 110. The pre-processor 122 converts the words constituting each sentence included in the text file into vector values using already learned word embedding to use them as input values for the artificial neural network 130. The preprocessor 122 vectorizes each sentence into words and outputs the numbers indexed for each word as the result.

The dimension (IVD) of the input vector value of the TCR (topic classification recognition) model is calculated according to Equation 1.

[Equation 1]

IVD=NP*MN*DS

Here, NP is the number of sentences input at once, MN is the maximum number of morphemes included in one sentence, and DS is the size of the dimension of word embedding of one morpheme.

The sentences vectorized by the preprocessor 122 are input to each layer 142, 144, and 150. TCR first inappropriate layer 1 (142) is a model that checks inappropriate expressions included in each sentence. TCR first inappropriate layer 2 (144) is a model that verifies values output as inappropriate expressions by TCR first inappropriate layer 1 (142).

The TCR section layer 150 is a model that checks the script (or content) that the counselor must notify the counselor (or customer).

Since various inappropriate expressions may exist, in order to implement a model with higher accuracy, the TCR first inappropriate layer 140, unlike the TCR section layer 150, has two layers 142 and 144.

When the result values of each of the TCR first inappropriate layers 142 and 144 are the same, at least one of the TCR first inappropriate layers 142 and 144 transmits the vector value of the corresponding sentence to the TCR second inappropriate layer 160. do.

The TCR second inappropriate layer 160 learns only the appropriate expressions (positive) and inappropriate expressions (negative) included in the received sentence, and finally determines whether the sentence contains an appropriate expression or a sentence containing an inappropriate expression. Judge (or derive)

The NER (Named Entity Recognition; 170) model is an entity name recognition model that classifies the category of a specific word contained in a sentence based on machine learning. The NER (170) model further inspects the sentences classified by the TCR section layer (150) by classifying them into categories by word.

Because each sentence in each consultation transcript passes through all layers 140, 150, and 160, each sentence has a unique key value.

The post processor 124 receives the output of the TCR second inappropriate layer 160 and the output of the NER model 170, and determines whether there is a sentence containing an inappropriate expression for each counseling transcript based on the unique key value and whether the counselor determines whether there is a sentence containing an inappropriate expression. Check for omissions in scripts (or content) that must be provided to the counselee.

Through the post processor 124, each consultation transcript is finally classified as normal, supplementary, or warning.

The results predicted through the sales quality assurance model explained with reference to FIG. 1 are delivered to the sales quality assurance team of a company (for example, an insurance company) and used as reference indicators when checking numerous consultation transcripts to It allows you to check a much larger amount of consultation transcripts compared to the amount of time spent. Through this, it can be used to reduce the company's losses or replace the position of the sales quality assurance team and automate the sales quality assurance system.

Figure 2 shows a block diagram of a quality assurance service system that performs a consultation service quality assurance method using an artificial neural network according to an embodiment of the present invention.

The quality assurance service system 200 includes a telephone 215 of a customer (also referred to as a 'counseee') 210, a telephone 221 of a counselor (or counselor; 220), a computing device 223 of the counselor 220, and an ARS. (Automatic Response System; 230), and a quality assurance service server (240). Each phone 215 and 221 may be a wired phone or a wireless phone (including, for example, a smart phone). ARS (230) refers to a recording device that records the conversation between the customer (210) and the counselor (220) and generates a recorded voice file (SF).

The quality assurance service system 200 includes a sales (or agent) quality assurance service system, an artificial intelligence-based analysis system, and a call center (e.g., an inbound call center and/or an outbound call center). ) or a contact center.

Quality assurance service system 200 further includes a media or data storage device (e.g., database; 250) accessible by quality assurance service server 240.

The contents of the consultation (i.e., the contents of the phone call) exchanged between the customer 210 and the counselor 220 using the phones 215 and 221 are automatically recorded by the ARS 230, and the recorded voice file (SF) is stored in a data storage device (e.g., database; 235) accessible by the ARS device 230 and the quality assurance service server 240.

Depending on embodiments, the recorded voice file (SF) may be transmitted (or streamed) to the quality assurance service server 240 automatically or in real time through a communication network.

The quality assurance service server 240 includes a processor 241, memory 243, and interface 248.

*The processor 241 can run a computer program that guarantees the counselor's counseling quality and provides recommended answers. A computer program is stored in a medium (e.g., memory 243) coupled with hardware (e.g., quality assurance service server 240 or processor 241) to provide consultation quality assurance and recommended answers; The hardware may perform steps executed by the computer program described herein.

The computer program provides instructions (or program codes) for hardware (e.g., 240 or 241) to perform each step performed (or executed) by each module shown and described in FIG. 3 described herein. ) includes. Each module 310 to 380 described in this specification refers to a functional set and/or structural set of instructions (or program codes).

The interface 245, which performs the function of a transmitting and receiving device, is the first interface necessary for the ARS device 230 and the quality assurance service server 240 to transmit or receive information, and the computing device of the quality assurance service server 240 and the counselor 220. It collectively refers to the second interface necessary for 223 to give or receive information, and the third interface necessary for the quality assurance service server 240 and the database 250 to give or receive information.

The computing device 223 of the counselor 220 is a display device that receives and displays information (e.g., information corresponding to QAR, EAA, and/or ANS in FIG. 3) transmitted from the quality assurance service server 240. (e.g., monitor 225). Computing device 223 includes a PC or a mobile device, such as a laptop computer, a mobile internet device (MID), or an enterprise digital assistant (EDA).

FIG. 3 is a diagram for explaining the operation of the computer program shown in FIG. 2.

The common controller 110 of FIG. 1 corresponds to the interface 245 of FIG. 3, the pre-processor 122 of FIG. 1 performs the functions of the modules 310, 320, and 330, and the post processor of FIG. 1 124 performs the function of the post-processing module 340 of FIG. 3.

The drawing number 130 in FIG. 1 represents the essential keyword recognition module 332 and the inappropriate expression recognition module 334 in FIG. 3 in more detail, and the essential keyword recognition module 332 in FIG. 3 has the configuration of FIG. 1. 150 and 170 may perform the functions of components 150 and 170, and the inappropriate expression recognition module 334 of FIG. 3 may perform the functions of components 140 and 160 of FIG. 1.

Each TCR (Topic Classification Recognize; 142, 144, 162, 164, and 166) in FIG. 1 is a deep learning model that distinguishes what type of text the input text is.

The components 140 and 160 corresponding to the inappropriate expression recognition module 334 go through a two-step filtering process to recognize (or determine) whether the expression contained in the sentence included in the input text is an inappropriate expression. . Each configuration 140 and 160 includes a plurality of TCR models.

The components 150 and 170 corresponding to the essential keyword recognition module 332 are components for determining (or recognizing) which essential keywords are present in sentences included in the input text.

The TCR section layer (150) for classifying required notices included in the input text is a TCR model that classifies which part of the required notices the input text is, and the NER model (170) for recognizing required keywords. is a NER model that determines (or recognizes) the presence or absence and location of essential keyword(s) that may be included in the text output from the TCR section layer 150.

Referring to Figures 2 and 3, the speech-to-text (STT) module 310 is a conversation between the customer 210 and the counselor 220 transmitted (e.g., transmitted in real time) from the ARS 230. A voice file (SF) with recorded content is received and converted into text (TF).

The STT module 310 learns by itself a large number of voice-text sets that match each voice with each text, and converts each of the sentences included in the voice file (SF) into texts based on the learning results. (TF) uses the STT deep learning model to convert each. Here, model may refer to an algorithm or software that implements the algorithm.

When a voice file (SF) for product (or service) consultation (or sales) is input to the STT module 310 through the interface 245, the STT module 310 sends texts corresponding to the voice file (SF) ( TF) is generated and output to the speaker separation module 320.

The speaker separation module 320 separates the utterances of the customer 210 from texts (TF) in which the utterances of the customer 210 (also referred to as 'words') and the utterances of the counselor 220 are mixed together. The corresponding text and the text corresponding to the counselor's utterance (220) are separated.

The speaker separation module 320 learns consultation log data that is pre-tagged in each text corresponding to the customer's utterance and the text corresponding to the counselor's utterance using a deep learning model.

The speaker separation module 320 divides each of the texts (TF) into utterances of the customer 210 based on consultation log data (e.g., insurance product sales consultation log data, etc.) already learned with a deep learning model. It is classified into text and text corresponding to the counselor's utterance (220). It is assumed that one text contains one sentence.

The speaker separation module 320 transmits the counselor utterance texts (CGP) classified as texts corresponding to the utterances of the counselor 220 among the texts (TF) to the topic classification module 330, and Customer utterance texts (CQP) classified as texts corresponding to utterances of the customer 210 are transmitted to the query element extraction module 350 and the meta vector extraction module 370.

The topic classification module 330 is an artificial intelligence module that classifies each of the counselor speech texts (CGP) as required notice text (CGP1) or inappropriate expression text (CGP2).

For example, in order to prevent misselling, mandatory disclaimers are required to be provided (or to be provided) by a counselor (or seller) recommending the purchase of a financial product or service, such as insurance or funds, to a customer who is the purchaser. It means necessary and sufficient information (or content).

Improper selling means that a financial product solicitor (e.g., counselor) sells the product without properly informing the consumer (e.g., customer) of the risk and possibility of loss of the product. In other words, when selling a financial product to a customer who is an investor, a financial company counselor ensures that the customer understands key details (e.g., required notices), such as the basic structure of the financial product, fund management method, and whether there is a loss of principal. There is a duty to explain. A sale that violates these obligations is an incomplete sale.

The mandatory notice text (CGP1) is a text containing essential notice(s) that the counselor 220 must utter to the customer 210, and the inappropriate expression text (CGP2) is a text that the counselor 220 must utter to the customer 210. ) is a text that contains expressions that are inappropriate to speak to people (for example, sales speech, etc.).

The topic classification module 330 uses sentences or expressions included in pre-learned counseling texts to classify each of the counselor speech texts (CGP) into one of the required notice text (CGP1) and the inappropriate expression text (CGP2). Classified into one, the required notice text (CGP1) is transmitted to the keyword recognition module 332, and the inappropriate expression text (CGP2) is transmitted to the inappropriate expression recognition module 334.

Depending on the embodiment, the topic classification module 330 classifies each of the counselor speech texts (CGP) as required notice text (CGP1) or inappropriate expression text (CGP2) using a sentence splitter deep learning model. .

According to embodiments, the topic classification module 330 is configured to classify the classification target text as the above through a classification deep learning model that learns existing counseling texts (or sentences included in the counseling texts). It is possible to predict which text matches among existing counseling texts and output the prediction result.

The essential keyword recognition module 332, which utilizes the NER (Named-Entity Recognition) artificial intelligence model, allows the counselor 220 to identify essential notices among the required notices in the required notice text (CGP1) transmitted from the subject classification module 330. Determine whether a word or keyword (or keywords) that needs to be uttered exists.

The essential keyword recognition module 332 is a NER that learns how many essential keywords (entities) are in the mandatory notice text (CGP1) transmitted from the topic classification module 330 and where the essential keywords are located in the sentence. Uses a deep learning model. The essential keyword recognition module 332 uses the NER deep learning model to determine how many essential keywords exist in the required notice text (CGP1) and the location of the required keywords, and sends the result of the judgment to a post-processing module ( 340).

The inappropriate expression recognition module 334 receives the inappropriate expression text (CGP2) transmitted from the topic classification module 330, determines whether there is an inappropriate expression in the inappropriate expression text (CGP2), and sends the result of the judgment to a post-processing module ( 340).

The inappropriate expression recognition module 334 learns texts containing inappropriate expressions using a classification deep learning model. The inappropriate expression recognition module 334 predicts whether an inappropriate expression (or expressions) exists in the inappropriate expression text (CGP2) using a classification deep learning model, and outputs the prediction result to the post-processing module 340.

The post-processing module 340 receives the judgment result of the essential keyword recognition module 332 and the judgment result of the inappropriate expression recognition module 334, and uses the received results to convert each of the counselor speech texts (CGP) to the required notice. Is it text (CGP1) or inappropriately worded text (CGP2), how many required keywords are included in the required disclaimer text (CGP1) and where in the sentence are those required keywords located, and/or inappropriately expressed text (CGP2)? Determine whether there are inappropriate expressions in .

According to embodiments, the post-processing module 340 uses the judgment result of the essential keyword recognition module 332 to check whether the counselor 220 has uttered all the content that must be essentially notified to the customer 210 during the real-time consultation. You can check it.

The post-processing module 340 uses the judgment result of the essential keyword recognition module 332, If there are keywords that must be uttered to the customer 210 in the required notice text (CGP1), it is possible to check whether the counselor 220 has uttered all the required keywords to the customer 210 during real-time consultation.

The post-processing module 340 uses the judgment result of the inappropriate expression recognition module 334 to determine whether the counselor 220 made an inappropriate expression to the customer 210 during a real-time consultation, and determines whether the counselor 220 made an inappropriate expression to the customer (210). 210) If an inappropriate expression is uttered, action may be taken.

The post-processing module 340 determines whether there are sentences that the counselor 220 speaking to the customer 210 failed to utter (or omitted) to the customer 210 and whether the necessary notices required for the contract were uttered to the customer 210. , it is determined whether the essential keywords required for the contract were uttered to the customer 210 and/or the counselor 220 made an inappropriate expression to the customer 210, and the judgment result (QAR) is sent to the counselor QC visualization module 245- Output as 1). The post-processing module 340 generates and outputs an analysis result (QAR) for quality assurance of the counselor 220's utterance for each sentence.

The counselor QC (Quality Control) visualization module 245-1 determines the presence or absence of sentences omitted by the counselor 220 according to the quality assurance analysis result (QAR) for the utterance output from the post-processing module 340. The presence or absence of missing required notices, the presence or absence of missing keywords, and/or the presence or absence of inappropriate expressions can be fed back to the counselor 220 in real time (i.e., the analysis results (QAR) are transmitted to the monitor 225). Content fed back to the counselor 220 is displayed on the monitor 225 of the computing device 223.

In order to determine the intention of the customer's utterance (or query) 210, the query element extraction module 350 extracts one or more query elements (e.g., words, intent, and/or emotion, etc.).

For example, the query element extraction module 350 may apply a named entity recognition (NER) deep learning model, a speech act recognition (SAR) deep learning model, and/or text emotion recognition (TER) to each of the customer speech texts (CQP). A deep learning model is applied to extract one or two or more query elements for each customer utterance text (CQP).

The NER deep learning model is an example of a deep learning model that extracts a word (or words) included in a sentence corresponding to a query from a customer 210, and the SAR deep learning model is an example of a deep learning model that extracts a word (or words) included in a sentence corresponding to a query from a customer 210. is an example of a deep learning model that extracts the querying intent of the customer 210, and the TER deep learning model is an example of a deep learning model that extracts the emotions of the querying customer 210.

Simultaneously or in parallel with the operation of the query element extraction module 350, the meta vector extraction module 370 applies a NER deep learning model, a SAR deep learning model, and/or a TER deep learning model to each customer utterance text (CQP). By applying this, a customer utterance meta vector value (MV1) corresponding to each customer utterance text (CQP) is generated and output to the meta vector similarity comparison module 380.

The meta vector extraction module 370 receives question-answer pairs (QnA) from the database 250, and applies a NER deep learning model, a SAR deep learning model, and/or to each of the queries included in the question-answer pairs (QnA). Alternatively, apply the TER deep learning model to generate query meta vector values (MV2) corresponding to each of the above queries, and compare the query meta vector values (MV2) and question-answer pairs (QnA) to the meta vector similarity comparison module 380. ) is output.

The suitable answer search module 360 provides a suitable answer to be provided to the counselor 350 who must answer the inquiry of the customer 210 based on one or two or more query elements (QE) extracted by the query element extraction module 350. Search for (or extract) answers. It is assumed that the appropriate answer search module 360 can include (or use) answers corresponding to the extracted query elements (QE).

The suitable answer search module 360 classifies the query type of the query elements (QE) extracted by the query element extraction module 350, and selects a query corresponding to the query type among pre-built query-answer pairs. Answer pairs are searched, and the answers (EAA) included in the searched question-answer pairs are output to the counselor recommended answer visualization module 245-2.

When a suitable answer (EAA) to be provided to the counselor 220 is searched for by the suitable answer search module 360, the suitable answer search module 360 displays the searched answer (EAA) in the counselor recommended answer visualization module 245-2. Output as

For example, when the customer 210's query is normal, the appropriate answer search module 360 does not refer to the question-answer pair (QnA) transmitted from the database 250 and provides an answer corresponding to the query by itself ( EAA) can be provided to the counselor (220).

However, if a suitable answer to be provided to the counselor 220 is not searched for by the suitable answer search module 360 (for example, the query of the customer 210 is abnormal, the suitable answer search module 360 (when the answer corresponding to the query cannot be provided to the counselor 220 on its own without referring to the question-answer pair (QnA) stored in the database 250), the appropriate answer search module 360 searches (or extracts) ) Output a failure message (FLR) to the meta vector similarity comparison module 380. A search failure message (FLR) indicates that the customer's query is atypical.

The meta-vector similarity comparison module 380 receives customer utterance meta-vector values (MV1), query meta-vector values (MV2), and question-answer pairs (QnA), and queries only when a search failure message (FLR) is received. By comparing each of the meta vector values (MV2) with the customer utterance meta vector value (MV1), the query with the most similar value to the customer utterance meta vector value (MV1) is selected among the query meta vector values (MV2) and paired with the selected query. The answers (ANS) that make up are output to the counselor's recommended answer visualization module (245-2) as answers to similar questions.

For example, when the number of question-answer pairs (QnA) output from the database 250 is 100, the meta vector extraction module 370 generates a NER deep learning model, a SAR deep learning model, and Apply the TER deep learning model to generate 100 query meta vector values (MV2).

For example, when the query metavector value for the 50th question-answer pair (QnA) from 1 to 100 has the value most similar to the customer utterance metavector value (MV1), the metavector The similarity comparison module 380 outputs the response (ANS) included in the 50th question-answer pair as a recommended answer to the counselor recommended answer visualization module 245-2.

When the search failure message (FLR) is not received, the meta-vector similarity comparison module 380 generates customer utterance meta-vector values (MV1), query meta-vector values (MV2), and question-answer pairs (QnA) for the customer utterance text. throw away At this time, the appropriate answer search module 360 outputs the searched answer (EAA) to the counselor recommended answer visualization module 245-2.

The counselor recommended answer visualization module 245-2 receives the searched answer (EAA) output from the suitable answer search module 360 or the similar question answer (ANS) output from the metavector similarity comparison module 380 and monitors ( Feedback is provided to the counselor (220) through 225).

Therefore, the counselor 220 can receive real-time feedback on the answer (EAA or ANS) provided by the computer program 300 to the customer 210's inquiry through the UI or UX screen configuration during the consultation with the customer 210. there is.

The present invention has been described with reference to the embodiments shown in the drawings, but these are merely illustrative, and those skilled in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true scope of technical protection of the present invention should be determined by the technical spirit of the attached registration claims.

200: Quality assurance service system 210: Customer or consultee
215: Customer phone 220: Sangsang company
221: Agent phone 223: Agent computing device
230: ARS (Automatic Response System; 230) 240: Quality assurance service server
241: Processor 243: Memory
245: Interface 300: Computer program
310: STT module 320: Speaker separation module
330: Topic separation module 332: Required keyword recognition module
334: Inappropriate expression recognition module 340: Post-processing module
350: Query element extraction module 360: Appropriate answer search module
370: Metavector extraction module
380: Metavector similarity comparison module

Claims (13)

STT module that converts voice files recording conversations between customers and counselors into text;
a speaker separation module that extracts a customer speech text corresponding to the customer's speech from the text;
a query element extraction module that receives the customer speech text and extracts query elements;
Meta-vector extraction for receiving the customer speech text and extracting customer speech meta-vector values corresponding to the customer speech text, and receiving question-answer pairs from a database to extract query meta-vector values corresponding to the question-answer pairs. module;
The extracted query elements are received, the query types of the query elements are classified, and the first answer corresponding to the query type is searched. If the first answer is determined to be impossible, a failure message is sent to the meta-vector similarity comparison module. Passing the appropriate answers to the navigation module; and
In response to receiving the failure message, receive the customer utterance meta vector value, the query meta vector values, and the question-answer pairs, and compare the query meta vector values with the customer utterance meta vector value to determine a final query. , the meta-vector similarity comparison module for determining a second answer corresponding to the final query based on the question-answer pairs.
Including,
The meta-vector similarity comparison module is
Includes at least one of a Named Entity Recognition (NER) deep learning model, a Speech Act Recognition (SAR) deep learning model, and a Text Emotion Recognition (TER) deep learning model, and when the failure message is not received, the customer utterance meta vector refuse to receive the value, the query meta-vector values and the question-answer pairs,
Simultaneously with the operation of the query element extraction module, the meta-vector extraction module extracts the customer utterance meta-vector value and transmits the customer utterance meta-vector value to the meta-vector similarity comparison module.
According to paragraph 1,
The speaker separation module is
From the text, extract the counselor utterance text corresponding to the counselor's utterance,
Sentence splitter A topic classification module that classifies the counselor's utterance text as required notice text or inappropriate expression text using a deep learning model.
Consultation quality assurance service server, including further.
According to paragraph 2,
A keyword recognition module that receives the required notice text and extracts information about the number and location of required keywords present in the required notice text using the NER deep learning model.
Consultation quality assurance service server, including further.
According to clause 3,
An inappropriate expression recognition module that receives the inappropriate expression text and predicts the presence or absence of inappropriate expressions in the inappropriate expression text using a classification deep learning model.
Consultation quality assurance service server, including further.
According to clause 4,
A post-processing module that receives information about the number and location of the essential keywords received from the keyword recognition module and the presence or absence of the inappropriate expressions received from the inappropriate expression recognition module, and outputs a decision result.
Consultation quality assurance service server, including further.
According to clause 4,
The inappropriate expression recognition module includes a plurality of TCR models,
The dimension of the input vector value of each of the TCR models is determined based on the number of sentences input at once, the maximum number of morphemes included in one sentence, and the size of the dimension of word embedding of one morpheme. Consultation quality assurance service. server.
Converting a voice file recording the conversation between the customer and the counselor into text;
extracting a customer speech text corresponding to the customer speech from the text;
Inputting the customer speech text into a query element extraction module to extract query elements from the customer speech text;
Inputting the customer speech text into a meta vector extraction module, and extracting a customer speech meta vector value corresponding to the customer speech text;
Receiving question-answer pairs from a database;
Inputting the question-answer pairs into the meta-vector extraction module to extract query meta-vector values corresponding to the question-answer pairs;
Inputting the extracted query elements into a suitable answer search module, classifying query types of the query elements, and searching for whether there is a first answer corresponding to the query type;
If the appropriate answer search module determines that the first answer is impossible, transmitting a failure message to the meta vector similarity comparison module;
The meta-vector similarity comparison module, in response to receiving the failure message, receives the customer utterance meta-vector value, the query meta-vector values, and the question-answer pairs;
The meta-vector similarity comparison module compares the query meta-vector values with the customer utterance meta-vector values to determine a final query, and determines a second answer corresponding to the final query based on the question-answer pairs; and
Providing the second answer to the counselor's computing device
Including,
Simultaneously with the operation of the query element extraction module, the meta-vector extraction module extracts the customer utterance meta-vector value and transmits the customer utterance meta-vector value to the meta-vector similarity comparison module.
In clause 7,
extracting a counselor utterance text corresponding to the counselor's utterance from the text; and
A step of classifying the text uttered by the counselor as a required notice text or an inappropriately expressed text using a sentence splitter deep learning model.
A method of operating a consultation quality assurance service server, further comprising:
According to clause 8,
Receiving the required notice text and extracting information about the number and location of required keywords present in the required notice text using the NER deep learning model.
A method of operating a consultation quality assurance service server, further comprising:
According to clause 9,
Receiving the inappropriate expression text from an inappropriate expression recognition module and predicting the presence or absence of inappropriate expressions present in the inappropriate expression text using a classification deep learning model.
A method of operating a consultation quality assurance service server, further comprising:
According to clause 10,
Receiving information on the number and location of the essential keywords and the presence or absence of the inappropriate expressions received from the inappropriate expression recognition module, and outputting a decision result.
A method of operating a consultation quality assurance service server, further comprising:
According to clause 10,
The inappropriate expression recognition module includes a plurality of TCR models,
The dimension of the input vector value of each of the TCR models is determined based on the number of sentences input at once, the maximum number of morphemes included in one sentence, and the size of the dimension of word embedding of one morpheme. Consultation quality assurance service. How the server operates.
A computer program combined with hardware and stored in a medium to execute the method of any one of claims 7 to 12.