KR102594629B1 - Apparatus for including speaking scoring artificial intelligence model to evaluate responses of answer to evaluation question - Google Patents

Abstract

According to the present invention, an apparatus including a speaking scoring artificial intelligence model for evaluating an evaluator's response to an evaluation question may include: a memory including a speaking scoring artificial intelligence model trained to determine an evaluation score of each of a pronunciation evaluation item for evaluating pronunciation, a sentence composition strength evaluation item for evaluating sentence composition strength, and a subject suitability evaluation item for evaluating subject suitability among a plurality of evaluation items for a speaking ability of an evaluator; and a processor configured to obtain evaluation data that is the evaluator's response to the evaluation question for evaluating the speaking ability, input the evaluation data to the speaking scoring artificial intelligence model as input data, and receive scoring data that is a result of determining the evaluation score of the evaluator from the speaking scoring artificial intelligence model as output data, wherein the pronunciation evaluation item may include one or more of a first pronunciation evaluation sub-item for evaluating pronunciation fluency, a second pronunciation evaluation sub-item for evaluating pronunciation accuracy, and a third pronunciation evaluation sub-item for evaluating an utterance speed, the sentence composition evaluation item may include a first sentence composition evaluation sub-item for evaluating a composition ability of a sentence, and the subject suitability evaluation item may include a first subject suitability evaluation sub-item for evaluating a subject matching degree.

Description

Apparatus for including speaking scoring artificial intelligence model to evaluate responses of answer to evaluation question}

The present invention relates to an apparatus including a speaking scoring artificial intelligence model for evaluating an evaluator's responses to evaluation items, and more specifically, a pronunciation evaluation item and a sentence composition evaluation item among a plurality of evaluation items for the evaluator's speaking ability. and a speaking scoring artificial intelligence model for evaluating the ratee's responses to the evaluation items, which can evaluate the ratee's responses to the evaluation questions using a speaking scoring artificial intelligence model learned to determine the evaluation score of the subject suitability evaluation item. It is about a device that includes.

As of 2022, there are approximately 356,000 applicants for the Test of Proficiency in Korean (TOPIK), implemented by the National Institute for International Education, from 81 countries. Currently, the TOPIK speaking evaluation requires applicants to visit the test site in person and is only being conducted in Korea.

Accordingly, a very burdensome and cumbersome form of evaluation continues for learners living abroad.

The one-time TOPIK speaking assessment fee is about 80,000 won, and it takes about 3 months or more from test application to administration and score announcement. Training is periodically conducted to train professional graders, and the test is conducted for 8 hours a day over two days. In the form of workshops, it still has many problems, including not only cost issues, but also security issues with evaluation criteria and grader management issues.

In addition, each time an evaluation is conducted, a grading fee must be paid to the evaluator, and the task requires high-skilled personnel with a certain level of experience in Korean language education and evaluation, which not only consumes a lot of money, but also raises issues about the fairness and objectivity of the evaluation.

In particular, the grading standards and distribution points for speaking evaluations are different for each institution (based on language institutes at university institutions), and there is a large difference in level even for the same grade depending on the institution to which the learner belongs, causing confusion among learners themselves about their own level.

Korean Patent Publication No. 10-2022-0032940

The purpose of the present invention is to use a speaking scoring artificial intelligence model learned to determine the evaluation scores of pronunciation evaluation items, sentence composition evaluation items, and topic suitability evaluation items among a plurality of evaluation items for the speaking ability of the person being evaluated. The aim is to provide a device that includes a speaking scoring artificial intelligence model for evaluating the appraisee's responses to evaluation questions, which can evaluate the appraisee's responses.

The objects of the present invention are not limited to the objects mentioned above, and other objects and advantages of the present invention that are not mentioned can be understood by the following description and will be more clearly understood by the examples of the present invention. Additionally, it will be readily apparent that the objects and advantages of the present invention can be realized by the means and combinations thereof indicated in the patent claims.

The device including a speaking scoring artificial intelligence model for evaluating the evaluator's response to the evaluation items according to the present invention includes a pronunciation evaluation item and sentence composition ability, which are evaluation items for evaluating pronunciation, among a plurality of evaluation items for the evaluator's speaking ability. A memory including a speech scoring artificial intelligence model trained to determine evaluation scores for each of the sentence composition evaluation items that are evaluated and the topic suitability evaluation item that evaluates topic suitability; And obtaining evaluation data that is the ratee's response to the evaluation questions for evaluating the speaking ability, inputting it as input data to the speaking scoring artificial intelligence model, and determining the evaluation score of the ratee from the speaking scoring artificial intelligence model. A processor that outputs the resulting scoring data as output data, wherein the pronunciation evaluation items include a first pronunciation evaluation sub-item for evaluating pronunciation fluency, a second pronunciation evaluation sub-item for evaluating pronunciation accuracy, and a third pronunciation evaluation sub-item for evaluating speech speed. It includes one or more of the pronunciation evaluation sub-items, wherein the sentence composition ability evaluation item includes a first sentence composition ability evaluation sub-item that evaluates sentence composition ability, and the topic suitability evaluation item includes a first topic suitability evaluation item that evaluates the degree of topic matching. May include sub-items.

The processor provides training evaluation data, which is evaluation data of other assessees for training evaluation items, which are evaluation items prepared for learning of the speaking scoring artificial intelligence model, and training evaluation scores that are a result of scoring by a rater based on the training evaluation data. The scoring data can be matched to form training data, and the speaking scoring artificial intelligence model can be learned using the training data.

The processor further configures training item type information indicating the type of the training evaluation item corresponding to the training evaluation data with the training data, and confirms evaluation item type information indicating the type of the evaluation item corresponding to the evaluation data. Then, the evaluation item type information along with the evaluation data can be input as the input data to the speaking scoring artificial intelligence model, and the scoring data of the assessee can be output as the output data from the speaking scoring artificial intelligence model.

If the item type is a type that describes a photo or picture provided to the appraisee, the processor acquires the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data and speaks the appraisee using the input data. The scoring data is voice characteristic difference data representing the difference in voice characteristics between the native speaker's voice data and the second ratee's voice data that is input to the scoring artificial intelligence model and read along with the original text data converted into text from the second ratee's voice data. can be output as the output data from the speaking scoring artificial intelligence model.

If the item type is a type that describes a photo or picture provided to the appraisee, the processor acquires the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data and speaks the appraisee using the input data. It is input to a scoring artificial intelligence model, and the scoring data, which is pronunciation difference data representing the pronunciation difference between the pronunciation data corresponding to the original text data converted into text from the second evaluator's voice data and the second evaluator's voice data, is input to the speaking scoring artificial intelligence. The above output data can be output from the intelligent model.

If the item type is a type that describes a photo or picture provided to the appraisee, the processor acquires the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data and speaks the appraisee using the input data. It can be input to the scoring artificial intelligence model, and the scoring data, which is voice speech rate data indicating the speech speech rate of the second evaluator's speech data, can be output as the output data from the speech scoring artificial intelligence model.

If the item type is a type that describes a photo or picture provided to the appraisee, the processor acquires the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data and speaks the appraisee using the input data. The scoring data is input to a scoring artificial intelligence model and is sentence composition data indicating dependency between words in sentences included in the second evaluator's voice data and sentence pattern suitability of the sentences included in the second evaluator's voice data. The above output data can be output from the scoring artificial intelligence model.

If the item type is a type that describes a photo or picture provided to the appraisee, the processor acquires the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data and speaks the appraisee using the input data. It can be input to the scoring artificial intelligence model, and the scoring data, which is topic agreement rate information indicating the match rate between words included in the second evaluator's speech data and words included in the answer data, can be output as the output data from the speaking scoring artificial intelligence model. there is.

According to the present invention, among a plurality of evaluation items for the speaking ability of the appraisee, the appraisee evaluates the evaluation items using a speaking scoring artificial intelligence model that has been learned to determine the evaluation scores of the pronunciation evaluation item, sentence composition evaluation item, and topic suitability evaluation item. By evaluating the responses, it is possible to automatically, quickly and accurately evaluate pronunciation, sentence composition, and topic appropriateness among speaking abilities.

Figure 1 is a configuration diagram of an apparatus including a speaking scoring artificial intelligence model for evaluating an evaluator's responses to evaluation questions according to an embodiment of the present invention.
Figure 2 is a system configuration diagram of a device including a speaking scoring artificial intelligence model for evaluating an evaluator's responses to evaluation questions according to an embodiment of the present invention.
Figure 3 is a configuration diagram of an apparatus including a speaking scoring artificial intelligence model for evaluating an evaluator's response to an evaluation item according to an embodiment of the present invention, which further includes an input/output unit.
Figure 4 is a diagram illustrating a process in which a speaking scoring artificial intelligence model is learned by a device including a speaking scoring artificial intelligence model for evaluating an evaluator's response to an evaluation item according to an embodiment of the present invention.
Figure 5 shows a scoring result in which an apparatus including a speaking scoring artificial intelligence model for evaluating an evaluator's response to an evaluation item according to an embodiment of the present invention determines the evaluation score of the evaluator using the speaking scoring artificial intelligence model. This is a diagram to explain the process of outputting data.
FIG. 6 is a diagram illustrating an example of evaluation items that can be evaluated by an apparatus including a speaking scoring artificial intelligence model for evaluating an evaluator's response to an evaluation item according to an embodiment of the present invention.
7 shows an apparatus including a speaking scoring artificial intelligence model for evaluating the evaluator's response to the evaluation items according to an embodiment of the present invention, and calculates the evaluation score of the pronunciation evaluation item among a plurality of evaluation items for the evaluator's speaking ability. This is a drawing to explain the decision-making process.
Figure 8 shows an apparatus including a speaking scoring artificial intelligence model for evaluating the evaluator's responses to evaluation items according to an embodiment of the present invention, which evaluates pronunciation evaluation items and sentence composition ability among a plurality of evaluation items for the evaluator's speaking ability. This is a diagram to explain the process of determining the evaluation score for each item and topic suitability evaluation item.

Hereinafter, various embodiments of the present invention are described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, and should be understood to include various modifications, equivalents, and/or alternatives to the embodiments of the present invention. In connection with the description of the drawings, similar reference signs may be used for similar components.

In this document, expressions such as “have,” “may have,” “includes,” or “may include” refer to the existence of the corresponding feature (e.g., a numerical value, function, operation, or component such as a part). , and does not rule out the existence of additional features.

In this document, expressions such as “A or B,” “at least one of A or/and B,” or “one or more of A or/and B” may include all possible combinations of the items listed together. . For example, “A or B,” “at least one of A and B,” or “at least one of A or B” (1) includes at least one A, (2) includes at least one B, or (3) It can refer to all cases including both at least one A and at least one B.

As used herein, expressions such as "first", "second", "first", or "second" may modify various elements in any order and/or importance, and may refer to one element. It is only used to distinguish from other components and does not limit the components. For example, a first user device and a second user device may represent different user devices regardless of order or importance. For example, a first component may be renamed a second component without departing from the scope of rights described in this document, and similarly, the second component may also be renamed to the first component.

A component (e.g., a first component) is “(operatively or communicatively) coupled with/to” another component (e.g., a second component). When “connected to” is mentioned, it should be understood that any component may be directly connected to the other component or may be connected through another component (e.g., a third component). On the other hand, when a component (e.g., a first component) is referred to as being “directly connected” or “directly connected” to another component (e.g., a second component), it is said that the component and the other component are It may be understood that no other components (e.g., third components) exist between components.

The expression "configured to" used in this document may be used depending on the context, for example, "suitable for," "having the capacity to." )", "designed to", "adapted to", "made to", or "capable of". . The term “configured (or set) to” may not necessarily mean “specifically designed to” in hardware. Instead, in some contexts, the expression “a device configured to” may mean that the device is “capable of” working with other devices or components. For example, the phrase "control configured (or set) to perform A, B, and C" means that a dedicated processor (e.g., an embedded processor) to perform the operations, or by executing one or more software programs stored in memory, , may refer to a general-purpose processor (e.g., CPU or application processor) capable of performing the corresponding operations.

In particular, in this specification, “~device” may include one or more of a central processing unit (CPU), an application processor (AP), and a communication processor (CP). .

In this specification, “˜device” refers to all types of hardware devices including at least one processor, and depending on the embodiment, it may be understood as encompassing software configurations that operate on the hardware device. For example, “~device” can be understood as including, but is not limited to, smartphones, tablet PCs, desktops, laptops, and user clients and applications running on each device.

Terms used in this document are merely used to describe specific embodiments and may not be intended to limit the scope of other embodiments. Singular expressions may include plural expressions, unless the context clearly indicates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the technical field described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted to have the same or similar meaning as the meaning they have in the context of related technology, and unless clearly defined in this document, they may be interpreted in an ideal or excessively formal sense. It is not interpreted. In some cases, even terms defined in this document cannot be interpreted to exclude embodiments of this document.

Figure 1 is a configuration diagram of an apparatus including a speaking scoring artificial intelligence model for evaluating an evaluator's responses to evaluation questions according to an embodiment of the present invention.

Referring to FIG. 1, an apparatus 100 that evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention provides multiple evaluations of the speaking ability of the person being evaluated. Memory 110 including a speaking scoring artificial intelligence model (S-AI) learned to determine an evaluation score for each of one or more of the evaluation items, and evaluation data that is the evaluator's response to the evaluation questions for evaluating speaking ability. A processor 120 that receives the scoring data, which is the result of determining the evaluation score of the assessee from the speaking scoring artificial intelligence model (S-AI), as output data by acquiring and inputting it as input data to the speaking scoring artificial intelligence model (S-AI). Includes.

The processor 120 is connected to the memory 110 and controls the overall operation of the device 100, which evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI).

In one embodiment, the device 100 that evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) is a server, a smartphone, or a tablet personal computer. , mobile phone, video phone, laptop PC, netbook computer, notebook computer, PDA (personal digital assistant), PMP (portable multimedia player), wearable device device).

Meanwhile, the memory 110 can store various programs and data necessary for the operation of the device 100, which evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI). The memory 110 may be implemented as non-volatile memory, volatile memory, flash-memory, hard disk drive (HDD), or solid state drive (SSD).

The processor 120 controls the overall operation of the device 100, which evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) using various programs stored in the memory 110. You can. The processor 120 may be comprised of RAM, ROM, a graphics processing unit, a main CPU, first to n interfaces, and a bus. At this time, RAM, ROM, graphics processing unit, main CPU, first to n interfaces, etc. may be connected to each other through a bus.

RAM stores O/S and application programs. Specifically, when the device 100, which evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI), is booted, the O/S is stored in RAM, and various application data selected by the user are stored in the RAM. Can be stored in RAM.

ROM stores a set of instructions for booting the system. When a turn-on command is input and power is supplied, the main CPU copies the O/S stored in the memory 110 to RAM according to the command stored in the ROM, executes the O/S, and boots the system. When booting is complete, the main CPU copies various application programs stored in the memory 110 to RAM and executes the application programs copied to RAM to perform various operations.

The graphics processing unit uses a calculation unit (not shown) and a rendering unit (not shown) to create a screen containing various objects such as items, images, and text. Here, the calculation unit may be configured to calculate attribute values such as coordinate values, shape, size, color, etc. for each object to be displayed according to the layout of the screen using a control command received from the input unit. Additionally, the rendering unit may be configured to generate screens of various layouts including objects based on attribute values calculated by the calculation unit. The screen generated by this rendering unit may be displayed within the display area of the display.

The main CPU accesses the memory 110 and performs booting using the OS stored in the memory 110. And, the main CPU performs various operations using various programs, contents, data, etc. stored in the memory 110.

The first to n interfaces are connected to the various components described above. One of the first to n interfaces may be a network interface connected to an external device through a network.

Meanwhile, furthermore, the processor 120 can control an artificial intelligence model. In this case, of course, the processor may include a dedicated graphics processor (e.g., GPU) for controlling the artificial intelligence model.

Meanwhile, the artificial intelligence model according to the present invention may be a model based on supervised learning or unsupervised learning. Furthermore, the artificial intelligence model according to the present invention may include SVM (support vector machine), decision tree, neural network, etc., and methodologies applied thereto.

In one embodiment, the artificial intelligence model according to the present invention may be an artificial intelligence model based on convolutional deep neural networks (CNN) that is learned by inputting learning data. However, it is not limited to this, and of course, various artificial intelligence models can be applied to the present invention. For example, models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN) can be used as artificial intelligence models, but are not limited to these.

At this time, Convolutional deep Neural Networks (CNN) are a type of multilayer perceptrons designed to use minimal preprocessing. A convolutional neural network consists of one or several convolutional layers and general artificial neural network layers on top of them, and additionally utilizes weights and pooling layers. Thanks to this structure, the convolutional neural network can fully utilize input data with a two-dimensional structure. Additionally, convolutional neural networks can be trained via standard back propagation. Convolutional neural networks have the advantage of being easier to train and using fewer parameters than other feedforward artificial neural network techniques.

Additionally, Deep Neural Networks (DNN) are Artificial Neural Networks (ANN) consisting of multiple hidden layers between an input layer and an output layer.

At this time, the structure of the deep neural network may be composed of a perceptron. A perceptron consists of several inputs, one processor, and one output. The processor multiplies multiple input values by their respective weights and then adds up all of the input values multiplied by the weights. Next, the processor substitutes the combined value into the activation function and outputs one output value. If you want a specific value to come out as the output value of the activation function, you can modify the weight multiplied by each input value and recalculate the output value using the modified weight. At this time, each perceptron can use a different activation function. Additionally, each perceptron accepts the outputs from the previous layer as input and then obtains the output using an activation function. The obtained output is passed to the input of the next layer. By going through the above-described process, you can finally obtain several output values.

Recurrent Neural Network (RNN) refers to a neural network in which the connections between units that make up the artificial neural network form a directed cycle. Unlike a forward neural network, a recurrent neural network can utilize the memory inside the neural network to process arbitrary input.

Deep Belief Networks (DBN) is a generative graphical model used in machine learning. In deep learning, it refers to a deep neural network composed of multiple layers of latent variables. It has the characteristic that there is a connection between layers, but no connection between units within the layer.

Due to the nature of the deep trust neural network as a generative model, it can be used for prior learning, and after learning the initial weights through prior learning, the weights can be fine-tuned through backpropagation or other discriminant algorithms. This characteristic is very useful when there is little training data, because the smaller the training data, the stronger the influence of the initial value of the weights on the resulting model. The pre-learned weight initial value becomes closer to the optimal weight compared to the arbitrarily set initial weight value, which enables improved performance and speed of the fine-tuning stage.

The content regarding the above-described artificial intelligence and its learning method is described for illustrative purposes, and the artificial intelligence and its learning method used in the above-described embodiments are not limited. For example, all types of artificial intelligence technologies and learning methods that can be applied by a person skilled in the art to solve the same problem can be used to implement the system according to the disclosed embodiments.

Meanwhile, the processor 120 may include one or more cores (not shown) and a graphics processing unit (not shown) and/or a connection path (e.g., bus, etc.) for transmitting and receiving signals with other components. You can.

Processor 120 according to one embodiment performs the method described in connection with the present invention by executing one or more instructions stored in memory 110.

For example, the processor 120 acquires new training data by executing one or more instructions stored in the memory 110, uses a learned model, performs a test on the acquired new training data, and provides test results, Extract first training data in which the labeled information is obtained with an accuracy higher than a predetermined first reference value, delete the extracted first training data from the new training data, and learn using the new training data from which the extracted training data was deleted. The model can be trained again.

Meanwhile, the processor 120 includes random access memory (RAM) (not shown) and read-only memory (ROM) that temporarily and/or permanently store signals (or data) processed within the processor 120. , not shown) may be further included. Additionally, the processor 120 may be implemented in the form of a system on chip (SoC) including at least one of a graphics processing unit, RAM, and ROM.

The memory 110 may store programs (one or more instructions) for processing and controlling the processor 120. Programs stored in the memory 110 may be divided into a plurality of modules according to their functions.

Figure 2 is a system configuration diagram of a device that evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention.

Referring to FIG. 2, the device 100 for evaluating speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention includes at least one terminal 200. It may further include a communication unit 130 that communicates with and obtains evaluation data.

The communication unit 130 can communicate with an external device. In particular, the communication unit 130 may include various communication chips such as a Wi-Fi chip, a Bluetooth chip, a wireless communication chip, an NFC chip, and a low-energy Bluetooth chip (BLE chip). At this time, the Wi-Fi chip, Bluetooth chip, and NFC chip communicate in the LAN method, WiFi method, Bluetooth method, and NFC method, respectively. When using a Wi-Fi chip or Bluetooth chip, various connection information such as SSID and session key are first transmitted and received, and various information can be transmitted and received after establishing a communication connection using this. A wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, ZigBee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), and 5G (5th Generation).

Meanwhile, the terminal 200 may be an electronic device owned by the person being evaluated.

In one embodiment, the terminal 200 is a server, a smartphone, a tablet personal computer, a mobile phone, a video phone, a laptop PC, or a netbook computer. , it may include at least one of a laptop computer, a personal digital assistant (PDA), a portable multimedia player (PMP), and a wearable device.

Figure 3 is a configuration diagram of a device that evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention, which further includes an input/output unit.

Referring to FIG. 3, the device 100 that evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention provides evaluation data based on a user command. It may further include an input/output unit 140 that obtains and provides at least one piece of information to the user.

The input/output unit 140 includes an external device connection port, a scanning device, a camera, a display, a mouse, a keyboard, a remote control, etc., obtains the user's input, and outputs information on the electronic device 100 to make the information visible to the user. can be provided.

The camera is configured to photograph at least one subject, and the processor 120 can recognize at least one object or identify the distance to the object through the camera.

A display is a configuration for visually outputting various information.

The display can be implemented as LCD (Liquid Crystal Display), PDP (Plasma Display Panel), OLED (Organic Light Emitting Diode), TOLED (Transparent OLED), Micro LED, etc., but is not limited to this and various other known forms. It may include a display of. The display may be implemented in the form of a touch screen capable of detecting a user's touch operation, or may be implemented as a flexible display that can be folded or bent.

The device 100, which evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention, provides evaluation data and evaluation item types through the input/output unit 140. Obtain information, training evaluation data (EP-T), training scoring data (SD-T), training item type information, and standard scoring data, or obtain evaluation data and evaluation from an external device or terminal 200 through the communication unit 130. Item type information, training evaluation data (EP-T), training scoring data (SD-T), training item type information, and standard scoring data can be obtained.

Figure 4 shows a speaking scoring artificial intelligence model (S-AI) learned by a device that evaluates speaking ability for each evaluation item using the speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention. It is a drawing to explain the process, and Figure 5 shows a device that evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention. It is a diagram to explain the process of outputting scoring data that is the result of determining the evaluation score of the assessee using a model (S-AI), and Figure 6 shows a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention. ) is a diagram illustrating an example of evaluation items that can be evaluated by a device that evaluates speaking ability for each evaluation item.

Referring further to FIGS. 4 to 6, the processor 120 configures training data for learning a speech scoring artificial intelligence model (S-AI), and uses the constructed training data to construct a speech scoring artificial intelligence model (S-AI). ) can be taught to determine an evaluation score for each one or more evaluation items among a plurality of evaluation items regarding the speaking ability of the person being evaluated.

For this purpose, the processor 120 uses training evaluation data (EP-T) and training evaluation data ( Based on EP-T), training scoring data (SD-T) representing the evaluation score that is the result of the grader's scoring can be matched to form training data.

At this time, the processor 120 may select only specific evaluation questions as training evaluation questions.

Specifically, the processor 120 may analyze each of the arbitrary evaluation items and identify evaluation items by which the speaking ability of the appraisee can be evaluated based on the arbitrary evaluation items. At this time, the processor 120 may check the evaluation items using information about the evaluation items previously matched to each arbitrary evaluation item.

Here, as shown in Figure 6, the evaluation items include a pronunciation evaluation item that is an evaluation item for evaluating pronunciation, a vocabulary evaluation item that is an evaluation item that evaluates vocabulary, a grammatical ability evaluation item that is an evaluation item that evaluates grammatical ability, and sentence composition ability. It may include a sentence composition evaluation item that evaluates and a topic suitability evaluation item that evaluates topic suitability.

Additionally, each evaluation item may include evaluation sub-items.

Specifically, the pronunciation evaluation item may include one or more of a first pronunciation evaluation sub-item for evaluating pronunciation fluency, a second pronunciation evaluation sub-item for evaluating pronunciation accuracy, and a third pronunciation evaluation sub-item for evaluating speech speed.

Additionally, the vocabulary evaluation items may include one or more of the first vocabulary evaluation subitem that evaluates vocabulary diversity, the second vocabulary evaluation subitem that evaluates vocabulary accuracy, and the third vocabulary evaluation subitem that evaluates vocabulary difficulty.

The grammatical proficiency evaluation items may include one or more of the first grammatical proficiency evaluation sub-item that evaluates the diversity of grammar, the second grammatical proficiency evaluation sub-item that evaluates grammatical accuracy, and the third grammatical proficiency evaluation sub-item that evaluates grammatical difficulty.

The sentence composition ability evaluation item may include a first sentence composition ability evaluation sub-item that evaluates the sentence composition ability.

The topic suitability evaluation item may include a first topic suitability evaluation sub-item that evaluates the degree of topic matching.

Accordingly, the speaking scoring artificial intelligence model (S-AI) is trained to determine the evaluation scores for each of the pronunciation evaluation items, vocabulary evaluation items, grammar evaluation items, sentence composition evaluation items, and topic suitability evaluation items regarding the speaking ability of the person being evaluated. It can be.

The processor 120 checks the learning difficulty of the speaking scoring artificial intelligence model (S-AI) corresponding to each confirmed evaluation item, and the evaluation items that evaluate the evaluation item whose confirmed learning difficulty exceeds the standard difficulty are evaluated for training. Rather than selecting as an item, only evaluation items that evaluate evaluation items whose confirmed learning difficulty level is less than the standard level of difficulty can be selected as training evaluation items.

Here, the learning difficulty may be the difficulty of learning that allows the speech scoring artificial intelligence model (S-AI) to determine the evaluation score of the corresponding evaluation item.

For example, the higher the learning difficulty, the greater the time required for learning and the amount of training data.

Meanwhile, the processor 120 uses the above-described training scoring data (SD-T), which represents an evaluation score that is a result of scoring by a rater, based on training evaluation data (EP-T), which is evaluation data of another ratee for the training evaluation items. Training evaluation data (EP-T) can be matched to form training data.

At this time, the processor 120 may generate scoring standard information that serves as a reference and basis for the grader when he or she performs scoring based on the training evaluation data (EP-T).

Specifically, the processor 120 may generate scoring standard information indicating a scoring standard based on reference scoring data that is different from the training scoring data (SD-T) and is scoring data obtained for setting the scoring standard.

More specifically, the processor 120 analyzes standard scoring data, which is a result of previous scoring by raters, identifies evaluation items that are frequently scored among a plurality of evaluation items, and provides scoring criteria information so that the identified evaluation items are scored. can be created.

This scoring criteria information is provided to the grader and can be referenced by the grader when scoring the training evaluation data (EP-T).

Meanwhile, the processor 120 performs preprocessing on the training evaluation data (EP-T), training scoring data (SD-T), and evaluation data (EP), and performs preprocessing on the training evaluation data (EP-T) and training scoring. Train the speaking scoring artificial intelligence model (S-AI) using training data consisting of data (SD-T), and input the preprocessed evaluation data (EP) into the speaking scoring artificial intelligence model (S-AI) as input data. You can do it.

Here, preprocessing means that the processor 120 improves the quality of the data by reducing noise, converts speech to text for the data, separates morphemes for the data, counts the total number of tokens and significant tokens for the data, and performs morpheme processing for the data. Part-of-speech tagging, if the language type of the data is Korean, it may mean performing one or more of the following matching vocabulary and grammar lists for grades 1 to 6 classified based on the internationally accepted Korean standard curriculum. .

Meanwhile, the processor 120 may further configure training item type information indicating the type of training evaluation item corresponding to the training evaluation data (EP-T) as training data.

Afterwards, when evaluation data (EP) is obtained from the evaluator who has received the evaluation items, the processor 120 inputs the evaluation data (EP) into the speaking scoring artificial intelligence model (S-AI) as input data, and the speaking scoring artificial intelligence model (S-AI) Scoring data (SD), which is the result of determining the evaluation score of the appraisee from the model (S-AI), can be output as output data.

At this time, the processor 120 checks the evaluation item type information indicating the type of evaluation item corresponding to the evaluation data (EP), and uses the evaluation item type information along with the evaluation data (EP) as input data to speak the scoring artificial intelligence model ( You can receive the evaluator's scoring data (SD) as output data from the speaking scoring artificial intelligence model (S-AI) by inputting it into S-AI.

Meanwhile, each of the training item type information and the evaluation item type information may be the same as the type information of the evaluation item asked in the speaking evaluation of the “Test of Proficiency in Korean (TOPIK).”

For example, the training item type information and the evaluation item type information respectively include the first type reading along the passage provided to the appraisee, the second type explaining a photo or drawing provided to the appraisee, and the simple answer in situations the appraisee frequently encounters in daily life. Type 3, where the appraisee listens to questions and answers appropriately; Type 4, where the appraisee listens to simple questions and performs the role appropriately while looking at pictures of situations that the appraisee often encounters in daily life; Type 4, where the appraisee responds appropriately to what the other person says according to the context of the conversation Type 5, where the appraisee responds appropriately, Type 6, where the appraisee critically states his/her opinion by looking at and interpreting data on social or abstract topics such as economy, science, mass media, culture, art, politics, and the environment, etc. It may be one of the 7th types, which presents one's opinion logically on a specialized field, abstract content, social problem, etc., or presents one's opinion from a pro or anti position.

Meanwhile, the scoring data (SD) may include evaluation scores for each evaluation item, or may include a total score obtained by adding up all evaluation scores for each evaluation item.

At this time, the processor 120 may determine the speaking ability evaluation grade of the person being evaluated based on the total score. Specifically, the processor 120 may determine the speaking ability evaluation grade so that the larger the sum score, the higher the speaking ability of the person being evaluated.

Thereafter, the processor 120 selects a low-ability evaluation item among the plurality of evaluation items as a low-ability evaluation item based on the scoring data, and evaluates a high-ability evaluation item among the plurality of evaluation items as a high ability evaluation item based on the scoring data. Items can be selected, and selection result data representing the selection results of low-ability evaluation items and high-ability evaluation items can be generated.

Subsequently, the processor 120 may generate learning content recommendation information that recommends learning content that can improve speaking skills corresponding to low-ability evaluation items based on the selection result data.

This learning content recommendation information may be output by the terminal 200.

Additionally, the processor 120 may generate learning method information that can improve the evaluation score of each evaluation item based on the selection result data. This learning method information may also be output by the terminal 200.

Meanwhile, the speaking scoring artificial intelligence model (S-AI) in the present invention is an explainable artificial intelligence that provides explanatory information that allows the evaluator to understand the results output from the artificial intelligence model (output data, scoring data of the present invention). It may be a model (eXplainable Artificial Intelligence (XAI)).

Additionally, the speaking scoring artificial intelligence model (S-AI) in the present invention may be an artificial intelligence model based on dynamic weights where the weights are not fixed.

Meanwhile, the processor 120 according to another embodiment acquires information on the answer submitted by an evaluator who took the speaking evaluation of the Test of Proficiency in Korean (TOPIK) as an answer to the speaking evaluation of the Test of Proficiency in Korean (TOPIK), and The information is used as evaluation data and input into the speaking grading artificial intelligence model (S-AI), and the speaking grading artificial intelligence model (S-AI) is used to score the speaking evaluation of the Test of Proficiency in Korean (TOPIK) taken by the assessee. The scoring data that is the result of the decision (hereinafter referred to as “topic scoring data”) can be output as output data.

Thereafter, the processor 120 according to another embodiment may determine the TOPIK-based speaking ability evaluation grade of the evaluator based on the total score included in the topic scoring data and the sum of the evaluation scores for each evaluation item of the TOPIK speaking evaluation.

Additionally, the processor 120 according to another embodiment may acquire actual scoring data of the speaking evaluation of the Test of Proficiency in Korean (TOPIK) taken by the person being evaluated.

Here, the actual scoring data may be the speaking ability evaluation grade of the evaluator, which is the result of the evaluation from the host organization that hosted the speaking evaluation of the Test of Proficiency in Korean (TOPIK).

Thereafter, the processor 120 according to another embodiment may calculate the grade difference between the speaking ability evaluation grade of the actual scoring data and the TOPIK-based speaking ability evaluation grade as a scoring error rate.

The processor 120 according to another embodiment may calculate the grade difference using the following equation.

Equation: R = | (G1 - G2) / N _G |

Here, R is the scoring error rate, G1 is the speaking ability evaluation grade of the actual scoring data, G2 is the TOPIK-based speaking ability evaluation grade, and N _G is the speaking ability evaluation grade determined in the speaking evaluation of the Test of Proficiency in Korean (TOPIK). It is a count.

Hereinafter, the process by which the device 100, which evaluates speaking ability for each evaluation item using the speaking scoring artificial intelligence model (S-AI) according to the present invention, determines the evaluation score of the pronunciation evaluation item, will be described. .

Figure 7 shows a device for evaluating speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention, which evaluates pronunciation among a plurality of evaluation items for the speaking ability of the person being evaluated. This is a drawing to explain the process of determining the evaluation score of an item.

Referring to FIG. 7, the processor 120 controls the communication unit 130 to transmit the fingerprint to the terminal 200 so that the evaluator can read the fingerprint when the item type is the first type that reads the fingerprint provided to the evaluator. You can.

The terminal 200 can output information about the question type, information about how to solve the question, and a fingerprint.

The appraised person can read the passage and speak it out loud. Accordingly, the terminal 200 acquires the first evaluator's voice data of the evaluator who reads the fingerprint and transmits it to the communication unit 130, and the processor 120 acquires the first evaluator's voice data as evaluation data and scores the speech using the input data. The scoring data, which is speech characteristic difference data representing the difference in speech characteristics between the native speaker's speech data and the first evaluator's speech data, is input into the artificial intelligence model (S-AI) and read along with the passage, and the speaking scoring artificial intelligence model (S-AI) ) can be output as output data.

Here, the voice characteristic difference may include one or more of voice frequency, voice intensity, voice pause period, and prosody.

Through this, the processor 120 can determine and obtain an evaluation score for the first pronunciation evaluation sub-item that evaluates pronunciation fluency among the plurality of evaluation sub-items of the pronunciation evaluation item, which is one of the plurality of evaluation items.

In addition, the processor 120 acquires the first evaluator's voice data as evaluation data, inputs it to the speech scoring artificial intelligence model (S-AI) as input data, and pronunciation difference between the pronunciation data corresponding to the fingerprint and the first evaluator's voice data. Scoring data, which is pronunciation difference data representing , can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Here, the pronunciation difference data is obtained using an Automatic Speech Recognition (ASR) method and may be information indicating the difference in pronunciation.

Through this, the processor 120 can determine and obtain an evaluation score for the second pronunciation evaluation sub-item that evaluates pronunciation accuracy among the plurality of evaluation sub-items of the pronunciation evaluation item, which is one of the plurality of evaluation items.

In addition, the processor 120 acquires the first evaluator's voice data as evaluation data, inputs it to the speech scoring artificial intelligence model (S-AI) as input data, and provides voice speech rate data indicating the voice speech rate of the first evaluator's voice data. The scoring data can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the third pronunciation evaluation sub-item that evaluates speech speed among the plurality of evaluation sub-items of the pronunciation evaluation item, which is one of the plurality of evaluation items.

Hereinafter, a process in which the apparatus 100, which performs evaluation for each evaluation item on speaking ability using the speaking scoring artificial intelligence model (S-AI) according to the present invention, determines the evaluation scores of the vocabulary evaluation item and the grammar evaluation item. Let me explain.

Figure 8 shows a device that evaluates speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention, and evaluates pronunciation among a plurality of evaluation items for the speaking ability of the person being evaluated. This is a diagram to explain the process of determining the evaluation score for each item, sentence composition evaluation item, and topic suitability evaluation item.

Referring to FIG. 8, if the item type indicated by the evaluation item type information is a second type that describes a photo or picture provided to the assessee, the processor 120 posts the photo to the terminal 200 so that the assessee can view the photo or picture. Alternatively, the communication unit 130 can be controlled to transmit a picture.

The terminal 200 can output information about the question type, information about how to solve the question, and a photo or drawing.

The person being evaluated can explain the photo or drawing by looking at it and speaking. Accordingly, the terminal 200 may obtain voice data of the second appraisee of the appraisee explaining the photo or picture and transmit it to the communication unit 130.

The processor 120 acquires the second evaluator's voice data as evaluation data, inputs it into a speaking scoring artificial intelligence model (S-AI) as input data, and native speaker reads the original text data converted into text from the second evaluator's voice data. Scoring data, which is voice characteristic difference data representing the difference in voice characteristics between the native speaker's voice data and the second evaluator's voice data, can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the first pronunciation evaluation sub-item that evaluates pronunciation fluency among the plurality of evaluation sub-items of the pronunciation evaluation item, which is one of the plurality of evaluation items.

In addition, the processor 120 acquires the voice data of the second evaluator as evaluation data, inputs it to the speech scoring artificial intelligence model (S-AI) as input data, and corresponds to the original text data converted into text from the voice data of the second evaluator. Scoring data, which is pronunciation difference data representing the pronunciation difference between the pronunciation data and the second evaluator's voice data, can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the second pronunciation evaluation sub-item that evaluates pronunciation accuracy among the plurality of evaluation sub-items of the pronunciation evaluation item, which is one of the plurality of evaluation items.

In addition, the processor 120 acquires the second evaluator's voice data as evaluation data, inputs it to the speech scoring artificial intelligence model (S-AI) as input data, and provides voice speech rate data indicating the voice speech rate of the second evaluator's voice data. The scoring data can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the third pronunciation evaluation sub-item that evaluates speech speed among the plurality of evaluation sub-items of the pronunciation evaluation item, which is one of the plurality of evaluation items.

Meanwhile, the processor 120 acquires the second evaluator's voice data as evaluation data and inputs it into a speaking scoring artificial intelligence model (S-AI) as input data, and determines the dependencies between words in sentences included in the second evaluator's voice data and the above. Scoring data, which is sentence composition data indicating the suitability of the sentence pattern of the sentence included in the voice data of the second evaluator, can be output as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the first sentence composition ability evaluation subitem that evaluates the sentence composition ability among the plurality of evaluation subitems of the sentence composition ability evaluation item, which is one of the plurality of evaluation items. .

To this end, the speech scoring artificial intelligence model (S-AI) uses word embeddings (e.g., BERT (Bidirectional Encoder Representations from Transformers) and Word2Vec) to output sentence composition data indicating the above-described dependencies and sentence pattern suitability. You can.

Meanwhile, the processor 120 acquires the voice data of the second evaluator as evaluation data, inputs it into the speaking scoring artificial intelligence model (S-AI) as input data, and inputs the words included in the voice data of the second evaluator and the words included in the answer data. Scoring data, which is topic agreement rate information indicating the agreement rate between subjects, can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the first topic suitability evaluation sub-item that evaluates the degree of topic matching among the plurality of evaluation sub-items of the topic suitability evaluation item, which is one of the plurality of evaluation items.

Hereinafter, the device 100, which includes a speaking scoring artificial intelligence model for evaluating the evaluator's responses to the evaluation items according to an embodiment of the present invention, uses vocabulary evaluation items and grammar among a plurality of evaluation items for the evaluator's speaking ability. Explain the process of determining the evaluation score for each competency evaluation item.

The processor 120 acquires the second evaluator's voice data as evaluation data, inputs it to the speech scoring artificial intelligence model (S-AI) as input data, and provides stem number information indicating the number of stems included in the second evaluator's voice data. Scoring data can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the first vocabulary evaluation sub-item that evaluates lexical diversity among the plurality of evaluation sub-items of the vocabulary evaluation item, which is one of the plurality of evaluation items.

In addition, the processor 120 acquires the second evaluator's voice data as evaluation data and inputs it into the speech scoring artificial intelligence model (S-AI) as input data, and the original text data converted into text from the second evaluator's voice data and the original Scoring data, which is the first text match rate information indicating the match rate between vocabulary-corrected text data that corrects vocabulary usage errors in text data, can be output as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the second vocabulary evaluation sub-item that evaluates vocabulary accuracy among the plurality of evaluation sub-items of the vocabulary evaluation item, which is one of the plurality of evaluation items.

In addition, the processor 120 acquires the second evaluator's voice data as evaluation data, inputs it to the speech scoring artificial intelligence model (S-AI) as input data, and selects the vocabulary included in the second evaluator's voice data among the plurality of vocabulary difficulty levels. Scoring data, which is vocabulary difficulty level information indicating the vocabulary difficulty level to which words belong, can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the third vocabulary evaluation sub-item that evaluates vocabulary difficulty among the plurality of evaluation sub-items of the vocabulary evaluation item, which is one of the plurality of evaluation items.

Here, the scoring data, which is vocabulary difficulty level information, may include a vocabulary count indicating the number of vocabulary (vocabularies included in the second evaluator's voice data) included in each of a plurality of vocabulary difficulty levels.

Here, the plurality of vocabulary difficulty levels may be levels 1 to 6, which are the difficulty levels of vocabulary classified based on the internationally accepted Korean standard curriculum, and the plurality of vocabulary difficulty levels may include vocabulary classified based on the internationally accepted standard Korean language curriculum. The vocabulary list may be matched to each difficulty level of level 1 to level 6.

Using this, the processor 120 outputs scoring data, which is vocabulary distribution data representing the distribution by plural vocabulary difficulty levels for vocabulary included in the second evaluator's voice data, from the speaking scoring artificial intelligence model (S-AI). You can print it out.

In addition, the processor 120 uses the scoring data, which is the result of determining the vocabulary difficulty level of the appraisee based on the result of comparing the vocabulary distribution data with the standard vocabulary distribution map set for each of the plurality of vocabulary difficulty levels, into a speaking scoring artificial intelligence model (S-AI). It can be output as output data from .

At this time, the speaking scoring artificial intelligence model (S-AI) determines the vocabulary difficulty level whose distribution indicated by the vocabulary distribution data exceeds the standard vocabulary distribution among the plurality of vocabulary difficulty levels as the vocabulary difficulty level of the evaluator, and the determined result is the scoring data. can be output as output data.

If the distribution shown by the vocabulary distribution data has multiple vocabulary difficulty levels that exceed the standard vocabulary distribution, the speaking scoring artificial intelligence model (S-AI) will determine the vocabulary difficulty level that the distribution shown by the vocabulary distribution data exceeds the standard vocabulary distribution. The most difficult vocabulary difficulty level among the grades can be determined as the vocabulary difficulty level of the evaluator.

To this end, the processor 120 may set the standard vocabulary distribution to a lower level for each of the plurality of vocabulary difficulty levels as the difficulty level increases, and set the standard vocabulary distribution to be the same for neighboring levels of difficulty. For example, for multiple levels of difficulty, the processor 120 may set the standard vocabulary distribution to a lower level for each level. The standard vocabulary distributions of 25%, 25%, 20%, 15%, 10%, and 5% can be set for each of the vocabulary difficulty levels of grades 1 to 6.

Accordingly, the speaking scoring artificial intelligence model (S-AI) determines if, among multiple vocabulary difficulty levels, the distribution of grade 5 exceeds the standard vocabulary distribution of 10% at 12%, and the distributions of the remaining grades 1 to 4 and 6 respectively If the vocabulary distribution is below the standard, the vocabulary difficulty level of the evaluator can be determined as level 5.

As another example, the speaking scoring artificial intelligence model (S-AI) determines that among multiple vocabulary difficulty levels, the distribution of level 5 is 12%, which exceeds the standard vocabulary distribution of 10%, and the distribution of level 6 is 7%, which is below the standard. If the vocabulary distribution exceeds 5%, and the distribution of each of the remaining grades 1 to 4 is below the standard vocabulary distribution, grade 6, the most difficult among the multiple vocabulary difficulty grades exceeding the standard vocabulary distribution, is determined as the vocabulary difficulty grade of the evaluator. You can.

In addition, the processor 120 acquires the second evaluator's voice data as evaluation data, inputs it to the speech scoring artificial intelligence model (S-AI) as input data, and indicates the number of endings and particles included in the second evaluator's voice data. Scoring data, which is information on the number of endings and particles, can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the first grammar evaluation sub-item that evaluates grammatical diversity among the plurality of evaluation sub-items of the grammar evaluation item, which is one of the plurality of evaluation items.

Meanwhile, the processor 120 acquires the second evaluator's voice data as evaluation data and inputs it into the speech scoring artificial intelligence model (S-AI) as input data, and the original text data converted into text from the second evaluator's voice data and the original Scoring data, which is second text match rate information indicating the match rate between grammar-corrected text data that corrects grammatical errors in text data, can be output as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the second grammar evaluation sub-item that evaluates grammatical accuracy among the plurality of evaluation sub-items of the vocabulary evaluation item, which is one of the plurality of evaluation items.

In addition, the processor 120 acquires the voice data of the second evaluator as evaluation data, inputs it to the speaking scoring artificial intelligence model (S-AI) as input data, and selects the voice data of the second evaluator among the plurality of grammar usage difficulty levels. Scoring data, which is grammar usage difficulty level information indicating the grammar usage difficulty level to which the grammar usages belong, can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor 120 can determine and obtain an evaluation score for the third grammar evaluation sub-item that evaluates grammar difficulty among the plurality of evaluation sub-items of the grammar evaluation item, which is one of the plurality of evaluation items.

Here, the scoring data, which is grammar usage difficulty level information, may include the number of grammar usages indicating the number of grammar usages (grammar usages included in the second ratee's voice data) included in each of a plurality of grammar usage difficulty levels.

Here, the plurality of grammar usage difficulty levels may be levels 1 to 6, which are the difficulty levels of grammar usage classified based on the internationally accepted Korean standard curriculum, and the plurality of grammar usage difficulty levels may be based on the internationally accepted Korean standard curriculum. A list of grammar usages may be matched to each of grades 1 to 6, which are the difficulty levels of classified grammar usage.

Using this, the processor 120 uses the scoring data, which is grammar usage distribution data indicating the distribution by difficulty level of a plurality of grammar usages for the grammar usages included in the voice data of the second evaluator, into a speaking scoring artificial intelligence model (S-AI). It can be output as output data from .

In addition, the processor 120 uses the scoring data, which is the result of determining the grammar usage difficulty level of the evaluator, based on the results of comparing the grammar usage distribution data with the standard grammar usage distribution map set for each of the plurality of grammar usage difficulty levels, using a speaking scoring artificial intelligence model ( It can be output as output data from S-AI).

At this time, the speaking scoring artificial intelligence model (S-AI) determines the grammar usage difficulty level whose distribution indicated by the grammar usage distribution data exceeds the standard grammar usage distribution among the plurality of grammar usage difficulty levels as the grammar usage difficulty level of the evaluator, The scoring data, which is the determined result, can be output as output data.

If there are multiple grammar usage difficulty levels in which the distribution shown by the grammar usage distribution data exceeds the standard grammar usage distribution, the speaking scoring artificial intelligence model (S-AI) determines that the distribution shown by the grammar usage distribution data exceeds the standard grammar usage distribution. Among the plurality of grammar usage difficulty levels, the most difficult grammar usage difficulty level can be determined as the grammar usage difficulty level of the evaluator.

To this end, the processor 120 may set the standard grammar usage distribution to a lower level for each of the plurality of grammar usage difficulty levels, if the difficulty level is higher, and set the standard grammar usage distribution to be the same if the difficulty levels are neighboring levels.

For example, standard grammar usage distributions of 25%, 25%, 20%, 15%, 10%, and 5% may be set for each of grades 1 to 6, which are plural grammar usage difficulty levels.

Accordingly, the speaking scoring artificial intelligence model (S-AI) determines if, among multiple grammar usage difficulty levels, the distribution of level 5 is 12%, which exceeds the standard grammar usage distribution of 10%, and the remaining levels 1 to 4 and 6 respectively. If the distribution of is less than the standard grammar usage distribution, the difficulty level of the evaluator's grammar usage can be determined as level 5.

For another example, the speaking scoring artificial intelligence model (S-AI) determines that among multiple grammar usage difficulty levels, the distribution of level 5 is 12%, which exceeds the standard grammar usage distribution of 10%, and the distribution of level 6 is 7%. If the distribution of standard grammar usage exceeds 5% and the distribution of each of the remaining grades 1 to 4 is below the standard grammar usage distribution, the most difficult grade 6 among the plurality of grammar usage difficulty levels that exceed the standard grammar usage distribution is assigned to the appraisee. It can be determined by the difficulty level of grammar usage.

Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

100: Device including a speaking scoring artificial intelligence model for evaluating the appraisee's responses to evaluation questions
110: memory
120: processor
130: Department of Communications
140: input/output unit

Claims (8)

Among the plurality of evaluation items for the speaking ability of the person being evaluated, the evaluation scores for each of the pronunciation evaluation item that evaluates pronunciation, the sentence composition ability evaluation item that evaluates sentence composition ability, and the topic suitability evaluation item that evaluates topic suitability are determined. A memory containing a speech scoring artificial intelligence model trained to do so; and
Evaluation data that is the ratee's responses to the evaluation questions for evaluating the speaking ability is obtained and input as input data to the speaking scoring artificial intelligence model, and the evaluation score of the ratee is determined from the speaking scoring artificial intelligence model. Includes a processor that outputs the scoring data as output data,
The above pronunciation evaluation items are
It includes at least one of a first pronunciation evaluation subitem that evaluates pronunciation fluency, a second pronunciation evaluation subitem that evaluates pronunciation accuracy, and a third pronunciation evaluation subitem that evaluates speech speed,
The above sentence composition evaluation items are
Includes the first sentence composition ability evaluation sub-item that evaluates sentence composition ability,
The above topic suitability evaluation items are
Includes a first topic suitability evaluation sub-item that evaluates the degree of topic matching,
The processor is
Training evaluation data, which is evaluation data of other ratees for the training evaluation items, which are evaluation questions prepared for learning of the speaking scoring artificial intelligence model, and training scoring data representing evaluation scores that are the result of scoring by a rater based on the training evaluation data. Match and configure training data, and use the training data to learn the speaking scoring artificial intelligence model,
The processor is
Further configuring training item type information indicating a type of the training evaluation item corresponding to the training evaluation data with the training data,
The processor is
If the item type is a type that describes a photo or picture provided to the appraisee, artificial intelligence scores the speech using the input data by obtaining the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data. The scoring data, which is speech characteristic difference data indicating the difference in speech characteristics between the native speaker's speech data and the second evaluator's speech data, is input to the model and read along with the original text data converted into text from the second evaluator's speech data. Receive the above output data from the scoring artificial intelligence model,
The difference in voice characteristics is
Includes one or more of the following: vocal frequency, vocal intensity, vocal pause period, and prosody;
The processor is
If the item type is a type that describes a photo or picture provided to the appraisee, artificial intelligence scores the speech using the input data by obtaining the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data. input into the model, and the scoring data, which is pronunciation difference data representing the pronunciation difference between the pronunciation data corresponding to the original text data converted into text from the second evaluator's voice data and the second evaluator's voice data, is derived from the speaking scoring artificial intelligence model. Receive output as the output data,
The processor is
Obtain answer information submitted by the evaluator who took the speaking evaluation, which is a test for evaluating the speaking ability, as an answer to the speaking evaluation, and use the answer information as the evaluation data as input data to create the speaking scoring artificial intelligence model. input, and the scoring data that is the result of determining the evaluation score of the speaking evaluation that the evaluator took from the speaking scoring artificial intelligence model is output as the output data, based on the evaluation score of the speaking evaluation included in the scoring data. Determine the speaking ability evaluation grade based on the speaking evaluation of the person being evaluated,
The processor is
Obtaining actual scoring data, which is the speaking ability evaluation grade of the person being evaluated, issued by the organization in charge of the speaking evaluation,
Calculating the grade difference between the speaking ability evaluation grade of the actual scoring data and the speaking ability evaluation grade based on the speaking evaluation as the scoring error rate,
The processor is
Characterized in calculating the scoring error rate using the following equation:
A device that includes a speaking scoring artificial intelligence model for evaluating the ratee's responses to evaluation questions.

<Equation>
R = | (G1 - G2) / N _G |
Here, R is the scoring error rate, G1 is the speaking ability evaluation grade of the actual scoring data, G2 is the speaking ability evaluation grade based on the speaking evaluation, and N _G is the number of speaking ability evaluation grades determined by the organization in charge of the speaking evaluation.
delete According to paragraph 1,
The processor is
The evaluation item type information indicating the type of the evaluation item corresponding to the evaluation data is confirmed, and the evaluation item type information along with the evaluation data are input to the speaking scoring artificial intelligence model as the input data, and the speaking scoring artificial intelligence model Characterized in that the scoring data of the evaluator is output from the model as the output data.
A device that includes a speaking scoring artificial intelligence model for evaluating the ratee's responses to evaluation questions.
delete According to paragraph 3,
The processor is
If the item type is a type that describes a photo or picture provided to the appraisee, artificial intelligence scores the speech using the input data by obtaining the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data. input into the model, and the scoring data, which is pronunciation difference data representing the pronunciation difference between the pronunciation data corresponding to the original text data converted into text from the second evaluator's voice data and the second evaluator's voice data, is derived from the speaking scoring artificial intelligence model. Characterized by being output as the output data
A device that includes a speaking scoring artificial intelligence model for evaluating the ratee's responses to evaluation questions.
According to paragraph 3,
The processor is
If the item type is a type that describes a photo or picture provided to the appraisee, artificial intelligence scores the speech using the input data by obtaining the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data. Characterized in that the scoring data, which is voice speech rate data representing the voice speech rate of the second ratee's voice data, is input to the model and output as the output data from the speaking scoring artificial intelligence model.
A device that includes a speaking scoring artificial intelligence model for evaluating the ratee's responses to evaluation questions.
According to paragraph 3,
The processor is
If the item type is a type that describes a photo or picture provided to the appraisee, artificial intelligence scores the speech using the input data by obtaining the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data. The speaking scoring artificial intelligence inputs the scoring data, which is sentence composition data that is input into a model and is sentence composition data indicating dependency between words in sentences included in the second evaluator's voice data and sentence pattern suitability of the sentences included in the second evaluator's voice data, Characterized in that the output data is output from the model.
A device that includes a speaking scoring artificial intelligence model for evaluating the ratee's responses to evaluation questions.
According to paragraph 3,
The processor is
If the item type is a type that describes a photo or picture provided to the appraisee, artificial intelligence scores the speech using the input data by obtaining the second appraisee voice data of the appraisee describing the photo or picture as the evaluation data. Inputting the scoring data into the model, and receiving the scoring data, which is topic agreement rate information indicating the match rate between words included in the second ratee's voice data and words included in the answer data, as the output data from the speaking scoring artificial intelligence model.
A device that includes a speaking scoring artificial intelligence model for evaluating the ratee's responses to evaluation questions.