KR102706374B1 - Apparatus for evaluating grammar and vocabulary using a speaking scoring artificial intelligence model - Google Patents

Abstract

The device for evaluating grammar and vocabulary using a speaking scoring artificial intelligence model according to the present invention comprises: a memory including a speaking scoring artificial intelligence model learned to determine evaluation scores for each of a vocabulary evaluation item, which is an evaluation item for evaluating vocabulary, and a grammar evaluation item, which is an evaluation item for evaluating grammar, among a plurality of evaluation items for the speaking ability of an assessee; and a processor for obtaining evaluation data, which is a response of the assessee to an evaluation item for evaluating the speaking ability, and inputting the evaluation data, which is a result of determining the evaluation score of the assessee, as input data from the speaking scoring artificial intelligence model; wherein the vocabulary evaluation items include at least one of a first vocabulary evaluation sub-item for evaluating vocabulary diversity, a second vocabulary evaluation sub-item for evaluating vocabulary accuracy, and a third vocabulary evaluation sub-item for evaluating vocabulary difficulty, and the grammar evaluation items may include at least one of a first grammar evaluation sub-item for evaluating grammar diversity, a second grammar evaluation sub-item for evaluating grammar accuracy, and a third grammar evaluation sub-item for evaluating grammar difficulty.

Description

{Apparatus for evaluating grammar and vocabulary using a speaking scoring artificial intelligence model}

The present invention relates to a device for evaluating grammar and vocabulary using a speaking scoring artificial intelligence model, and more specifically, to a device for evaluating grammar and vocabulary using a speaking scoring artificial intelligence model, which is capable of obtaining scoring data, which are evaluation scores for a vocabulary evaluation item and a grammar evaluation item, by using a speaking scoring artificial intelligence model learned to determine evaluation scores for a vocabulary evaluation item and a grammar evaluation item among a plurality of evaluation items for the speaking ability of an evaluatee.

As of 2022, there are approximately 356,000 applicants from 81 countries for the Test of Proficiency in Korean (TOPIK) administered by the National Institute for International Education. Currently, the TOPIK speaking test requires test takers to visit the test site in person and is only administered domestically.

Accordingly, for foreign learners, evaluations that are very burdensome and cumbersome continue to be used.

The TOPIK speaking test fee per test is approximately 80,000 won, and it takes about 3 months or more from test registration to administration and score announcement. Training for professional raters is conducted periodically in the form of a workshop that lasts 8 hours a day for two days. This still has many problems, such as not only cost issues, but also security issues with the evaluation criteria and rater management issues.

In addition, each time an evaluation is conducted, a grading fee must be paid to the evaluator, and this task requires highly skilled personnel with a certain level of experience in Korean language education and evaluation, which not only consumes a lot of money, but also raises questions about the fairness and objectivity of the evaluation.

In particular, the grading criteria and distribution of points for speaking assessments differ for each institution (based on university language schools), and even for the same grade, there is a large difference in level depending on the institution to which the learner belongs, and learners themselves are confused about their own level.

Korean Patent Publication No. 10-2022-0032940

The purpose of the present invention is to provide a device for evaluating grammar and vocabulary skills using a speaking scoring artificial intelligence model, which can obtain scoring data, which are evaluation scores for vocabulary and grammar evaluation items, by using a speaking scoring artificial intelligence model learned to determine evaluation scores for vocabulary and grammar evaluation items among multiple evaluation items for the speaking ability of an assessee.

The purposes of the present invention are not limited to the purposes mentioned above, and other purposes and advantages of the present invention which are not mentioned can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. In addition, it will be easily understood that the purposes and advantages of the present invention can be realized by the means and combinations thereof indicated in the claims.

The device for evaluating grammar and vocabulary using a speaking scoring artificial intelligence model according to the present invention comprises: a memory including a speaking scoring artificial intelligence model learned to determine evaluation scores for each of a vocabulary evaluation item, which is an evaluation item for evaluating vocabulary, and a grammar evaluation item, which is an evaluation item for evaluating grammar, among a plurality of evaluation items for the speaking ability of an assessee; and a processor for obtaining evaluation data, which is a response of the assessee to an evaluation item for evaluating the speaking ability, and inputting the evaluation data, which is a result of determining the evaluation score of the assessee, as input data from the speaking scoring artificial intelligence model; wherein the vocabulary evaluation items include at least one of a first vocabulary evaluation sub-item for evaluating vocabulary diversity, a second vocabulary evaluation sub-item for evaluating vocabulary accuracy, and a third vocabulary evaluation sub-item for evaluating vocabulary difficulty, and the grammar evaluation items may include at least one of a first grammar evaluation sub-item for evaluating grammar diversity, a second grammar evaluation sub-item for evaluating grammar accuracy, and a third grammar evaluation sub-item for evaluating grammar difficulty.

The above processor matches training evaluation data, which is evaluation data of another evaluator for training evaluation items, which are evaluation items prepared for learning the speaking scoring artificial intelligence model, and training scoring data, which represents evaluation scores scored by an evaluator based on the training evaluation data, to form training data, and can train the speaking scoring artificial intelligence model using the training data.

The above processor further configures training item type information indicating the type of the training evaluation item corresponding to the training evaluation data into the training data, verifies evaluation item type information indicating the type of the evaluation item corresponding to the evaluation data, and inputs the evaluation item type information together with the evaluation data into the speaking scoring artificial intelligence model as the input data, so as to receive the scoring data of the evaluated person as the output data from the speaking scoring artificial intelligence model.

The above processor may, if the type of the question is a type that describes a photograph or picture provided to the assessee, obtain second assessee voice data of the assessee describing the photograph or picture as the evaluation data and input it as the input data to the speaking scoring artificial intelligence model, and receive the scoring data, which is stem count information indicating the number of stems included in the second assessee voice data, as output data from the speaking scoring artificial intelligence model.

The above processor may, if the type of the question is a type that describes a photograph or picture provided to the assessee, obtain second assessee voice data of the assessee describing the photograph or picture as the evaluation data, input the second assessee voice data as the input data to the speaking scoring artificial intelligence model, and receive the scoring data, which is first text matching rate information indicating the matching rate between original text data converted into text from the second assessee voice data and vocabulary-corrected text data that corrects vocabulary usage errors in the original text data, as output data from the speaking scoring artificial intelligence model.

The above processor may, if the above question type is a type that describes a photograph or picture provided to the above assessee, obtain second assessee voice data of the above assessee describing the photograph or picture as the evaluation data and input it as the input data to the speaking scoring artificial intelligence model, and receive the scoring data, which is vocabulary difficulty level information indicating a vocabulary difficulty level to which vocabulary included in the second assessee voice data belongs among a plurality of vocabulary difficulty levels, as output data from the speaking scoring artificial intelligence model.

The above processor may, if the type of the question is a type that describes a photograph or picture provided to the assessee, obtain second assessee voice data of the assessee describing the photograph or picture as the evaluation data and input it as the input data to the speaking scoring artificial intelligence model, and receive the scoring data, which is information on the number of suffixes and particle numbers indicating the number of suffixes and particle numbers included in the second assessee voice data, as output data from the speaking scoring artificial intelligence model.

The above processor may, if the type of the question is a type that describes a photograph or picture provided to the assessee, obtain second assessee voice data of the assessee describing the photograph or picture as the evaluation data, input the second assessee voice data as the input data to the speaking scoring artificial intelligence model, and receive the scoring data, which is second text matching rate information indicating the matching rate between original text data converted into text from the second assessee voice data and grammar-corrected text data that corrects grammatical errors in the original text data, as output data from the speaking scoring artificial intelligence model.

The above processor may, if the above question type is a type that describes a photograph or picture provided to the above assessee, obtain second assessee voice data of the above assessee describing the photograph or picture as the evaluation data and input the same into the speaking scoring artificial intelligence model as the input data, and receive the scoring data, which is grammar usage difficulty grade information indicating a grammar usage difficulty grade to which grammar usages included in the second assessee voice data belong among a plurality of grammar usage difficulty grades, from the speaking scoring artificial intelligence model as output data.

According to the present invention, by obtaining scoring data, which are evaluation scores for vocabulary and grammar evaluation items, using a speaking scoring artificial intelligence model learned to determine evaluation scores for vocabulary and grammar evaluation items among multiple evaluation items for the speaking ability of an assessee, it is possible to automatically, quickly and accurately perform an ability evaluation for vocabulary and grammar among speaking abilities.

Figure 1 is a configuration diagram of a device for evaluating grammar and vocabulary skills using a speaking scoring artificial intelligence model according to one embodiment of the present invention.
FIG. 2 is a system configuration diagram of a device for evaluating grammar and vocabulary skills using a speaking scoring artificial intelligence model according to one embodiment of the present invention.
FIG. 3 is a block diagram of a device for evaluating grammar and vocabulary skills using a speaking scoring artificial intelligence model according to an embodiment of the present invention that further includes an input/output unit.
FIG. 4 is a diagram for explaining a process in which a speaking scoring artificial intelligence model is learned by a device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model according to an embodiment of the present invention.
FIG. 5 is a diagram for explaining a process in which a device for evaluating grammar and vocabulary using a speaking scoring artificial intelligence model according to one embodiment of the present invention outputs scoring data, which is a result of determining the evaluation score of an assessee using a speaking scoring artificial intelligence model.
FIG. 6 is a diagram illustrating an example of evaluation items that can be evaluated by a device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model according to an embodiment of the present invention.
FIG. 7 is a diagram for explaining the process of determining an evaluation score for a pronunciation evaluation item, which is an evaluation item that evaluates pronunciation among multiple evaluation items for the speaking ability of an assessee, by a device for evaluating grammar and vocabulary using a speaking scoring artificial intelligence model according to one embodiment of the present invention.
FIG. 8 is a diagram for explaining the process of determining evaluation scores for each of the vocabulary evaluation item, which is an evaluation item evaluating vocabulary, and the grammar evaluation item, which is an evaluation item evaluating grammar, among multiple evaluation items for the speaking ability of an assessee, by a device evaluating grammar and vocabulary using a speaking scoring artificial intelligence model according to one embodiment of the present invention.

Hereinafter, various embodiments of the present invention will be described with reference to the accompanying drawings. However, this is not intended to limit the present invention to specific embodiments, but should be understood to include various modifications, equivalents, and/or alternatives of the embodiments of the present invention. In connection with the description of the drawings, similar reference numerals may be used for similar components.

In this document, the expressions "have", "can have", "include", or "may include" indicate the presence of a feature (e.g., a numerical value, function, operation, or component such as a part), but do not exclude the presence of additional features.

In this document, the expressions "A or B", "at least one of A and/or B", or "one or more of A or/and B" can include all possible combinations of the items listed together. For example, "A or B", "at least one of A and B", or "at least one of A or B" can all refer to cases where (1) at least one A is included, (2) at least one B is included, or (3) both at least one A and at least one B are included.

The expressions "first", "second", "first", or "second" used in this document can describe various components, regardless of order and/or importance, and are only used to distinguish one component from another, but do not limit the components. For example, a first user device and a second user device can represent different user devices, regardless of order or importance. For example, without departing from the scope of the rights set forth in this document, a first component can be named a second component, and similarly, a second component can also be renamed as a first component.

When it is stated that a component (e.g., a first component) is "(operatively or communicatively) coupled with/to" or "connected to" another component (e.g., a second component), it should be understood that the component can be directly coupled to the other component, or can be connected via another component (e.g., a third component). Conversely, when it is stated that a component (e.g., a first component) is "directly coupled to" or "directly connected" to another component (e.g., a second component), it should be understood that no other component (e.g., a third component) exists between the component and the other component.

The expression "configured to" as used herein can be used interchangeably with, for example, "suitable for", "having the capacity to", "designed to", "adapted to", "made to", or "capable of". The term "configured to" does not necessarily mean only that which is "specifically designed to" in terms of hardware. Instead, in some contexts, the expression "a device configured to" can mean that the device is "capable of" doing something together with other devices or components. For example, the phrase "a control unit configured (or set) to perform A, B, and C" can mean a dedicated processor (e.g., an embedded processor) to perform those operations, or a generic-purpose processor (e.g., a CPU or an application processor) that can perform those operations by executing one or more software programs stored in memory.

In particular, in this specification, “~device” may include one or more of a Central Processing Unit (CPU), an Application Processor (AP), and a Communication Processor (CP).

In this specification, “~device” means any kind of hardware device including at least one processor, and may be understood to encompass software configurations operating on the hardware device according to an embodiment. For example, “~device” may be understood to encompass, but is not limited to, smartphones, tablet PCs, desktops, laptops, and all user clients and applications running on each device.

The terms used in this document are used only to describe specific embodiments and may not be intended to limit the scope of other embodiments. The singular expression may include the plural expression unless the context clearly indicates otherwise. The terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by a person of ordinary skill in the art described in this document. Among the terms used in this document, terms defined in general dictionaries may be interpreted as having the same or similar meaning in the context of the related technology, and shall not be interpreted in an ideal or excessively formal meaning unless explicitly defined in this document. In some cases, even if a term is defined in this document, it cannot be interpreted to exclude the embodiments of this document.

Figure 1 is a configuration diagram of a device for evaluating grammar and vocabulary skills using a speaking scoring artificial intelligence model according to one embodiment of the present invention.

Referring to FIG. 1, a device (100) for evaluating speaking ability by evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention includes a memory (110) including a speaking scoring artificial intelligence model (S-AI) learned to determine an evaluation score by evaluation item among a plurality of evaluation items for the speaking ability of an evaluatee, and a processor (120) for obtaining evaluation data, which is a response of an evaluatee to an evaluation item for evaluating speaking ability, and inputting the evaluation data as input data into the speaking scoring artificial intelligence model (S-AI), thereby receiving, as output data, the evaluation data as a result of determining the evaluation score of the evaluatee from the speaking scoring artificial intelligence model (S-AI).

The processor (120) is connected to the memory (110) and controls the overall operation of the device (100) that performs an evaluation on speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI).

In one embodiment, a device (100) that performs an evaluation on speaking ability by each evaluation item using a speaking scoring artificial intelligence model (S-AI) may include at least one of a server, a smartphone, a tablet personal computer, a mobile phone, a video phone, a laptop PC, a netbook computer, a notebook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), and a wearable device.

Meanwhile, the memory (110) can store various programs and data necessary for the operation of the device (100) that performs evaluation on speaking ability by evaluation item using the speaking scoring artificial intelligence model (S-AI). The memory (110) can be implemented as a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

The processor (120) can control the overall operation of the device (100) that performs an evaluation of speaking ability by evaluation item using a speaking scoring artificial intelligence model (S-AI) using various programs stored in the memory (110). The processor (120) can be composed of a RAM, a ROM, a graphic processing unit, a main CPU, first to n interfaces, and a bus. At this time, the RAM, the ROM, the graphic processing unit, the main CPU, the first to n interfaces, etc. can be connected to each other through a bus.

RAM stores O/S and application programs. Specifically, when a device (100) that performs evaluation on speaking ability by evaluation item using a speaking scoring artificial intelligence model (S-AI) is booted, the O/S is stored in RAM, and various application data selected by the user can be stored in RAM.

The ROM stores a set of commands for system booting, etc. When a turn-on command is input and power is supplied, the main CPU copies the O/S stored in the memory (110) to the RAM according to the commands stored in the ROM, and executes the O/S to boot the system. When booting is complete, the main CPU copies various application programs stored in the memory (110) to the RAM, and executes the application programs copied to the RAM to perform various operations.

The graphic processing unit generates a screen including various objects such as items, images, and texts using a calculation unit (not shown) and a rendering unit (not shown). Here, the calculation unit may be configured to calculate attribute values such as coordinate values, shape, size, and color of each object to be displayed according to the layout of the screen using a control command received from the input unit. In addition, the rendering unit may be configured to generate screens of various layouts including objects based on the attribute values calculated by the calculation unit. The screen generated by the rendering unit may be displayed within the display area of the display.

The main CPU accesses the memory (110) and performs booting using the OS stored in the memory (110). Then, the main CPU performs various operations using various programs, contents, data, etc. stored in the memory (110).

The first to n interfaces are connected to the various components described above. One of the first to n interfaces may be a network interface that connects to an external device via a network.

Meanwhile, furthermore, the processor (120) can control an artificial intelligence model. In this case, it goes without saying that the processor can include a graphics-only processor (e.g., GPU) for controlling the artificial intelligence model.

Meanwhile, the artificial intelligence model according to the present invention may be a model based on supervised learning or unsupervised learning. Furthermore, the artificial intelligence model according to the present invention may include a support vector machine (SVM), a decision tree, a neural network, and the like, and methodologies applied thereto.

In one embodiment, the artificial intelligence model according to the present invention may be an artificial intelligence model based on a convolutional deep neural network (CNN) trained by inputting learning data. However, it is not limited thereto, and it goes without saying that various artificial intelligence models may be applied to the present invention. For example, models such as a DNN (Deep Neural Network), an RNN (Recurrent Neural Network), and a BRDNN (Bidirectional Recurrent Deep Neural Network) may be used as the artificial intelligence model, but are not limited thereto.

At this time, convolutional deep neural networks (CNNs) are a type of multilayer perceptrons designed to use minimal preprocessing. Convolutional neural networks consist of one or more convolutional layers and general artificial neural network layers placed on top of them, and additionally utilize weight and pooling layers. Thanks to this structure, convolutional neural networks can sufficiently utilize two-dimensional structured input data. In addition, convolutional neural networks can be trained through standard backpropagation. Convolutional neural networks have the advantage of being easier to train than other feedforward artificial neural network techniques and using a small number of parameters.

Additionally, deep neural networks (DNNs) are artificial neural networks (ANNs) that consist of multiple hidden layers between the input layer and the output layer.

At this time, the structure of the deep neural network can be composed of a perceptron. The perceptron consists of multiple input values, one processor, and one output value. The processor multiplies each input value by a weight, and then adds all the input values that have been multiplied by the weights. Then, the processor substitutes the combined value into the activation function and outputs one output value. If a specific value is desired as the output value of the activation function, the weight multiplied by each input value can be modified, and the output value can be recalculated using the modified weight. At this time, each perceptron can use a different activation function. In addition, each perceptron receives the outputs transmitted from the previous layer as input, and then uses the activation function to obtain the output. The obtained output is transmitted as the input of the next layer. After going through the above-described process, several output values can be obtained in the end.

A recurrent neural network (RNN) is a neural network in which the connections between the units that make up the artificial neural network form a directed cycle. Unlike a feed-forward neural network, a recurrent neural network can utilize the memory within the neural network to process arbitrary inputs.

Deep Belief Networks (DBN) are a generative graphical model used in machine learning, and in deep learning, they refer to a deep neural network consisting of multiple layers of latent variables. They have the characteristic that there are connections between layers, but no connections between units within a layer.

Deep belief neural networks can be used for pre-learning due to their generative nature, and after learning the initial weights through pre-learning, the weights can be fine-tuned through backpropagation or other discriminative algorithms. This characteristic is very useful when there is little training data, because the less training data there is, the more the initial values of the weights affect the resulting model. The pre-learned initial values of the weights are closer to the optimal weights than the randomly set initial values of the weights, and this enables improved performance and speed in the fine-tuning stage.

The above-described artificial intelligence and its learning method are described for illustrative purposes, and the artificial intelligence and its learning method used in the above-described embodiments are not limited. For example, all types of artificial intelligence technologies and their learning methods that can be applied by a person skilled in the art to solve the same problem can be utilized to implement the system according to the disclosed embodiment.

Meanwhile, the processor (120) may include one or more cores (not shown) and a graphics processing unit (not shown) and/or a connection path (e.g., a bus, etc.) for transmitting and receiving signals with other components.

A processor (120) according to one embodiment performs a method described in connection with the present invention by executing one or more instructions stored in a memory (110).

For example, the processor (120) may acquire new learning data by executing one or more instructions stored in the memory (110), perform a test on the acquired new learning data using a learned model, extract first learning data for which labeled information is acquired with an accuracy higher than a predetermined first reference value as a result of the test, delete the extracted first learning data from the new learning data, and re-learn the learned model using the new learning data from which the extracted learning data has been deleted.

Meanwhile, the processor (120) may further include a RAM (Random Access Memory, not shown) and a ROM (Read-Only Memory, not shown) that temporarily and/or permanently store signals (or data) processed within the processor (120). In addition, the processor (120) may be implemented in the form of a system on chip (SoC) that includes at least one of a graphics processing unit, a RAM, and a ROM.

The memory (110) can store programs (one or more instructions) for processing and controlling the processor (120). The programs stored in the memory (110) can be divided into multiple modules according to function.

FIG. 2 is a system configuration diagram of a device that performs an evaluation for each evaluation item on speaking ability using a speaking scoring artificial intelligence model (S-AI) according to one embodiment of the present invention.

Referring to FIG. 2, a device (100) that performs an evaluation on speaking ability by evaluation item using a speaking scoring artificial intelligence model (S-AI) according to one embodiment of the present invention may further include a communication unit (130) that communicates with at least one terminal (200) to obtain evaluation data.

The communication unit (130) can perform communication with an external device. In particular, the communication unit (130) can include various communication chips such as a WiFi chip, a Bluetooth chip, a wireless communication chip, an NFC chip, a low-power Bluetooth chip (BLE chip), etc. At this time, the WiFi chip, the Bluetooth chip, and the NFC chip perform communication in a LAN method, a WiFi method, a Bluetooth method, and an NFC method, respectively. When a WiFi chip or a Bluetooth chip is used, various connection information such as an SSID and a session key are first transmitted and received, and then communication is connected using this, and then various pieces of information can be transmitted and received. The wireless communication chip refers to a chip that performs communication according to various communication standards such as IEEE, Zigbee, 3G (3rd Generation), 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), 5G (5th Generation), etc.

Meanwhile, the terminal (200) may be an electronic device possessed by the subject of the evaluation.

In one embodiment, the terminal (200) may include at least one of a server, a smartphone, a tablet personal computer, a mobile phone, a video phone, a laptop PC, a netbook computer, a notebook computer, a personal digital assistant (PDA), a portable multimedia player (PMP), and a wearable device.

FIG. 3 is a configuration diagram of a device that performs an evaluation on speaking ability for each evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention that further includes an input/output unit.

Referring to FIG. 3, a device (100) that performs an evaluation on speaking ability by evaluation item using a speaking scoring artificial intelligence model (S-AI) according to one embodiment of the present invention may further include an input/output unit (140) that obtains evaluation data based on a user command and provides at least one piece of information to the user.

The input/output unit (140) can obtain user input and output information from the electronic device (100) by including an external device connection port, a scanning device, a camera, a display, a mouse, a keyboard, a remote control, etc., to provide visualized information to the user.

The camera is configured to capture at least one subject, and the processor (120) can recognize at least one object or identify a distance to the object through the camera.

A display is a device that visually outputs various information.

The display may be implemented as a liquid crystal display (LCD), a plasma display panel (PDP), an organic light emitting diode (OLED), a transparent OLED (TOLED), a micro LED, etc., but is not limited thereto and may include various other forms of displays known in the art. The display may be implemented as a touch screen capable of detecting a user's touch operation, and may also be implemented as a flexible display capable of being folded or bent.

A device (100) for evaluating speaking ability by evaluation item using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention may obtain evaluation data, evaluation item type information, training evaluation data (EP-T), training scoring data (SD-T), training item type information, and standard scoring data through an input/output unit (140), or may obtain evaluation data, evaluation item type information, training evaluation data (EP-T), training scoring data (SD-T), training item type information, and standard scoring data from an external device or terminal (200) through a communication unit (130).

FIG. 4 is a diagram for explaining a process in which a speaking scoring artificial intelligence model (S-AI) is learned by a device that performs an evaluation for each evaluation item of speaking ability using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention, and FIG. 5 is a diagram for explaining a process in which a device that performs an evaluation for each evaluation item of speaking ability using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention outputs scoring data, which is a result of determining an evaluation score of an evaluatee using a speaking scoring artificial intelligence model (S-AI), and FIG. 6 is a diagram illustrating an example of evaluation items that can be evaluated by a device that performs an evaluation for each evaluation item of speaking ability using a speaking scoring artificial intelligence model (S-AI) according to an embodiment of the present invention.

Referring further to FIGS. 4 to 6, the processor (120) configures training data for training a speaking scoring artificial intelligence model (S-AI), and uses the configured training data to train the speaking scoring artificial intelligence model (S-AI) to determine an evaluation score for each of one or more evaluation items among a plurality of evaluation items for the speaking ability of the evaluator.

To this end, the processor (120) can match training evaluation data (EP-T), which is evaluation data of other evaluators for training evaluation items, which are evaluation items prepared for learning of a speaking scoring artificial intelligence model (S-AI), and training scoring data (SD-T), which represents evaluation scores scored by a scorer based on the training evaluation data (EP-T), to form training data.

At this time, the processor (120) can select only specific evaluation items as training evaluation items.

Specifically, the processor (120) can analyze each of the arbitrary evaluation items to identify the evaluation items by which the speaking ability of the assessee can be evaluated. At this time, the processor (120) can identify the evaluation items by using information about the evaluation items that are matched in advance to each of the arbitrary evaluation items.

Here, the evaluation items may include, as illustrated in Figure 6, a pronunciation evaluation item that evaluates pronunciation, a vocabulary evaluation item that evaluates vocabulary, a grammar evaluation item that evaluates grammar, a sentence composition evaluation item that evaluates sentence composition, and a topic suitability evaluation item that evaluates topic suitability.

Additionally, each evaluation item may include evaluation subitems.

Specifically, the pronunciation assessment items may include at least one of a first pronunciation assessment subitem assessing pronunciation fluency, a second pronunciation assessment subitem assessing pronunciation accuracy, and a third pronunciation assessment subitem assessing speech rate.

Additionally, the vocabulary assessment items may include at least one of a first vocabulary assessment sub-item assessing vocabulary diversity, a second vocabulary assessment sub-item assessing vocabulary accuracy, and a third vocabulary assessment sub-item assessing vocabulary difficulty.

The grammar evaluation items may include one or more of the first grammar evaluation sub-item evaluating grammatical diversity, the second grammar evaluation sub-item evaluating grammatical accuracy, and the third grammar evaluation sub-item evaluating grammatical difficulty.

The sentence composition ability assessment items may include the first sentence composition ability assessment subitem that evaluates the ability to construct sentences.

The topic suitability assessment items may include a first topic suitability assessment subitem that evaluates the degree of topic match.

Accordingly, the speaking scoring artificial intelligence model (S-AI) can be trained to determine evaluation scores for each of the pronunciation evaluation items, vocabulary evaluation items, grammar evaluation items, sentence structure evaluation items, and topic appropriateness evaluation items for the speaking ability of the assessee.

The processor (120) can check the learning difficulty of the speaking scoring artificial intelligence model (S-AI) corresponding to each of the identified evaluation items, and select only the evaluation items that evaluate the evaluation items whose identified learning difficulty exceeds the standard difficulty as training evaluation items, while not selecting the evaluation items that evaluate the evaluation items whose identified learning difficulty is lower than the standard difficulty as training evaluation items.

Here, learning difficulty can be the learning difficulty that allows the speaking scoring artificial intelligence model (S-AI) to determine the evaluation score of the corresponding evaluation item.

For example, higher learning difficulty may require more time to learn and more training data.

Meanwhile, the processor (120) can match the training evaluation data (EP-T) described above with the training scoring data (SD-T) representing the evaluation score, which is the result of scoring by the scorer, based on the training evaluation data (EP-T), which is the evaluation data of another assessee for the training evaluation items, to form training data.

At this time, the processor (120) can generate scoring criteria information that serves as a reference and basis for the scorer when the scorer performs scoring based on the training evaluation data (EP-T).

Specifically, the processor (120) can generate scoring criteria information indicating scoring criteria based on reference scoring data, which is scoring data obtained for setting scoring criteria and is different from training scoring data (SD-T).

More specifically, the processor (120) can analyze the criterion scoring data, which is a result of previous scoring by scorers, to identify evaluation items with a high scoring frequency among a plurality of evaluation items, and generate scoring criterion information so that the identified evaluation items are scored.

This scoring criteria information is provided to the scorer so that the scorer can refer to it when scoring the training evaluation data (EP-T).

Meanwhile, the processor (120) performs preprocessing on training evaluation data (EP-T), training scoring data (SD-T), and evaluation data (EP), trains a speaking scoring artificial intelligence model (S-AI) using training data composed of the preprocessed training evaluation data (EP-T) and training scoring data (SD-T), and inputs the preprocessed evaluation data (EP) as input data to the speaking scoring artificial intelligence model (S-AI).

Here, preprocessing may mean that the processor (120) performs at least one of the following: quality improvement for reducing noise in the data, converting voice into text in the data, morpheme separation in the data, counting the total number of tokens and meaningful tokens in the data, part-of-speech tagging in the data by morpheme, and matching the data with vocabulary and grammar lists of each of levels 1 to 6 classified based on the international standard Korean language curriculum when the language type of the data is Korean.

Meanwhile, the processor (120) can further configure training item type information indicating the type of training evaluation item corresponding to the training evaluation data (EP-T) as training data.

Thereafter, when evaluation data (EP) is obtained from the evaluator who has been provided with evaluation items, the processor (120) can input the evaluation data (EP) as input data to the speaking scoring artificial intelligence model (S-AI), and receive the scoring data (SD), which is the result of determining the evaluation score of the evaluator, as output data from the speaking scoring artificial intelligence model (S-AI).

At this time, the processor (120) checks the evaluation item type information indicating the type of evaluation item corresponding to the evaluation data (EP), and inputs the evaluation item type information together with the evaluation data (EP) as input data to the speaking scoring artificial intelligence model (S-AI) so that the evaluation subject's scoring data (SD) can be output as output data from the speaking scoring artificial intelligence model (S-AI).

Meanwhile, each of the training item type information and the evaluation item type information may be identical to the evaluation item type information given in the speaking test of the “Test of Proficiency in Korean (TOPIK).”

For example, the training item type information and the evaluation item type information can each be one of the following: type 1, in which the assessee reads along with the text provided to the assessee; type 2, in which the assessee describes a photograph or drawing provided to the assessee; type 3, in which the assessee listens to simple questions in situations that they frequently encounter in their daily lives and answers them appropriately; type 4, in which the assessee listens to simple questions while looking at pictures of situations that they frequently encounter in their daily lives and plays a role appropriately; type 5, in which the assessee responds appropriately to what the other person says in accordance with the context of the conversation; type 6, in which the assessee reports and interprets materials on social or abstract topics such as economics, science, mass media, culture, art, politics, and the environment, and critically states his or her own opinion; and type 7, in which the assessee logically presents his or her own opinion on specialized fields, abstract content, social issues, etc., or presents his or her opinion from a pro or con perspective.

Meanwhile, the scoring data (SD) may include evaluation scores for each evaluation item, or may include a combined score that is the sum of all evaluation scores for each evaluation item.

At this time, the processor (120) can determine the speaking ability evaluation grade of the person being evaluated based on the total score. Specifically, the processor (120) can determine the speaking ability evaluation grade so that a higher total score indicates a higher speaking ability of the person being evaluated.

Thereafter, the processor (120) may select a low-ability evaluation item among a plurality of evaluation items as a low-ability evaluation item based on the scoring data, select a high-ability evaluation item among a plurality of evaluation items as a high-ability evaluation item based on the scoring data, and generate selection result data indicating the selection results of the low-ability evaluation items and the selection results of the high-ability evaluation items.

Next, the processor (120) can generate learning content recommendation information that recommends learning content that can improve speaking ability corresponding to the low-ability evaluation item based on the selection result data.

Such learning content recommendation information can be output by the terminal (200).

In addition, the processor (120) can generate learning method information that can improve the evaluation score of each evaluation item based on the selection result data. This learning method information can also be output by the terminal (200).

Meanwhile, the speaking scoring artificial intelligence model (S-AI) of the present invention may be an explainable artificial intelligence model (eXplainable Artificial Intelligence; XAI) that provides explanatory information that enables the evaluator to understand the results (output data, scoring data of the present invention) output from the artificial intelligence model.

In addition, the speaking scoring artificial intelligence model (S-AI) in the present invention may be an artificial intelligence model based on dynamic weights with non-fixed weights.

Meanwhile, a processor (120) according to another embodiment may obtain answer information submitted by an examinee who took the speaking evaluation of the “Test of Proficiency in Korean (TOPIK)” as an answer to the speaking evaluation of the “Test of Proficiency in Korean (TOPIK)”, input the answer information as evaluation data into a speaking scoring artificial intelligence model (S-AI) as input data, and receive, from the speaking scoring artificial intelligence model (S-AI), as output data, the scoring data (hereinafter referred to as “TOPIK scoring data”), which is a result of determining an evaluation score of the speaking evaluation of the Test of Proficiency in Korean (TOPIK) taken by the examinee.

Thereafter, a processor (120) according to another embodiment may determine a TOPIK-based speaking ability evaluation grade of the assessee based on a combined score of evaluation scores for each evaluation item of the TOPIK speaking evaluation, which is included in the topic scoring data.

Additionally, a processor (120) according to another embodiment can obtain actual scoring data of a speaking test of the Test of Proficiency in Korean (TOPIK) taken by the examinee.

Here, the actual scoring data may be the speaking ability evaluation grade of the examinee, which is the result of an evaluation by the organization that administered the speaking test of the Test of Proficiency in Korean (TOPIK).

Thereafter, a processor (120) according to another embodiment can calculate the difference in grade between the speaking ability evaluation grade of the actual scoring data and the speaking ability evaluation grade based on TOPIK as a scoring error rate.

A processor (120) according to another embodiment can calculate the grade difference using the following mathematical formula.

Mathematical formula: R = | (G1 - G2) / N _G |

Here, R is the scoring error rate, G1 is the speaking ability evaluation grade of the actual scoring data, G2 is the speaking ability evaluation grade based on TOPIK, and N _G is the number of speaking ability evaluation grades determined in the speaking evaluation of the Test of Proficiency in Korean (TOPIK).

Hereinafter, a process of determining an evaluation score for a pronunciation evaluation item by a device (100) that performs an evaluation for each evaluation item on speaking ability using a speaking scoring artificial intelligence model (S-AI) according to the present invention will be described.

FIG. 7 is a diagram for explaining the process of determining an evaluation score for a pronunciation evaluation item, which is an evaluation item that evaluates pronunciation among a plurality of evaluation items for the speaking ability of an assessee, by a device that performs an evaluation for each evaluation item of speaking ability using a speaking scoring artificial intelligence model (S-AI) according to one embodiment of the present invention.

Referring to FIG. 7, if the question type is the first type that reads along with the fingerprint provided to the assessee, the processor (120) can control the communication unit (130) to transmit the fingerprint to the terminal (200) so that the assessee can read the fingerprint.

The terminal (200) can output information on question types, information on question solving methods, and fingerprints.

The assessee can read the fingerprint and speak out loud to read the fingerprint. Accordingly, the terminal (200) obtains the first assessee voice data of the assessee reading along the fingerprint and transmits it to the communication unit (130), and the processor (120) obtains the first assessee voice data as evaluation data and inputs it as input data to the speech scoring artificial intelligence model (S-AI), and can receive scoring data, which is voice characteristic difference data representing the voice characteristic difference between the native speaker voice data of the native speaker reading along the fingerprint and the first assessee voice data, as output data from the speech scoring artificial intelligence model (S-AI).

Here, the voice characteristic difference may include one or more of voice frequency, voice loudness, voice pause period, and prosody.

Through this, the processor (120) can determine and obtain an evaluation score for the first pronunciation evaluation sub-item that evaluates pronunciation fluency among the multiple evaluation sub-items of the pronunciation evaluation item, which is one of the multiple evaluation items.

In addition, the processor (120) may obtain the first evaluator's voice data as evaluation data and input it as input data to a speech scoring artificial intelligence model (S-AI), and output the scoring data, which is pronunciation difference data indicating the pronunciation difference between the pronunciation data corresponding to the fingerprint and the first evaluator's voice data, as output data from the speech scoring artificial intelligence model (S-AI).

Here, pronunciation difference data may be information that is obtained using the ASR (Automatic Speech Recognition) method and represents differences in pronunciation.

Through this, the processor (120) can determine and obtain an evaluation score for a second pronunciation evaluation sub-item that evaluates pronunciation accuracy among multiple evaluation sub-items of a pronunciation evaluation item, which is one of multiple evaluation items.

In addition, the processor (120) may obtain the first evaluator's voice data as evaluation data and input it as input data to a speech scoring artificial intelligence model (S-AI), and receive the scoring data, which is voice speaking rate data indicating the voice speaking rate of the first evaluator's voice data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for a third pronunciation evaluation sub-item that evaluates speech speed among multiple evaluation sub-items of a pronunciation evaluation item, which is one of multiple evaluation items.

Hereinafter, a process of determining evaluation scores for vocabulary and grammar evaluation items by a device (100) that performs evaluation for each evaluation item on speaking ability using a speaking scoring artificial intelligence model (S-AI) according to the present invention will be described.

FIG. 8 is a diagram for explaining the process of determining evaluation scores for each of the vocabulary evaluation item, which evaluates vocabulary, and the grammar evaluation item, which evaluates grammar, among multiple evaluation items for the speaking ability of an assessee, by a device that performs an evaluation for each evaluation item of speaking ability using a speaking scoring artificial intelligence model (S-AI) according to one embodiment of the present invention.

Referring to FIG. 8, if the type of item indicated by the evaluation item type information is the second type that describes a photo or picture provided to the evaluator, the processor (120) can control the communication unit (130) to transmit the photo or picture to the terminal (200) so that the evaluator can view the photo or picture.

The terminal (200) can output information about the question type, information about how to solve the question, and a photo or picture.

The evaluator can describe the photo or picture by speaking while looking at the photo or picture. Accordingly, the terminal (200) can obtain the second evaluator voice data of the evaluator describing the photo or picture and transmit it to the communication unit (130).

At this time, the processor (120) may obtain the second evaluator's voice data as evaluation data and input it as input data to the speech scoring artificial intelligence model (S-AI), and may receive the scoring data, which is stem count information indicating the number of stems included in the second evaluator's voice data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for the first vocabulary evaluation sub-item that evaluates vocabulary diversity among the multiple evaluation sub-items of the vocabulary evaluation item, which is one of the multiple evaluation items.

In addition, the processor (120) may obtain second evaluator voice data as evaluation data, input it as input data to a speech scoring artificial intelligence model (S-AI), and output scoring data, which is first text matching rate information indicating a matching rate between original text data converted into text from the second evaluator voice data and vocabulary-corrected text data that corrects vocabulary usage errors in the original text data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for a second vocabulary evaluation sub-item that evaluates vocabulary accuracy among multiple evaluation sub-items of a vocabulary evaluation item, which is one of multiple evaluation items.

In addition, the processor (120) may obtain the second evaluator's voice data as evaluation data and input it as input data to the speech scoring artificial intelligence model (S-AI), and output the scoring data, which is vocabulary difficulty level information indicating the vocabulary difficulty level to which the vocabulary included in the second evaluator's voice data belongs among a plurality of vocabulary difficulty levels, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for a third vocabulary evaluation sub-item that evaluates vocabulary difficulty among multiple evaluation sub-items of a vocabulary evaluation item, which is one of multiple evaluation items.

Here, the scoring data, which is vocabulary difficulty level information, may include a vocabulary count indicating the number of vocabulary (vocabularies included in the second evaluator's speech data) included in each of multiple vocabulary difficulty levels.

Here, the multiple vocabulary difficulty levels may be levels 1 to 6, which are difficulty levels of vocabulary classified based on the international standard Korean language curriculum, and the multiple vocabulary difficulty levels may be matched with vocabulary lists that are each matched with levels 1 to 6, which are difficulty levels of vocabulary classified based on the international standard Korean language curriculum.

Using this, the processor (120) can receive scoring data, which is vocabulary distribution data indicating the distribution of multiple vocabulary difficulty levels for vocabulary included in the second evaluator's voice data, as output data from the speaking scoring artificial intelligence model (S-AI).

In addition, the processor (120) can output scoring data, which is a result of determining the vocabulary difficulty level of the evaluator based on the results of comparing the vocabulary distribution data with the reference vocabulary distribution set for each of a plurality of vocabulary difficulty levels, as output data from the speaking scoring artificial intelligence model (S-AI).

At this time, the speaking scoring artificial intelligence model (S-AI) can determine the vocabulary difficulty level of the examinee among multiple vocabulary difficulty levels in which the vocabulary distribution data indicates a vocabulary difficulty level that exceeds the standard vocabulary distribution level, and output the scoring data, which is the determined result, as output data.

If there are multiple vocabulary difficulty levels for which the vocabulary distribution data indicates a distribution level that exceeds the standard vocabulary distribution level, the speaking scoring artificial intelligence model (S-AI) can determine the vocabulary difficulty level with the highest difficulty level among the multiple vocabulary difficulty levels for which the vocabulary distribution data indicates a distribution level that exceeds the standard vocabulary distribution level as the vocabulary difficulty level of the assessee.

To this end, the processor (120) may set a lower standard vocabulary distribution for each of a plurality of vocabulary difficulty levels as the difficulty level increases, and may set the same standard vocabulary distribution for adjacent difficulty levels. For example, standard vocabulary distributions of 25%, 25%, 20%, 15%, 10%, and 5% may be set for each of a plurality of vocabulary difficulty levels, i.e., levels 1 to 6.

Accordingly, the speaking scoring artificial intelligence model (S-AI) can determine the vocabulary difficulty level of the assessee as level 5 if the distribution of level 5 among multiple vocabulary difficulty levels exceeds the standard vocabulary distribution of 10% at 12%, and the distributions of the remaining levels 1 to 4 and level 6 are each lower than the standard vocabulary distribution.

As another example, the speaking scoring artificial intelligence model (S-AI) can determine that among multiple vocabulary difficulty levels, if the distribution of level 5 exceeds the standard vocabulary distribution of 10% at 12%, the distribution of level 6 exceeds the standard vocabulary distribution of 5% at 7%, and the distributions of the remaining levels 1 to 4 are each lower than the standard vocabulary distribution, the vocabulary difficulty level of the assessee can be level 6, which has the highest difficulty level among the multiple vocabulary difficulty levels exceeding the standard vocabulary distribution.

In addition, the processor (120) may obtain the second evaluator's voice data as evaluation data and input it as input data to a speech scoring artificial intelligence model (S-AI), and receive scoring data, which is information on the number of endings and particles included in the second evaluator's voice data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for the first grammar evaluation sub-item that evaluates grammar diversity among the multiple evaluation sub-items of the grammar evaluation item, which is one of the multiple evaluation items.

Meanwhile, the processor (120) may obtain second evaluator voice data as evaluation data and input it as input data to a speech scoring artificial intelligence model (S-AI), and output scoring data, which is second text matching rate information indicating the matching rate between original text data converted into text from the second evaluator voice data and grammar-corrected text data that corrects grammatical errors in the original text data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for a second grammar evaluation sub-item that evaluates grammatical accuracy among multiple evaluation sub-items of a vocabulary evaluation item, which is one of multiple evaluation items.

In addition, the processor (120) may obtain the second evaluator's voice data as evaluation data and input it as input data to the speaking scoring artificial intelligence model (S-AI), and may receive scoring data, which is grammar usage difficulty level information indicating the grammar usage difficulty level to which the grammar usages included in the second evaluator's voice data belong among a plurality of grammar usage difficulty levels, as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for a third grammar evaluation sub-item that evaluates grammar difficulty among multiple evaluation sub-items of a grammar evaluation item, which is one of multiple evaluation items.

Here, the scoring data, which is information on the difficulty level of grammar usage, may include the number of grammar usages indicating the number of grammar usages (grammar usages included in the second evaluator's speech data) included in each of the multiple grammar usage difficulty levels.

Here, the plurality of grammar usage difficulty levels may be levels 1 to 6, which are difficulty levels of grammar usage classified based on the international standard Korean language curriculum, and the plurality of grammar usage difficulty levels may be matched with lists of grammar usages that are each matched with levels 1 to 6, which are difficulty levels of grammar usage classified based on the international standard Korean language curriculum.

Using this, the processor (120) can receive scoring data, which is grammar usage distribution data indicating the distribution of multiple grammar usage difficulty levels for grammar usages included in the second evaluator's voice data, as output data from the speaking scoring artificial intelligence model (S-AI).

In addition, the processor (120) can output scoring data, which is a result of determining the grammar usage difficulty level of the evaluator based on the results of comparing the grammar usage distribution data with the standard grammar usage distribution map set for each of a plurality of grammar usage difficulty levels, as output data from the speaking scoring artificial intelligence model (S-AI).

At this time, the speaking scoring artificial intelligence model (S-AI) can determine the grammar usage difficulty level of the assessee as the grammar usage difficulty level whose distribution data indicates a grammar usage distribution level that exceeds the standard grammar usage distribution level among multiple grammar usage difficulty levels, and output the scoring data, which is the determined result, as output data.

If there are multiple grammar usage difficulty grades for which the distribution indicated by the grammar usage distribution data exceeds the standard grammar usage distribution, the speaking scoring artificial intelligence model (S-AI) can determine the grammar usage difficulty grade with the highest difficulty among the multiple grammar usage difficulty grades for which the distribution indicated by the grammar usage distribution data exceeds the standard grammar usage distribution as the grammar usage difficulty grade of the assessee.

To this end, the processor (120) may set the standard grammar usage distribution to be lower for each of a plurality of grammar usage difficulty levels as the difficulty level increases, and may set the standard grammar usage distribution to be the same for adjacent difficulty levels.

For example, for each of the multiple grammar usage difficulty levels 1 to 6, the standard grammar usage distributions can be set to 25%, 25%, 20%, 15%, 10%, and 5%, respectively.

Accordingly, the speaking scoring artificial intelligence model (S-AI) can determine the grammar usage difficulty level of the assessee as level 5 if the distribution of level 5 among multiple grammar usage difficulty levels is 12%, which exceeds the standard grammar usage distribution of 10%, and the distributions of the remaining levels 1 to 4 and level 6 are each lower than the standard grammar usage distribution.

As another example, the speaking scoring artificial intelligence model (S-AI) can determine that among multiple grammar usage difficulty levels, if the distribution of level 5 is 12%, which exceeds the standard grammar usage distribution of 10%, the distribution of level 6 is 7%, which exceeds the standard grammar usage distribution of 5%, and the distributions of the remaining levels 1 to 4 are each lower than the standard grammar usage distribution, the grammar usage difficulty level of the assessee can be level 6, which has the highest difficulty level among the multiple grammar usage difficulty levels that exceed the standard grammar usage distribution.

Hereinafter, a process of determining evaluation scores for each of a pronunciation evaluation item, a sentence composition evaluation item, and a topic appropriateness evaluation item among multiple evaluation items for the speaking ability of the evaluatee, by a device (100) including a speaking scoring artificial intelligence model for evaluating the response of an evaluatee to an evaluation item according to an embodiment of the present invention, will be described.

The processor (120) obtains the second evaluator's voice data as evaluation data and inputs it as input data to a speaking scoring artificial intelligence model (S-AI), and outputs scoring data, which is voice characteristic difference data indicating the voice characteristic difference between the second evaluator's voice data and the native speaker's voice data that reads the original text data converted into text from the second evaluator's voice data, as output data from the speaking scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for the first pronunciation evaluation sub-item that evaluates pronunciation fluency among the multiple evaluation sub-items of the pronunciation evaluation item, which is one of the multiple evaluation items.

In addition, the processor (120) may obtain second evaluator voice data as evaluation data and input it as input data to a speech scoring artificial intelligence model (S-AI), and output scoring data, which is pronunciation difference data indicating a pronunciation difference between the second evaluator voice data and pronunciation data corresponding to original text data converted into text from the second evaluator voice data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for a second pronunciation evaluation sub-item that evaluates pronunciation accuracy among multiple evaluation sub-items of a pronunciation evaluation item, which is one of multiple evaluation items.

In addition, the processor (120) may obtain the second evaluator's voice data as evaluation data and input it as input data to the speech scoring artificial intelligence model (S-AI), and receive the scoring data, which is voice speaking rate data indicating the voice speaking rate of the second evaluator's voice data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for a third pronunciation evaluation sub-item that evaluates speech speed among multiple evaluation sub-items of a pronunciation evaluation item, which is one of multiple evaluation items.

Meanwhile, the processor (120) may obtain the second evaluator's voice data as evaluation data, input it as input data to the speech scoring artificial intelligence model (S-AI), and receive the scoring data, which is sentence composition data indicating the dependencies between words in sentences included in the second evaluator's voice data and the sentence pattern suitability of sentences included in the second evaluator's voice data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for the first sentence composition ability evaluation sub-item that evaluates sentence composition ability among the multiple evaluation sub-items of the sentence composition ability evaluation item, which is one of the multiple evaluation items.

To this end, the speech scoring AI model (S-AI) can output sentence composition data that represents the above-described dependency and sentence pattern fit using word embeddings (e.g., BERT (Bidirectional Encoder Representations from Transformers) and Word2Vec).

Meanwhile, the processor (120) may obtain the second evaluator's voice data as evaluation data, input it as input data to the speech scoring artificial intelligence model (S-AI), and receive the scoring data, which is subject match rate information indicating the match rate between words included in the second evaluator's voice data and words included in the answer data, as output data from the speech scoring artificial intelligence model (S-AI).

Through this, the processor (120) can determine and obtain an evaluation score for the first subject suitability evaluation sub-item that evaluates the degree of subject consistency among the multiple evaluation sub-items of the subject suitability evaluation item, which is one of the multiple evaluation items.

Above, while the embodiments of the present invention have been described with reference to the attached drawings, it will be understood by those skilled in the art that the present invention may be implemented in other specific forms without changing the technical idea or essential features thereof. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

100: A device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model
110 : Memory
120 : Processor
130 : Communication Department
140 : Input/output section

Claims (9)

A memory including a speaking scoring artificial intelligence model that is trained to determine an evaluation score for each of the vocabulary evaluation item, which is an evaluation item evaluating vocabulary, and the grammar evaluation item, which is an evaluation item evaluating grammar, among multiple evaluation items for the speaking ability of the assessee; and
A processor for obtaining evaluation data, which is the response of the assessee to the evaluation items for evaluating the speaking ability, inputting the evaluation data as input data into the speaking scoring artificial intelligence model, and receiving the evaluation data, which is the result of determining the evaluation score of the assessee, as output data from the speaking scoring artificial intelligence model;
The above vocabulary assessment items are
Contains at least one of the following vocabulary assessment sub-items: the first vocabulary assessment sub-item that evaluates vocabulary diversity, the second vocabulary assessment sub-item that evaluates vocabulary accuracy, and the third vocabulary assessment sub-item that evaluates vocabulary difficulty;
The above grammar assessment items are
Contains at least one of the first grammar assessment sub-item that evaluates grammatical diversity, the second grammar assessment sub-item that evaluates grammatical accuracy, and the third grammar assessment sub-item that evaluates grammatical difficulty.
The above processor
Training data is configured by matching training evaluation data, which is evaluation data of other evaluators for training evaluation items, which are evaluation items prepared for learning the above-mentioned speaking scoring artificial intelligence model, and training scoring data, which is an evaluation score scored by a scorer based on the above-mentioned training evaluation data, and training the above-mentioned speaking scoring artificial intelligence model using the above-mentioned training data.
The above processor
Further configuring training item type information indicating the type of the training evaluation item corresponding to the above training evaluation data into the above training data,
Confirm the evaluation item type information indicating the type of the evaluation item corresponding to the evaluation data, input the evaluation item type information together with the evaluation data into the speaking scoring artificial intelligence model as the input data, and output the evaluation data of the evaluated person as the output data from the speaking scoring artificial intelligence model.
The above processor
If the above question type is a type that describes a photo or picture provided to the above assessee, the second assessee voice data of the above assessee describing the photo or picture is acquired as the evaluation data and inputted into the speaking scoring artificial intelligence model as the input data, and the scoring data, which is the number of suffixes and particle numbers indicating the number of suffixes and particle numbers included in the second assessee voice data, is output as output data from the speaking scoring artificial intelligence model (S-AI).
A device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model.
delete delete In the first paragraph,
The above processor
If the above question type is a type that describes a photo or picture provided to the above-mentioned assessee, the second assessee voice data of the above-mentioned assessee describing the photo or picture is acquired as the evaluation data and inputted into the above-mentioned speaking scoring artificial intelligence model as the input data, and the scoring data, which is the stem count information indicating the number of stems included in the above-mentioned second assessee voice data, is output as output data from the above-mentioned speaking scoring artificial intelligence model.
A device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model.
In the first paragraph,
The above processor
If the above question type is a type that describes a photo or picture provided to the above assessee, the second assessee voice data of the above assessee describing the photo or picture is acquired as the evaluation data and inputted into the speaking scoring artificial intelligence model as the input data, and the scoring data, which is first text matching rate information indicating the matching rate between the original text data converted into text from the second assessee voice data and the vocabulary-corrected text data that corrects vocabulary usage errors in the original text data, is output as output data from the speaking scoring artificial intelligence model (S-AI).
A device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model.
In the first paragraph,
The above processor
If the above question type is a type that describes a photo or picture provided to the above-mentioned assessee, the second assessee voice data of the above-mentioned assessee describing the photo or picture is acquired as the evaluation data and inputted into the speaking scoring artificial intelligence model as the input data, and the scoring data, which is vocabulary difficulty level information indicating the vocabulary difficulty level to which the words included in the second assessee voice data belong among a plurality of vocabulary difficulty levels, is output as output data from the speaking scoring artificial intelligence model (S-AI).
A device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model.
delete In the first paragraph,
The above processor
If the above question type is a second type that describes a photo or picture provided to the above assessee, the second assessee voice data of the above assessee describing the photo or picture is acquired as the evaluation data and inputted into the speaking scoring artificial intelligence model as the input data, and the scoring data, which is second text matching rate information indicating the matching rate between the original text data converted into text from the second assessee voice data and the grammar-corrected text data that corrects grammatical errors in the original text data, is output as output data from the speaking scoring artificial intelligence model.
A device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model.
In the first paragraph,
The above processor
If the above question type is a type that describes a photo or picture provided to the above assessee, the second assessee voice data of the above assessee describing the photo or picture is acquired as the evaluation data and inputted into the speaking scoring artificial intelligence model as the input data, and the scoring data, which is grammar usage difficulty grade information indicating the grammar usage difficulty grade to which the grammar usages included in the second assessee voice data belong among a plurality of grammar usage difficulty grades, is output as output data from the speaking scoring artificial intelligence model (S-AI).
A device that evaluates grammar and vocabulary skills using a speaking scoring artificial intelligence model.