JP7559762B2 - Information processing device, information processing method, and program - Google Patents

Description

This technology relates to an information processing device, an information processing method, and a program that can be applied to the learning process of a predictive model using machine learning.

Conventionally, technology has been developed to build predictive models using machine learning. By constructing a predictive model appropriately, it becomes possible to perform a variety of predictive processing. A predictive model is built by training a large amount of data, but this training process can take time.

For example, Patent Document 1 describes a system that can add hardware resources during deep learning processing. In this system, an add button for adding hardware resources is presented to the user along with the progress of the learning processing. This allows the user to add hardware resources to improve the speed of the learning processing, for example, when the learning processing is not progressing well (see paragraphs [0030], [0034], [0035], Figure 4, etc. of the specification of Patent Document 1).

JP 2017-182114 A

In the learning process of predictive models, as described above, increasing the computational resources makes it possible to train a large amount of data in a short period of time. On the other hand, in cases where increasing the amount of data to be trained does not improve prediction accuracy, time and money may be wasted. For this reason, there is a demand for technology that can train predictive models efficiently.

In view of the above circumstances, the object of the present technology is to provide an information processing device, an information processing method, and a program capable of efficiently learning a predictive model.

In order to achieve the above object, an information processing device according to an embodiment of the present technology includes an acquisition unit and an estimation processing unit.
The acquisition unit acquires features of a partial dataset that is a part of an entire dataset used to generate a prediction model.
The estimation processing unit estimates accuracy information representing a prediction accuracy of the prediction model generated using the entire data set, based on the feature amount of the partial data set.

In this information processing device, feature quantities of a partial dataset from among the entire dataset are acquired. Based on these feature quantities, accuracy information is estimated that represents the prediction accuracy when a predictive model is generated using the entire dataset. This makes it possible to determine, for example, whether or not the entire dataset should be used, and to efficiently generate a predictive model.

The estimation processing unit may estimate, as the accuracy information, a change in the prediction accuracy of the predictive model generated using the entire dataset relative to the prediction accuracy of the predictive model generated using the partial dataset.

The estimation processing unit may be configured using an estimation model that estimates changes in the prediction accuracy.

The estimation model may be a model that has learned the relationship between features of a portion of a specified dataset and the change in prediction accuracy that occurs when a specified prediction model is generated using all or part of the specified dataset.

The estimation model may be a classification model that classifies the amount of change in prediction accuracy into multiple levels.

The estimation model may be a rule-based approximation of a classification model that classifies the amount of change in prediction accuracy into multiple levels.

The estimation model may be a regression model that estimates the amount of change in the prediction accuracy.

The features of the partial data set may include a first feature corresponding to the content of the partial data set. In this case, the acquisition unit may calculate the first feature by analyzing the partial data set.

The first feature may include at least one of the number of data included in the partial data set, the number of features included in the data, and the ratio between the number of data and the number of features included in the data.

The features of the partial dataset may include a second feature corresponding to the predictive model generated using the partial dataset. In this case, the acquisition unit may calculate the second feature by executing a generation process of the predictive model using the partial dataset.

The partial data set may include a plurality of data groups having different uses. In this case, the second feature may include at least one of an evaluation value for evaluating a predicted value of the prediction model generated using the partial data set for each of the plurality of data groups, or a comparison value for comparing the evaluation values.

The plurality of data groups may include a group of training data, a group of validation data, and a group of test data.

The evaluation values may include at least one of the median error, mean squared error, and median error rate for the predicted values of the predictive model generated using the partial dataset.

The comparison value may include at least one of the difference or ratio of the evaluation values calculated for two of the plurality of data groups.

The information processing device may further include a screen generation unit that generates a screen presenting the accuracy information.

The estimation processing unit may estimate, as the accuracy information, a change in the prediction accuracy of the prediction model generated using the entire data set relative to the prediction accuracy of the prediction model generated using the partial data set. In this case, the screen generation unit may generate at least one of a screen that presents the amount of change in the prediction accuracy divided into a plurality of levels, or a screen that presents the value of the amount of change in the prediction accuracy.

The screen generation unit may generate a selection screen for selecting execution of a generation process of the predictive model using the partial dataset. In this case, when execution of the generation process is selected, the acquisition unit may execute the generation process to calculate features of the partial dataset. Furthermore, the estimation processing unit may estimate the accuracy information based on the features of the partial dataset.

An information processing method according to one embodiment of the present technology is an information processing method executed by a computer system, and includes acquiring features of a partial dataset that is a part of an entire dataset used to generate a predictive model.
Accuracy information representing the prediction accuracy of the prediction model generated using the entire dataset is estimated based on the feature amounts of the partial dataset.

A program according to an embodiment of the present technology causes a computer system to execute the following steps.
A step of obtaining features of a partial dataset that is a part of the entire dataset used to generate a predictive model.
A step of estimating accuracy information representing the prediction accuracy of the prediction model generated using the entire dataset, based on the feature quantities of the partial dataset.

1 is a block diagram showing an example configuration of a model generation system according to an embodiment of the present technology. 2 is a block diagram showing an example of the configuration of a terminal device shown in FIG. 1 . FIG. 2 is a schematic diagram showing an overview of a generation process of an estimation model. FIG. 1 is a schematic diagram for explaining an overview of a model generation system. 11 is a table showing specific examples of meta features. 1 is a flowchart illustrating an example of a basic operation of the model generation system. FIG. 13 is a schematic diagram showing an example of a setting screen. FIG. 13 is a schematic diagram showing an example of a selection area interface. FIG. 13 is a schematic diagram showing an example of an evaluation screen regarding a first prediction model. FIG. 13 is a schematic diagram showing an example of an interface of a display area of the improvement width α. 11 is a time chart showing an example of a learning process including calculations in a server device.

Below, an embodiment of the present technology is described with reference to the drawings.

[System Configuration]
1 is a block diagram illustrating an example of a configuration of a model generation system according to an embodiment of the present technology. The model generation system 100 is a system that generates a prediction model that performs prediction processing using a machine learning technique. The prediction model enables prediction analysis of a prediction target.
An application for generating a predictive model (hereinafter, referred to as a predictive analysis tool) runs in the model generation system 100. By using the predictive analysis tool, a user can generate a predictive model that performs a desired predictive process.
There are no limitations on the type of prediction model or the prediction target, and the user can set them as desired.

The model generation system 100 includes a terminal device 10 and a server device 30. The terminal device 10 and the server device 30 are connected to each other via a communication network 31 so as to be able to communicate with each other.
The terminal device 10 is an information processing device that is directly operated by a user, and functions as an operation terminal for the predictive analysis tool. A personal computer (PC) or the like is used as the terminal device 10. Alternatively, a tablet terminal, a smartphone, or the like may be used as the terminal device 10.
The server device 30 is an information processing device that is remotely connected to the terminal device 10. The server device 30 executes, for example, a predetermined process (e.g., a learning process for a predictive model, etc.) designated by the terminal device 10, and transmits the processing result to the terminal device 10. The server device 30 may be, for example, a network server that can be connected via a predetermined network, or a cloud server that can be connected to the cloud. Here, it is assumed that a pay-as-you-go server device 30 is used.
The communication network 31 is a network that connects the terminal device 10 and the server device 30 so that they can communicate with each other, and may be, for example, an Internet line or a dedicated local network.

Figure 2 is a block diagram showing an example of the configuration of the terminal device 10 shown in Figure 1. The terminal device 10 has a display unit 11, an operation unit 12, a communication unit 13, a memory unit 14, and a control unit 15.

The display unit 11 is a display that displays various information, for example, a UI (User Interface) screen of a predictive analysis tool. For example, a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display is used as the display unit 11. The specific configuration of the display unit 11 is not limited, and for example, a display equipped with a touch panel or the like that functions as the operation unit 12 may be used. Furthermore, an HMD (Head Mounted Display) may be used as the display unit 11.

The operation unit 12 includes an operation device for the user to input various information. As the operation unit 12, for example, a device capable of inputting information, such as a mouse or a keyboard, is used. The specific configuration of the operation unit 12 is not limited to this. For example, a touch panel or the like may be used as the operation unit 12. Furthermore, as the operation unit 12, a camera or the like that photographs the user may be used, and input may be possible through gaze or gestures.

The communication unit 13 is a module that performs communication processing between the terminal device 10 and other devices (e.g., the server device 30). The communication unit 13 is configured, for example, by a wireless LAN (Local Area Network) module such as Wi-Fi, or a wired LAN module. In addition, a communication module capable of short-range wireless communication such as Bluetooth (registered trademark), optical communication, etc. may be used.

The storage unit 14 is a non-volatile storage device, and may be, for example, a hard disk drive (HDD) or a solid state drive (SSD). In addition, the type of recording medium used as the storage unit 14 is not limited, and any recording medium that non-temporarily records data may be used.
A control program 16 according to the present embodiment is stored in the storage unit 14. The control program 16 is a program that controls the overall operation of the terminal device 10, for example.

The memory unit 14 also stores a learning dataset 17 used to generate the predictive model. The learning dataset 17 is a dataset including multiple data used for machine learning of the predictive model. The learning dataset 17 is generated as appropriate to match the target (prediction item) of the predictive model 50, and stored in the memory unit 14. When constructing a predictive model, the data included in the learning dataset 17 is read and used as appropriate.

The data in the learning dataset 17 is, for example, data in which a plurality of attribute values (feature values) are associated with corresponding correct labels. In this case, it is possible to train a prediction model that predicts items with correct labels.
For example, assume that customer data is used as the learning data set 17 to generate a model that predicts products that customers like. In this case, for example, among the customer data, items that indicate products that customers like (e.g., products that customers have purchased or viewed) become correct labels. Items regarding other attributes (such as the customer's age, sex, and frequency of product purchases) become input items for training the prediction model.
In addition, the type of the learning dataset 17 is not limited, and any dataset according to the prediction model may be used.

As described below, the terminal device 10 executes a process using a partial dataset of the training dataset 17. Hereinafter, a dataset that is a part of the training dataset 17 will be referred to as a partial dataset 18.
The partial data set 18 is composed of, for example, a plurality of data sampled from the learning data set 17. The data to be the partial data set 18 is sampled as appropriate, for example, every time the partial data set 18 is required. Alternatively, the data to be the partial data set 18 may be set in advance.
In this embodiment, the learning dataset 17 corresponds to the entire dataset used to generate a prediction model, and the partial dataset 18 corresponds to a partial dataset that is a part of the entire dataset.

The control unit 15 controls the operation of each block of the terminal device 10. The control unit 15 has the hardware configuration necessary for a computer, such as a CPU and memory (RAM, ROM). The CPU loads a program stored in the storage unit 14 into the RAM and executes it, thereby executing various processes. The control unit 15 may be, for example, a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array), or other devices such as an ASIC (Application Specific Integrated Circuit).

In this embodiment, the CPU of the control unit 15 executes the program according to this embodiment to realize functional blocks including a UI generation unit 20, a prediction model generation unit 21, a meta-feature calculation unit 22, and an accuracy estimation unit 23. These functional blocks then execute the information processing method according to this embodiment. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

The UI generation unit 20 generates a UI for exchanging information between the user and the terminal device 10 (or the server device 30). Specifically, the UI generation unit 20 generates a UI screen (see FIGS. 7 and 9) to be displayed on the display unit 11 when generating the prediction model 50. This UI screen becomes the screen of the above-mentioned prediction analysis tool.
The UI screen displays, for example, information to be presented to the user and input fields for the user to input information. The user can specify various settings, values, etc. by operating an operation unit (keyboard, etc.) while viewing the UI screen. The UI generation unit 20 accepts information specified by the user via the UI screen in this manner.
In this embodiment, the UI generating unit corresponds to a screen generating unit.

The prediction model generation unit 21 executes a process of generating a prediction model. In this embodiment, the prediction model generation unit 21 executes a process of generating a prediction model using the partial data set 18. This process can be executed in a short time compared to the process of generating a prediction model using all of the learning data set 17. Note that the generation process using all of the learning data set 17 is executed by, for example, the server device 30.
Hereinafter, a prediction model generated using the partial data set 18 will be referred to as a first prediction model, and a prediction model generated using the entire training data set 17 will be referred to as a second prediction model.

The process of generating a predictive model includes a series of processes required to construct a predictive model. For example, in a predictive analysis tool, the process of generating a predictive model includes a learning process for training a predictive model, a verification process for verifying the state of the predictive model (learning tendency, etc.), a test process for checking the prediction accuracy of the predictive model, etc., as appropriate.
Therefore, in the prediction model generation unit 21, learning processing, verification processing, test processing, etc. are each performed using the partial data set 18.
The machine learning algorithm used in the prediction model is not limited, and any algorithm may be used according to the processing content of the prediction model. The present technology is applicable regardless of the type of algorithm.
Hereinafter, the process of generating a predictive model may be simply referred to as a learning process.

The meta-feature calculation unit 22 acquires features of a partial dataset 18, which is a part of a learning dataset 17 used to generate a predictive model.
Here, the feature of the partial dataset 18 is a feature that represents the properties of the partial dataset 18 itself. Hereinafter, such a feature of the dataset itself will be referred to as a meta-feature. That is, the meta-feature calculation unit 22 acquires the meta-feature of the partial dataset 18.
Note that meta features are different from attribute values (hereinafter referred to as data features) recorded in the data constituting the partial dataset 18. For example, features possessed by the dataset itself, such as the number of data included in the dataset or the number of data features, are meta features.

The meta feature includes a feature (first feature) obtained by analyzing the partial data set 18. This feature is calculated by analyzing the partial data set 18.
The meta features also include features (second features) obtained by actually using the partial data set 18. These features are calculated using the prediction model generated by the prediction model generation unit 21 described above.
The meta-features will be described in detail later with reference to FIG. 5 etc.
In this embodiment, the predictive model generating unit 21 and the meta-feature calculating unit 22 work together to realize an acquiring unit.

The accuracy estimation unit 23 estimates accuracy information representing the prediction accuracy of a prediction model (second prediction model) generated using the training dataset 17 based on the features (meta features) of the partial dataset 18.
The accuracy information is information capable of expressing the prediction accuracy of the second prediction model. By referring to this accuracy information, it is possible to determine the degree of prediction accuracy that can be achieved when the second prediction model is constructed using all of the learning data sets 17.

In this embodiment, the accuracy estimation unit 23 estimates, as accuracy information, the change in the prediction accuracy of a prediction model (second prediction model) generated using the learning dataset 17 relative to the prediction accuracy of a prediction model (first prediction model) generated using the partial dataset 18.
In machine learning, it is believed that the more data that is used for learning, the more accurate the prediction will be. However, simply increasing the amount of data does not necessarily improve the prediction accuracy.
The accuracy estimation unit 23 estimates an expected improvement in prediction accuracy when a second prediction model is generated using all of the learning data set 17, based on the first prediction model trained with the partial data set 18. This improvement in prediction accuracy corresponds to the change in prediction accuracy described above.

The accuracy estimation unit 23 is configured using an estimation model that estimates a change in prediction accuracy. The estimation model is a learning model that is trained to input meta-features of the partial dataset 18 and output a change in prediction accuracy of the second prediction model relative to the first prediction model. In this way, the accuracy estimation unit 23 can also be said to be a module that implements the estimation model (estimation module).
The estimation model is constructed by learning from meta-features of a large number of data sets available from the web, etc. A method for generating the estimation model will be described in detail later.

In the predictive analysis tool, first, an estimation model (estimation module) is configured to estimate the improvement range of prediction accuracy. The estimation model is generated, for example, according to the type of prediction model. Alternatively, a versatile estimation model that can be used with different types of prediction models may be generated. Data of the estimation model is, for example, stored in advance in the storage unit 14, and is appropriately read and used each time the accuracy estimation unit 23 is operated.
The accuracy estimation unit 23 uses the estimation model configured in this manner to estimate the improvement in estimation accuracy (change in estimation accuracy) when all data sets of the actually used learning data set 17 are used for learning.

[Estimation model generation process]
3 is a schematic diagram showing an overview of a process for generating an estimation model. A method for generating an estimation model 40 for estimating an improvement in prediction accuracy will be described below with reference to FIG.
In recent years, many datasets that can be used for machine learning have become available on the web, etc. By using information learned from the meta-features of these datasets, it is possible to predict the properties of new datasets.
In this embodiment, a large number of already existing data sets are used to construct the estimation model 40. Hereinafter, the data sets used to construct the estimation model 40 will be referred to as estimation data sets.

For example, suppose a prediction model (hereinafter, referred to as an estimation prediction model) is generated using an estimation dataset. In this case, the prediction accuracy differs between a model trained using a part of the estimation dataset and a model trained using all of the estimation dataset. By training the model on such a difference in prediction accuracy for each of a plurality of estimation datasets, an estimation model 40 is constructed that estimates the improvement range of the prediction accuracy.
The estimation prediction model may be arbitrarily set in accordance with the estimation data set, for example.
In this embodiment, the estimation data set corresponds to a predetermined data set, and the estimation prediction model corresponds to a predetermined prediction model.

As shown in FIG. 3, the process of generating an estimation model uses a set of input data 25 and answer data 26 corresponding to the input data 25. A set of input data 25 and answer data 26 is generated for each estimation dataset. The number of estimation datasets (the number of sets of input data 25 and answer data 26) used in the process of generating an estimation model is, for example, approximately several hundred sets.

The input data 25 is a metafeature of the estimation dataset. Specifically, metafeatures of a portion of the dataset (e.g., 10%) included in the target estimation dataset are used as the input data 25.
Examples of meta-features included in the input data 25 include the number of data, the number of data features, or predicted evaluation values for training data (train), validation data (validation), and test data (test), which will be described later. The number and types of these meta-features are set in the same way as the meta-features that the estimation model 40 actually refers to when estimating the improvement range (see FIG. 5).
Furthermore, when generating the input data 25, a method similar to the method used by the metafeature calculation unit 22 to calculate the metafeature of the partial data set 18 is used.

The answer data 26 is a correct answer label of an item (improvement in prediction accuracy) to be learned by the estimation model 40. Specifically, the difference (improvement) between the prediction accuracy of the estimation prediction model when learning is performed using a part (e.g., 10%) of the estimation dataset and the prediction accuracy of the estimation prediction model when learning is performed using the entire estimation dataset is used as the correct answer label.
Therefore, the answer data 26 is calculated by actually learning the estimation prediction model using a part or all of the estimation data set. Note that a part of the input data 25 is calculated from the estimation prediction model generated at this time.

In the process of generating the estimation model 40, the above-mentioned input data 25 and answer data 26 are generated. That is, for a plurality of estimation data sets, their meta-features (input data 25) and the improvement in prediction accuracy when actually learning with all the data (answer data 26) are calculated.
Machine learning is executed based on the input data 25 and answer data 26 calculated in this manner. Specifically, learning processing and the like are executed using the improvement range of prediction accuracy as a correct answer label and meta features as features.
This process can also be said to be a process of learning the features of a dataset that will increase the improvement in prediction accuracy when all data is used. That is, the estimation model 40 learns the features (meta-features) of a dataset that will increase the prediction accuracy when the amount of data is increased. This allows the constructed trained estimation model 40 to estimate the improvement in prediction accuracy from the meta-features of the dataset.

In this way, the estimation model 40 is a model that learns the relationship between the features of a portion of the estimation dataset and the change in prediction accuracy that occurs when the estimation prediction model is trained using all and part of the estimation dataset.
By using the estimation model 40, even when an unknown learning dataset 17 is used, it is possible to accurately and easily estimate the improvement in the prediction accuracy of the prediction model.
The estimation model 40 may be a learning model obtained by learning from meta-features and correct labels, or may be a model that approximates the learning model. Types of the estimation model 40 will be described below.

For example, the estimation model 40 is a classification model that classifies the amount of change in prediction accuracy into a plurality of levels. In this case, for example, the correct answer label (answer data 26) is a binary classification of each level representing the amount of change in prediction accuracy.
The level of change is set to indicate the degree to which prediction accuracy changes when learning with the entire data set compared to when learning with only a portion of the data set. For example, the correct answer label is set in three levels: when prediction accuracy is "significantly improved (5% or more)," when it is "improved to some extent (2-5%)," and when it is "almost no improvement (less than 2%)." This makes it possible to estimate the improvement in prediction accuracy in multiple levels.

Furthermore, for example, the estimation model 40 may be a rule-based approximation of a classification model that classifies the amount of change in prediction accuracy into a plurality of levels. In this case, the estimation model 40 becomes a rule-based classifier that is a simplification of the classification model.
For example, the above-described classification model is approximated by a predetermined algorithm to calculate the final estimation model 40. As an algorithm for approximating the classification model, a decision tree algorithm, a random forest in which decision trees are randomly combined, or a rule fit in which processing by the classification model is replaced with a set of rules, etc., are used.
By using a rule-based model, it is possible to reduce the amount of calculation and the time required to estimate the improvement amount. It is also possible to explain the details of the estimation process in a way that is easy for users to understand.

Furthermore, for example, the estimation model 40 may be a regression model that estimates the amount of change in prediction accuracy. In this case, for example, the correct answer label (answer data 26) is set to the value of the amount of change in prediction accuracy (for example, improvement range X%).
In this manner, the estimation model 40 may be constructed so as to directly regress the amount of change (improvement) in the prediction accuracy as a specific numerical value, thereby making it possible to present a specific estimated value of the improvement in the prediction accuracy to the user.
Besides, the specific configuration of the estimation model 40 is not limited.

[Overview of the model generation system]
4 is a schematic diagram for explaining an overview of the model generation system 100. Here, the flow of processing is illustrated, from estimating the improvement range of prediction accuracy using the estimation model 40 described above to presenting the estimation result.
4 includes a process for generating a prediction model 50 (first prediction model 51) (step 1), a process for calculating meta-features (step 2), a process for estimating an improvement amount (step 3), and a process for presenting a UI (step 4). Each process will be described in order below.

[Prediction model generation process]
When estimating the improvement range of prediction accuracy, a process of generating a first prediction model 51 is executed using the partial data set 18. This process is a preliminary generation process executed before learning is performed using the entire training data set 17 (a process of generating a second prediction model 52).
Specifically, the prediction model generation unit 21 samples a partial dataset 18 included in the learning dataset 17. Then, machine learning is performed using the partial dataset 18.

In this generation process, the partial data set 18 is divided into a plurality of data groups each having a different purpose for use, that is, it can be said that the partial data set 18 includes a plurality of data groups each having a different purpose.
Each data group includes at least one piece of data, and each group is used for a different purpose. Note that the method for setting the data groups is not limited.

In this embodiment, the multiple data groups are a group of training data, a group of validation data, and a group of test data.
The training data is data used when performing a learning process for the prediction model 50, and is data that the prediction model 50 actually learns (trains) from. The more training data there is, the more the accuracy of the prediction model 50 tends to improve.
The validation data is data used when performing a validation process to validate the learning state (learning tendency, etc.) of the prediction model 50. Therefore, it can be said that the validation data is data for checking the learning of the prediction model 50.
The test data is data used when performing a test process to confirm the final prediction accuracy, etc., of the prediction model 50 trained with the training data. Therefore, the test data can be said to be data for evaluating the prediction model 50.
Depending on the type of learning or settings, validation data may not be necessary. In this case, the validation data group may be omitted.

In the prediction model generation unit 21, the above-mentioned learning process, verification process, test process, etc. are appropriately executed using these data groups, whereby a trained prediction model 50 (first prediction model 51) trained from the partial data set 18 is generated.
Information on each data group and data of the first prediction model 51 are output to the meta-feature calculation unit 22.

[Meta feature amount calculation process]
The meta-feature calculation unit 22 calculates the meta-feature F of the partial data set 18 used to generate the first prediction model 51.
First, data required for calculating meta-features is appropriately read. Specifically, each data group included in the partial dataset 18 and a first prediction model 51 generated using the partial dataset 18 are read. Fig. 4 shows a schematic diagram of the learning data and test data groups of the partial dataset 18 and the first prediction model 51. Although not shown, a validation data group is also appropriately read.
In this way, for a dataset (learning dataset 17) for which the degree of improvement in prediction accuracy is desired, a partial dataset 18 (learning data, validation data, test data) sampled from the dataset and a first prediction model that has been trained using the partial dataset 18 are prepared.
The meta feature amount F calculated based on these data will be specifically described below.

Fig. 5 is a table showing specific examples of meta-features. Fig. 5 shows the items of each meta-feature and their specific contents for a plurality of meta-features. These meta-features are used, for example, when a regression model is used as a prediction model.
Here, the meta-features are numbered (F1 to F16) for explanation. Note that the table shown in Fig. 5 is an example, and the number and types of meta-features are not limited.

The meta feature of the partial data set 18 includes a first feature according to the content of the partial data set 18. The first feature is a feature that the partial data set 18 itself has.
In this embodiment, the meta-feature calculation unit 22 calculates the first feature by analyzing the partial data set 18. In the table shown in Fig. 5, the meta-feature F1 to F4 and F9 correspond to the first feature.

The meta-feature F1 (number of data) is the number of data included in the partial dataset 18. For example, the total number of data included in the partial dataset 18 is calculated as the meta-feature. Alternatively, the total number of training data included in the partial dataset 18 may be used.
The meta feature F2 (number of features) is the number of features (data features) included in the data of the partial data set 18. For example, the total number of data features set in each data is calculated as the meta feature. In addition, when the number (type) of features differs for each data, the total number or the like may be calculated.
The meta-feature F3 (number of features/number of data) is the ratio between the number of data included in the partial data set 18 and the number of data features included in the data. For example, a value obtained by dividing the meta-feature F2 by the meta-feature F1 is calculated as a new meta-feature.
The meta feature F4 (number of features after expansion) is the number of data features used for the learning data after a predetermined preprocessing is completed. For example, when performing preprocessing such as OneHot Encoding, the number of data features changes before and after the processing due to the use of dummy variables. The total number of data features after this processing is calculated as the meta feature.
The meta-feature F9 (variance of correct values) is the variance of the prediction target label (correct label). For example, a variance value (e.g., standard deviation) of the value of the prediction target label to be predicted by the regression model is calculated as the meta-feature.

Furthermore, the meta features of the partial dataset 18 include second features corresponding to the prediction model 50 (first prediction model 51) generated using the partial dataset 18. In other words, the second features can be said to be features obtained by actually using the partial dataset 18.
In this embodiment, the second feature is calculated by the prediction model generation unit 21 executing a generation process of the first prediction model 51 using the partial data set 18. In the table shown in Fig. 5, the meta features F5 to F8 and F10 to F16 correspond to the second feature.

Here, the second feature is an evaluation value that evaluates a predicted value of the first prediction model 51 generated using the partial data set 18 for each of the multiple data groups. Here, the evaluation value is a parameter that can evaluate a predicted value output from the first prediction model 51 when a certain data group (a group of learning data, validation data, and test data) is input.
As a parameter for evaluating a predicted value, for example, a mean absolute error (MAE), a root mean squared error (RMSE), a mean absolute percentage error (MAPE), etc. related to the predicted value are used. Alternatively, the variance of the predicted value, etc. may be used as an evaluation value. The parameters used as the evaluation value are not limited, and other indices may be used.

Metafeature F5 (change in median error of test data according to the number of iterations) is the change in median error in iteration processing for test data. Iteration processing is, for example, a process (cross-validation) in which the prediction accuracy of the model is verified multiple times by changing the way the test data is selected, and has the effect of reducing evaluation bias due to the way the test data is selected. Specifically, the difference between the median error at half the number of iterations when it converges and the final converged median error is calculated as the metafeature. Metafeature F5 is an example of an evaluation value for test data.

The meta-feature F6 (median error of training/validation/test data) is the median error (MAE) value for each group of training data, validation data, and test data when predictions are made using the trained first prediction model 51.
The meta-feature F7 (root mean square error of training/validation/test data) is the root mean square error (RMSE) value for each group of training data, validation data, and test data when predictions are made using the trained first prediction model 51.
The meta-feature F8 (median error rate of training/validation/test data) is the median error rate (MAPE) value for each group of training data, validation data, and test data when predictions are made using the trained first prediction model 51.
The meta-feature F10 (variance of predicted values) is the variance (standard deviation, etc.) of the predicted values predicted by the trained first prediction model 51.
All or some of these evaluation values may be used.

In addition, a comparison value obtained by comparing the above-mentioned evaluation values is used as the second feature amount. Here, the comparison value is a value obtained by comparing the evaluation values calculated for each data group (groups of training data, validation data, and test data) between groups.
Specifically, at least one of the difference or ratio between evaluation values calculated for two of the plurality of data groups is used as the comparison value.

The meta-feature F11 (the difference in the median error between the training data and the test data) is the difference between the median error for the training data and the median error for the test data.
The meta-feature F12 (the ratio of the median errors between the training data and the test data) is the ratio of the median error for the training data to the median error for the test data.
The meta-feature F13 (the difference in the median error between the validation data and the test data) is the difference between the median error for the validation data and the median error for the test data.
The meta-feature F14 (ratio of median errors between validation data and test data) is the ratio of the median error for validation data to the median error for test data.
The meta-feature F15 (the difference in the median error between the training data and the validation data) is the difference between the median error for the training data and the median error for the validation data.
The meta-feature F16 (ratio of median errors between training data and validation data) is the ratio of the median error for the training data to the median error for the validation data.

These meta-feature amounts F11 to F16 are calculated based on the results of the meta-feature amount F6 described above, for example.
The criteria for calculating the difference and ratio may be set arbitrarily. For example, in the meta-feature F11, the median error for the training data may be subtracted from the median error for the test data, or vice versa. Alternatively, the absolute value of the difference may be used. Also, for example, in the meta-feature F12, the median error for the training data may be divided by the median error for the test data to calculate the ratio, or vice versa.
Also, instead of the median error, a comparison value obtained by comparing the mean square error or the median error rate may be used as meta information.

In this way, the median error and the like can be calculated if the trained first prediction model and the partial data set 18 used for training are given. All other feature amounts can also be calculated if the sampled training data, validation data, test data, and first prediction model 51 are used.
Most of these values are calculated in the process of creating the first prediction model 51, and no additional calculation is required.

[Improvement Estimation Process]
Returning to FIG. 4, the accuracy estimation unit 23 calculates the improvement α of the prediction accuracy of the prediction model 50 based on the meta-feature F calculated as described above.
To estimate the improvement range α of the prediction accuracy, an estimation model 40 that estimates the improvement range of the prediction accuracy, constructed by learning from meta-features, is used (see FIG. 3). Specifically, the meta-feature F of the partial data set 18 is input as input data to the estimation model 40. Then, a calculation is performed using the estimation model 40, and a classification value or value of the improvement range α is output.

For example, when the estimation model 40 is a classification model or a rule-based model that approximates a classification model, a classification result in which the improvement range α is classified into a plurality of levels is output. In this case, the output value is the prediction probability for each level, such as "great improvement (5% or more),""some improvement (2-5%)," and "little improvement (less than 2%)." In other words, the probability that the improvement range will be 5% or more is calculated.
Furthermore, for example, when the estimation model 40 is a regression model, the value of the improvement range α of the prediction accuracy is directly estimated by solving a regression problem. In this case, the output value is a value that specifically represents the improvement range α (for example, α=4%).
The output of the estimation model 40 is output to the UI generation unit 20 .

[UI Presentation Processing]
The estimated improvement range α of the prediction accuracy is displayed by the UI generating unit 20. Specifically, the UI generating unit 20 generates a screen presenting an estimation result of the improvement range α of the prediction accuracy (change in the prediction accuracy). The generated screen is then displayed on the display unit 11.
This makes it possible to present to the user the expected improvement α in prediction accuracy when a prediction model 50 (second prediction model 52) is generated using all of the learning data sets 17, thereby assisting the user in deciding whether or not to generate the second prediction model 52.

In this way, the model generation system 100 (predictive analysis tool) can perform learning in a short period of time using a portion of the learning dataset 17, and estimate from the information at that time how much the prediction accuracy will improve when learning using the entire dataset. In other words, by learning only once from a portion of the dataset, it is possible to estimate the improvement α in prediction accuracy when all data is used for learning.

The inventors actually constructed an estimation model 40 that estimates the improvement range α of prediction accuracy and verified its accuracy. As a result, it was found that the AUC (evaluation index for classification problems) of the estimation model 40 that classifies the improvement range α was 0.75, and that the improvement range α could be predicted with high accuracy. This means that it is possible to properly predict datasets in which prediction accuracy will improve, from meta-features.

The inventors also obtained knowledge from actual experimental results about the tendency of datasets for which accuracy improves when the amount of data is increased. Specifically, they found that the greater the difference in the evaluation index (e.g., the above-mentioned evaluation value) of the predicted value for the training data and the test data, the greater the improvement in accuracy when training with all the data. For example, the difference in the evaluation index between the training data and the test data is an index that indicates the degree to which the prediction model 50 has overfitted to the training data, and when the difference between these is large, it is often possible to expect an improvement in accuracy as the amount of data increases.

Therefore, among meta-features, the evaluation index (evaluation value) of the predicted value for the training data and the test data and the value obtained by comparing the evaluation index (comparison value) are particularly important features. By using an estimation model 40 that uses such meta-features as input, it becomes possible to accurately estimate the improvement range α of the prediction accuracy.

[Basic operation of the model generation system]
6 is a flowchart showing an example of a basic operation of the model generation system. The process shown in Fig. 6 is executed when a user of the terminal device 10 generates a predictive model 50 using a predictive analysis tool, for example.
First, each setting value of the prediction model 50 is read (step 101). Specifically, the UI generating unit 20 generates a setting screen related to the prediction model 50 and outputs it to the display unit 11. Then, the contents (setting values) input by the user via the setting screen are read.

Figure 7 is a schematic diagram showing an example of a settings screen. As shown in Figure 7, the settings screen 35 is provided with a plurality of setting fields. Here, a case will be described in which customer data including product purchase records is used as a learning dataset 17 to generate a predictive model 50 that predicts whether or not a product is purchased.

In the "Input Items" setting field (right side of the screen), it is possible to specify the items (data features) included in the learning dataset 17 that will be used to train the predictive model 50. Here, items related to the customer such as "age", "gender", "customer rank", "past purchase amount", "number of coupons used", "email address registration", and "option purchase" are presented as selectable items. In addition, the data type and unique number for each item are also displayed.

In the "Prediction type" setting field, the type of prediction model 50 can be specified. Here, the items "binary classification", "multiple-value classification", and "numerical prediction" (regression prediction) are displayed as selectable items. Here, binary classification is selected as the type of prediction model 50.
In the "Predicted value" setting field, it is possible to specify the prediction target (target item) of the prediction model 50. Here, of the items "Purchase made" and "No purchase", "Purchase made" is selected as the prediction target. Note that the percentage of each item in the learning dataset 17 is displayed next to the item names ("Purchase made" and "No purchase").

The area indicated by the dotted line on the setting screen 35 is a selection area 36 for selecting learning using the partial dataset 18. In the example shown in Fig. 7, a setting field for "proportion of data to be used" is provided in the selection area 36. In this setting field, it is possible to select and specify the proportion of data to be used as the partial dataset 18 from several candidates.
For example, the ratio of the partial data set 18 to the learning data set 17 is presented selectable in the range of 0% to 100% (10% is selected here). In this UI, when the ratio of the partial data set 18 is a finite value greater than 0%, learning using the partial data set 18 is selected. Note that when the ratio of the partial data set 18 is 0%, learning using the partial data set 18 is not selected.

After entering the necessary information in each setting field, pressing the execute button labeled "Execute learning and evaluation" will start the learning process for the predictive model 50. Pressing the cancel button labeled "Cancel" will cancel the input of each setting value on the setting screen 35 and display the previous screen.

FIG. 8 is a schematic diagram showing an example of an interface of the selection area 36. As shown in FIG.
8A is provided with a setting field for "proportion of data to be used." In this setting field, the proportion of data to be used as the partial data set 18 can be freely input and specified within the range of 0% to 100%. In this case, when the input value is greater than 0, learning using the partial data set 18 is selected.

8B, a setting field for "learning mode" is provided. In this setting field, the items "quick mode" and "learning with all data" are presented as selectable items.
Here, the quick mode is a mode in which learning is performed using the partial data set 18, and the improvement in prediction accuracy is calculated in a short time before actual learning. In the quick mode, for example, the partial data set 18 is selected and used at a preset default ratio. Note that the ratio of the partial data set 18 may be selectable. In this way, by having the user select the learning mode, it may be set whether or not learning is performed using the partial data set 18.

8C includes a setting field for "Device to study on." In this setting field, the options "Study on this device" and "Study on the cloud" are presented as selectable items.
The item "learn on this terminal" is selected when learning using the partial data set 18 (part of data) is executed on the terminal device 10. The item "learn on the cloud" is selected when learning using the entire learning data set 17 is executed on the server device 30. In this way, by selecting the device that will perform the learning process, it may be possible to set whether or not to learn using the partial data set 18.

In this manner, the UI generation unit 20 generates the setting screen 35 for selecting execution of the generation process of the first prediction model 51 using the partial data set 18. In this embodiment, the setting screen 35 corresponds to a selection screen.
This allows the user to appropriately select whether or not to perform learning with the partial data set 18, that is, whether or not to estimate the improvement range α.

Returning to FIG. 6, when the setting values inputted from the setting screen 35 are read, it is determined whether or not to start the learning process for the partial data set 18 (step 102).
For example, assume that learning with the partial data set 18 is selected in the UI displayed in the selection area 36. In this state, when the execute button shown in Fig. 7 is pressed, it is determined that learning with the partial data set 18 is to be performed (Yes in step 102), and learning with the partial data set 18 and a process of estimating the improvement range α using the learning result are started.
Also, for example, if the execute button is pressed when learning with partial data set 18 has not been selected, it is determined that learning with partial data set 18 will not be performed (No in step 102), and step 107 described below is executed.

When it is determined that learning is to be performed using the partial data set 18, the prediction model generation unit 21 executes a process of generating a first prediction model 51 using the partial data set 18 (step 103). This process corresponds to the process of generating a prediction model in step 1 described with reference to FIG.
For example, a model is constructed that outputs a predicted value from the input item selected by the setting value of the setting screen 35, and learning processing, verification processing, test processing, etc. are performed using the partial dataset 18, and a learned first prediction model 51 is constructed.

Once the first prediction model 51 is constructed, the meta-feature calculation unit 22 calculates the meta-feature F of the partial data set 18 (step 104). This process corresponds to the meta-feature calculation process of step 2 described with reference to FIG. 4.
For example, data of the first prediction model 51 and the partial dataset 18 used for learning the first prediction model 51 are read, and meta-features F that serve as inputs to the estimation model 40 that has already been prepared are calculated.

Once the first prediction model 51 is constructed, the meta-feature calculation unit 22 calculates the meta-feature F of the partial data set 18 (step 104). This process corresponds to the meta-feature calculation process of step 2 described with reference to FIG. 4.
For example, data of the first prediction model 51 and the partial dataset 18 used for learning the first prediction model 51 are read, and meta-features F that serve as inputs to the estimation model 40 that has already been prepared are calculated.
In this manner, in this embodiment, when execution of the generation process of the first prediction model 51 is selected, the generation process is executed to calculate the meta-feature value F of the partial data set 18 .

After the meta-feature F is calculated, the accuracy estimation unit 23 estimates an improvement range α (accuracy information) of the prediction accuracy based on the meta-feature F of the partial data set 18 (step 105). This process corresponds to the improvement range estimation process of step 3 described with reference to FIG. 4.
For example, each meta-feature F calculated in the previous step is input to the estimation model 40, and the classification level and value of the improvement range α of the prediction accuracy are calculated.

When the improvement range α is calculated, the UI generating unit 20 generates a screen for presenting the improvement range α (step 106). This process corresponds to the UI presenting process in step 4 described with reference to FIG.
In this embodiment, an evaluation screen presenting the improvement range α together with the evaluation result of the first prediction model 51 is generated and displayed on the display unit 11 .

FIG. 9 is a schematic diagram showing an example of an evaluation screen for the first prediction model 51. As shown in FIG.
A model selection area 36 is provided on the left side of the evaluation screen 37 shown in Fig. 9. In the selection area, first prediction models 51 that have already been evaluated are presented in a selectable manner together with their evaluation values, generation dates and times, and the names of the data used.
Also provided on the right side of the evaluation screen 37 are a display field for "prediction accuracy level" indicating the level of prediction accuracy of the first prediction model 51 selected in the selection area 36, and a display field for "contribution of item." Also provided on the right side of the evaluation screen 37 is a display area 38 presenting the estimated result of the improvement range α.

In the "Prediction accuracy level" display field, for example, the AUC (Area Under the Curve) of a ROC (Receiver Operating Characteristic) curve is displayed as an evaluation index indicating the performance of the first prediction model 51. The AUC is an index indicating the classification accuracy of a classification model. In addition, explanatory items related to the evaluation index (comments on the accuracy of the model, etc.) are displayed.
In addition, the "Item Contribution" display field displays a bar graph showing the contribution of each item that influenced the classification. This makes it possible to compare items, for example, items that influenced the classification "Purchase" and items that influenced the classification "No Purchase".

9, text explaining the improvement range α is presented. Here, an explanation is used that presents the improvement range α of the prediction accuracy expected by using all the data (all the learning data sets 17) together with the proportion of the data (partial data sets 18) used in training the first prediction model 51. Here, an explanation is used that presents the improvement range α as a specific value (X%), but an explanation that presents the improvement range α in multiple levels (e.g., large, medium, small, etc.) may also be used.
In this way, by presenting the estimation results in text such as an explanatory sentence, it is possible to explicitly give advice as to whether or not learning should be performed using all of the learning datasets 17.

FIG. 10 is a schematic diagram showing an example of an interface of a display area 38 for the improvement width α.
10A includes an execute button 39 for executing a process for generating a second prediction model 52 using all of the learning data sets 17. The processing time required for the process for generating the second prediction model 52 and the expected improvement α in prediction accuracy are displayed near the execute button 39.
This allows the user to refer to the improvement range α and the processing time to determine whether or not to perform the generation process of the second prediction model 52. In addition, by selecting the execute button 39, the generation process using all the learning data sets 17 can be started as is, so there is no need to re-enter the setting values.

10B, the estimation result of the improvement width α is displayed as it is in a display area 38. Here, the estimation result of the improvement width α classified into a plurality of levels is displayed using character data.
For example, the level of the improvement range α may be represented using graphics or the like that represent a plurality of levels. Alternatively, a gauge or a graph that represents the value of the improvement range α may be used.
This allows the user to easily grasp the level and value of the improvement amount α.
In this manner, the UI generating unit 20 generates the evaluation screen 37 that presents the improvement range α of the prediction accuracy divided into a plurality of levels, and the evaluation screen 37 that presents the value of the improvement range α of the prediction accuracy.

6, when the improvement range α (evaluation screen 37) is presented, it is determined whether or not to start a learning process for all the learning data sets 17 (all data sets) (step 107). In other words, it is determined whether or not to generate a second prediction model 52.
For example, if the improvement range α is high and the user selects learning with all the learning data sets 17 (Yes in step 107), learning and evaluation with all the learning data sets 17 is started (step 108). In this embodiment, for example, the learning data set 17 and the setting values are output to the server device 30, and the server device 30 executes a series of processes for generating the second prediction model 52. Alternatively, the second prediction model 52 may be generated by the prediction model generation unit 21 of the terminal device 10. After the second prediction model 52 is generated, an evaluation screen or the like is displayed.
Also, for example, if the improvement range α is low and the user does not select learning with all of the learning data sets 17 (Yes in step 107), the process of generating the prediction model 50 ends.

This makes it possible to know the prediction model 50 and its prediction accuracy, as well as an estimate of the accuracy when learning from the entire data set, in a short time, for example, without occupying a local PC (terminal device 10) for a long time when learning from a large data set, and without using a pay-as-you-go server (server device 30) on the cloud for a long time. This makes it possible to efficiently generate the prediction model 50.

An application example of the model generation system 100 (predictive analysis tool) according to the present technology will be described below using a specific example.
[Application Example 1]
In this example, when training a predictive model using large-scale data, after identifying a useful combination of features, training is performed on all data using a pay-as-you-go server device 30 on the cloud.
Here, it is assumed that an insurance company constructs a prediction model 50 that predicts what type of insurance products customers will prefer.

In this case, for example, the customer data used as the learning dataset 17 is a huge amount of data that includes logs of customer deposits and procedures for insurance products. For this reason, if learning is performed using all of the learning dataset 17, it takes about 6 to 12 hours, making it difficult to complete the process within business hours.
Furthermore, there are hundreds of different types of procedure logs, making it difficult to specify the type of behavior (combination of features) to set as features for learning.

For example, a user prepares about 10 patterns of combinations of features to be used as input data based on hypotheses about daily work. Then, using the model generation system 100, preliminary learning is performed on the partial data set 18 for each pattern prepared by the user. As a result, the prediction accuracy (improvement range α) when learning with all the learning data sets 17 is estimated for each pattern.
This preliminary learning (e.g., step 103 in FIG. 6 ) is performed on the partial data set 18 sampled from the learning data set 17, and therefore takes about 30 minutes to complete each time. By learning 10 patterns of feature combinations every 30 minutes, it is possible to narrow down the patterns of feature combinations that are particularly useful during business hours to about three.

For each of the approximately three useful feature combination patterns narrowed down in the above preliminary learning, a second prediction model 52 is generated using all of the learning data sets 17. This process is executed over a long period of time, for example, during nighttime or on weekends, using a pay-as-you-go server device 30.
For example, when a user comes to work the next morning (or at the beginning of the next week), the learning result of the second prediction model 52 trained by the server device 30 is confirmed. Then, the model with the highest prediction accuracy, etc. is determined to be the model to be finally used.

In this way, by using the model generation system 100, it is possible to narrow down promising parameter candidates in a short time. This makes it possible to efficiently generate the predictive model 50 without wasting work time.

[Application Example 2]
An example of trial and error to see if it is possible to predict the amount a customer will pay for a service from customer logs.
Here, we will build a prediction model that predicts the amount of money that a customer will spend on a service provided on the web in a month. By building such a prediction model, it is expected that measures such as issuing coupons to users who spend a small amount of money will be taken, and that this will encourage customers to use the service.

In this case, customer log data that records, for example, access times, etc., is used as the learning data set 17 .
However, log data is a huge amount of data and is thought to contain a lot of noise. For this reason, it is unclear whether learning from log data can actually predict the amount of money a customer will spend.
On the other hand, the amount of money used by customers is one of the KPIs (Key Performance Indicators) for a service. Therefore, if the amount of money used by customers can be predicted, it will have great business value, and we will try to build a prediction model as much as possible.

For example, assume that there are several tens of gigabytes of log data, and it is found that if learning is performed using all of the log data (learning dataset 17), the learning process will take several days.
Therefore, first, learning processing and the like are executed on a sampled portion of the entire log data (partial data set 18) using the model generation system 100. This learning processing is executed for, for example, about six hours.

With reference to the learning results using the partial data set 18, it was found that the median error rate for the first prediction model 51 was 120%, indicating that sufficient prediction accuracy was not achieved. Furthermore, it was suggested that the accuracy when all data sets were used would also have a median error rate of approximately 100%, indicating that the expected accuracy could not be achieved.
In such a case, even if the amount of data is increased and processing is performed on the local terminal device 10 or the cloud server device 30, it will be a waste of time and money. For this reason, the construction of a prediction model for predicting the amount of money used by customers from log data is abandoned.

As mentioned above, we found that it is not possible to directly predict the amount of money a customer will spend. Therefore, we changed the problem setting and considered a predictive model that performs binary classification to classify whether a customer spends more than 1,000 yen per month on the service.
Specifically, when the model generation system 100 was used to train a prediction model that performs the above binary classification using a partial data set sampled from all log data, the AUC was 0.65. Furthermore, it was suggested that the AUC could be increased to 0.7 by training using all log data.
Since this is a practically feasible level of accuracy, a learning process was actually performed using all the log data using the server device 30 on the cloud, and a prediction model with an AUC of 0.71 was obtained.

In this way, we found that practical predictions can be made by changing the problem setting (the target of the prediction model) to a binary classification of whether or not a customer spends more than 1,000 yen a month on services. This makes it possible to start measures to encourage customers to spend more, such as issuing coupons or discounts to customers who are likely to spend less than 1,000 yen a month on services.
In this application example, the model generation system 100 is used while finding an appropriate problem setting through trial and error. This allows prediction accuracy to be estimated without actually performing learning using all the data, making it possible to build a model without spending unnecessary learning time and money.

[Application Example 3]
When training on large-scale data, the accuracy of training on all training data sets 17 is first estimated using a local terminal device 10, and if it is deemed feasible, training on all training data sets 17 is performed on a pay-as-you-go server device 30.

For example, a predictive model is trained on a partial data set 18 using the terminal device 10. After that, the learning results etc. are presented, and the user checks whether a practical prediction accuracy is obtained when learning is performed on all of the learning data sets 17. For example, assume that it is predicted that the AUC will be 0.72 when learning is performed on all of the learning data sets 17. In this case, it is determined that the expected prediction accuracy has reached a practical level, and a decision is made to perform a learning process using all of the learning data sets 17 using the server device 30 on the cloud.

In fact, it is assumed that a prediction model with an AUC of 0.71 is constructed as expected as a result of performing a learning process using the server device 30. In this case, it is determined that the prediction model is a model suitable for practical use and is to be introduced into a production environment.
In this way, the model generation system 100 can estimate the accuracy when all data is used beforehand when learning with large-scale data. By referring to the estimation result, the user can efficiently use the computing resources of the server device 30 and the like.

Fig. 11 is a time chart showing an example of a learning process including calculations in the server device 30. Fig. 11 shows a flow of processing in the model generation system 100 in a case where learning is performed using large-scale data as described in Application Example 3, for example.
First, with the learning data set 17 to be used for learning loaded, the user presses the learning button and inputs an instruction to start the learning process to the terminal device 10 (step 201).
At this time, the terminal device 10 calculates the data capacity of the learning data set 17, the learning time, etc. Then, if the data capacity or the learning time exceeds a threshold, a message is displayed to inform the user that learning will be performed using only a portion of the data (partial data set 18) because the data is huge (step 202).

The terminal device 10 executes a learning process on the partial data set 18 (step 203). As described above, in the example shown in Fig. 11, the terminal device 10 automatically selects and executes the learning process on the partial data set 18 depending on the size of the data in the learning data set 17, etc. Note that the learning on the partial data set 18 may be executed after confirmation by the user.
When the learning process on the partial data set 18 is completed, the learning results (evaluation results of the first prediction model 51) and the estimated prediction accuracy (improvement range α) expected when learning on all data are displayed (step 204).

Assume that the estimated prediction accuracy is high and the user decides to execute a learning process using all of the learning data sets 17. In this case, a predetermined execution button is pressed and an instruction is input to the terminal device 10 to execute learning in the cloud (server device 30) (step 205). Then, the terminal device 10 uploads all of the learning data sets 17 and data such as the setting values of the prediction model to the server device 30 (step 206).

The server device 30 executes the learning process for all the learning data sets 17 (step 207). The server device 30 generally has high computing power, and therefore is able to complete the learning process in a shorter time than the terminal device 10. Note that while the server device 30 is executing the learning process, no computational load is placed on the terminal device 10. Therefore, the user can use this time to have the terminal device 10 execute other processes, etc.

When the learning process is completed for all the learning data sets 17, the learning results (the evaluation results of the second prediction model 52) are transmitted from the server device 30 to the terminal device 10 (step 208). Then, the terminal device 10 generates an evaluation screen including the learning results for all the learning data sets 17, and displays it on the display unit (step 209).
In this way, the estimated prediction accuracy is presented before actual learning using all the data is performed. This allows the user to determine whether or not to perform actual learning. In particular, when learning using large amounts of data, it is possible to reduce unnecessary calculation time and costs and execute only necessary calculations. This makes it possible to significantly improve the efficiency of the generation process of the prediction model.

As described above, in the control unit 15 according to this embodiment, the meta-feature F of the partial dataset 18 from the learning dataset 17 is acquired. Based on this meta-feature F, accuracy information (improvement range α) representing the prediction accuracy when a prediction model 50 (first prediction model 51) is generated using the learning dataset 17 is estimated. This makes it possible to determine, for example, whether or not the learning dataset 17 should be used, and makes it possible to efficiently generate the prediction model 50.

In machine learning, it is generally known that the more training data there is, the more accurate the predictions become. However, as the amount of data increases, the time required for training also increases.
For example, when performing parameter search or feature search, the increase in learning time is often a problem. For example, in predictive analysis services provided to non-experts, parameter and feature search is essential. For this reason, when learning a large data set exceeding several hundred megabytes, for example, it is considered that a lot of time is required for the parameter search process.

One method for estimating the degree of improvement in prediction accuracy by increasing the amount of data is to train on multiple data sets of different sizes. In this case, the improvement in prediction accuracy when the amount of data is increased can be estimated by examining the relationship between the amount of data in each data set and the prediction accuracy for the test data. However, this method requires multiple training sessions (e.g., 5-10 times) to train on multiple data sets. For this reason, despite the goal of grasping prediction accuracy in a short training time, there is a risk that estimating prediction accuracy itself will take a long time.

In this embodiment, the improvement α in the prediction accuracy of the prediction model trained on the training dataset 17 is estimated from the meta-feature F of the partial dataset 18, which is a part of the training dataset 17. The meta-feature F is calculated from a single training run using the partial dataset 18.

This makes it possible to estimate in a short time the side accuracy of the prediction model when trained using all of the training data sets 17. This allows the user to immediately know an indication of the prediction result on the local terminal device 10, and thus makes it possible to appropriately determine whether or not to perform training using all of the data.
For example, when the data is large, when parameters or features are being explored, or when problem setting is being done through trial and error, it is possible to estimate the prediction accuracy when learning from the entire data set in a short time without performing unnecessary learning.

In addition, the user can know the estimated accuracy when learning with all the data without occupying the terminal device 10 for a long period of time. This makes it possible to use the system in such a way that, for example, learning is performed with a portion of the data (partial data set 18) during work hours to grasp the approximate prediction accuracy when all the data is used, and then learning is performed with all the data at night or on holidays.

In addition, there is no need to run learning on the cloud for data sets that are estimated in advance for which learning will not go well (low prediction accuracy, no improvement expected, etc.). Therefore, users can run learning on the server device 30 only when it is estimated to be effective. This eliminates the need to pay unnecessary fees to the pay-as-you-go server device 30, making it possible to reduce development costs.

In this manner, in this embodiment, it is possible to obtain an estimate of prediction accuracy without occupying the local terminal device 10 for a long period of time or the server device 30 on the cloud for a long period of time.
This makes it possible to avoid a situation in which, for example, learning is performed for a long period of time, for example, half a day to a day, on all learning data sets 17, but the expected accuracy is not achieved, resulting in a waste of time and server fees.
In addition, if an estimate of the prediction accuracy when the number of data items is increased can be obtained to improve the accuracy of the data set, a guideline for improving the accuracy can be obtained. In other words, it is possible to develop a data set that increases the improvement α of the prediction accuracy by referring to the improvement α of the prediction accuracy.

<Other embodiments>
The present technology is not limited to the above-described embodiment, and various other embodiments can be realized.

In the above, a standalone control unit 15 (terminal device 10) has been given as an example of one embodiment of an information processing device related to the present technology. However, the information processing device related to the present technology may be realized by any computer that is configured separately from the control unit 15 and connected to the control unit 15 via a wired or wireless connection. For example, the information processing method related to the present technology may be executed by a cloud server. Alternatively, the control unit 15 may be linked to another computer to execute the information processing method related to the present technology.

In other words, the information processing method and program related to the present technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction with each other. In this disclosure, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device in which multiple modules are housed in a single housing, are both systems.

The information processing method and program execution related to the present technology by a computer system includes both cases where, for example, the acquisition of features of a partial dataset and the estimation of accuracy information are executed by a single computer, and cases where each process is executed by a different computer. Furthermore, the execution of each process by a specific computer includes having another computer execute part or all of the process and obtaining the results.

In other words, the information processing method and program related to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed collaboratively by multiple devices via a network.

It is also possible to combine at least two of the characteristic features of the present technology described above. In other words, the various characteristic features described in each embodiment may be combined in any manner, without distinction between the embodiments. Furthermore, the various effects described above are merely examples and are not limiting, and other effects may be achieved.

In this disclosure, the terms "same," "equal," "orthogonal," etc. are concepts that include "substantially the same," "substantially equal," "substantially orthogonal," etc. For example, this also includes a state that falls within a specified range (e.g., a range of ±10%) based on "completely the same," "completely equal," "completely orthogonal," etc.

The present technology can also be configured as follows.
(1) an acquisition unit that acquires features of a partial dataset that is a part of an entire dataset used to generate a predictive model;
and an estimation processing unit that estimates accuracy information representing a prediction accuracy of the prediction model generated using the entire data set, based on the feature amount of the partial data set.
(2) The information processing device according to (1),
The estimation processing unit estimates, as the accuracy information, a change in prediction accuracy of the prediction model generated using the entire data set relative to the prediction accuracy of the prediction model generated using the partial data set.
(3) The information processing device according to (2),
The information processing device, wherein the estimation processing unit is configured using an estimation model that estimates a change in the prediction accuracy.
(4) The information processing device according to (3),
The information processing device, wherein the estimation model is a model that learns the relationship between features of a portion of a specified dataset and a change in prediction accuracy that occurs when a specified prediction model is generated using all or a portion of the specified dataset.
(5) The information processing device according to (3) or (4),
The information processing device, wherein the estimation model is a classification model that classifies the amount of change in the prediction accuracy into a plurality of levels.
(6) The information processing device according to (3) or (4),
The information processing device, wherein the estimation model is a rule-based approximation of a classification model that classifies the amount of change in prediction accuracy into a plurality of levels.
(7) The information processing device according to (3) or (4),
The information processing device, wherein the estimation model is a regression model that estimates a change in the prediction accuracy.
(8) An information processing device according to any one of (1) to (7),
the feature amount of the partial data set includes a first feature amount according to the content of the partial data set;
The acquisition unit calculates the first feature amount by analyzing the partial data set.
(9) The information processing device according to (8),
the first feature amount includes at least one of a number of data items included in the partial data set, a number of feature amounts included in the data, and a ratio between the number of data items and the number of feature amounts included in the data.
(10) An information processing device according to any one of (1) to (9),
the feature amount of the partial data set includes a second feature amount corresponding to the prediction model generated using the partial data set;
The information processing device, wherein the acquisition unit calculates the second feature amount by executing a generation process of the prediction model using the partial data set.
(11) The information processing device according to (10),
The partial data set includes a plurality of data groups each having a different purpose,
the second feature amount includes at least one of an evaluation value that evaluates a predicted value of the prediction model generated using the partial data set for each of the plurality of data groups, or a comparison value that compares the evaluation values.
(12) The information processing device according to (11),
The information processing device, wherein the plurality of data groups include a group of training data, a group of validation data, and a group of test data.
(13) The information processing device according to (11) or (12),
The information processing device, wherein the evaluation value includes at least one of a median error, a mean squared error, and a median error rate regarding a predicted value of the prediction model generated using the partial data set.
(14) An information processing device according to any one of (11) to (13),
The information processing apparatus, wherein the comparison value includes at least one of a difference or a ratio of the evaluation values calculated for two of the plurality of data groups.
(15) The information processing device according to any one of (1) to (14), further comprising:
An information processing device comprising: a screen generating unit that generates a screen for presenting the accuracy information.
(16) The information processing device according to (15),
The estimation processing unit estimates, as the accuracy information, a change in prediction accuracy of the prediction model generated using the entire data set relative to a prediction accuracy of the prediction model generated using the partial data set;
The screen generation unit generates at least one of a screen presenting the amount of change in prediction accuracy divided into a plurality of levels, or a screen presenting a value of the amount of change in prediction accuracy.
(17) The information processing device according to (15) or (16),
The screen generation unit generates a selection screen for selecting execution of a generation process of the prediction model using the partial data set;
the acquisition unit, when execution of the generation process is selected, executes the generation process to calculate features of the partial data set;
The information processing device, wherein the estimation processing unit estimates the accuracy information based on feature amounts of the partial data set.
(18) acquiring features of a partial dataset that is a part of the entire dataset used to generate a predictive model;
An information processing method executed by a computer system, comprising: estimating accuracy information representing a prediction accuracy of the prediction model generated using the entire dataset, based on the feature amounts of the partial dataset.
(19) acquiring features of a partial dataset that is a part of the entire dataset used to generate a predictive model;
and estimating accuracy information representing a prediction accuracy of the prediction model generated using the entire dataset, based on the feature amounts of the partial dataset.

F... meta-feature 10... terminal device 14... storage unit 15... control unit 16... control program 17... learning dataset 18... partial dataset 20... UI generation unit 21... prediction model generation unit 22... meta-feature calculation unit 23... accuracy estimation unit 30... server device 35... setting screen 37... evaluation screen 40... estimation model 50... prediction model 51... first prediction model 52... second prediction model 100... model generation system

Claims (18)

An acquisition unit that acquires features of a partial dataset that is a part of an entire dataset used to generate a predictive model;
and an estimation processing unit that estimates accuracy information representing a prediction accuracy of the prediction model generated using the entire data set based on the feature amount of the partial data set,
the feature amount of the partial data set includes a first feature amount according to the content of the partial data set;
The acquisition unit calculates the first feature amount by analyzing the partial data set.
Information processing device. 2. The information processing device according to claim 1,
The estimation processing unit estimates, as the accuracy information, a change in prediction accuracy of the prediction model generated using the entire data set relative to the prediction accuracy of the prediction model generated using the partial data set. 3. The information processing device according to claim 2,
The information processing device, wherein the estimation processing unit is configured using an estimation model that estimates a change in the prediction accuracy. 4. The information processing device according to claim 3,
The information processing device, wherein the estimation model is a model that learns the relationship between features of a portion of a specified dataset and a change in prediction accuracy that occurs when a specified prediction model is generated using all or a portion of the specified dataset. 4. The information processing device according to claim 3,
The information processing device, wherein the estimation model is a classification model that classifies the amount of change in the prediction accuracy into a plurality of levels. 4. The information processing device according to claim 3,
The information processing device, wherein the estimation model is a rule-based approximation of a classification model that classifies the amount of change in prediction accuracy into a plurality of levels. 4. The information processing device according to claim 3,
The information processing device, wherein the estimation model is a regression model that estimates a change in the prediction accuracy. 2. The information processing device according to claim 1 ,
the first feature amount includes at least one of a number of data items included in the partial data set, a number of feature amounts included in the data, and a ratio between the number of data items and the number of feature amounts included in the data. 2. The information processing device according to claim 1,
the feature amount of the partial data set includes a second feature amount corresponding to the prediction model generated using the partial data set;
The information processing device, wherein the acquisition unit calculates the second feature amount by executing a generation process of the prediction model using the partial data set. The information processing device according to claim 9 ,
The partial data set includes a plurality of data groups each having a different purpose,
the second feature amount includes at least one of an evaluation value that evaluates a predicted value of the prediction model generated using the partial data set for each of the plurality of data groups, or a comparison value that compares the evaluation values. The information processing device according to claim 10 ,
The information processing device, wherein the plurality of data groups include a group of training data, a group of validation data, and a group of test data. The information processing device according to claim 10 ,
The information processing device, wherein the evaluation value includes at least one of a median error, a mean squared error, and a median error rate regarding a predicted value of the prediction model generated using the partial data set. The information processing device according to claim 10 ,
The information processing apparatus, wherein the comparison value includes at least one of a difference or a ratio of the evaluation values calculated for two of the plurality of data groups. The information processing device according to claim 1 , further comprising:
An information processing device comprising: a screen generating unit that generates a screen for presenting the accuracy information. The information processing device according to claim 14 ,
The estimation processing unit estimates, as the accuracy information, a change in prediction accuracy of the prediction model generated using the entire data set relative to a prediction accuracy of the prediction model generated using the partial data set;
The screen generation unit generates at least one of a screen presenting the amount of change in prediction accuracy divided into a plurality of levels, or a screen presenting a value of the amount of change in prediction accuracy. The information processing device according to claim 14 ,
The screen generation unit generates a selection screen for selecting execution of a generation process of the prediction model using the partial data set;
the acquisition unit, when execution of the generation process is selected, executes the generation process to calculate features of the partial data set;
The information processing device, wherein the estimation processing unit estimates the accuracy information based on feature amounts of the partial data set. A step of acquiring features of a partial dataset that is a part of an entire dataset used to generate a predictive model;
estimating accuracy information representing a prediction accuracy of the prediction model generated using the entire data set based on the feature amount of the partial data set ;
An information processing method executed by a computer system , comprising:
the feature amount of the partial data set includes a first feature amount according to the content of the partial data set;
The step of acquiring the feature amount of the partial data set includes calculating the first feature amount by analyzing the partial data set.
Information processing methods. A step of acquiring features of a partial dataset that is a part of an entire dataset used to generate a predictive model;
and estimating accuracy information representing a prediction accuracy of the prediction model generated using the entire data set based on the feature amount of the partial data set ,
the feature amount of the partial data set includes a first feature amount according to the content of the partial data set;
The step of acquiring the feature amount of the partial data set includes calculating the first feature amount by analyzing the partial data set.
program.

Images