CN114902239A - Bias detection and interpretability of deep learning models - Google Patents

Abstract

Systems and methods for detecting potential deviations by artificial intelligence modeling human decision making using time series prediction data and event data of survey participants and personal trait data of the participants. The deep bayesian model solves a deviation distribution that fits a modeled prediction distribution of the time series event data and the personal characteristic data to a prediction probability distribution derived by the recurrent neural network. A set of population bias clusters is evaluated for key features of related personal characteristics. The causal graph is defined by a dependency graph of key features. Bias interpretability is inferred by perturbations in a deep bayesian model of feature subsets from the causal graph, determining which causal relationships are most sensitive to changing participant population membership.

Description

Bias detection and interpretability of deep learning models

Technical Field

The application relates to deep learning models. More particularly, the present application relates to a system that infers potential population bias from a deep learning model of human decisions to improve interpretability of the deep learning model.

Background

The Deep Learning (DL) modeling branch of Artificial Intelligence (AI) has revolutionized pattern recognition over the last decade, providing near instantaneous high quality detection and classification of objects in complex dynamic scenes. When used in an autonomous system or as a tactical decision aid, DL can improve the effectiveness of decision skills by improving the speed and quality of task detection and classification.

The interpretability of the DL model helps to improve the confidence of the prediction. For example, after training a "black box" DL network, it is still uncertain whether the loss function performs accurately in finding the most similar match between the test input and the known input. In the field of human decision making models, one area of uncertainty is the presence of potential human bias in the training data. For example, in an attempt to develop a DL model for human prediction, the training data may consist of thousands of event predictions. While DL models can parameterize various known impacts on predictions to learn the human decision-making process, potential aspects (e.g., bias) create gaps in achieving complete modeling. As an illustrative example, when attempting to model decisions, for example for a particular task, there are various sources of bias that may skew the data-driven model. The sources of such deviations may include implicit and unobserved deviations (perhaps even subconscious or unconscious) from the traits and characteristics of individuals who view themselves as part of the population, resulting in implicit deviation of the behavior of the population. Since no such implicit bias was observed with respect to common population traits and biased population members, no model with interpretable bias has been developed.

In existing work, a traditional Bayesian Network (BN) is used to build models representing human cognition and decision making. It allows an expert to specify a model of the decision-making process from a probabilistic story generated, which is generally consistent with human intuition of underlying cognitive processes. Typically, the structure of the BN is pre-specified and the parameters of the probabilistic model are a-priori selected. However, performing inference and learning in such models is NP-complete (i.e., time consuming and often solved by using heuristic methods and approximation methods). This is especially a problem for very complex BN's.

A class of Deep Probabilistic Models (DPMs), known as Deep Bayes Models (DBMs), can be employed to model human decision making. Interpretable AI of DBM is based on mutual information, gradient-based techniques, and correlation-based analysis. Thus, unlike conventional BNs, there is no prior art that can perform causal inferences about DBMs or the results of DBMs.

Disclosure of Invention

The disclosed methods and systems may address all of the above challenges using causal reasoning and perturbation analysis to determine potential deviations present in decision data to improve decision prediction models. A Deep Bayesian Model (DBM) learns prediction distributions from modeled prediction data and associated personal trait data and identifies population deviation clusters. Key features in the personal trait data related to population bias clustering are correlated in a fully connected dependency graph. A causal graph is constructed based on the dependency graph to identify causal relationships of the key features to the cluster of population deviations. Perturbation of individual key features with causal relationships reveals the sensitivity of the features to produce more robust correlations to bias or preference of the prediction data according to specific features of the personal traits, providing interpretability of potential bias in deep learning models.

The results model may enhance interpretability by: (1) providing detailed information why the DBM predicts a certain response for the individual; (2) directly providing data descriptors (e.g., personal characteristics) that cause the deviation; (3) the rationale for how data descriptors are related and which descriptors can produce the largest response variation is provided directly.

Drawings

Non-limiting and non-exhaustive embodiments of the present embodiments are described with reference to the following figures, wherein like reference numerals refer to like elements throughout the various figures unless otherwise specified.

Fig. 1 shows an example of a computer vision system with improved retrieval of a best matching 3D model of a target object according to an embodiment of the invention.

FIG. 2A illustrates an example of a Deep Bayesian Model (DBM) in accordance with embodiments of the present invention.

FIG. 2B illustrates a modified version of the DBM shown in FIG. 2B for estimating population bias clusters according to an embodiment of the present invention.

FIG. 3 shows an example of a flow chart of a process of modeling interpretable potential deviations extracted from decision modeling according to an embodiment of the invention.

FIG. 4 illustrates an example of a computing environment in which embodiments of the invention may be implemented.

Detailed Description

The disclosed methods and systems address the problem of understanding human bias in decision making, which has a variety of applications in industry, such as learning the contribution of bias to user preferences when building design software that can predict or estimate user preferences from past data of the user. Other examples include predictive software and root cause inference, which can be affected by human bias resulting from repeated experience in the past. The observations of the decision data are examined using artificial intelligence to determine patterns to form clusters of populations that indicate population bias, from which deeper inspection and perturbation can be made to explain which key features define the population and which factors will force members out of the population. From this understanding, correlations and causality can be extracted to detect whether there is a bias or preference in the observed decisions. In addition to detecting deviations, the disclosed framework also identifies which human characteristics or traits (culture, ability, gender, education, work experience, technical background, etc.) are the greatest causes of the detected deviations or preferences. As a simple example, in an industrial environment, it is advantageous to understand that operators with a mechanical context tend to make decisions that may infer a troubleshooting decision, which, subject to technical context bias, may be primarily inclined to mechanical causes, which may hinder the troubleshooting process.

FIG. 1 shows an example of a system for data-driven modeling to infer population bias for improved interpretable model construction for human decision-making according to an embodiment of the invention. In one embodiment, the system includes a computing device 110, the computing device 110 including a processor 115 and a memory 111 (e.g., a non-transitory computer readable medium) with various computer applications, modules, or executable programs stored on the memory 111. These modules include a preprocessing module 112, a local Depth Bayesian Model (DBM) module 114, a topic module 116, a correlation module 117, a causal module 118, and a perturbation module 119.

The local DBM module 114 is a client-side module for interacting with a cloud-based or web-based DBM150 to determine population-bias clusters from biased human decision data and event data 120 and personal characteristics/demographics 125 (i.e., data descriptors of the data 120). A network 130, such as a Local Area Network (LAN), Wide Area Network (WAN), or internet-based, is connected to the computing device 110, DBM150, and data stores 120, 125.

FIG. 2A illustrates an example of a DBM according to an embodiment of the invention. The Bayesian Network (BN) can be used for modeling predictions, anWith the characteristics defined by the distribution. In the present invention, a Deep Bayesian Model (DBM) is applied that implements a Deep Learning (DL) network to parameterize the distribution of the bayesian network and predict the parameters. DBM201 includes DL network 210 and BN 212, where some (or all) relationships between variables, model parameters and data are represented using DL algorithms as function approximators. As shown in fig. 2A, the variable p that is not observed represents the true probability of a certain event x. Human expert generates a time series of "predictions" f ═ f ₁ ，...，f _t-1 . A series of auxiliary information n ═ n ₀ ，...，n _t (e.g., news headlines) are processed by a Recurrent Neural Network (RNN)210 together with f, where h-h ₀ ，h ₁ ，...，h _t The probability distribution p can be predicted. In DBM, each f _i And n _i Are all random variables and their relationship to p is a probability distribution, parameterized by RNN 210. For this simplified case, a Bayesian model 212 of the predictor's decision (prediction) is shown, where age affects ability, which in turn affects f _t . The variables and parameters in this example may be defined as follows:

P(p|n，f)＝Dirichlet[α＝RNN(n，f)]

P(x＝0|p)＝Bernoulli(p)

P(c|age)＝Normal[μ＝NN(age)，σ＝NN(age)]

P(f _t |p，c)＝Normal[μ＝p，σ＝NN(c)]

the relationship between these variables is a probability distribution, the parameters of which are represented by the DL neural network 211. In general, such predictive models 212 may additionally include multiple predictors, complex predictor models, and any auxiliary data (time series or non-time series).

Using DL to represent the probability parameters of the DBM increases the flexibility of the models and reduces their bias, since human experts do not need to limit the probability relationships to a simple functional form. The complex non-linear relationship between the large heterogeneous data and the probability parameters of the interpretable model is expressed by the DL algorithm. Otherwise, the DBM can be used as any BN so that given values of certain variables, distributions of other variables can be estimated and maximum likelihood values obtained, etc.

A bayesian network can be interpreted by design. The visual DL algorithm for estimating the functional relationship is very challenging. In one embodiment, the DL component can be hidden while only the BN is exposed to the user. The DBM decision model may be executed to generate useful operation recommendations. The DBM can calculate the posterior p (x | data, model), which is the probability of the target variable x given the decision model and any available data. The maximum d posteriori value of x is the optimal decision, based on the model and data. The basis of each suggestion can be tracked over the network. Metrics such as mutual information or Average Causal Effect (ACE) quantify the strength of a connection in the DBM. The disclosed framework supports the interpretability of its recommendations by back-tracking the impact of the BN on the decision node. One of the main benefits of using a bayesian framework is the possibility to evaluate the model in a strict, unbiased way in terms of evidence, i.e. the data given the model assumptions. In addition to the simplest models, computational model evidence is involved in solving a difficult non-analytical integration problem. Traditional methods, such as markov chain monte carlo or nested sampling, are very time consuming and often require adjustments to be made for a particular task. In contrast, variational reasoning with evidence of a DBM model is a class of objects. In the disclosed framework, approximate model evidence is directly optimized during training. Its approximation is readily available during training and reasoning. This enables the disclosed framework to support comparison and evaluation of competitive decision models. The framework uses streaming data to continually reevaluate evidence for multiple competition models.

FIG. 2B illustrates a modified version of the DBM201 used to estimate population bias clustering in accordance with an embodiment of the invention. DBM 220 as a variant of the DBM shown in FIG. 2A, RNN 2 was learned using temporal deep learningThe observed time series event data x is modeled 21. The response at time t is used to parameterize the distribution for the actual event probability p according to the final outcome of the survey prediction. The probability distribution p is input to the BN222 for biased behavior prediction. In one embodiment, the RNN 221 models the event data (e.g., questions and correct options) that occurred, such as investigating a question X ∈ X ₀ ,...,x _t And participant response Y e Y ₀ ,...,y _t . The RNN 221 models the true probability p of an event given the prediction history data. Response y at time t _t For parameterizing the distribution of true event probabilities p. The BN222 inputs the probabilities p of the historical events from the RNN model 221, the potential deviation estimates and the personal trait data PD (e.g., applying a potential dirichlet distribution or a hierarchical bayesian model) to construct a prediction model f _t The model models the distribution of observed decision data as predicted behavior F ∈ F ₀ ,...,f _t . The BN222 models an estimated deviation distribution, which represents potential deviations over time, modeled as hidden node deviations. The initial distribution for the deviant nodes is modeled by one or more a priori parameters θ. Distribution input prediction model f relating to personal characteristic data PD, deviation distribution deviation and event probability p _t . The relationship between these variables is a probability distribution, the parameters of which are represented by the DL neural network 211. In one embodiment, separate nodes are modeled for each category of personal trait data (e.g., competency, gender, technical experience). Deviation distribution values representing characteristics (reflecting age, ability, education, etc.) of the deviation cluster survey participants are estimated. In one embodiment, curve fitting analysis is applied to solve for the most suitable prediction distribution f _t Deviation distribution from the actual predicted distribution p. Once the curve fit converges, the final parameter values of the curve fit function (e.g., parameters of a potential dirichlet analysis (LDA)) associated with each participant are collectively checked for the presence of clusters of similar values. From these cluster values, a population bias cluster 224 is defined.

In one embodiment, the BN222 includes the LDA algorithm as described above. Traditionally, LDA is useful for extracting topics from documents, where a document is a mixture of potential topics, each word being sampled from a distribution corresponding to one of the topics. This functionality of LDA is extended to the goal of the invention at hand, which is to sample each decision from the distribution corresponding to one of the potential deviations. In one embodiment, an LDA algorithm is applied to the time series data 320 to group related tasks together such that the DBM

FIG. 3 shows an example of a flow chart for constructing an interpretable model of human decision-making, which includes population bias inference. In one embodiment, a virtual (computer-based) decision model is found for a particular task or topic, where the decision is critical to the task. For a decision model with the best confidence in the prediction decision, the potential bias or preference will be an element of inclusion. The process of the disclosed framework involves modeling predicted or decision events for a given task domain based on data collected from numerous human predictions or decisions. From the prediction/decision data model, a deep bayesian model can be used to derive and associate population deviation clusters with common key features of personal characteristics and demographic data. Further processing includes perturbation of the causal graph and sensitivity, which will generate interpretable models for potential bias or preference present in the prediction or decision data.

In embodiments involving human predictive tasks, two forms of data are collected for modeling: (1) human decision data and event data from which potential deviations are found, and (2) personal trait data collected as data descriptors for the decision data that characterize capabilities and other traits that help in finding clustering patterns that can be used to infer population-related deviations. The time series human decision and event data 320 may be collected from a survey (e.g., question/answer format) of multiple participants, which is relevant to the prediction of future events. Decision and event data 320 may capture participant prediction decisions over time to collect future event related data useful to the predictive model. Participants may be asked questions related to predictions or predictive decisions for the target task or topic. For example, the questions may relate to voting of the options, or/and the option is not. Each question may have a probability value (e.g., "how certain do you vote for you. Some surveys may be conducted for a long period of time to produce a distribution of changes. For example, the survey may be repeated monthly over the year until a selected date for predicting the event. The actual results for the predicted events are recorded and included in the archived time series event data 320, which is useful for modeling the prediction data and tracking the probability of whether the prediction is true. In some embodiments, data set 320 may include up to 1000000 to 3000000 predicted data.

In other embodiments, the time series and event data 320 relates to the behavior observed by the participant when performing other types of tasks besides prediction. For example, the DL model may learn predictive binary decisions to perform or not perform a task in a given situation. In this case, the interpretability of the DL model is sought in terms of potential bias affecting such decisions.

The personal characteristics/demographics 325 are data descriptors for the time series event data 320 and may include a range of personal characteristics such as gender, educational level, and competency test scores for the individual being investigated. The goal in collecting data may be to understand cultural influences (e.g., food, religion, region, language) that can identify common group traits of individuals, often with deviations that are implicit and the reasons for decision or prediction are not observable. Examples of other traits found from the prediction data that lead to implicit bias may include one or more of the following: experience may change voting behavior, age or gender may affect the predictive decision for a given topic, and training may change in response to questions. The personal characteristics/demographics 325 may characterize the ability and may be used to identify bias traits. In one aspect, the detailed psychological and cognitive assessment of the decision-maker (e.g., about 20 measures) may include a progressive matrix of the script, cognitive retch tests, berlin calculations, cispril abstract and lexical test scores, political and financial knowledge, arithmetic, working memory and similar test scores, demographic data (e.g., gender, age, educational level), self-assessment (e.g., heart of responsibility, experience openness, extroversion, courage, social value orientation, cultural world view, need for closure).

Data pre-processing 312 is performed on the time series data 320 and the personal characteristics/demographics 325, and includes: (1) data cleanup errors, inconsistencies, and missing data; (2) data integration to integrate data in multiple files and map the data using relationships between the files; (3) feature extraction for dimensionality reduction, such as depth feature mapping (e.g., Word2vec) and feature dimensionality reduction (e.g., PCA, tSNE); and (4) data transformation and temporal data visualization, such as normalization.

The topic grouping module 316 performs exploratory topic data analysis that generates results indicative of event topic groups 321 for survey questions x and can identify similar questions that explain the impact of tasks on population bias clustering. As with the cultural model, it is assumed that the behavior (e.g., decision) of the population bias cluster will be dictated by the scenario under consideration (i.e., the topic associated with the task in the context of the data set). The topic grouping module 316 groups related questions and event tasks together using LDA analysis.

The DBM module 314 receives data from the task-based model 313 and the personal characteristics/demographics 325, determines a prediction probability p from the event data using the process described in fig. 2B, where the event data x corresponds to the time series data 320 and the PD corresponds to the personal characteristics/demographics 325, and determines an estimated population bias cluster 335. In one embodiment, the cluster identifier function of the DBM module 314 applies a parametric curve fit analysis (e.g., a latent dirichlet distribution analysis) to identify which participant belongs to which population deviation cluster, and determines a set of population deviation clusters from the input data. The key feature extractor of the DBM module 314 identifies key features 336 from the population clusters and related data descriptors (personal characteristics/demographics) as features of personal characteristics common among the participants in the group.

Since the DBM model is not a classification model, but is inspired by a subject model, such as Latent Dirichlet Analysis (LDA), evaluation criteria such as accuracy, precision recall, and area under the curve are not applicable. For topic models involving documents, the evaluation first determines the topic of each document using LDA, and then evaluates the appropriateness of the obtained topics. In one embodiment of DBM analysis, a similar method is performed, where population deviation clusters with shared personal characteristic features indicate key features that explain the grouping. In one aspect, cross-validation and "cosine similarity measures" are performed on the packets to obtain a numerical score. For example, by randomly combining 90% of participants, the participants are divided into 50 equal parts. A population bias model is determined on a per-segment basis using DBM 314. For each model, an instance feature selection is made for each user by the personal trait data. The co-selected features under each population are determined for each model using cosine similarity. Next, DBM 314 determines whether the same population bias clusters found by different data share similar common characteristics. A population match may be determined by mapping the population to the population having the highest mahins correlation coefficient.

The correlation module 317 obtains each of the identified population bias clusters 335 and estimates the correlation between the identified key features 336 by using a dependent network analysis to arrive at a cluster having

Full connection dependency graph of connections. In one embodiment, the dependency analysis network utilizes singular value decomposition to compute partial dependencies between features of the dependency network (e.g., by performing partial dependencies between columns of data sets or between network nodes). The computation of the dependency is based on finding the region of highest "node liveness" in the dependency network, which is defined by the impact of the node relative to other network nodes. These node liveness represent the average effect of node j on the pairwise dependencies C (i, k) of all nodes i, k ∈ N. The correlation effect is derived from the difference between the correlation C (i, k) and the partial correlation PC, as shown by the following relationship:

d(i，k|j)＝C(i，k)-PC(i，k|j)

where i, j, k represents the number of nodes in the network.

The total influence D (i, j) represents the total influence of node j on node i, and is defined as the average influence of node j on the correlation C (i, k) of all nodes k, and is expressed as follows:

the node activity of node j is then calculated as the sum of D (i, j):

a fixed number of top features (e.g., top 10, 20, or 50) with the highest node liveness are selected and a causal analysis is performed using these features.

The causal graph module 318 derives a causal graph 322 from the dependency graph for each cluster of population deviations by reducing the non-causal relationships of the dependency graph using the feature subset of the results from the correlation module 317. The causal graph 322 provides a causal relationship between participant characteristics/data descriptors (i.e., dependency graph features) in the data set for each cluster deviation cluster and all cluster deviation clusters combined. In one embodiment, the causal relationship analysis uses a Greedy Equivalent Search (GES) algorithm to obtain causal relationships and build a causal graph. GES is a score-based algorithm that greedily maximizes a scoring function (typically a Bayesian Information Criterion (BIC) score) in the basic (i.e., observation) graph space in three stages, starting with a blank graph: a forward phase, a backward phase and a turn phase. In the forward phase, the GES algorithm moves in the space of the base graph in steps corresponding to the addition of a single edge in the Directed Acyclic Graph (DAGs) space, and once the score can no longer be increased, the phase terminates. In the backward phase, the algorithm performs moves corresponding to deleting a single edge in the DAGs space until the score can no longer be increased. In the turn phase, the algorithm performs a move corresponding to the reversal of a single arrow in the DAGs space until the score can no longer be increased. The GES algorithm loops through these three phases until it is no longer possible to increase the score. In short, the GES algorithm maximizes the scoring function in the graph space. Since the graphics space is too large, a "greedy" approach is applied. The reason for using the GES score for the causal relationship evaluation is as follows. To estimate an accurate causal DAG, two key assumptions need to be kept in theory: (1) causal sufficiency refers to the absence of hidden (or potential) variables, and (2) causal loyalty is defined as follows: if X _ A and X _ B are conditionally independent of a given X _ S, then A and B are separated by S in the causal DAG. However, empirically, if these assumptions do not hold, the performance of the GES algorithm is still acceptable when the number of nodes is not large. In one embodiment, the causal relationships may be pre-specified based on expert knowledge prior to obtaining the data-driven causal network using the GES algorithm. In one embodiment, the causal relationship module 318 is configured to additionally perform a counterfactual analysis to determine the effect of enforcing a particular edge on the causal graph 322. In one aspect, a graphical user interface may be used to be implemented by a user-specified edge, on which changes in response to a Graphical User Interface (GUI) of a network may be observed based on observation data using a GES algorithm.

The perturbation module 319 refines the results of the causal relationship module 318 so that bias interpretability 375 can be inferred. Although the causal graph 322 gives relationships between nodes, it does not provide a group bias cluster membership as to how large the change (node sensitivity) of each node is enough to change the survey response and/or participants. To estimate the variation, the perturbation module 319 selects a single feature (i.e., a subset of features determined to be causal) from the causal graph 322, perturbs the selected feature in the DBM 314, and evaluates the response to the group membership. If the perturbation of a particular feature X causes the problem response of most population members to change, then the feature X may become an explanation for the population bias clustering behavior for that particular topic. Bias interpretability 375 indicates one or more personal characteristic features as the most influential factors, most likely the cause of group bias in decision and event data 310. For example, a sensitivity score may be assigned to each perturbed feature based on the number of population members for which the population membership has changed (e.g., by changing the answers to the predictive survey questions).

As with the cultural model, it is assumed that the behavior of the population bias clusters will be dictated by the scenario, i.e., the topic related to the task in the context of the dataset being considered. Perturbations of individual features derived from the causal graph may indicate a change in the individual's perspective or preference for a particular task belonging to a given topic. In one embodiment, the preference change also helps to infer a bias interpretation 375, indicating additional factors of potential bias. To detect this preference-based topic, the perturbation module 319 includes the event topic population 321 in interpretability inferences 375. If an observation that a change in a certain characteristic causes the individual's opinion or preference to change continuously in most tasks belonging to a certain topic, the observation identifies the relationship between the individual's characteristics, the topic associated with the event under consideration, and the deviation from the model, and provides a probabilistic estimate for the confidence associated with these estimates.

The advantages provided by the modeling described above are numerous. Any field of application decision or prediction can be greatly improved by understanding potential deviations in the process. Once the group bias for a given task or topic is known from the above systems and processes, any computer-based decision modeling can better predict the outcome. For example, the design of an automated assistance system with an emergency model, such as in an automobile or other automated assistance vehicle, may be improved to predict potential deviations or preferences for an operator under different operating conditions and different demographic conditions. These models may be adapted according to the personal characteristics of the driver, e.g. taking into account the learning preferences of such a person. Other such decision modeling applications are well known.

FIG. 4 illustrates an example of a computing environment in which embodiments of the invention may be implemented. Computing environment 400 includes a computer system 410. the computer system 410 may include a communication mechanism such as a system bus 421 or other communication mechanism for communicating information within the computer system 410. Computer system 410 also includes one or more processors 420 coupled with system bus 421 for processing information. In one embodiment, computing environment 400 corresponds to a public system that infers deviations from human decision data, where computer system 410 involves a computer as described in detail below.

Processor 420 may include one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), or any other processor known in the art. More generally, a processor as described herein is a device for executing machine-readable instructions stored on a computer-readable medium for performing tasks and may include any one or combination of hardware and firmware. The processor may also include a memory storing machine-readable instructions executable to perform tasks. Processors operate on information by manipulating, analyzing, modifying, converting, or transmitting information for use by an executable process or an information device, and/or by routing information to an output device. For example, a processor may use or include the functionality of a computer, controller, or microprocessor, and may use executable instructions to perform special purpose functions not performed by a general purpose computer. The processor may include any type of suitable processing unit, including but not limited to a central processing unit, microprocessor, Reduced Instruction Set Computer (RISC) microprocessor, Complex Instruction Set Computer (CISC) microprocessor, microcontroller, Application Specific Integrated Circuit (ASIC), Field Programmable Gate Array (FPGA), system on chip (SoC), Digital Signal Processor (DSP), or the like. Further, processor 420 may have any suitable micro-architectural design that includes any number of constituent components, such as registers, multiplexers, arithmetic logic units, cache controllers to control read/write operations to the cache, branch predictors, and the like. The micro-architectural design of a processor is capable of supporting any of a variety of instruction sets. The processor may be coupled (electrically coupled and/or include executable components) with any other processor to enable interaction and/or communication therebetween. The user interface processor or generator is a known element comprising electronic circuitry or software or a combination of both for generating a display image or a part thereof. The user interface includes one or more display images that enable a user to interact with the processor or other device.

The system bus 421 may include at least one of a system bus, memory bus, address bus, or message bus, and may allow information (e.g., data (including computer executable code), signaling, etc.) to be exchanged between the various components of the computer system 410. The system bus 421 may include, but is not limited to, a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and the like.

With continued reference to FIG. 4, computer system 410 may also include a system memory 430 coupled to system bus 421 for storing information and instructions to be executed by processor 420. The system memory 430 may include computer-readable storage media in the form of volatile and/or nonvolatile memory such as Read Only Memory (ROM)431 and/or Random Access Memory (RAM) 432. The RAM 432 can include other dynamic storage devices (e.g., dynamic RAM, static RAM, and synchronous DRAM). ROM 431 can include other static storage devices (e.g., programmable ROM, erasable PROM, and electrically erasable PROM). In addition, system memory 430 may be used to store temporary variables or other intermediate information during execution of instructions by processor 420. A basic input/output system 433(BIOS), containing the basic routines that help to transfer information between elements within computer system 410, such as during start-up, may be stored in ROM 431. RAM 432 may contain data and/or program modules that are immediately accessible to and/or presently being operated on by processor 420. The system memory 430 may also include, for example, an operating system 434, application modules 435, and other program modules 436. The application modules 435 may include the aforementioned modules described with respect to fig. 1, and may also include a user portal for developing applications, allowing input parameters to be entered and modified as needed.

Operating system 434 may be loaded into memory 430 and may provide an interface between other application software executing on computer system 410 and the hardware resources of computer system 410. More specifically, operating system 434 may include a set of computer-executable instructions for managing the hardware resources of computer system 410 and providing common services to other applications (e.g., managing memory allocation among various applications). In some example embodiments, operating system 434 may control the execution of one or more program modules depicted as stored in data storage 440. Operating system 434 may include any operating system now known or later developed, including but not limited to any server operating system, any host operating system, or any other proprietary or non-proprietary operating system.

The computer system 410 may also include a disk/media controller 443 coupled to the system bus 421 to control one or more storage devices, such as a magnetic hard disk 441 and/or a removable media drive 442 (e.g., a floppy disk drive, an optical disk drive, a tape drive, a flash drive, and/or a solid state drive), for storing information and instructions. The storage device 440 may be added to the computer system 410 using an appropriate device interface (e.g., Small Computer System Interface (SCSI), Integrated Device Electronics (IDE), Universal Serial Bus (USB), or firewire). The storage devices 441, 442 may be located external to the computer system 410.

Computer system 410 can include a user input interface 460 for a Graphical User Interface (GUI)461, which can include one or more input devices such as a keyboard, touch screen, tablet and/or pointing device for interacting with a computer user and providing information to processor 420.

Computer system 410 may perform some or all of the process steps of an embodiment of the invention in response to processor 420 executing one or more sequences of one or more instructions contained in a memory, such as system memory 430. Such instructions may be read into system memory 430 from another computer-readable medium, such as magnetic hard disk 441 or removable media drive 442, of memory 440. Magnetic hard disk 441 and/or removable media drive 442 may contain one or more data stores and data files used by embodiments of the present invention. The data store 440 may include, but is not limited to, a database (e.g., relational, object-oriented, etc.), a file system, a flat file, a distributed data store in which data is stored on multiple nodes of a computer network, a peer-to-peer network data store, and the like. The data storage content and data files may be encrypted to improve security. Processor 420 may also be used in a multi-processing arrangement to execute one or more sequences of instructions contained in system memory 430. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions. Thus, embodiments are not limited to any specific combination of hardware circuitry and software.

As described above, computer system 410 may include at least one computer-readable medium or memory for holding instructions programmed according to embodiments of the invention and for containing data structures, tables, records, or other data described herein. The term "computer-readable medium" as used herein refers to any medium that participates in providing instructions to processor 420 for execution. A computer-readable medium may take many forms, including but not limited to, non-transitory, non-volatile media, and transmission media. Non-limiting examples of non-volatile media include optical disks, solid-state drives, magnetic disks, and magneto-optical disks, such as the magnetic hard disk 441 or the removable media drive 442. Non-limiting examples of volatile media include dynamic memory, such as system memory 430. Non-limiting examples of transmission media include coaxial cables, copper wire and fiber optics, including the wires that comprise system bus 421. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

The computer-readable medium instructions for carrying out operations of the present invention may be assembler instructions, Instruction Set Architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C + +, or the like, as well as conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet service provider). In some embodiments, an electronic circuit comprising, for example, a programmable logic circuit, a Field Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), can execute computer-readable program instructions by personalizing the electronic circuit with state information of the computer-readable program instructions to perform various aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable medium instructions.

Computing environment 400 may also include computer system 410 operating in a networked environment using logical connections to one or more remote computers, such as a remote computing device 473. The network interface 470 may enable communication with other remote devices 473 or systems and/or storage devices 441, 442, e.g., via a network 471. The remote computing device 473 may be a personal computer (laptop or desktop), a mobile device, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to computer system 410. When used in a networking environment, the computer system 410 may include a modem 472 for establishing communications over a network 471, such as the Internet. The modem 472 can be connected to the system bus 421 via the user network interface 470, or via another suitable mechanism.

Network 471 can be any network or system generally known in the art, including the internet, an intranet, a Local Area Network (LAN), a Wide Area Network (WAN), a Metropolitan Area Network (MAN), a direct or serial connection, a cellular telephone network, or any other network or medium capable of facilitating communication between computer system 410 and other computers, e.g., remote computing device 473. The network 471 may be wired, wireless, or a combination thereof. The wired connection may be implemented using ethernet, Universal Serial Bus (USB), RJ-6, or any other wired connection commonly known in the art. The wireless connection may be implemented using Wi-Fi, Worldwide Interoperability for Microwave Access (WiMAX), and bluetooth, infrared, cellular networks, satellite, or any other wireless connection method generally known in the art. Further, multiple networks may operate independently or in communication with each other to facilitate communications in the network 471.

It should be appreciated that the program modules, applications, computer-executable instructions, code, etc., depicted in FIG. 4 as being stored in system memory 430 are merely illustrative and not exhaustive, and that the processes depicted as being supported by any particular module may alternatively be distributed among multiple modules, or performed by different modules. In addition, various program modules, scripts, plug-ins, Application Programming Interfaces (APIs), or any other suitable computer-executable code locally hosted on computer system 410, remote device 473, and/or hosted on other computing devices accessible via one or more networks 471 may be provided to support the functions provided by the program modules, the application programs, or the computer-executable code illustrated in fig. 4, and/or additional or alternative functions. Further, the functionality may be variously modular such that processes described as being commonly supported by a collection of program modules illustrated in FIG. 4 may be performed by a fewer or greater number of modules, or the functionality described as being supported by any particular module may be at least partially supported by additional modules. Further, program modules supporting the functionality described herein may form part of one or more application programs that are executable on any number of systems or devices according to any suitable computing model (e.g., a client-server model, a peer-to-peer model, etc.). Further, any functionality described as being supported by any program modules illustrated in FIG. 4 may be implemented at least partially in hardware and/or firmware across any number of devices.

It should also be understood that the computer system 410 may include alternative and/or additional hardware, software, or firmware components than those described or depicted without departing from the scope of the present invention. More specifically, it should be understood that software, firmware, or hardware components depicted as forming part of computer system 410 are merely illustrative, and that in various embodiments certain components may not be present or additional components may be provided. While various illustrative program modules have been depicted and described as software modules stored in system memory 430, it will be understood that the functions described as being supported by program modules may be enabled by any combination of hardware, software, and/or firmware. It should be further appreciated that, in various embodiments, each of the aforementioned modules may represent a logical partition of supported functionality. The logical partitions are described for ease of explanation of functionality and may not represent the structure of software, hardware, and/or firmware for implementing the functionality. Thus, it is to be understood that in various embodiments, functionality described as being provided by a particular module may be provided, at least in part, by one or more other modules. Further, in some embodiments one or more of the depicted modules may not be present, while in other embodiments additional modules may be present that are not depicted and may support at least a portion of the described functionality and/or additional functionality. Further, while certain modules may be depicted and described as sub-modules of additional modules, in certain embodiments, such modules may be provided as standalone modules or sub-modules of other modules.

While specific embodiments of the invention have been described, those of ordinary skill in the art will recognize that many other modifications and alternative embodiments exist within the scope of the invention. For example, any of the functions and/or processing capabilities described with respect to a particular device or component may be performed by any other device or component. In addition, while various illustrative implementations and architectures have been described in accordance with embodiments of the invention, those of ordinary skill in the art will appreciate that many other modifications to the illustrative implementations and architectures described herein are also within the scope of the invention. Thus, the phrase "based on" or variations thereof should be interpreted as "based, at least in part, on.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present aspect. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. Each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Claims (14)

1. A system for detecting potential deviations through human decision-making artificial intelligence modeling, the system comprising:

a processor; and

non-transitory memory having stored thereon modules for execution by the processor, the modules comprising:

a data store of time series event data, the data store comprising predictions of future events by survey participants and event outcomes, the predictions having potential deviations;

a data repository of personal characteristics data for each survey participant;

a deep bayesian model module comprising:

a recurrent neural network configured to model the time series event data as a predictive probability distribution p,

a Bayesian network having hidden nodes representing at least an estimated deviation distribution and personal data nodes representing the personal characteristic data, the Bayesian network configured to receive the probability distribution and solve a deviation distribution that best fits a model prediction distribution f to the prediction probability distribution p;

a cluster identifier configured to define a set of population bias clusters from the bias distribution;

a key feature extractor configured to identify key features from common personal characteristics within the cluster of population deviations;

a correlation module configured to receive information relating to each of the population deviation clusters and to estimate correlations between the identified key features using a dependency analysis network to construct a dependency graph for each of the population deviation clusters based on singular value decomposition;

a causal graph module configured to perform causal graph analysis to derive a causal graph from the dependency graph for each of the population deviation clusters using a greedy equivalence search algorithm to construct the causal graph across the space of the base graph, the causal graph providing causal relationships between individual characteristics in each population deviation cluster and all combined population deviation clusters; and

a perturbation module configured to infer bias interpretability by perturbing features derived from the causal graph to determine which causal relationships are most sensitive to changing participant population membership, wherein the bias interpretability includes a most likely cause indicating which personal characteristics are cluster of population bias identified based on a highest sensitivity value.

2. The system of claim 1, wherein the cluster identifier function is configured to apply a curve-fitting analysis to solve for the deviation distribution that best fits the predicted distribution fto the actual predicted distribution p, and to collectively examine current parameter values of the curve-fitting function associated with each participant once the curve-fitting converges to determine whether there is a cluster of similar values, the cluster defining a set of population deviation clusters.

3. The system of claim 1, wherein the curve-fitting analysis is latent dirichlet analysis.

4. The system of claim 1, further comprising a topic module configured to determine a set of event topics from the time series event data using a latent dirichlet allocation analysis; wherein the perturbation module is further configured to include a set of event topics for inferring bias interpretability.

5. The system of claim 1, wherein the causality module is further configured to perform a counterfactual analysis to determine an effect of enforcing a particular edge on the causality map.

6. The system of claim 1, wherein the causal graph module is further configured to derive the causal graph by pruning non-causal relationships of the dependency graph.

7. The system of claim 1, wherein the correlation module is further configured to determine a plurality of top features from the dependency graph, the dependency graph including a network of nodes representing the features, the top features being features having a highest node liveness defined by an effect of a node relative to other nodes, the top features being sent to the causal relationship module for the causal relationship analysis.

8. A method of detecting potential deviations through human decision-making artificial intelligence modeling, the method comprising:

modeling, by a recurrent neural network, time series event data as a prediction probability distribution p, wherein the time series event data includes predictions of future events by survey participants and event outcomes, the predictions having potential deviations;

receiving the probability distribution over a bayesian network having hidden nodes representing at least an estimated deviation distribution and personal data nodes representing personal characteristic data of each survey participant, and solving a deviation distribution that best fits a model prediction distribution f to the prediction probability distribution p;

defining a set of population bias clusters from the bias distribution;

identifying key features from common personal characteristics within the population bias cluster;

estimating correlations between the identified key features using a dependency analysis network to construct a dependency graph for each of the population deviation clusters based on singular value decomposition;

performing a causal relationship analysis to derive a causal graph from the dependency graph for each of the population deviation clusters using a greedy equivalence search algorithm to construct the causal graph across the space of the base graph, the causal graph providing causal relationships between individual characteristics in each population deviation cluster and all combined population deviation clusters; and

inferring bias interpretability by perturbing features derived from the causal graph to determine which causal relationships are most sensitive to changing participant population membership, wherein the bias interpretability includes a most likely cause indicating which personal characteristics are population bias clusters identified based on a highest sensitivity value.

9. The method of claim 8, further comprising: applying a curve-fitting analysis to solve for a deviation distribution that best fits the predicted distribution f to the actual predicted distribution p, and, once the curve-fitting converges, collectively examining current parameter values of the curve-fitting function associated with each participant to determine whether there is a cluster of similar values, the cluster defining a set of population deviation clusters.

10. The method of claim 8, wherein the curve-fitting analysis is latent dirichlet analysis.

11. The method of claim 8, further comprising: determining a set of event topics from the time series event data using a potential dirichlet allocation analysis; and including a set of event topics for use in inferring interpretability of the deviation.

12. The method of claim 8, further comprising: a counterfactual analysis is performed to determine the effect of enforcing a particular edge on the cause and effect graph.

13. The method of claim 8, further comprising: deriving the causal graph by compacting non-causal relationships of the dependency graph.

14. The method of claim 8, further comprising: determining a plurality of top features from the dependency graph, the dependency graph including a network of nodes representing the features, the top features being features having a highest node liveness defined by an impact of a node with respect to other nodes, the top features being sent to the causal relationship module for the causal relationship analysis.

Images