CN113835947B

CN113835947B - Method and system for determining abnormality cause based on abnormality recognition result

Info

Publication number: CN113835947B
Application number: CN202010514155.2A
Authority: CN
Inventors: 吴新琪; 苏煜; 章鹏; 车延辙
Original assignee: Alipay Hangzhou Information Technology Co Ltd
Current assignee: Sasi Digital Technology Beijing Co ltd
Priority date: 2020-06-08
Filing date: 2020-06-08
Publication date: 2024-01-26
Anticipated expiration: 2040-06-08
Also published as: CN113835947A; CN117827593A

Abstract

The specification discloses a method and a system for determining an abnormality cause based on an abnormality recognition result. The method comprises the following steps: acquiring at least one index associated with the abnormal recognition result, wherein each index comprises a plurality of fields, and each field is associated with a certain preset business meaning; determining influence factors of each field on the abnormal recognition result based on each field; the influencing factors comprise the abnormality degree and the contribution degree of each field; and determining at least one field as an abnormal field in the plurality of fields based on the influence factors, and determining an abnormal reason based on the abnormal field.

Description

Method and system for determining abnormality cause based on abnormality recognition result

Technical Field

The present disclosure relates to the field of data monitoring, and in particular, to a method and system for determining an abnormality cause based on an abnormality recognition result.

Background

The function of data monitoring is to find the potential risk of data in the data platform and can give an alarm to business personnel in time, so that the system is an important auxiliary tool for fault diagnosis and anomaly analysis, and the importance of the monitoring system to various data platforms is self-evident.

However, after the abnormality is found in the data monitoring process, the abnormal data needs to be checked by using service personnel to find the reason of the abnormality, and the time consumed by the checking largely determines whether the risk can be handled in time.

Disclosure of Invention

One of the embodiments of the present specification provides a method of determining a cause of an abnormality based on an abnormality recognition result, the method including: acquiring at least one index associated with the abnormal recognition result, wherein each index comprises a plurality of fields, and each field is associated with a certain preset business meaning; determining influence factors of each field on the abnormal recognition result based on each field; the influencing factors comprise the abnormality degree and the contribution degree of each field; and determining at least one field as an abnormal field in the plurality of fields based on the influence factors, and determining an abnormal reason based on the abnormal field.

One of the embodiments of the present specification provides a system for determining a cause of an abnormality based on an abnormality recognition result, the system including: the abnormal recognition result acquisition module is used for acquiring at least one index associated with the abnormal recognition result, wherein each index comprises a plurality of fields, and each field is associated with a certain preset business meaning; the influence factor determining module is used for determining influence factors of each field on the abnormal recognition result based on each field; the influencing factors comprise the abnormality degree and the contribution degree of each field; and the abnormality cause determining module is used for determining at least one field as an abnormality field in the fields based on the influence factors and determining an abnormality cause based on the abnormality field.

One of the embodiments of the present specification provides an apparatus for determining a cause of an abnormality based on an abnormality recognition result, including a processor and a storage medium storing computer instructions, the processor being configured to execute at least a part of the computer instructions to implement a method as described above.

Drawings

The present specification will be further elucidated by way of example embodiments, which will be described in detail by means of the accompanying drawings. The embodiments are not limiting, in which like numerals represent like structures, wherein:

FIG. 1 is a schematic illustration of an application scenario of a system for determining a cause of an anomaly based on anomaly recognition results, according to some embodiments of the present disclosure;

FIG. 2 is an exemplary flow chart of a method of determining a cause of an anomaly based on anomaly identification results, according to some embodiments of the present disclosure;

FIG. 3 is an exemplary flow chart for obtaining at least one indicator associated with an anomaly identification result according to some embodiments of the present disclosure;

FIG. 4 is an exemplary flow chart of another method of determining the cause of an anomaly based on the results of anomaly identification according to some embodiments of the present disclosure;

FIG. 5 is an exemplary system block diagram of a system for determining a cause of an anomaly based on an anomaly identification result, according to some embodiments of the present description;

FIG. 6 is a system block diagram of an anomaly recognition result acquisition module shown in accordance with some embodiments of the present specification.

Detailed Description

In order to more clearly illustrate the technical solutions of the embodiments of the present specification, the drawings that are required to be used in the description of the embodiments will be briefly described below. It is apparent that the drawings in the following description are only some examples or embodiments of the present specification, and it is possible for those of ordinary skill in the art to apply the present specification to other similar situations according to the drawings without inventive effort. Unless otherwise apparent from the context of the language or otherwise specified, like reference numerals in the figures refer to like structures or operations.

It should be understood that as used herein, "server," "platform," "background," "server," etc. may be interchanged, "user," "user terminal," "requestor," "front-end," "user device," etc. may be interchanged. As used herein, a "system," "apparatus," "unit," and/or "module" is a means for distinguishing between different components, elements, parts, portions, or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

As used in this specification and the claims, the terms "a," "an," "the," and/or "the" are not specific to a singular, but may include a plurality, unless the context clearly dictates otherwise. In general, the terms "comprises" and "comprising" merely indicate that the steps and elements are explicitly identified, and they do not constitute an exclusive list, as other steps or elements may be included in a method or apparatus.

A flowchart is used in this specification to describe the operations performed by the system according to embodiments of the present specification. It should be appreciated that the preceding or following operations are not necessarily performed in order precisely. Rather, the steps may be processed in reverse order or simultaneously. Also, other operations may be added to or removed from these processes.

Fig. 1 is a schematic view of an application scenario of a system for determining an abnormality cause based on an abnormality recognition result according to some embodiments of the present specification.

In some embodiments, the application scenario of fig. 1 may include a server 110, a processor 120, a network 130, and a storage device 140.

In some application scenarios, the system 100 for determining an abnormality cause based on an abnormality recognition result may be widely applied to the back ends of various service platforms, for example, an e-commerce platform, a payment platform, a security monitoring platform, and the like.

Server 110 may retrieve data from storage device 140 or save data to a storage device while processing. In some embodiments, the operations in this specification may be performed by processor 120 executing program instructions. The foregoing is merely for convenience of understanding, and the system may also be implemented in other possible operating modes.

In some embodiments, a storage device 140 may be included in server 110 or other possible system components. In some embodiments, the processor 120 may be included in the server 110 or other possible system components.

In some examples, different functions, such as screening of data, preprocessing, execution of modules, etc., may be performed on different devices, respectively, which is not limited in this specification.

Server 110 may be used to manage resources and process data and/or information from at least one component of the present system or external data sources (e.g., a cloud data center). In some embodiments, the server 110 may be a single server or a group of servers. The server farm may be centralized or distributed (e.g., server 110 may be a distributed system), may be dedicated, or may be serviced concurrently by other devices or systems. In some embodiments, server 110 may be regional or remote. In some embodiments, server 110 may be implemented on a cloud platform or provided in a virtual manner. For example only, the cloud platform may include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an internal cloud, a multi-layer cloud, or the like, or any combination thereof.

Processor 120 may process data and/or information obtained from other devices or system components. The processor may execute program instructions to perform one or more of the functions described herein based on such data, information, and/or processing results. In some embodiments, processor 120 may include one or more sub-processing devices (e.g., single-core processing devices or multi-core processing devices). By way of example only, the processor 120 may include a Central Processing Unit (CPU), an Application Specific Integrated Circuit (ASIC), an Application Specific Instruction Processor (ASIP), a Graphics Processor (GPU), a Physical Processor (PPU), a Digital Signal Processor (DSP), a Field Programmable Gate Array (FPGA), an editable logic circuit (PLD), a controller, a microcontroller unit, a Reduced Instruction Set Computer (RISC), a microprocessor, or the like, or any combination thereof.

The network 130 may connect components of the system and/or connect the system with external resource components. The network 130 enables communication between the various components and with other components outside the system to facilitate the exchange of data and/or information. In some embodiments, the network 130 may be any one or more of a wired network or a wireless network. For example, the network 130 may include a cable network, a fiber-optic network, a telecommunications network, the internet, a Local Area Network (LAN), a Wide Area Network (WAN), a Wireless Local Area Network (WLAN), a Metropolitan Area Network (MAN), a Public Switched Telephone Network (PSTN), a bluetooth network, a ZigBee network, a Near Field Communication (NFC), an intra-device bus, an intra-device line, a cable connection, and the like, or any combination thereof. The network connection between the parts can be in one of the above-mentioned ways or in a plurality of ways. In some embodiments, the network may be a point-to-point, shared, centralized, etc. variety of topologies or a combination of topologies. In some embodiments, network 130 may include one or more network access points. For example, the network 130 may include wired or wireless network access points, such as base stations and/or network switching points 130-1, 130-2, …, through which one or more components of the access system 100 may connect to the network 130 to exchange data and/or information.

Storage device 140 may be used to store data and/or instructions. Storage device 140 may include one or more storage components, each of which may be a separate device or may be part of a database or other device. In some embodiments, the storage device 140 may include Random Access Memory (RAM), read Only Memory (ROM), mass storage, removable memory, volatile read-write memory, and the like, or any combination thereof. By way of example, mass storage may include magnetic disks, optical disks, solid state disks, and the like. In some embodiments, the storage device 140 may be implemented on a cloud platform.

Data refers to a digitized representation of information and may include various types such as binary data, text data, image data, video data, and the like. Instructions refer to programs that may control a device or apparatus to perform a particular function.

In some embodiments, a service platform (e.g., shopping platform, transaction platform, paymate or banking institution, etc.) may collect a large amount of streamlines over time, which typically include a number of information related to the services provided by the service platform. In order to ensure stable and safe operation of the platform, risk monitoring is required to be carried out on the collected flow data, and the monitored abnormality is reported to related departments in time. By way of example only, the service platform acts as a banking institution or payoff platform, and it is desirable to follow up with identifying whether there is a risk such as interface abuse, gambling, money laundering, etc. for users within the platform, to locate which merchants are behaving abnormally (e.g., sudden increases in transactions, focusing on channels), which buyers are behaving abnormally (re-shopping, suspicious nighttime, integer transactions), with highly aggregated device environments, etc., in order for the regulatory authorities to push results back to the regulatory authorities, following corresponding risk exposure governments. In another example, the service platform is a shopping platform or a payment platform, and when a certain large marketing campaign is going on, if a lot of malicious behavior "pull wool" needs to be identified, information such as a specific scene, a region, a crowd, a medium main body and the like of the behavior needs to be positioned according to the flow data so as to accurately follow up the attack and defense of the corresponding wind control strategy in real time.

In some embodiments, for the risk, the server 110 may obtain the historical summary index (such as daily transaction number) at the same time, and monitor the historical summary index by setting a threshold in a same-ratio or ring-ratio manner, for example, suspicious behavior generally occurs between 12 nights and 1 early morning, and the server 110 may obtain an average value of the indexes in the 12 nights and 1 early morning in the previous day or week to compare with the current flowing data, and perform an anomaly prompt when the data float exceeds the preset threshold (such as 20%). However, due to the limitation of the same ratio or the ring ratio, risks may be missed in some cases, such as the transaction number which looks smooth overall, the severe fluctuation of the merchant with small transaction amount may be covered by the relatively smooth of the merchant with large transaction amount, and for example, the "Mi Lane" order is usually paid for a smaller amount (such as not more than 20 yuan), and statistics is carried out together with other orders with larger amount, and obvious abnormality may not be reflected in the indexes such as the transaction total amount, thereby the risks are missed. In addition, due to the adoption of the historical summary index, the scheme also lacks attention on other indexes which are not easy to quantify, such as transaction strokes which are smooth in overall appearance, and payment channels (such as card transaction, mobile payment and the like) and payment types (such as gateway, shortcut, payment, presentation and the like) behind the scheme can be changed.

In some embodiments, when an abnormality is found by monitoring the flow data, the data is typically presented to a business person, and the abnormality data is judged by the experience of the business person to determine the cause of the abnormality. For example, by monitoring that the "daily transaction count" is reduced by 50% in the same way, after the business personnel receives the abnormality, the business personnel need to analyze according to the business understanding degree and the secondary exploration data, so that the reason for taking the practical measures can be gradually located, which is time-consuming and labor-consuming, and the risk of leakage is also present.

In some embodiments, in consideration of multi-index analysis in the flowing water data, a system for determining the cause of the abnormality based on the abnormality identification result is provided, and the abnormality cause can be determined more accurately through the cooperation of algorithms, so that the defects in some other embodiments can be overcome.

FIG. 2 is an exemplary flow chart of a method of determining a cause of an anomaly based on an anomaly identification result, according to some embodiments of the present description.

One or more operations of a method 200 of determining a cause of an anomaly based on the anomaly identification results shown in FIG. 2 may be implemented by the system 100 of FIG. 1.

Step 210, obtaining at least one index associated with the anomaly identification result. In some embodiments, step 210 may be performed by the anomaly identification result acquisition module 510.

The anomaly recognition result indicates that one type of data has anomalies relative to past data or a preset attention value, and it can be understood that the occurrence of anomalies in the data is not equivalent to the occurrence of risks, for example, in the popularization process of mobile payment, if the fact that the cash payment proportion in the payment channel of the merchant is reduced is detected, the occurrence of risks cannot be represented, and excessive attention is not required to be paid to the payment channel.

In some embodiments, the anomaly identification results may be obtained by employing a conventional model, may be obtained based on pipelined data using a specific algorithm, or may be obtained by receiving other systems via the network 130. The manner of acquiring the abnormality recognition result is described in detail later.

In some embodiments, the anomaly identification result generally has an associated indicator that can represent a class of data, such as a time indicator, a payment channel indicator, a payment amount indicator, a high risk period indicator, a merchant account indicator, a transaction count indicator, and the like.

In some embodiments, the metrics associated with each anomaly identification result include a plurality of fields, each field being associated with a certain preset business meaning. It will be appreciated that in some embodiments, the field is a piece of data in the storage device 140, and when the index associated with the abnormal recognition result is a time index, the preset business meaning associated with the field represents a certain time period, and specifically, the time index includes 24 evenly-separated fields, that is, the preset business meaning associated with each field is transaction-related data of one hour in a day. It should be noted that in some other embodiments, the time index may also include other numbers of fields, such as 3 or 4, and the fields are not evenly separated, such as 6 hours into one field during early morning hours when the transaction is less and one hour into one field during prime hours when the transaction is active.

In some embodiments, the metrics are at least data discretized, i.e., the multiple fields included in the metrics are pre-processed to ensure that the subsequent interpretable dimensions are all enumerated. Discretization is a common mode in data processing, and can effectively reduce time complexity and improve the space-time efficiency of an algorithm. By way of example only, if the transaction amount is typically large in span and has a fraction as a result of anomaly identification, then to reduce the algorithm time complexity, the transaction amount is represented in intervals by discretization, such as an enumeration value within ten-yuan being 1, an enumeration value within ten-yuan being 2, an enumeration value within hundred-yuan being 3, an enumeration value within thousand-yuan being 4, and an enumeration value above ten-thousand-yuan being 5.

In some embodiments, the index may also perform processing such as missing value filling, continuous discrete judgment, value normalization, etc., for example only, if no transaction occurs in the early 4 o 'clock, then the transaction amount in the early 4 o' clock is added to be 0 by missing value filling, so that the dimension may be enumerated. In another example, whether the time index is high-risk time index is determined by normalization processing, the time from 22 to 6 on the next day is represented as 1, and the rest of time is represented as 0. In some embodiments, the index may perform one or more of missing value filling, continuous discrete judgment, numerical normalization processing, and other forms of preprocessing, such as regularization or dimension reduction processing, according to different service types.

In some embodiments, to refer toFor example, labeled daily transaction number, the i-th transaction after processing may be represented as (X _i ，Y _i )＝(X _i,1 ,X _i,2 ,…,X _i,p ,Y _i ) Wherein each dimension field X _i Are all enumerated, Y _i The transaction number (1 for single stroke or total daily value after splitting each dimension), Y is transaction number, i.e. index related to abnormal recognition result, X is other index (such as transaction time, transaction amount interval, etc.), X is the transaction number _i,p For the enumerated value of the field, p is the number of other indices.

And 220, determining influence factors of each field on the anomaly identification result based on each field. In some embodiments, step 220 may be performed by the influence factor determination module 520.

The index is associated with the abnormality recognition result, which means that there is an abnormality in at least one field among the indexes including a plurality of fields, that is, it is necessary to determine the influence factor of each field on the abnormality recognition result (index).

In some embodiments, the influencing factors include an abnormality degree and a contribution degree of each field. Contribution is used to indicate how much the local variation of a field can account for the overall variation. By way of example only, if the amount of transactions occurring during the day is significantly higher than that occurring at night, then the contribution of the day may be significantly higher in the transaction period index than at night, subject to error due to the daytime condition, based on existing data. The degree of abnormality is used to represent the degree of change of the field in the overall distribution, and is used to make up for the shortage of the contribution degree in some cases.

In some embodiments, the anomaly is at least one of a group stability index (PSI), an information divergence (KL divergence), or a Jensen shannon difference (Jensen-Shannon divergence, also known as JS divergence). Specifically, in this embodiment, since the excellent characteristics of symmetry and value range distribution at [0,1] are provided, the value of the jensen shannon difference is selected as the degree of anomaly, and the jensen shannon difference can be calculated by the following formula:

wherein P and Q in formula (1) represent two probability distributions, i is the field label, P _i And q _i The outlier and normal values of the field are represented, respectively.

It will be appreciated that when the jensen shannon difference is low, the field may be considered as unable to distinguish between normal and abnormal distributions, whereas when the jensen shannon difference is high, it is indicated whether the current field is more capable of distinguishing between abnormalities. In determining the cause of the abnormality, the index having the abnormality discrimination capability is more prone to be searched down. For example, the internet banking payment channel jensen shannon variance is 2% and the mobile payment channel jensen shannon variance is 8%, at which time the mobile payment channel is more prone to be selected for inferior detection.

In some embodiments, the contribution is typically calculated using a contribution analysis, i.e. the contribution is determined based on the field and an indicator to which the field belongs, and in some embodiments the contribution may be calculated by the following formula:

C _ij ＝(A _ij -N _ij )/(A _m -N _m ) (2)

Wherein A is _ij Represents the abnormal value corresponding to the dimension value j under the dimension i (i.e. the ith field), N _ij Representing the normal value A corresponding to the dimension value j in the dimension i _m And N _m Then the total normal and abnormal values are indicated. For example only, assume that in the index of the trade period, there is the following data:

daytime: the normal user quantity is 980, and the abnormal user quantity is 490;

at night: the normal user quantity is 20, and the abnormal user quantity is 10;

from the above data, it can be seen that A corresponds to daytime _ij 490, N _ij 980A _m 500, N _m For 1000, the contribution degree in the trade period index in the daytime can be obtained by the formula (2) to be 98%, and the contribution degree in the nighttime can be calculated to be 2% in the same way. Indeed, in some scenarios the risk is more likely to occur at night, while the technical responsible personnel would be more desirableAt night the data is normal, so this data also reveals that in some embodiments, the mere use of contribution may cause error-attribution problems as previously mentioned.

In some embodiments, the risk is often judged with a priori knowledge to a certain extent by using past experience, for example only, if the risk is more prone to be detected when a large amount of fields in the transaction amount index are involved in the transaction, and if the risk is more prone to be detected at night in the transaction time period index, the risk is better interpreted and focused on the business by using the priori knowledge, so that the priori risk is introduced. In some embodiments, the influencing factors further include a priori risk, and it can be understood that the a priori risk is a weight that is preset by the field before determining the cause of the abnormality according to the business meaning thereof. For example only, the a priori risk level may be set to a different value depending on its weight, and if a double focus on the night field is desired, the night field weight may be set to 2 and the other fields in the trade period index weight may be set to 1. It should be noted that, in other embodiments, the weight may be 0.5, 0.8, 1.5, or 3.

And step 230, determining at least one field as an abnormal field in the fields based on the influence factors, and determining an abnormal reason based on the abnormal field. In some embodiments, step 230 may be performed by the anomaly cause determination module 530.

In some embodiments, at least one field can be determined from among the plurality of fields included in the index based on the influencing factors obtained in step 220. It will be appreciated that the determined field or fields greatly affect the indicator, thereby causing the indicator to be identified as anomalous. And taking the determined field as an abnormal field, and analyzing and determining the reason of the abnormality based on the abnormal field.

In some embodiments, the method of determining the reason of the abnormality based on the abnormality field may be analyzed by using manual experience, for example, the reason that the reason of the abnormality is located to the decrease of the daily transaction count is that the transaction count of a certain transaction channel drops by zero during the night transaction period, so that the business responsible person can more closely follow up the butt joint investigation work of the corresponding channel, and meanwhile, the main problem of the channel can be provided to occur at night, so that the investigation range is reduced, and the determination and solution efficiency of the reason of the abnormality is improved.

In some embodiments, when each field is established in the database to be an abnormal field, the corresponding reason may be set up, and after the abnormal field is determined, a matching operation is performed to obtain the reason of the abnormality.

In some embodiments, at least one field having the greatest impact on the anomaly identification result is determined as an anomaly field from among the plurality of fields based on the anomaly degree, the contribution degree, and the a priori risk degree. Illustratively, the determination mode with the greatest influence is that the abnormality degree, the contribution degree and the prior risk degree are multiplied, and one or more fields with the greatest products are used as abnormality fields.

In some embodiments, the contribution is derived from equation (1) _ij Obtaining the anomaly degree JS by the formula (2) _ij Simultaneously, the prior risk degree obtained in the step 220 is expressed as Weigt _ij The influencing factors can be expressed as:

Score _ij ＝C _ij *JS _ij *Weigt _ij (3)

i and j in equation (3) are the same as equation (1) and are used to represent the outlier of the dimension value j in dimension i (i.e., the i-th field).

It should be noted that, in some embodiments, the determination of the influencing factor may also be a measure of adding the abnormality degree, the contribution degree, and the a priori risk degree or making a distribution difference.

FIG. 3 is an exemplary flow chart for obtaining at least one indicator associated with an anomaly identification result according to some embodiments of the present disclosure.

In some embodiments, the anomaly identification results in the method 200 of determining the cause of the anomaly based on the anomaly identification results may also be obtained by the server 110 in the system 100. For illustrative purposes only, the disclosure describes the disclosed solution in detail with the server 110 obtaining the anomaly identification result, and is not intended to limit the scope of the disclosure, and in some embodiments, the anomaly identification result may be other servers or sent to the server 110 through the network 130.

Referring to FIG. 3, in some embodiments, the obtaining at least one anomaly index in step 210 includes the steps of:

step 211, obtaining a plurality of monitoring indexes. In some embodiments, step 211 may be performed by the monitoring index acquisition unit 511.

In some embodiments, the monitoring index is similar to the index in step 210, that is, it can represent a class of data, except that the monitoring index in step 211 generally includes a plurality of monitoring indexes, and it is required to identify an abnormal result in a plurality of data, and it is generally required to monitor a plurality of types of data, and the monitoring of the plurality of types of data forms a plurality of monitoring indexes according to different actual scenes. For example only, such as for anomaly identification for a trading platform, the monitoring metrics may include a time metric, a payment channel metric, a payment amount metric, a high risk time period metric, a merchant account metric, a trading count metric, a last two-digit metric, a trading date metric, a merchant account id, and the like.

In some embodiments, each of the monitoring metrics includes a plurality of fields, each field being associated with a certain preset business meaning. The field is the same as the field in the index associated with the anomaly identification result, and specifically, reference may be made to the description related to step 210, which is not repeated herein.

And step 213, removing the periodic component in the monitoring index based on a time sequence decomposition algorithm to obtain the inspection monitoring index. In some embodiments, step 213 may be performed by the timing decomposition unit 513.

The timing decomposition algorithm is for a time series, assuming it is an additive model (Additive decomposition), then for the time series Total _t Can be decomposed into a periodic component (seasonal component), a trend component (trend component), and a remainder (remainder component), which in some embodiments can be expressed as:

Total _t ＝Seasonal _t +Trend _t +Residual _t ,t＝1,2,…,n (4)

in some embodiments, the monitoring index is a time series, and the periodic component Seaseal can be removed based on the above formula (4) _t I.e. Trend component Trend _t And remainder Residual _t And adding to obtain the inspection and monitoring index. After the periodic components in the monitoring index are removed, the influence of the received periodicity is reduced in the subsequent data processing process, and other anomalies except the periodic influence can be more focused.

In some embodiments, the temporal decomposition algorithm is a Seasonal and trend decomposition method (Seasonal-Trend decomposition procedure based on Loess, STL) based on locally weighted regression. Seasonal and trending decomposition methods based on locally weighted regression are a versatile and more robust method of time series decomposition. The algorithm is a more mature scheme in the prior art, and uses local weighted regression (Loess) as a regression algorithm to perform decomposition of the time sequence, which is not described in detail herein. In some embodiments, the timing decomposition algorithm may also be an MSTL algorithm or the like.

And step 215, processing the inspection and monitoring index based on an abnormality detection algorithm to obtain an abnormality identification result. In some embodiments, step 215 may be performed by anomaly detection unit 515.

In some embodiments, for the inspection and monitoring index obtained after removing the periodic component, the anomaly identification result in which the anomaly occurs needs to be found, and it should be noted that the occurrence of the anomaly in one index does not represent that the risk is necessarily present, and it can be understood that the anomaly has good anomaly and bad anomaly, for example, when the marketing campaign is performed, the transaction amount index may rise greatly for a short time to generate anomalies, and at this time, the anomaly of the transaction amount can represent the success of the marketing campaign to a certain extent.

In some embodiments, the anomaly detection algorithm may be an anomaly detection algorithm based on a statistical hypothesis test, or may be an anomaly detection common algorithm using a time series model such as a 3-Sigma model, an isolated Forest (Isolation Forest), etc., and in some embodiments of the present specification, the anomaly detection algorithm based on a statistical hypothesis test is further described below.

Step 217, determining at least one index associated with the abnormality recognition result in the monitoring indexes based on the abnormality recognition result. In some embodiments, step 217 may be performed by the anomaly identification result determination unit 517.

In some embodiments, the anomaly identification result may be in the form of a value or probability, etc., and at least one indicator associated with the anomaly identification result is determined from the monitored indicators. As can be seen in fig. 4, in some embodiments, the index associated with the anomaly identification result determined in step 217 is an index for further determining the cause of the anomaly in step 210.

In some embodiments, the anomaly detection algorithm in step 215 is a hypothesis testing algorithm. Further, in the hypothesis test algorithm, the anomaly identification result is a test statistic (test statistical), and at least one index associated with the anomaly identification result is obtained based on the test statistic in step 217.

In some embodiments, grubbs' Test is chosen as a hypothesis testing method, which is often used to Test single outliers in a univariate dataset (univariate data set) Y subject to normal distribution, i.e. to Test the above mentioned Test monitor indicators, if any, which must be the maximum or minimum value in the dataset. In some embodiments, the Test statistics used for Grubbs' Test assumptions may be expressed as:

in the formula (5) of the present invention,and s is the standard deviation. But in real-world datasets, outliers tend to be multiple rather than single. To extend Grubbs' Test to k outlier detection, it is necessary to gradually delete the value (maximum or minimum) that deviates most from the mean in the dataset, update the corresponding t-distribution threshold synchronously,checking whether the original assumption is true. Based on this, a generalized version ESD (Extreme Studentized Deviate Test) of Grubbs' Test is proposed, which can be expressed as:

in the formula (6), calculate and averageThe furthest offset residuals.

The generalized version of Grubbs' Test ESD scheme does not capture part of the outliers well, resulting in low recall, as individual outliers can stretch the mean and variance significantly. Further, in some embodiments, the hypothesis testing algorithm is chosen to be the Hybrid gend extreme student variance (Hybrid GESD), which uses the median and absolute median difference (Median Absolute Deviation, MAD) with greater robustness to replace the mean and standard deviation in equation (6), which in some embodiments may be expressed as:

Mad=media (|y) in formula (7) _t –median(Y|)。

In some embodiments, the test statistics defined above are used to verify whether and how many (or at most, at least) outliers are present in the test monitor indicator.

In some embodiments, test statistic R in equation (7) _j For the following hypothesis testing problem:

h0 (original assumption): no outliers in the dataset;

h1 (alternative hypothesis): there are at most k outliers in the dataset.

Calculating based on formula (7) to obtain test statistic R _j Thereafter, a threshold value (critical value) is derived based on the number of fields included in the monitoring indicator, which in some embodiments may representThe method comprises the following steps:

in the formula (8), n is the number of samples of the data set, t _p,n-j-1 T distribution threshold for significance (significance level) equal to p and freedom (degrees of freedom) equal to (n-j-1).

Calculating a critical value lambda based on the formula (8) _j After that, the original hypothesis is checked, and the checking statistic R is compared _j And a critical value lambda _j If the test statistic is greater than the critical value, the original assumption H0 is not established, the sample point at the corresponding moment is an abnormal point, and the steps k times are repeated until the algorithm is finished. Accordingly, it is understood that when the test statistic is not greater than the threshold value, the current monitoring index is not abnormal.

FIG. 4 is an exemplary flow chart of another method of determining the cause of an anomaly based on the results of anomaly identification according to some embodiments of the present specification.

Referring to fig. 4, in some embodiments of the present disclosure, after determining the cause of the abnormality based on the abnormality identification result, since only one abnormality is not guaranteed in all data, the method 400 further includes, according to actual needs, performing a next round of obtaining the abnormality identification result based on the monitoring index and determining the abnormality cause or determining the abnormality cause based on the abnormality identification result on the remaining data, where the method 400 further includes:

removing the abnormal field from the monitoring index; determining at least one abnormality index based on the remaining fields in the monitoring index; and determining a new abnormal field based on the abnormal index until an iteration cut-off condition is met.

More than one anomaly field may be present in the same indicator associated with the anomaly identification result, so in some embodiments, after determining the cause of the anomaly based on the anomaly field, only the anomaly field is removed from the indicator associated with the anomaly identification result. In some embodiments, as can be seen from the foregoing, the index associated with the anomaly identification result is one of the monitored indexes, so the anomaly field is removed from the monitored indexes here in order to continue to identify anomalies based on the remaining fields in the monitored indexes.

The determination of at least one anomaly indicator based on the remaining fields in the monitored indicators, i.e., to find the indicator associated with the next anomaly identification result, may be performed in some embodiments by determining the indicator associated with the anomaly identification result in fig. 2 and 3, and specifically, reference may be made to step 210 and the descriptions related to steps 211-217, which are not repeated herein.

The determination of the new anomaly field based on the anomaly metrics may be performed in some embodiments by determining the anomaly field in FIG. 2, and is described in detail with reference to steps 210-230, which are not repeated herein.

It can be seen that in some embodiments, after the abnormal field is removed from the monitoring indicator, a new iteration is performed for the remaining fields in the monitoring indicator to implement attribution downscaling of the cause of the abnormality until the iteration cutoff condition is satisfied. And at each iteration, the abnormality reasons and related data acquired in the previous round are stored. Continuing with the example in step 230, if the first iteration determines that the cause of the abnormality is an abnormal occurrence of the online banking index in the trade channel index, the second iteration determines that the cause of the abnormality is a night occurrence of the trade time index, and the third iteration determines that the cause of the abnormality is an abnormal occurrence of the trade count index, the cause that can be determined for the decrease of the daily trade count is that the trade count of a certain trade channel drops by zero over the night trade period.

In some embodiments, the iteration cutoff condition may be a preset number of times (e.g., 3 times, 5 times, 15 times, etc.), or may be a cutoff when at least one index associated with the abnormality recognition result cannot be determined in the monitored index based on the abnormality recognition result.

In some embodiments, when the iteration is terminated, the system may be caused to output the running result at the time granularity (e.g., daily/hour), including the abnormality attribution details (the lower probe index and field, the abnormality index, etc.) and the corresponding intermediate results (e.g., the calculation results of the contribution degree, the JS divergence, etc. of each round) of each round, so as to facilitate the business responsible person to analyze the data.

It should be noted that the descriptions of the flows in fig. 2-4 above are for illustration and description only, and are not intended to limit the scope of applicability of some embodiments of the present disclosure. Various modifications and changes to the flow may be made by those skilled in the art under the guidance of some embodiments of the present description. However, such modifications and variations are still within the scope of the present description. For example, steps 211-217 and step 210 may be performed independently, with no necessarily sequential order of the two steps.

FIG. 5 is an exemplary system block diagram of a system for determining a cause of an anomaly based on an anomaly identification result, according to some embodiments of the present specification.

As shown in fig. 5, a system 500 for determining an abnormality cause based on an abnormality recognition result may include an abnormality recognition result acquisition module 510, an influence factor determination module 520, and an abnormality cause determination module 530. These modules may also be implemented as an application program or as a set of instructions for execution by a processing engine. Furthermore, a module may be any combination of hardware circuitry and applications/instructions. For example, a module may be part of a processor when the processing engine or processor executes an application/set of instructions.

The anomaly identification result obtaining module 510 may be configured to obtain at least one index associated with the anomaly identification result, where each index includes a plurality of fields, and each field is associated with a preset business meaning.

Further description of the anomaly identification results may be found elsewhere in this specification (e.g., in step 210 and its associated description), and will not be described in detail herein.

The influencing factor determining module 520 may be configured to determine, based on the each field, a influencing factor of the each field on the anomaly identification result; the influencing factors comprise the degree of abnormality and the degree of contribution of each field.

Further description of influencing factors may be found elsewhere in this specification (e.g., in step 220 and its associated description), and will not be described in detail herein.

The anomaly cause determination module 530 may be configured to determine at least one field of the plurality of fields as an anomaly field based on the influencing factor, and determine an anomaly cause based on the anomaly field.

Further description of the anomaly field and the cause of the anomaly may be found elsewhere in this specification (e.g., in step 230 and its associated description), and will not be repeated here.

In some embodiments, in the influencing factor determination module 520, the anomaly is at least one of a population stability index, an information divergence, or a jensen shannon variance; the contribution is determined based on the field and an indicator to which the field belongs.

In some embodiments, the influencing factor determination module 520, the influencing factor includes an a priori risk level; the prior risk degree is a risk preset weight of the field.

In some embodiments, in the influence factor determining module 520, at least one field having the greatest influence on the anomaly identification result is determined as an anomaly field from the plurality of fields based on the anomaly degree, the contribution degree, and the a priori risk degree.

In some embodiments, in the anomaly identification result obtaining module 510, the index is at least subjected to data discretization.

As shown in fig. 6, in some embodiments, the abnormality recognition result acquisition module 510 may include a monitoring index acquisition unit 511, a timing decomposition unit 513, an abnormality detection unit 515, and an abnormality recognition result determination unit 517. These units may also be implemented as an application or as a set of instructions read and executed by a processing engine. Furthermore, a unit may be any combination of hardware circuitry and applications/instructions.

The monitor index obtaining unit 511 may be configured to obtain a plurality of monitor indexes; each monitoring index comprises a plurality of fields, and each field is associated with a certain preset business meaning.

Further description of the monitoring index may be found elsewhere in this specification (e.g., in step 211 and its associated description), and will not be described in detail herein.

The time sequence decomposition unit 513 may be configured to remove the periodic component in the monitoring index based on a time sequence decomposition algorithm to obtain an inspection monitoring index;

further description of the timing resolution algorithm and the inspection monitor indicator may be found elsewhere in this specification (e.g., in step 213 and its associated description), and will not be repeated here.

The anomaly detection unit 515 may be configured to process the inspection and monitoring indicator based on an anomaly detection algorithm to obtain an anomaly identification result;

further description of the anomaly detection algorithm and anomaly recognition results may be found elsewhere in this specification (e.g., in step 215 and its associated description), and will not be repeated here.

The abnormality recognition result determination unit 517 may determine at least one index associated with the abnormality recognition result among the monitor indexes based on the abnormality recognition result.

Further description of the indicators associated with the anomaly identification results may be found elsewhere in this specification (e.g., in step 217, step 210, and their associated descriptions), and will not be described in detail herein.

In some embodiments, the anomaly detection algorithm is a hypothesis test algorithm, and the anomaly identification result is a test statistic; at least one indicator associated with the anomaly identification result is derived based on the test statistic.

In some embodiments, a threshold is derived based on the number of fields included in the monitoring indicator; and when the test statistic is not greater than the critical value, the current monitoring index is not abnormal.

In some embodiments, the hypothesis testing algorithm is a hybrid generalized version extreme student variance.

In some embodiments, the temporal decomposition algorithm is a seasonal and trend decomposition method based on locally weighted regression.

In some embodiments, the exception field is removed from the monitoring indicator; determining at least one abnormality index based on the remaining fields in the monitoring index; and determining a new abnormal field based on the abnormal index until an iteration cut-off condition is met.

It should be understood that the apparatus shown in fig. 5 and 6, and the modules and units thereof, may be implemented in various ways. For example, in some embodiments, the apparatus and its modules may be implemented in hardware, software, or a combination of software and hardware. Wherein the hardware portion may be implemented using dedicated logic; the software portions may then be stored in a memory and executed by a suitable instruction execution device, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the methods and apparatus described above may be implemented using computer executable instructions and/or embodied in processor control code, such as provided on a carrier medium such as a magnetic disk, CD or DVD-ROM, a programmable memory such as read only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The apparatus of the present specification and its modules may be implemented not only with hardware circuits such as very large scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., but also with software executed by various types of processors, for example, and with a combination of the above hardware circuits and software (e.g., firmware).

It should be noted that the above description of the privacy-based encryption system and its modules is for convenience of description only and is not intended to limit the present description to the scope of the illustrated embodiments. It will be appreciated by those skilled in the art that, given the principles of the apparatus, it is possible to combine individual modules or units arbitrarily or to construct a sub-apparatus in connection with other modules without departing from such principles. For example, the timing decomposition unit 513 and the abnormality detection unit 515 in fig. 6 may be the same unit having computing power, and the same computing unit executes two algorithms. For another example, each module in the system for determining the cause of the abnormality based on the result of the abnormality identification may be located on the same server or may belong to different servers. Such variations are within the scope of the present description.

The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

Possible benefits of embodiments of the present description include, but are not limited to: (1) By introducing the concepts of anomaly degree, contribution degree and priori risk degree, the real reasons behind the anomalies are searched heuristically, and reasonable anomaly reasons are rapidly positioned from the data anomalies gathered in each dimension; (2) The data anomalies gathered by each dimension are further explored, recursive search is carried out, and the potential reasons of the anomalies of the dimension are given layer by layer; (3) By adopting the improved abnormality detection algorithm, on one hand, a new time sequence is generated after decomposing the time sequence and eliminating the period influence; on the other hand, the detection statistic is improved, so that the abnormality detection achieves a steady effect.

It should be noted that, the advantages that may be generated by different embodiments may be different, and in different embodiments, the advantages that may be generated may be any one or a combination of several of the above, or any other possible advantages that may be obtained.

While the basic concepts have been described above, it will be apparent to those skilled in the art that the foregoing detailed disclosure is by way of example only and is not intended to be limiting. Although not explicitly described herein, various modifications, improvements, and adaptations to the present disclosure may occur to one skilled in the art. Such modifications, improvements, and modifications are intended to be suggested within this specification, and therefore, such modifications, improvements, and modifications are intended to be included within the spirit and scope of the exemplary embodiments of the present invention.

Meanwhile, the specification uses specific words to describe the embodiments of the specification. Reference to "one embodiment," "an embodiment," and/or "some embodiments" means that a particular feature, structure, or characteristic is associated with at least one embodiment of the present description. Thus, it should be emphasized and should be appreciated that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various positions in this specification are not necessarily referring to the same embodiment. Furthermore, certain features, structures, or characteristics of one or more embodiments of the present description may be combined as suitable.

Furthermore, the order in which the elements and sequences are processed, the use of numerical letters, or other designations in the description are not intended to limit the order in which the processes and methods of the description are performed unless explicitly recited in the claims. While certain presently useful inventive embodiments have been discussed in the foregoing disclosure, by way of various examples, it is to be understood that such details are merely illustrative and that the appended claims are not limited to the disclosed embodiments, but, on the contrary, are intended to cover all modifications and equivalent arrangements included within the spirit and scope of the embodiments of the present disclosure. For example, while the system components described above may be implemented by hardware devices, they may also be implemented solely by software solutions, such as installing the described system on an existing server or mobile device.

Likewise, it should be noted that in order to simplify the presentation disclosed in this specification and thereby aid in understanding one or more inventive embodiments, various features are sometimes grouped together in a single embodiment, figure, or description thereof. This method of disclosure, however, is not intended to imply that more features than are presented in the claims are required for the present description. Indeed, less than all of the features of a single embodiment disclosed above.

In some embodiments, numbers describing the components, number of attributes are used, it being understood that such numbers being used in the description of embodiments are modified in some examples by the modifier "about," approximately, "or" substantially. Unless otherwise indicated, "about," "approximately," or "substantially" indicate that the number allows for a 20% variation. Accordingly, in some embodiments, numerical parameters set forth in the specification and claims are approximations that may vary depending upon the desired properties sought to be obtained by the individual embodiments. In some embodiments, the numerical parameters should take into account the specified significant digits and employ a method for preserving the general number of digits. Although the numerical ranges and parameters set forth herein are approximations that may be employed in some embodiments to confirm the breadth of the range, in particular embodiments, the setting of such numerical values is as precise as possible.

Each patent, patent application publication, and other material, such as articles, books, specifications, publications, documents, etc., referred to in this specification is incorporated herein by reference in its entirety. Except for application history documents that are inconsistent or conflicting with the content of this specification, documents that are currently or later attached to this specification in which the broadest scope of the claims to this specification is limited are also. It is noted that, if the description, definition, and/or use of a term in an attached material in this specification does not conform to or conflict with what is described in this specification, the description, definition, and/or use of the term in this specification controls.

Finally, it should be understood that the embodiments described in this specification are merely illustrative of the principles of the embodiments of this specification. Other variations are possible within the scope of this description. Thus, by way of example, and not limitation, alternative configurations of embodiments of the present specification may be considered as consistent with the teachings of the present specification. Accordingly, the embodiments of the present specification are not limited to only the embodiments explicitly described and depicted in the present specification.

Claims

1. A method of determining a cause of an anomaly based on an anomaly identification result, comprising:

Acquiring at least one index associated with the abnormal recognition result, wherein each index comprises a plurality of fields, and each field is associated with a certain preset business meaning;

determining influence factors of each field on the abnormal recognition result based on each field; the influencing factors comprise the abnormality degree, the contribution degree and the priori risk degree of each field; the anomaly is at least one of a population stability index, information divergence, or jensen shannon variance; the prior risk degree is a risk preset weight of the field; the contribution degree is determined based on the field and an index to which the field belongs; the contribution degree is calculated by:determining; wherein (1)>Represents the abnormal value corresponding to the dimension value j in the dimension i,representing the normal value corresponding to dimension value j in dimension i, < >>And->Then the total normal and abnormal values are indicated;

determining at least one field with the greatest influence on the abnormal recognition result from the plurality of fields as an abnormal field based on the influence factors, and determining an abnormal reason based on the abnormal field; the field with the largest influence is one or more fields with the largest product obtained by multiplying the anomaly degree, the contribution degree and the prior risk degree.

2. The method of claim 1, wherein:

the index is subjected to at least data discretization.

3. The method of claim 1, wherein the obtaining at least one indicator associated with the anomaly identification result comprises:

acquiring a plurality of monitoring indexes; each monitoring index comprises a plurality of fields, and each field is associated with a certain preset business meaning;

removing periodic components in the monitoring index based on a time sequence decomposition algorithm to obtain a detection monitoring index;

processing the inspection and monitoring index based on an abnormality detection algorithm to obtain an abnormality identification result;

at least one index associated with the anomaly identification result is determined in the monitoring index based on the anomaly identification result.

4. A method as claimed in claim 3, wherein:

the abnormality detection algorithm is a hypothesis test algorithm, and the abnormality identification result is a test statistic;

at least one indicator associated with the anomaly identification result is derived based on the test statistic.

5. The method of claim 4, further comprising:

obtaining a critical value based on the number of the fields included in the monitoring index;

and when the test statistic is not greater than the critical value, the current monitoring index is not abnormal.

6. The method of claim 4, wherein:

the hypothesis testing algorithm is a hybrid generalized version extreme student variance.

7. A method as claimed in claim 3, wherein:

the time sequence decomposition algorithm is a seasonal and trend decomposition method based on local weighted regression.

8. A method as claimed in claim 3, comprising:

removing the abnormal field from the monitoring index;

determining at least one abnormality index based on the remaining fields in the monitoring index;

and determining a new abnormal field based on the abnormal index until an iteration cut-off condition is met.

9. A system for determining a cause of an anomaly based on an anomaly identification result, comprising:

the abnormal recognition result acquisition module is used for acquiring at least one index associated with the abnormal recognition result, wherein each index comprises a plurality of fields, and each field is associated with a certain preset business meaning;

the influence factor determining module is used for determining influence factors of each field on the abnormal recognition result based on each field; the influencing factors comprise the abnormality degree, the contribution degree and the priori risk degree of each field; the anomaly is at least one of a population stability index, information divergence, or jensen shannon variance; the prior risk degree is a risk preset weight of the field; the contribution degree is determined based on the field and an index to which the field belongs; the contribution degree is calculated by: Determining; wherein (1)>Representing the abnormal value corresponding to dimension j in dimension i, < >>Representing the normal value corresponding to dimension value j in dimension i, < >>And->Then the total normal and abnormal values are indicated;

the abnormal cause determining module is used for determining at least one field with the greatest influence on the abnormal recognition result from the plurality of fields as an abnormal field based on the influence factors, and determining an abnormal cause based on the abnormal field; the field with the largest influence is one or more fields with the largest product obtained by multiplying the anomaly degree, the contribution degree and the prior risk degree.

10. The system of claim 9, wherein:

the index is subjected to at least data discretization.

11. The system of claim 9, the anomaly recognition result acquisition module comprising:

the monitoring index acquisition unit is used for acquiring a plurality of monitoring indexes; each monitoring index comprises a plurality of fields, and each field is associated with a certain preset business meaning;

the time sequence decomposition unit is used for removing periodic components in the monitoring index based on a time sequence decomposition algorithm to obtain a detection monitoring index;

the abnormality detection unit is used for processing the inspection and monitoring indexes based on an abnormality detection algorithm to obtain an abnormality identification result;

An abnormality recognition result determination unit that determines at least one index associated with the abnormality recognition result among the monitor indexes based on the abnormality recognition result.

12. The system of claim 11, wherein:

13. The system of claim 12, further comprising:

14. The system of claim 12, wherein:

15. The system of claim 11, wherein:

16. The system of claim 11, comprising:

removing the abnormal field from the monitoring index;

17. An apparatus for determining a cause of an anomaly based on an anomaly identification result, comprising a processor and a storage medium for storing computer instructions, the processor for executing at least a portion of the computer instructions to implement the method of any one of claims 1-8.